Visar inlägg med etikett PYTHON. Visa alla inlägg
Visar inlägg med etikett PYTHON. Visa alla inlägg

Distutils: ValueError: [u'path']

This is a really stupid error that occurs on Windows 64bit when trying to build extension modules using distutils:

  File "C:\Python27\lib\distutils\command\build_ext.py", line 340, in run
    self.build_extensions()
  File "setup.py", line 213, in build_extensions
    with_ext = self.check_extension_availability(ext)
  File "setup.py", line 239, in check_extension_availability
    depends=ext.depends)
  File "C:\Python27\lib\distutils\msvc9compiler.py", line 473, in compile
    self.initialize()
  File "C:\Python27\lib\distutils\msvc9compiler.py", line 383, in initialize
    vc_env = query_vcvarsall(VERSION, plat_spec)
  File "C:\Python27\lib\distutils\msvc9compiler.py", line 299, in query_vcvarsall
    raise ValueError(str(list(result.keys())))
ValueError: [u'path']

The workaround is to take the bat-file in C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\vcvars32.bat and copy it to C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\vcvarsamd64.bat. Not that easy to figure out!

The greatest Django CMS

There are a lot of Django CMS solutions to choose from. I've spent most of the day comparing each of them and trying to figure out which work and which doesn't. My conclusion is that the good one is django-cms which is the only one that didn't bug out on me and also had full and easy to use i18n support. I believe it is a very good CMS tool when you understand how to use it. It has a steep learning curve and poor documentation you have to suffer through first though.

Looking for a Django Multilingual Solution

One that actually works that is. django-multilingual is quite nice when it works, but has some quite severe bugs. It doesn't work with Django 1.2 due to Issue 1114 and it seems that it wont be fixed before 1.2 is out.

It's trunk doesnt work either: Issue 95. django-multilingual-ng looks nice and apparently was created to solve these problems, but it has its own issues and doesn't work with Django 1.1.

Jumpstarting Waf

Introduction

This document is an introduction to the build-tool waf. Waf is a tool written in Python for setting up automated build systems for a programming projects.

Intended Audience and Prerequisite Knowledge

It is assumed that the reader is interested in using waf for setting up the build for a project. The instructions given were recorded on a Linux system and might not be suitable for Windows users.

Why is a Build System Needed?

The build and distribution problem is one of the hardest problems in applied computer science. You have to make sure that your software works in a totally unknown environment radically different from the one in which it was developed.

Additionally if the build fails, the user of the software must be informed in a graceful way with a description of why.

Why Waf?

As previously described, building software is hard. There are many build systems out there, but only waf and autotools provides a holistic system that takes care of every step from configure to distribution. waf has several advantages over autotools that makes it preferable:

  • waf uses Python while autotools mixes shell scripting, M4 and Perl.
  • waf does not generate intermediate files that confuses users like autotools does.
  • waf is many times faster than autotools.

Installing Waf

The first step in using Waf is to install it. I recommend checking it out from Subversion instead of downloading a tarball as the former usually is much more up to date and contains the latest bugfixes. As of this writing, the repository is at revision 4813. Create a checkout using:

$ svn checkout http://waf.googlecode.com/svn/trunk/ waf-read-only

To build waf, enter the waf-read-only directory generate the waf script:

$ cd waf-read-only
$ ./waf-light --make-waf

This creates an exectuable Python script called waf. This script should be copied to the root directory of any project you want to use waf for. It is also possible to install waf globally, although that is not recommended. The README file has more details.

The First wscript

Each build tool has its own kind of configuration files. make has Makefiles, autoconf has configure.in-files, scons has SConstruct files and so on. waf has wscripts. So the first step is to create an initial bare-bone wscript file. Copy this to a file named wscript in your project root directory:

srcdir = '.' blddir = 'build' def set_options(opt): pass def configure(conf): pass def build(bld): pass

Note that this a regular Python program and that you can enter whatever Python code you want in it.

Explanation

srcdir = '.'

The srcdir variable informs waf of where the source files are in relation to the wscript. Its value will almost always be '.' which is the current directory.

blddir = 'build'

This variable tells waf where to put the built files. By default, waf uses an out-of-tree build process, in contrast to most autotooled projects which are built in-tree, which means that waf will not litter your source directories.

def set_options(opt): pass

The set_options function is used for modifying the global options object passed in as the opt parameter. Here you can add extra options both for configuring and building.

def configure(conf): pass

This function performs the exact same task as the configure script in autotooled projects -- it checks that all the requirements for the software is fullfilled.

def build(bld): pass

build defines how the software is to be built. Target rules are specified here.

Running the Build

Provided that the waf program is in the same directory, it is now possible to run this empty build process:

$ ./waf configure
Configuration finished successfully (00:00:00); project is now ready to build.
$ ./waf build
Compilation finished successfully (00:00:00)

We haven't told waf what to do yet, so it outputs nothing interesting. waf creatures a directory called build/ which contains the result of the build process:

$ find build/
build/
build/config.log
build/c4che
build/c4che/build.config.py
build/c4che/default.cache.py
build/default
build/.wafpickle-6

Each built file is placed in build/default, the other files are just house-keeping for waf. Truth be told, this isn't all that interesting, but it gets better in the next section where we make waf actually do something useful.

wafing a Project

Lets talk about the imagined project we are using waf for for a while. We are creating a very simple shared library written in C complete with documentation, tests and example programs.

File Structure

Our project contains the following files and directories:

$ find
.
./wscript
./waf
./src
./src/hello.c
./src/hello.h
./src/wscript_build
./tests/test.c
./tests/demo.c
./tests/wscript_build
build/
As mentioned before, this is where waf puts the built files.
docs/
Project documentation.
src/
C sources and header files for the shared library.
tests/
Tests and example programs.

The hello.c and hello.h files are the C sources. First hello.c:

#include <stdio.h> #include "hello.h" void say_hi () { printf ("Ho\n"); }

And hello.h:

#ifndef HELLO_H #define HELLO_H void say_hi(); #endif

The other files will be introduced as needed in later sections.

Source Targets

Since this a C project, we must add configure checks to waf to ensure that the user has a C compiler installed before the project can be built. Lets begin with that:

def set_options(opt): opt.tool_options('compiler_cc') def configure(conf): conf.check_tool('compiler_cc')

These lines adds C compiler options and configure checks. Rerunning ./waf configure outputs the following:

$ ./waf configure
Checking for program gcc                 : ok /usr/bin/gcc
Checking for compiler version            : ok 4.2.4
Checking for program cpp                 : ok /usr/bin/cpp
Checking for program ar                  : ok /usr/bin/ar
Checking for program ranlib              : ok /usr/bin/ranlib
Checking for gcc                         : ok
Configuration finished successfully (00:00:00); project is now ready to build.

This proves that waf is now verifying that the user has a C compiler installed. Next we add target definitions for the source files in the src directory. Create a file called wscript_build to the src directory.

obj = bld.new_task_gen('cc', 'shlib') obj.source = ['hello.c'] obj.target = 'hello'

These three lines work similar to a build rule in a makefile. The file hello.c is used as input to build the shared library libhello.so. Note that waf automatically adds the prefix "lib" and the suffix ".so" because we are using Linux. To add this target to the build process change the build function in the wscript file:

def build(bld): bld.add_subdirs('src')

This instructs waf to add the rules from any wscript or wscript_build file located in src. The code serves the same purpose as the SUBDIRS variable in Makefile.am variables. Naturally, it is also possible to put all build information in the main wscript file but that might get messy because all paths gets longer. On the other hand, fewer files might feel less cluttered.

Let's test the build:

$ ./waf build
[1/2] cc: src/hello.c -> build/default/src/hello_1.o
[2/2] cc_link: build/default/src/hello_1.o -> build/default/src/libhello.so
Compilation finished successfully (00:00:00)

Excellent! waf first executed the compilation creating hello_1.o and then linked it to produce libhello.so. Both stored in build/default/src/:

$ ls build/default/src/
hello_1.o  libhello.so

Preprocessor Defines

A preprocessor define instructs the compiler to replace all subsequent occurences of a macro with specified replacement tokens. The canonical way to pass compile time options is to let the build system create a "config.h" header file that all C source files includes.

Using config.h

Let's say we wan't to add a function to our hello library that returns its version. First we add the necessary code to the wscript file to create the config.h file with the necessary define:

VERSION = '1.0.0' srcdir = '.' blddir = 'build' def set_options(opt): opt.tool_options('compiler_cc') def configure(conf): conf.check_tool('compiler_cc') conf.define('PACKAGE_VERSION', VERSION) conf.write_config_header('config.h') def build(bld): bld.add_subdirs('src')

The two new lines in the configure function creates the macro PACKAGE_VERSION and writes it to the config.h file. Configuring the project creates the config.h file in the build/default/ directory:

/* Configuration header created by Waf - do not edit */ #ifndef _CONFIG_H_WAF #define _CONFIG_H_WAF #define PACKAGE_VERSION "1.0.0" #endif /* _CONFIG_H_WAF */

By including config.h in hello.c, the macro can be used in the new function:

#include <stdio.h> #include "hello.h" #include "config.h" void say_hi () { printf ("Ho\n"); } const char * get_version () { return PACKAGE_VERSION; }

The function is also added to the header:

#ifndef HELLO_H #define HELLO_H void say_hi(); const char *get_version () #endif

The defines attribute

It is also possible to pass the defines as compiler arguments using gcc's -D option (or the equivalent for other compilers). Each target object in waf has a "defines" attribute where the defines are listed.

So to accomplish what we did in the previous chapter, without using a config.h file, change src/wscript_build to the following:

obj = bld.new_task_gen('cc', 'shlib') obj.source = ['hello.c'] obj.target = 'hello' obj.defines = 'PACKAGE_VERSION=\\"%s\\"' % VERSION

Note that an extra level of string escaping is needed to get the string literal correctly passed to the compiler from the shell.

Which ever way you prefer to use is a matter of taste. But when the list of defines grow large, the config.h method certainly becomes more pleasant. Even if it sometimes can increase recompile times.

Building GTK-Doc Documentation

Integrating the documentation building step into the build tool is usually tricky. End users usually does not need to rebuild the documentation and most documentation tools such as Javadoc and Doxygen does not integrate very easily into a target-based system.

Never the less, the popular documentation extraction tool GTK-Doc for GTK+ libraries contains a bunch of autotools macros so that it can be built using make. waf can do that too, ofcourse.

First, we have to write some documentation for our two functions in hello.c. GTK-Doc uses a pretty self-explanatory doc comment syntax reminiscent of Javadoc. We also add a third function, just for fun. The resulting hello.c and hello.c files are:

#include <stdio.h> #include "hello.h" /** * say_hi: * * Prints "Ho\n" on standard out. **/ void say_hi () { printf ("Ho\n"); } /** * get_version () * @returns: a string containing the version of this library. * * Returns a string on the format "major.minor.micro" describing which * is this librarys version. **/ const char * get_version () { return PACKAGE_VERSION; } /** * get_day_name: * @day_index: index of a week-day, in the range 0-6. * @sunday_first: one if a calendar with the week starting on Sunday * is desired, zero otherwise. * @returns: the name of the day, or %NULL if @day_index is out of * range. * * Returns the name of the day corresponding to the specified day * index. **/ const char * get_day_name (int day_index, int sunday_first) { const char *days[] = {"Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"}; if (day_index < 0 || day_index > 6) return NULL; if (sunday_first) day_index = (day_index + 6) % 7; return days[day_index]; }

and hello.h:

void say_hi (); const char *get_version (); const char *get_day_name (int day_index, int sunday_first); #endif

Adding Documentation Option

Autotooled projects employing GTK-Doc all have a --enable-gtk-doc configure option that determines whether the build process will build the documentation or not. It is easily added to waf by modifying the set_options function in the wscript file:

def set_options(opt): opt.tool_options('compiler_cc') opt.add_option('--enable-gtk-doc', action = 'store_true', default = False, help = 'use gtk-doc to build documentation ' + '[default: %default]')

Waf uses the standard library optparse module for parsing the command line, so any option format accepted by optparse will work for waf. See the documentation for that module for more information.

We can check that the extra option has become available:

$ ./waf configure --help
usage: waf [options] [commands ...]

* Main commands: configure build install clean dist distclean uninstall distcheck
* Example: ./waf build -j4

options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -j JOBS, --jobs=JOBS  amount of parallel jobs [Default: 2]
  -f, --force           force file installation
  -k, --keep            keep running happily on independent task groups
  -p, --progress        -p: progress bar; -pp: ide output
  -v, --verbose         verbosity level -v -vv or -vvv [Default: 0]
  --destdir=DESTDIR     installation root [Default: '']
  --nocache             compile everything, even if WAFCACHE is set
  -b BLDDIR, --blddir=BLDDIR
                        build dir for the project (configuration)
  -s SRCDIR, --srcdir=SRCDIR
                        src dir for the project (configuration)
  --prefix=PREFIX       installation prefix (configuration only) [Default: '/usr/local/']
  --zones=ZONES         debugging zones (task_gen, deps, tasks, etc)
  --targets=COMPILE_TARGETS
                        compile the targets given only [targets in CSV format, e.g. "target1,target2"]
  -d DEBUG_LEVEL, --debug-level=DEBUG_LEVEL
                        Specify the debug level, does nothing if CFLAGS is set in the environment. [Allowed
                        Values: 'ultradebug', 'debug', 'release', 'optimized', 'custom']
  --enable-gtk-doc      use gtk-doc to build documentation [default: False]
  ^<- here it is!

  C Compiler Options:
    --check-c-compiler=CHECK_C_COMPILER
                        On this platform (linux) the following C-Compiler will be checked by default: "gcc
                        suncc"

Python-devvers rock!

I don't know exactly why, but this thread made me really happy and warm inside. :) The people developing Python are the most pleasant free software developers imaginable.

Joining stringlists

Here is how you join a list of strings in elisp:

(mapconcat (lambda (x) x) load-path "\n")

It is equivalent to this Python code:

'\n'.join(stringlist)

Example of the LZW algorithm

This is probably old stuff for most of you, but here is my Python implementation of the LZW algorithm.

# The initial LZW tables that maps codes to strings and vice-versa. code_to_str = [chr(i) for i in range(256)] str_to_code = dict((chr(i), i) for i in range(256)) def compress(seq): ''' Returns an LZW compressed list of the input character sequence. ''' output = [] table = dict(str_to_code) s = '' for ch in seq: it = s + ch if it in table: s = it else: output.append(table[s]) table[it] = len(table) s = ch output.append(table[s]) return output def decompress(seq): ''' Returns a decompressed list of the LZW compressed input. ''' table = code_to_str[:] prevcode = seq[0] output = [] output.append(table[prevcode]) for code in seq[1:]: try: entry = table[code] except IndexError: # The lzw special case when code is not yet defined. entry = table[prevcode] entry += entry[0] output.append(entry) table.append(table[prevcode] + entry[0]) prevcode = code return output

Used like this:

data = open('somefile.txt').read() data = compress(data) print decompress(data)

Explanation for how the algorithm works can be found in other places on the net.

Grabbing URL:s with style

urlgrabber is a fairly nice Python package for grabbing URL:s. The documentation is kind of sparse, but reading the source is easy enough.

Here is a simple example:

from urlgrabber.grabber import URLGrabber
from urlgrabber.progress import TextMeter
    
grabber = URLGrabber(progress_obj = TextMeter())
grabber.urlgrab('http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tar.bz2')

Others mistakes

Seems like I am not the only one who has had problems with Python's evalution of default arguments :

class Namespace(object): def __init__(self, __ns={}, **kwargs): if kwargs: __ns.update(kwargs) self.__dict__ = __ns

This particular example was found in comp.lang.python .

Evaluation times in Python

One of Python's greatest strengths is its tendancy to reduce many of the common mistakes you do in other languages. But the language is not perfect, and it introduces its own bug sources. For example, try finding the bug in the following snippet:

class Vector: def __init__(self, x, y, z): self.x = x self.y = y self.z = z class Ray: def __init__(self, direction, origin = Vector(0, 0, 0)): self.direction = direction self.origin = origin ray1 = Ray(Vector(0, 0, -1)) ray2 = Ray(Vector(0, 0, 1)) ray3 = Ray(Vector(-1, 0, 0), Vector(2, 3, 4))

Does it look correct to you too?

It kind of does to me, but on closer inspection one finds that there is an "obvious" bug in there. The line:

def __init__(self, direction, origin = Vector(0, 0, 0)):

is wrong. Default arguments, such as the origin parameter is evaluated only when the module is loaded. Therefore, all Ray objects, whose origin Vector is created from the default argument, will share the same Vector instance.

>>> r1 = Ray(Vector(1, 2, 3)) >>> r2 = Ray(Vector(3, 2, 1)) >>> r1.origin.x = 33 >>> print r2.origin.x 33

That is certainly not what the programmer desired. Ray's __init__ must be fixed so that a new origin Vector is created every time the constructor is called with a default argument:

class Ray: def __init__(self, direction, origin = None): if origin is None: origin = Vector(0, 0, 0) self.origin = origin self.direction = direction

This is a fair bit uglier than the previous example. For me, these types of bugs are caused because I really, really do not like the if blaha is None: idiom. But I know it is time for me to just accept that you must never use mutable default arguments.

The behaviour is one of Python's warts which bites all newcomes to the language. Coming from languages where the norm is that function definitions are compiled, it is hard to grasp that Python actually executes them. The gotcha is documented in numerous sources:

So someone got the pretty logical idea that changing how default arguments is evaluated should be changed. It would both help newbies, reduce many lines of code and help developers write more correct code.

The idea was brought up on the Python-3000 mailing list. But was quickly rejected by the BFDL since it was a too big change and breaks orthogonality with class variables. And that was that. Well, there was also a few who even defended the current behaviour and thought that changing it would not help anyone. It must have been a long time since these people read introductionary material to Python and encountered the "do not use mutable default argument values!" warning. :)

But the proposed semantical change definitely was to big and it would break orthogonality with other language features. I just hope that someone else will invent a new and better way to solve the problem. Python's incapability of handling mutable default values nicely is a big wart that someone hopefully can find a solution for.

Quick guide to PyGTK on Windows

I found this to be non-trivial. Here are the steps you need to go through to get PyGTK to work in Windows:
  1. Download the Python 2.5 MSI installer. Direct link

    Install it to C:\Python25 and do not forget to add C:\Python25\bin to your PATH.

  2. Download the PyGTK EXE installer. Direct Link
  3. Download the GTK EXE installer. Direct link

    Install it in C:\GTK. It should automatically add C:\GTK\bin to your PATH. If it does not, do it manually.

  4. Download the PyCairo EXE installer. Direct link.

  5. Download the pygobject EXE installer. Direct link.
Maybe it is not that complicated. :)

AVL Tree in Python

Here is an AVL tree implemented in Python! Note that it is supposed to be "beautiful" and easy to read and not efficient, that is why both balance() and height() are recursive. I learnt the algorithm from C++ an Introduction to Data Structures which is an excellent book that I recommend. I hope this snippet is useful to someone:
# -*- coding: utf-8 -*- class Node: def __init__(self, data): self.data = data self.set_childs(None, None) def set_childs(self, left, right): self.left = left self.right = right def balance(self): lheight = 0 if self.left: lheight = self.left.height() rheight = 0 if self.right: rheight = self.right.height() return lheight - rheight def height(self): lheight = 0 if self.left: lheight = self.left.height() rheight = 0 if self.right: rheight = self.right.height() return 1 + max(lheight, rheight) def rotate_left(self): self.data, self.right.data = self.right.data, self.data old_left = self.left self.set_childs(self.right, self.right.right) self.left.set_childs(old_left, self.left.left) def rotate_right(self): self.data, self.left.data = self.left.data, self.data old_right = self.right self.set_childs(self.left.left, self.left) self.right.set_childs(self.right.right, old_right) def rotate_left_right(self): self.left.rotate_left() self.rotate_right() def rotate_right_left(self): self.right.rotate_right() self.rotate_left() def do_balance(self): bal = self.balance() if bal > 1: if self.left.balance() > 0: self.rotate_right() else: self.rotate_left_right() elif bal < -1: if self.right.balance() < 0: self.rotate_left() else: self.rotate_right_left() def insert(self, data): if data <= self.data: if not self.left: self.left = Node(data) else: self.left.insert(data) else: if not self.right: self.right = Node(data) else: self.right.insert(data) self.do_balance() def print_tree(self, indent = 0): print " " * indent + str(self.data) if self.left: self.left.print_tree(indent + 2) if self.right: self.right.print_tree(indent + 2) if __name__ == "__main__": tree = Node(5) tree.insert(7) tree.insert(9) tree.print_tree()
I just had to update this so that it would use COLORS!

Bloggarkiv