Building Stackless Python on a PS3 (PPC Linux)

I have recently gotten my hands on a PlayStation 3 that I am trying to use as a surrogate[1] in my Scavenger system. The first thing I did was to install Ubuntu Linux 9.10 on it, and the next step was then getting Stackless Python to compile... Which turned out to be a bit of a bitch ;-)

But after some heavy debugging and googling I found out how to do it:

1. Download the Stackless source distribution. I used version 2.6.4.


2. Unpack it somewhere and enter the newly created directory.


3. $ ./configure --prefix=/where/you/want/it
   


4. Edit the file called "Makefile" and change the variable SLPFLAGS to "SLPFLAGS=-fomit-frame-pointer -O2"


5. make


6. make install

It's very simple actually, just took me some time to find the solution :-) By the way, if you do not edit the Makefile you get the following error:

./Stackless/core/slp_transfer.c: In function 'slp_transfer':
./Stackless/core/slp_transfer.c:144: error: 31 cannot be used in asm here
./Stackless/core/slp_transfer.c:144: error: 31 cannot be used in asm here
make: *** [Stackless/core/slp_transfer.o] Error 1

Looking at that error the 31 hints at that the save registers code for PPC is trying to save register r31. This is reminiscent of the error you get when compiling Stackless under Mac OS X, but in this case adding the --enable-stacklessfewerregisters option does not work.

[1]: A surrogate in a cyber foraging system is a strong machine that performs heavy work on behalf of the weaker (often mobile) clients.

New Presence2 Beta

I have just released the fourth Presence2 beta. In it I have fixed some serious bugs, so if you are using the older betas please upgrade now ;-)

Grab the files from the usual place.

Binding keyboard shortcuts to Subclipse commands in Eclipse

I have just created a small screencast about how to bind keyboard shortcuts to Subclipse commands within Eclipse. Now I can add, commit, and diff files without having to use my mouse - and I'm using the same Emacs shortcuts that I know and love :-)



B.t.w., it is cool how products like iShowU HD makes it incredibly easy to create screencasts. I've probably spent more time writing this blog entry than actually creating and uploading the video ;-)

Presence2 beta 2 released

I found some bugs in the Python lib for the new version of Presence. These have now been fixed and therefore a second beta has been released. Grab it at the Presence Google Code page.

TeXlipse build that actually works...

I am currently writing an article - which under normal circumstances means that I am working in Emacs (Aquamacs f.t.w.!) using AUCTeX, RefTeX, flyspell etc. But this time around I wanted to try something new, so I installed the TeXlipse plugin for Eclipse and started writing.

I quickly faced a major problem though: If you, like any sane person, split up your document into smaller files and then \include or \input these files in some master file, then the auto completion of BiBTeX references does not work in TeXlipse. This is a known bug - it has been known for almost six months, and yet they haven't released a bugfix release!

I have therefore built TeXlipse from CVS instead, and that seems to be working so far. You can fetch this build here. It was built using the CVS head as of today...

Note: This build may have other bugs - it is _just_ a build of the TeXlipse CVS head.

Presence 2.0.0-b1 Released

I have just released the first beta of Presence2 (or qPresence as I also call it because of its Qt dependencies). I have put up downloadables for the daemon, the Python API, and the "daemonless" Python API (the one that allows for listening in on Presence announcements without running the Presence daemon).

Because of the new Qt dependencies you need to install the Qt library and SDK in order to build the daemon. You can fetch these at Nokia's Qt page.

Grab 'em while they're hot!

Byte Order Conversion in Qt

For my recent re-writing of Presence to use the Qt library, I needed to be able to do byte order swapping on two and four byte integers (short and int). This I used when serializing/marshalling/packing stuff up before transmitting them over the wire. I know that the "right" way of doing that in Qt would be to use QDataStream, which will then handle stuff like byte order conversion, but I needed to be able to pack my own data, so that a QDataStream-capable reader was not required on the other side of the socket connection.

Byte order conversion is done using the four functions (macros really) called htons, ntohs, htonl, and ntohl that may be found in inet/arpa.h on any Unix system. So of course I included that file and went happily on with my coding. But... when I tried to compile my new version of Presence under Windows I realized that inet/arpa.h does not exist on Windows. It probably resides in winsock.h I guess, but I _really_ do not want to have platform specific code in my project, so I decided to implement them myself instead :-)

Using Qt you can ask for your systems endianness by using the QSysInfo class, so a portable implementation of the needed functions is easy to do:

#define BYTE_SWAP4(x) \
    (((x & 0xFF000000) >> 24) | \
     ((x & 0x00FF0000) >> 8)  | \
     ((x & 0x0000FF00) << 8)  | \
     ((x & 0x000000FF) << 24))

#define BYTE_SWAP2(x) \
    (((x & 0xFF00) >> 8) | \
     ((x & 0x00FF) << 8))

quint16 _htons(quint16 x) {
    if (QSysInfo::ByteOrder == QSysInfo::BigEndian) {
        return x;
    }
    else {
        return BYTE_SWAP2(x);
    }
}

quint16 _ntohs(quint16 x) {
    if (QSysInfo::ByteOrder == QSysInfo::BigEndian) {
        return x;
    }
    else {
        return BYTE_SWAP2(x);
    }
}

quint32 _htonl(quint32 x) {
    if (QSysInfo::ByteOrder == QSysInfo::BigEndian) {
        return x;
    }
    else {
        return BYTE_SWAP4(x);
    }
}

quint32 _ntohl(quint32 x) {
    if (QSysInfo::ByteOrder == QSysInfo::BigEndian) {
        return x;
    }
    else {
        return BYTE_SWAP4(x);
    }
}

Note that the functions have a '_' prepended - otherwise some compilers may get confused.

Using these functions I do not need to have any platform specific code in my project. This comes at the cost of actually doing a function call when byte order conversion is needed, as opposed to simply using a macro. So if your code is very performance critical you should not use this code - but then again, if your code is that performance critical you should probably not be using C++ ;-)

Complete rebuild of Presence

I have spent the past week re-writing Presence from scratch using Qt. "Why?" you may ask. The previous version of Presence has been working very well for a long time, but it had a bug that only appeared on 64-bit operating systems. I spent a small amount of time debugging the old code base to find that bug, and then finally decided that a complete re-write would be better. Not that the old code base was that horrible, it was just written in a mixture of C and C++ that made it a bit hard to maintain. E.g., the network related code was written in pure C using a large select() statement and multiplexing over a large number of sockets - it seems obvious that this could be coded _much_ nicer using an event based architecture.

So I decided to recode it in Qt, and let me tell you: Qt = happiness! I am no fan of C++, but when using Qt it is actually a nice language to work with. The new version of Presence is easily maintainable, because it is so simple to read and understand Qt code (at least compared to reading and understanding C). No more waiting for data on a bunch of sockets, finding out which socket has activity, forwarding the task of handling the activity to someone else, and then back to the listening... In Qt all socket activity can of course be subscribed to as callbacks, so now the Announcer class takes care of all announcement packets, the ControlConnection class talks to application(s), and so forth.

In the process of porting the daemon I have altered the protocol slightly, which means that I also had to update the Python libraries. In doing this these have also been redesigned, making them a bit easier to read as well. They both work now, though I must admit that I am not completely satisfied with the main Python lib - but seeing as it works I will probably not get to updating it any time soon.

Controlling PDF Minor Version in LaTeX

I just had to submit a paper to a conference that was using some web based submission system. As part of that system, the PDF version of the paper I submitted was validated. The system refused my paper because it was seemingly a version 1.4 PDF (w.t.f. do I know about PDF versions?). I spent some time trying to update my LaTeX installation on my Mac (didn't work), compiling the paper on my university's Linux machines (also didn't work), and then I hit Google...

It seems that you can control the PDF minor version by using this magical LaTeX command

\pdfminorversion=5

This will make pdflatex create a version 1.5 PDF document (you can also create version 1.6 and 1.7 PDFs, but 1.5 was all I needed).

Native Mac PyGTK for Snow Leopard

I have just made a fresh build of native PyGTK for Mac on my new Snow Leopard install. It seems to be working - I have only tested my simple hello_pygtk.py script, and that works at least ;-)

You can grab the new build here: MacPyGTK_SnowLeopard_091809.dmg

Remember to follow the three simple instructions in README.txt in order to make it work. B.t.w., one has to be running Python in 32-bit mode for it to work. This can be done in two ways:

1. By setting this environment variable VERSIONER_PYTHON_PREFER_32_BIT=yes
2. By issuing this command (once): defaults write com.apple.versioner.python Prefer-32-Bit -bool yes

I would prefer setting the environment variable in a script starting my PyGTK application. That way I am changing as little as possible in the standard setup of the machine...

Piping output of shell commands into the clipboard

I love my terminal! The availability of a proper shell and set of *nix tools is one of the main reasons that I dared switch to Mac OS X two years ago. I have been used to having the full GNU tool chain on the Linux systems I have been using the last 10 years, and there is no way I will ever give that up :-)

Today I learned something new about my terminal on OS X; I learned about the pbcopy and pbpaste commands. These two shell commands copy shell output into the clipboard, and pastes the clipboard content into the shell, respectively. I.e., today I needed to grep+cut through some large log files, and the results of these commands I wanted to paste to a spreadsheet. Instead of having to 1) perform the command, and then 2) use the mouse to scroll up and select all of the results to copy-paste them into the spreadsheet, I found that I could use pbcopy to pipe stdout directly to the clipboard:

grep foo logfile.txt | cut -d " " -f1,2,4 | pbcopy

After running this I could just go to my spreadsheet application and insert the three selected rows.

This little command is a real time saver I can tell you! I must have saved at least an hours work today alone ;-)

Fresh PyGTK build for Leopard/Intel

I have just built a fresh version of native Mac OS X PyGTK and all of its dependencies. This time I have only made an Intel build for Leopard, so no PPC version this time around.

You can grab the new dmg file here: MacPyGTK_Leopard_Intel_091309.dmg

The install procedure is the usual:

1. Download and mount the dmg
2. Drag the MacPyGTK folder to /Developer
3. Copy the contents of site-packages into the site packages of your Python installation (on 10.5 this should be /Library/Python/2.5/site-packages)
4. Test that it works by running the hello_pygtk.py file with your Python interpreter.

That's it - now you should have a fully functional PyGTK installation to play around with.

New Logo for Locusts?

I have for some time now been searching for a nice picture of a grasshopper to use as a logo for the Locusts project that I am working on. I never found one that I was satisfied with though, but yesterday I got lucky. While I was out walking in Lappeenranta this little fellow was sitting on the side of the road - and he really liked being photographed.

B.t.w., I edited this image using Pixelmator to remove the background. This little app is really turning out to be an excellent image editor - who needs Photoshop anyway? :-)

Excited about Scavenger...

Just wanted to let you know that I have posted something about Scavenger on my "work" blog. I have conducted some benchmarks of Scavengers parallel processing abilities and it is looking very promising I must say.

Making your OS X hostname stick

If you are both a Mac user and a terminal junkie, as I am, you will probably have noticed that your computers hostname changes whenever you enter a new network environment[1]. More precisely, when you join a network through a DHCP server in some cases it also assigns you a new hostname. This has been a major annoyance to me for some time now, so I finally decided to find out how to make my hostname stick. And, as it turns out, that is of course quite simple to do:

1) Issue the command "sudo vi /etc/hostconfig"
2) Add the line HOSTNAME=my-hostname to this file (pressing i to enter edit mode).
3) Save and close (that's Esc to quit editing mode, and then :wq + Enter for you vi rookies)

The HOSTNAME line may already exist on your system in which case its value will be "-AUTOMATIC-". In that case you just replace that with your hostname. It was not there on my Leopard (client) machine.

Oh yeah, thanks to Robert Brewer for teaching me how to do this. Take a look at his article if you are interested in getting some more detailed information about the whole hostname determination business on OS X.

[1]: For those of you not used to using a terminal a new command-line in a terminal typically starts with "username@hostname:top-level-path$". E.g., my prompt right now shows "madsk@leela:Development$".

Scavenger tutorial available

I have finally gotten around to writing some documentation of Scavenger, so that others may in fact start writing Scavenger-enabled applications. I have spent the entire day updating the wiki pages of Scavengers GoogleCode homepage so now you will find a tutorial there along with some other related information about scavenger. Enjoy!

Python in the Android Emulator

I have recently started playing around with the Android platform; more specifically I have begun experimenting with Python development on Android through the Android Scripting Environment (or ASE, as its called).

In ASE you are able to run Python 2.6.2 scripts that, as far as I can tell so far, have full access to the underlying Python libraries. Furthermore, through the "android" module, you gain access to some Android specific things like location services, sensor data, sms messaging, and more.

One problem with ASE though is the fact that it is impossible to run a Python program within the Android Emulator. When you try to start a Python script all you get is a black screen. This is due to some sensor-thingy not being available in the emulator, and the suggested fix is to rebuild ASE with the sensor service disabled.

To spare others of the pain involved rebuilding ASE from source (it is really not that hard, just takes some valuable time away from coding in Python), I will share my build of ASE that actually works in the emulator. You can download it here: ASE Emulator build (do not use this on a real device!). This build is provided as is; I will not be held liable if it blows up your computer, burns down your house, or runs away with your wife ;-)

Scavenger demo application released!

I have just released the first Scavenger demo app: AugIM - the Augmented Image Manager. AugIM is an old demonstrator that I made a long time ago, and now I have ported it to use Scavenger for its cyber foraging tasks.


AugIM is an image browser/editor for Maemo (running on Nokia's N800/N810 devices) that allows you to do simple image manipulation tasks such as sharpening, resizing, adjusting color/contrast/brightness, rotating, flipping etc. It also allows you to share your images with the world through email or Google Picasa Web. The N800 and N810 devices are capable of performing these services on their own, but it takes a long time for them to do so and it consumes a very large amount of energy. Using Scavenger, AugIM makes sure that heavy processing is offloaded to larger surrogates whenever possible, which means that the image manipulation tasks are performed faster and using less energy on the mobile device.

Yet another Scavenger build...

I have just uploaded a new self-contained Scavenger daemon build to Scavengers Google Code page. This build is for Mac OS X 10.5 (Leopard) and it contains the same components as the Ubuntu Linux version I released yesterday.

From your Mac simply download the archive, unpack it, and start the start_daemon script from a terminal. Then a Scavenger daemon should be running on your machine, and you are ready to start playing with cyber foraging! :-)

Scavenger build available for Ubuntu 9.04

I have finally gotten around to creating a Scavenger build that you can use to easily experiment with Scavenger, without having to install Stackless Python and other dependencies (e.g., Presence).

The build, which is created and tested to run on Ubuntu 9.04, is available here. All you need to do is 1) download the archive, 2) unpack it, and 3) run the "start_daemon" script.

The "bundle" contains Stackless Python 2.6.2, the Presence python lib and daemon, PIL (Python Imaging Library), Google Data API, and the Scavenger daemon.

Timelapse photography using CHDK

I thought it could be fun to do some time-lapse photography now that I have a programmable camera. So I have quickly whipped up a small time-lapse script that works on my Ixus 750:

@title Timelapse
@param a Number of steps 
@default a 30
@param b Period (seconds)
@default b 60

rem Sanity check.
if a<1 then let a=1
if b<5 then let b=5
let b=b*1000

rem Take the pictures!
for s=1 to a+1
    rem Take a single picture.
    print "Shoot", s, "of", a
    press "shoot_half"
    sleep 2500
    press "shoot_full"
    release "shoot_full"
    release "shoot_half"
    print "shot done"

    rem Sleep until its time to take the next picture.
    sleep b
next s

At the standard setting it takes thirty pictures in roughly 30 minutes.

CHDK oddities

The script I posted in my last post, the one that does exposure bracketing, used to work - I swear it did! But all of a sudden it decided to stop working, and I can't find an good explanation for that at all ;-)

But, instead of an explanation i offer you a fix. The following script _does_ work on my Ixus 750 camera (right now it does anyway):

@title EXP bracketing 
@param a Number of +/- steps 
@default a 2
@param b Step size (1/3EV)
@default b 3

if a<1 then let a=2
if b<1 then let b=3

print "Preparing..."
for n=1 to a*b
    click "left"
next n

for s=1 to a*2
    print "Shoot", s, "of", a*2+1
    press "shoot_half"
    sleep 2500
    press "shoot_full"
    release "shoot_full"
    release "shoot_half"
    print "shot done"
    sleep 5000
    for n=1 to b
        click "right"
    next n
next s

print "Shoot", a*2+1, "of", a*2+1
press "shoot_half"
sleep 2500
press "shoot_full"
release "shoot_full"
release "shoot_half"
sleep 5000

print "Finalizing..."
for n=1 to a*b
    click "left"
next n

CHDK on an Ixus 750

While on paternity leave I have been playing around with my Canon Ixus 750 digital camera. Primarily I have been doing macro (close-up) shots of insects etc., but I also wanted to experiment with other kinds of photography, e.g., High Dynamic Range images (HDR). In HDR you take pictures of the same scene at different exposure levels, and after that a computer program stacks these images to create the HDR image. When taking such pictures it is vital that you do not move the camera, which can be quite hard even if you have a stand for it. I was therefore looking for a software solution to HDR, that would automatically take 3 to 5 pictures at different exposure levels for me, so that Idid not need to touch the camera in between shots.

For that I installed CHDK on my Ixus 750 and I must say that I am impressed. Using the CHDK firmware extension a horde of new features are unlocked (for example the possibility to take RAW images instead of JPEGs). And most importantly CHDK is scriptable, so you may write your own scripts to control your camera.

Based on an example in the tutorial I quickly wrote an HDR script that works on an Ixus 750. When I activate this script my camera snaps five pictures at -2, 1, 0, +1, +2 exposure.

@title EXP bracketing 
@param a Number of +/- steps 
@default a 2
@param b Step size (1/3EV)
@default b 3

if a<1 then let a=2
if b<1 then let b=3

sleep 1000

print "Preparing..."
rem click "erase"
for n=1 to a*b
    click "left"
next n

click "shoot_half"
sleep 3000
for s=1 to a*2
    print "Shoot", s, "of", a*2+1
    click "shoot_full"
    print "shot done"
    sleep 5000
    for n=1 to b
        click "right"
    next n
next s

print "Shoot", a*2+1, "of", a*2+1
click "shoot_full"
sleep 5000

print "Finalizing..."
for n=1 to a*b
    click "left"
next n
rem click "erase"

end

In order to use this script the camera must be in manual mode and the exposure menu must be opened...

Controlling the firewall on Mac OS X 10.4 (Tiger) from the command line

I have just had the need to control (actually disable) the firewall on a Mac running OS X 10.4 (Tiger). This should be no problem; just start System Preferences go to Sharing... The problem was that the only connection I had to the machine was through SSH, so going to System Preferences was not an option.

Luckily the firewall can be controlled with the ipfw command on the command line. Take a look at this article at macdevcenter for an example of how to use ipfw.

And, to disable the firewall all that is needed is thus:

sudo ipfw disable firewall

Update: To disable the firewall on Mac OS X 10.5 (Leopard) you can issue the following command:

sudo defaults write /Library/Preferences/com.apple.alf globalstate -int 0

Scavenger - Now With History Based Profiling

I have just added yet another scheduler to Scavenger (yes, there is three of them now). This scheduler is the "smartest" of the three as it uses history based profiling in order to be able to choose the right surrogate for the job. History based profiling in a cyber foraging system is nothing new - the Chroma system used that some 6-7 years ago. What is new is the way that the profiling information is used within the Scavenger system. In Scavenger the profiling information can be used not only to tell you something about how the task will perform on certain surrogates (i.e., the ones where you have earlier performed the task), it is also capable of making an informed estimate of the running time on an unknown surrogate... more on that later ;-)

Building the Python Imaging Library (PIL) for iPhone/iPod Touch

As my last post hints at I have started experimenting with building PIL for my iPod Touch. Doing so is non-trivial and I have therefore decided to write a small tutorial here, so that others may be spared of wasting their time on this project ;-)

What I did was to compile the thing directly on the iPod, and to do that I went through the following steps:

1. Jailbreak: I used QuickPWN 2.2.5 under Windows (it did not work on my Mac for some reason) and that was quite easy.
2. Install SSH on the iPod using Cydia (there is an entry in the main menu showing you how to do that).
3. SSH in to your iPod as the root user with password "alpine".
4. Install Python, the gcc compiler, the code signing tool, and make by issuing this command: apt-get install python iphone-gcc ldid make (alternatively you could install it using Cydia it you like).
5. Download the SDK headers, you may find them along with a description of how to use them here.
6. [Optional] Download the libjpeg source if you want jpeg support. Untar the source bundle, enter the directory containing the source and issue the following commands:

   export C_INCLUDE_PATH=/var/include/
   ./configure --prefix=/usr/
   make
   make install
   make install-lib

7. Download the PIL source and unpack it.
8. Enter the source directory and issue the following commands:

   python setup.py build
   python setup.py install
   ldid -S /usr/lib/python2.5/site-packages/PIL/*.dylib

   
   

After going through all of these steps I ended up with a working PIL library on my iPod. Oh, and thanks to Renaud Blanch for pointing out to me that I needed to sign the dynamic libraries for them to work.

Scavenger on iPhone/iPod Touch

The first thing I did after implementing the new Presence lib was of course to install Scavenger on my iPod Touch. And I am happy to report that it works! :-)

The Python version installable on a jailbroken iPhone/iPod is version 2.5.1, and since Scavenger is developed using Python 2.6 in a "backwards compatible" way everything worked like a charm.

The next step is getting the Python Imaging Library working on the iPod...

New Presence Consumer Lib

I have just created a new Presence library - one that does not depend on the Presence daemon. This lib may be used by Python applications that only act as consumers of Presence services, i.e., application that do not advertise services through Presence but merely uses services announced by other peers.

So why did I make this new consumer-only lib? Because it enables devices that are not capable of running the Presence daemon to use Presence anyway (as a client/consumer right now). This means for example, that I can discover and use Presence services from my jailbroken iPod.

The new lib can be found in /trunk/libs/python_consumer_lib.

More about the __cmp__ method in Python

I just stumbled upon this little something about overloading the __cmp__ method in a Python class. If you are not careful, you will end up having a class whose instances can not be compared to None (as I always do to see if some optional parameter has been passed to a method or function). Lets take a small example:

class Foo(object):

def __init__(self, x):

    self._x = x

def __cmp__(self, other):

    if type(other) == Foo:

        return cmp(self._x, other._x)

    else:

        raise TypeError("Foo cannot be compared to %s"%str(type(other))) f1 = Foo(1)

f2 = Foo(2)

print 'f1 is smaller than f2:', f1 < f2 # True

print 'f1 is larger than f2:', f1 > f2  # False

print 'f1 is equal to f2:', f1 == f2    # False

print 'f1 is None:', f1 == None         # Crash!

This code crashes in the final statement, because the comparison to None calls the __cmp__ method, and it has no way of comparing to None. This could of course be fixed by adding "if other == None: return -1" but the right way to fix this would be by overloading another method called __eq__ that is used to test for object equality.

class Foo(object):

def __init__(self, x):

    self._x = x

def __eq__(self, other):

    if type(other) == Foo:

        return cmp(self, other) == 0

    else:

        return False

def __cmp__(self, other):

    if type(other) == Foo:

        return cmp(self._x, other._x)

     else:

         raise TypeError("Foo cannot be compared to %s"%str(type(other)))



f1 = Foo(1)

f2 = Foo(2)

print 'f1 is smaller than f2:', f1 < f2 # True print 'f1 is larger than f2:', f1 > f2  # False

print 'f1 is equal to f2:', f1 == f2    # False

print 'f1 is None:', f1 == None         # False

Now the comparison to None (or in fact to any object that is not of type Foo) is caught by the __eq__ method, and comparisons to other type Foo objects are correctly forwarded to the __cmp__ method.

Emacs ido-mode (minor mode)

I have just spent some time play around with the Emacs "InteractivelyDoThings"-mode; ido-mode. Ido-mode replaces your regular "find-file" and "switch-buffer" mini-buffer with a much more flexible version; a version where you are allowed to do fussy matching, tab-complete your way to opening files on remote servers, and much much more...

Check out this screencast by Stuart Halloway if you have the time, or just dive into this EmacsWiki-page for a short explanation of what ido-mode does for you.

The "with"-statement in Python

In Python 2.5 and onwards a new "with" statement has been introduced. I recently read a really great article by Fredrik Lundh about the inner workings of this statement, so please read it if you are interested in all the dreary details. The syntax of the with statement is shown below:

with expression [as name]
    code block

What it boils down to is this: The with statement expects its expression to evaluate to an object that has two methods: __enter__ and __exit__. These will then be called before and after executing the code within the statement. To quote the article "Now, when the “with” statement is executed, Python evaluates the expression, calls the __enter__ method on the resulting value (which is called a “context guard”), and assigns whatever __enter__ returns to the variable given by as. Python will then execute the code body, and no matter what happens in that code, call the guard object’s __exit__ method. "

This means, that you can use the with statement for other things than merely locking (which is probably the mostly know use). Consider the code below:

with open('foo.txt', 'r') as infile:
    print infile.read()

This would open the file, assign it the name infile in the statement code block, read and print the file, and finally close the file again. This is all because a file objects __enter__ method returns the opened file, and its __exit__ method closes the file. Pretty neat, huh? :-)

Reading battery status on an N800/N810

For one of my projects I need to be able to read out the energy levels of the battery on my Nokia Internet Tablet (N810) - and I need to be able to do so from within a Python program, so that I may continuously monitor the energy usage of my program. After reading this blog post and talking to my friend Jussi, who had spent some days implementing such functionality in a C++ program of his, I spent about five minutes porting it to Python (thanks Jussi!). The result is shown below:

import dbus

bus = dbus.SystemBus()
hal_obj = bus.get_object ('org.freedesktop.Hal', 
                          '/org/freedesktop/Hal/Manager')
hal = dbus.Interface (hal_obj, 'org.freedesktop.Hal.Manager')
uids = hal.FindDeviceByCapability('battery')
dev_obj = bus.get_object ('org.freedesktop.Hal', uids[0])

print 'charge level percentage',\
      dev_obj.GetProperty('battery.charge_level.percentage')
print 'charge current', dev_obj.GetProperty('battery.reporting.current')
print 'charge design', dev_obj.GetProperty('battery.reporting.design')
print 'charge last full',\
      dev_obj.GetProperty('battery.reporting.last_full')
print 'charge unit', dev_obj.GetProperty('battery.reporting.unit')
print 'voltage current', dev_obj.GetProperty('battery.voltage.current')
print 'voltage design', dev_obj.GetProperty('battery.voltage.design')
print 'voltage unit', dev_obj.GetProperty('battery.voltage.unit')

This little snippet of Python code talks to the HAL layer through D-Bus, and in that way is capable of coming up with quite a lot of information about the battery. Check the complete HAL battery API for more information about what can be read through these calls.

Excellent article about securing Python

It seems that there is in fact people out there that are actively trying to create a secure Python; as in a Python that can be used to execute untrusted code within, without endangering your trusted code (or filesystem for that matter).

I was google "securing python" out of interest, to see if something has happened in this respect since the time when I was looking into it when designing my Scavenger system, and it seems that there have been some important progress since then. I found this great article by Tav (Vageesan Sivapathasundaram). It describes how a security system based on object capabilities can be built (and is in fact being built) within the Python interpreter. Some of the ideas are reminiscent of my approach - but what they are doing here is way cooler than what I was playing around with ;-)

By the way they are having a hack-attack, exactly like I did when I was testing Scavenger's security system :-)

Building Stackless Python on a Mac

I use Stackless Python for some of my work (and also for my open source project Scavenger). Seeing as I do most of my development on a Mac I have a number of times had the need to build Stackless Python from source. Doing that there is one thing that you should know: you need to add the configure option called --enable-stacklessfewerregisters, otherwise you will get the following error when compiling:

./Stackless/platf/switch_ppc_macosx.h:51: error: PIC register 'r31' 
clobbered in 'asm'

YASnippet LaTeX bundle

I have started using YASnippet for Emacs and one thing that I immediately noticed was the shocking lack of LaTeX snippets. Yes, I know that the auxtex package lets you do everything by using some keyboard shortcuts, but I find it much more intuitive working with tab completed snippets. I have therefore begun a project of creating a proper LaTeX bundle for YASnippet - you may find the snippets I create here [old link!].

Update: The snippets have been moved to a GitHub project - you may find it here http://github.com/madsdk/yasnippets-latex/

It is still a work in progress, but as soon as I find my bundle to be complete enough I will start releasing versioned "builds". As of right now you have to download the snippets individually from the download section of the homepage.

Great article about the Python GIL

I just stumbled upon this article about the Global Interpreter Lock in Python by Jesse Noller.

I was benchmarking my Stackless Python execution environment to see whether it was actually able to utilize my multi-core CPU fully - and it wasn't... In it I spawn a thread per CPU core and thus maintain separate stackless schedulers so that I may schedule multiple tasklets for execution at the same time - or so I thought :-)

The article by Jesse Noller is very nice, and I suggest that anyone interested in using threading in Python to do heavy lifting should read it. If you are using threads solely to get more responsiveness when doing blocking I/O calls there is no need to read it; just continue doing what it is you are doing ;-)

More Python YASnippets

I found that snippets were missing for try- and with-statements as well, so I created some:

# contributor: Mads D. Kristensen 
# name: try
# --
try:
    $0
except ${1:Exception}, ${2:e}:
    pass

# contributor: Mads D. Kristensen 
# name: with
# --
with ${1:lock}:
     $0

These are very simple snippets but I find them immensely useful and I hope you will too :-)

Python property snippet for YASnippet

I have recently begun using YASnippet for Emacs (AquaMacs on Mac actually). YASnippet adds powerful snippet support to Emacs reminiscent of the snippet support in TextMate. Using YASnippet you may for example in Python mode type "class" and then hit the Tab-key which would expand into a class-definition snippet where all of the boiler-plate code has already been written for you.

While YASnippet is quite powerful is does lack a number of standard snippets (or templates) in its standard library. For example I was missing a proper Python property snippet, so I wrote one:

# contributor: Mads D. Kristensen 
# name: prop
# --
def ${1:foo}():
   doc = """${2:Doc string}"""
   def fget(self):
       return self._$1
   def fset(self, value):
       self._$1 = value
   def fdel(self):
       del self._$1
   return locals()
$1 = property(**$1())

$0

Using this snippet you can write "prop" and press the Tab-key. Now you only need to provide a name for the property and, if you like, a doc string for the property. The snippet then creates a getter, setter, and deleter method for the property.

New blog

I have decided to move part of my blogging to this blogspot.com site instead of my original blog. My old blog will still hold all my research related blogging, but any tech/development/personal blogging will from now on reside on this blog.

The first thing I will do, when I get the time, is to migrate old (relevant) blog posts from my old blog to this new site.

By the way, I will be writing on this blog in both English and Danish. I have therefore created both an "english" and a "danish" category, and I will try to remember to apply one of these to all posts. This way you may e.g. create an English-only feed if you should so desire.

Comparing and the __cmp__ method in Python

Sometimes it would be nice to be able to ask questions such as “does this array hold an object that has its instance variable ‘foo’ set to 42?” in a simpler way than by iterating through the entire array whilst comparing each objects ‘foo’ attribute to 42. This can be done quite easily in Python by overloading the __cmp__ method in your classes. The following example shows how that works in practice:

class Foo(object):
    def __init__(self, a_string, an_integer):
        self.my_string = a_string
        self.my_integer = an_integer

    def __cmp__(self, other):
        if type(other) == type(self):
            return cmp(self.my_string, other.my_string) and\
            cmp(self.my_integer, other.my_integer)
        elif type(other) == int:
            return cmp(self.my_integer, other)
        elif type(other) == str:
            return cmp(self.my_string, other)
        else:
            raise TypeError('Unable to compare %s to %s'%(type(self), type(other)))

f1 = Foo('hello', 1)
f2 = Foo('world', 2)
f3 = Foo('!', 3)
my_list = [f1, f2]
print f1 in my_list # Prints "True"
print f3 in my_list # Prints "False"
print 1 in my_list  # Prints "True"
print 42 in my_list # Prints "False"
print 'hello' in my_list  # Prints "True"
print 'mojn' in my_list  # Prints "False"
try:
    (1+0j) in my_list # Raises a TypeError
except TypeError, e:
    print e.message