Kedelig Data

Python: Beware of standard parameter values

2011-07-05T13:04:00.002+02:00

I just spent an hour hunting for a strange bug in my Python code, and once I found it it made perfect sense (it often does when one is on that side of a bug hunt), but it was still obscure enough that I want to share it with you.

The bug occurred because of my (wrong) use of standard parameter values. Look at the following piece of code

class Foo(object):
    def __init__(self, mylist = []):
        self._mylist = mylist
    
    def addSomethingToMyList(self, something):
        self._mylist.append(something)
        
    def __str__(self):
        return str(self._mylist)
        
if __name__ == '__main__':
    f = Foo()
    f.addSomethingToMyList(42)
    print "I just added 42 to f's mylist, now it contains", f
    f2 = Foo()
    f2.addSomethingToMyList(117)
    print "I just added 117 to f2's mylist, now it contains", f2

Just by looking at it, one could expect that this would print:

I just added 42 to f's mylist, now it contains [42]
I just added 117 to f2's mylist, now it contains [117]

It doesn't though. What it prints is this:

I just added 42 to f's mylist, now it contains [42]
I just added 117 to f2's mylist, now it contains [42, 117]

Now why is that, you may ask? Well it has something to do with when standard parameter values are instantiated - which must be on class definition time. So if you use a standard parameter value that is a reference type (objects, list, dicts, sets, etc) it is that one instance that is passed on to all instances of your class.

The lesson therefore is: never use reference types in standard parameter values!

Building the Scavenger Python library

2011-03-22T21:56:00.004+01:00

This is the second "building scavenger" post -- see the post immediately before this one for a description of how to build the Scavenger daemon. In this post I will describe how you can install the Scavenger Python library on a client device, so that you may start offloading tasks to your Scavenger daemon instance.

When installing the Scavenger library you designate a Python interpreter that you wish to install the Python modules into, so the first dependency is of course Python. Secondly, the script building the library uses git to fetch the source code from GitHub, so git is a dependency too.

Once you have these two dependencies installed, building and installing the Scavenger library (on a Mac or Linux machine) is as simple as 1) downloading this script, and 2) running the script with the path to the chosen Python interpreter as an argument.

$ wget https://github.com/madsdk/scavenger-library/raw/master/build/build_and_install.sh
$ sh build_and_install.sh /usr/bin/python

If you have any questions about getting Scavenger to run on your system, please don't hesitate to contact me. The best thing to do is to leave a comment on this page, but you are also welcome to send me a mail.

Building the Scavenger daemon

2011-03-22T21:34:00.004+01:00

I recently moved the development of Scavenger onto GitHub. In that process I created two build scripts: One for building the Scavenger daemon on a Linux or Mac machine, and another for building and installing the Scavenger library on the same platforms. In this post I will describe the installation of the Scavenger daemon.

In order to build the Scavenger daemon, you need to install a couple of dependencies. You need git, the gcc compiler and, in order to compile the Presence daemon used for discovery, you also need to install the Qt SDK. If you are on a Mac, getting gcc installed means that you should install Xcode and getting git installed can e.g., be done using MacPorts.

After having installed these dependencies building the Scavenger daemon is as simple of downloading and running the Mac script or the Linux script. The script should be downloaded to and run from the folder in which you would like to build the Scavenger daemon. When run it will download and build Stackless Python, checkout and build the relevant GitHub projects, and configure a Scavenger daemon instance.

After having successfully run the build script, you can start a Scavenger daemon instance by entering the scavenger_daemon directory and running the start_daemon.sh script.

If you have any questions about getting Scavenger to run on your system, please don't hesitate to contact me. The best thing to do is to leave a comment on this page, but you are also welcome to send me a mail.

Building PyGTK for Mac

2011-03-02T10:31:00.005+01:00

In the past couple of months a number of people have asked me to provide new native Mac PyGTK builds. I do not use PyGTK anymore, so I don't have time to maintain these builds - sorry. Out of curiosity I recently built PyGTK on my Snow Leopard (10.6.6) machine, just to see if the process has gotten any easier, and I must say that it has. I was able to build the entire thing without any hacking. So, to those of you asking me to provide new builds, here you have a complete HOWTO of how to build a native Mac PyGTK:

Create a new user account (to make sure that no MacPorts/Fink/other libraries are available in the environment)

Install git (http://code.google.com/p/git-osx-installer/downloads/list?can=3)

Download the build script (https://github.com/jralls/gtk-osx-build/raw/master/gtk-osx-build-setup.sh)

Add ~/.local/bin to PATH

Edit ~/.jhbuildrc-custom setting setup_sdk like so: setup_sdk(target="10.6", sdk_version="10.6", architectures=["i386"])

Run these build commands:
1. $ jhbuild bootstrap
2. $ jhbuild build meta-gtk-osx-bootstrap
3. $ jhbuild build meta-gtk-osx-core
4. $ jhbuild build pygtk

Download the PyGTK hello world script (http://www.pygtk.org/pygtk2tutorial/examples/helloworld.py)

Update your PYTHONPATH: export export PYTHONPATH=$PYTHONPATH:~/gtk/inst/lib/python2.6/site-packages

Make sure that 32-bit Python is used: export VERSIONER_PYTHON_PREFER_32_BIT=yes

Test your build by running the hello world script: python helloworld.py

Note that to perform the last step you need to be properly logged in (not just through SSH).

I hope that this helps some of the people that have asked me about PyGTK builds. Happy hacking ;-)

Testing Restlet webservices from telnet

2011-01-11T10:04:00.004+01:00

For the upcoming P2P course at Aarhus University (which I have been co-teaching for the past four years) I am currently playing around with the Restlet REST API. We want the students to learn something about peer-to-peer algorithms, protocols, and techniques, and seeing as we like a hands-on approach we always ask them to code a small P2P system as an ongoing exercise throughout the course. In past years we have asked them to do this programming using regular sockets, but that is about to change... While learning about the socket abstraction may be instrumental in teaching a coming computer scientist about the underlying constructs of the network stack, it may be time to teach them this stuff only cursory and then skip ahead to a higher level of abstraction. I mean, who (besides myself) actually works with sockets anymore?

We have therefore decided to teach the students about REST and let them use the Restlet API to create their own P2P system. But before throwing a new API at my students, I like to experiment with it myself, and I therefore dived into a little bit of Restlet programming. In the process I needed to be able to test a HTTP PUT call to my Restlet, which can not (afaik) be done easily from a webbrowser. I therefore went to my old friend telnet for help. So, if you need to test a HTTP PUT call to a webserver from telnet do the following:

telnet localhost 8182
PUT /peerlist HTTP/1.1
Host: localhost
Content-Length: 13

Hello, World!

Substitute host, URL, and content to your hearts contend.

Scavenger and Presence are moving to GitHub

2011-01-05T16:51:00.003+01:00

I am currently re-booting my work on Scavenger (and therefore also on the Presence service discovery framework) as I need it for some upcoming research. In the process I have decided to move the source code from the Google Code pages and onto GitHub instead. In the last couple of months I have become increasingly fond of git---especially as a collaborative tool where other developers can freely fork my repositories, play around with the source on their own, and then send me a pull request when they are satisfied with their changes. This way of working has worked out extremely well in my small LaTeX YaSnippet project. I hope that switching to GitHub means that more people will consider adding to both Scavenger and Presence.

While moving the code I have also done quite a bit of refactoing and "spring cleaning". This means that the code has now been split into six (more or less) independent projects. All of these can be found here https://github.com/madsdk/

Installing the Scavenger client library or the Scavenger daemon has also been simplified quite a bit. Both the lib and the daemon now have build scripts that work on Mac and Linux. If I e.g., would like a Scavenger daemon running om my laptop, I would simply download and run this script. The script must be placed in the folder where you would like to install the Scavenger daemon, and it will then download and build all dependencies (Stackless Python, Presence daemon, diverse Python modules). Notice that in order to build the Presence daemon you need to have the Qt development tools installed.

Happy hacking!

Building Stackless Python on Snow Leopard

2010-12-06T10:42:00.003+01:00

I have had a hard time debugging this problem I have had with running Stackless Python on my Snow Leopard Mac - that is until I stumbled upon this post. It seems that the problem is with using Stackless on a 64-bit Mac OS, and the solution is thus to build Stackless in a 32-bit version. So, if you're having problems with Stackless Python crashing with a "Bus error", do this when you configure during the build process:

CC="gcc -arch i386" ./configure --enable-stacklessfewerregisters

That seems to have solved the problem for me anyway ;-)

My LaTeX snippets for YaSnippet/Emacs are still alive

2010-11-25T09:20:00.003+01:00

I have very few words for you this time. All I want to do, is to point out that my LaTeX snippets for YaSnippet (an Emacs snippet plugin) is still alive. I have not updated it myself for quite some time, since I seem to be satisfied with the collection of snippets I already have, but others have been doing the updating for me! Oh the joys of open source ;-)

The snippet collection is therefore still growing, thanks to additions made by Bjorn Reese, Song Qiang, and Claudio Marforio. Thanks you guys!

If you are interested in adding snippets to the collection please do one of the following (listed in prioritized order):
1) Create your own branch on github and send me a pull request, or
2) Send me the snippets by email.

SQLite with stdev (standard deviation) and more!

2010-09-10T10:40:00.005+02:00

At the moment I am collecting data - hoards of data - my test suite is a veritable cornucopia of boring data. Normally I would be logging such data into flat files, collecting some giga bytes of log files that I would then need to write a handful of Python, Perl, or Ruby scripts to make any sense of. I have done this a thousand times before ... and I'm getting tired of it!

In comes my new best friend - SQLite. This small relational-database-in-a-file is excruciatingly easy to use from almost any programming language you could think of. So instead of writing a lot of data handling scripts, this time I have written only one - a script that imports the log data into an SQLite database. All other questions about the data, and how the different test results relate to each other, can thus be formulated in simple SQL instead of through increasingly complex Python scripts.

"When are you coming to the stdev part!", you say? Calm down, I'm getting there. Working with my data in a relational database I found that there were only one piece of functionality that I was missing, and that was the possibility to select out both an average _and_ the standard deviation of a data set. Luckily, this is possible in SQLite through a simple extension.

This is what you do:

Fetch the source code for the extension here [direct link]

Compile it:

Mac$ gcc -fno-common -dynamiclib extension-functions.c -o libsqlitefunctions.dylib
Linux$ gcc -fPIC -lm -shared extension-functions.c -o libsqlitefunctions.so

Copy the resulting library to some location where you want to store it. In my case that would be $HOME/opt/lib

Before executing your select statements be sure to load the extension in SQLite by issuing the following command (within SQLite):
```
select load_extension('/full/path/to/libsqlitefunctions.dylib');
```

And you're done! Now you can issue select statements with stdev as an aggregate function. E.g.:

SELECT name, stdev(seconds) 
FROM benchmarks 
GROUP BY name

Note: The SQLite version included in Mac OS X 10.6 does not allow extensions, so you will have to build it yourself. That is luckily easily done - fetch the source here.

On CPU profiling

2010-06-29T09:05:00.002+02:00

CPUs are different ... no really, they are. And when working with task scheduling over heterogeneous peers, which is the metier of Scavenger, this heterogeneity of the peers makes it very hard to come up with precise running time estimates. When estimating a task's running time on a peer one needs two things: 1) the running time of the task (with the given input) on a known peer, and 2) a "strength" rating on each peer that says something about how strong the peer is w.r.t. processing work. Having this information an estimated running time can be derived simply by assuming that a peer of strength x can perform a given task twice as fast as a peer of strength x/2.

This is all fairly straight forward, but the tricky part is finding that magical "strength" value for each peer. In related systems, such as Spectra and Chroma, the CPUs clock speed and its Linux BogoMIPS rating has been used, respectively. I expected that these values were pretty useless at estimating peer strength, and my experiments up until now seem to confirm that suspicion. Because of that I in Scavenger chose to use a real CPU benchmark for the strength value, thinking that a CPU benchmark would be written specifically to exercise all the various parts of a CPU and therefore would be more apt at describing its relative strength. I have used the nbench benchmarking suite - primarily because it is available as source code, so that I can compile and run it on any platform that I would like to test. Nbench returns two scores, an integer and a floating point score, so I chose to use the average of these two scores as peer strength. My experiments show that this value does indeed model CPU strength much better than clock speed or BogoMIPS - but it is far from perfect!

Because of that I have started work on a new profiling approach - one that will either 1) increase the accuracy when estimating running time on hitherto unknown peers, or 2) show that the performance of different CPU architectures can not be scored on a linear scale... Of course I am hoping for 1 but I somehow expect to conclude 2 :-)

Getting EyeTV to Put Your Mac to Sleep ... Intelligently!

2010-05-05T20:47:00.004+02:00

One feature that I _really_ miss in EyeTV is the ability to put my Mac to sleep after a recording. It is possible to write an AppleScript that reacts to the RecordingDone signal in EyeTV, but my previous foray into that area ended up with a simple script that said something like

on RecordingDone(recordingID)
   tell application "Finder" to sleep
end RecordingDone

This works to some extent but it does have some serious flaws. First and foremost it ruins your second recording if you have scheduled to record two programs back-to-back. In that case EyeTV records the first program, emits the RecordingDone signal, starts recording the second program, and about five seconds into that recording the computer goes to sleep ;-) Another problem with this approach is when you are recording something while you are working on the computer - in that case your computer suddenly goes to sleep while you are working.

This has annoyed me quite a bit, but seemingly not enough for me to actively search for a better solution - until today. Googling around a bit I found this script: http://forums.elgato.com/viewtopic.php?f=91&t=1045#p18834. I have altered it a tiny bit to change the timeouts and to put the computer to sleep instead of powering it off. The result is here:

on RecordingDone(recordingID)
   set isCurrentlyRecording to check_eyetv_is_recording()
   if isCurrentlyRecording is false then
       with timeout of 300 seconds
           tell me to activate
           display dialog "Warning: EyeTV has now finished recording and the system will automatically shut down in 5 minutes unless you click Stop!" with icon 2 buttons {"Stop!"} giving up after 300 default button "Stop!"
           set theresults to the result
           if gave up of theresults is true then
               tell application "System Events" to sleep
           end if
       end timeout
   else
       quit
   end if
end RecordingDone

to check_eyetv_is_recording()
   delay 30
   tell application "EyeTV"
       if is_recording then
           return true
       else
           return false
       end if
   end tell
end check_eyetv_is_recording

This script works perfectly! When the recording is done a dialog is displayed telling you that the computer will be put to sleep in five minutes, and if you press the stop button you can keep the computer awake.

Update: A guy called Michael asked we where to put the script to make it work. You are supposed to save the scripts as /Library/Application Support/EyeTV/Scripts/TriggeredScripts/RecordingDone.scpt

YASnippet LaTeX bundle GitHub Project

2010-03-19T16:09:00.003+01:00

A long time ago I wrote about a LaTeX YASnippet bundle that I was building - but I never really got around to publishing it in a way so that anybody except me could get at them. I did create a Google Sites homepage for it, but that's a terrible way to share source code :-)

I have therefore just created a GitHub project called yasnippets-latex. Now my snippets are freely available - released under the GPL of course. So grab them, play with them, branch the project, create your own snippets, improve my snippets, etc.

Playing around with MacJournal

2010-03-11T13:08:00.002+01:00

I have recently bought the MacHeist nanoBundle2 which included a license for MacJournal - yet another journaling software for Mac. I already have Evernote, which seems to be far better than MacJournal for journalling, but MacJournal has some blogging features that I may be interested in. So I guess I’ll give it a try. This entry is written in MacJournal b.t.w. ;-)

This image was just copied into the editor window - and then resized a bit to try that out as well. It’s a photo I took during my recent vacation in Sweden b.t.w.

Now lets try some pre-formatted text (I need that for displaying code examples). I did this by changing the font in MacJournal to Courier 12pt - I would have liked to edit HTML tags instead...:

def compare_us(x, y):
return true if x == y else false

And then some regular text again...

Edit: Well, that did not work at all! I guess MacJournal is not for me then :-) My initial experiences with the software have not been very good. The first time I added a blog the application crashed, and I had to try five times before the auto update function actually worked. When adding a blog entry it did not append any of the tags I chose, the editor did not allow me to edit the HTML source etc. Conclusion: it is a crappy blog editor.

XeLaTeX on a Mac

2010-02-09T13:21:00.002+01:00

As I wrote a couple of hours ago, I am currently writing my thesis using XeLaTeX. In the process I have found that the MacPorts version of XeLaTeX seems to be broken when it comes to accepting/finding user-installed fonts. My LaTeX template needs a font called Fontin Sans that I have downloaded and installed into my "Font Book" on my Mac. After doing that the MacPorts version of XeLaTeX still couldn't find the font, so I uninstalled the texlive packages fro MacPorts and installed the MacTeX 2009 package instead. And now everything works :-)

XeLaTeX and Aquamacs Emacs

2010-02-09T10:48:00.003+01:00

I have finally begun writing my thesis... or sort of anyway - I have definitely started playing around with a LaTeX skeleton for it :-P

My friend Clemens gave me the LaTeX template that he used for his thesis, and it creates really pretty documents so I definitely wanted to use it for my thesis as well. But, this template makes use of XeLaTeX, a LaTeX variant that supports use of Unicode and "modern font technologies". I found that I already had xelatex installed on my system; apparently it comes with the MacPorts LaTeX installation, but it did not work immediately in Aquamacs. There _was_ a XeLaTeX command in Aquamacs, but in the background it was calling pdflatex, which in turn went into some never ending loop using a full CPU core...

In order to fix that you need to do change the way the xelatex command is called in Aquamacs, as described here.

Open Aquamacs in LaTeX mode (by opening a .tex file).
Go the the LaTeX menu, choose "Customize AUCTeX", choose "TeX Command", choose "TeX Command List".
Navigate to the XeLaTeX command (just search for xelatex).
Alter the "Command" part of that section to "%`xelatex --synctex=1%(mode)%' %t"
Set the "How" part to "TeX-run-command".
Make sure that LaTeX, ConTeXt, and AmSTeX have been checked off in the bottom part of the section.
Save the settings (Ctrl+x, Ctrl+s as usual).

And now it works!

Scavenger Daemon - VMWare Fusion Image

2010-02-03T14:18:00.004+01:00

I have just uploaded a VMWare Fusion 3 virtual machine image that you may use to tryout Scavenger. The image is a minimal Ubuntu 9.10 server install with Presence and Scavenger pre-installed - you can fetch it here.

Here's how you use it:

Download and unpack the zip file.

Open the VM in VMWare Fusion 3. When it prompts you the first time select "Take ownership", and the second time you must choose "I copied it".

Go to $HOME/scavenger (i.e., "cd scavenger").

Start the Scavenger daemon by issuing this command: "./start_daemon.sh"

Now you have a fully functional Scavenger daemon running. The Scavenger daemon is running from an SVN checkout, so you may update it to the most current version by issuing the command "svn up daemon scavenger_common" from within the $HOME/scavenger directory.

In order to make the daemon running within the VM discoverable you of course have to configure the networking options to fit your environment. I think it is set to use bridged networking at the moment, which should fit in most circumstances.

AugIm v. 2 released!

2010-02-02T13:26:00.004+01:00

I have just released an entirely new version of my Scavenger demonstrator AugIm (The Augmented Image Manager). AugIm is now Maemo 5 only - and it is written in PyQt so you need to add the extras-devel repository in order to install it.

You can grab the demonstrator at Scavenger's Google Code page, or you could just use this direct link.

B.t.w., if you look in 'branches' in Scavenger's SVN you will find an alternative version of AugIm. This version does _not_ use Scavenger, so it may be used as a comparison to the fully-fledged, Scavenger-enabled version.

I have also created a VMWare Fusion image of a Scavenger daemon to make testing Scavenger easier, but I have yet to find out how to make such a VM distributable using Fusion... I'll probably link to the current Fusion-only image in the near future, and if I get the time I will try to create an OVF compliant version.

PyQt: Emitting events from non-QObject subclasses.

2010-01-29T23:04:00.002+01:00

In order to emit events in PyQt the emitting class must inherit directly from QObject. This can be quite a nuisance, especially so because multiple inheritance is impossible when using PyQt. E.g., I recently had the problem that I had task running in a QThreadPool that I would like to emit a signal when progress had been made, so that I could update a progressbar in my UI. To make the task runnable in a QThreadPool the task class had to inherit from QRunnable, and as QRunnable is not a subclass of QObject it was impossible to emit signals from within the task class. I found a solution to this problem though - one that I find quite elegant, so I'd like to share it with you :-)

The trick is to use another object as the mediator for emitting signals, and then using Python properties to hide the fact that a mediator is being used. Consider the example below:

class Mediator(QObject):
    mySignal = pyqtSignal(int)
    def __init__(self):
        super(Mediator, self).__init__()

class Task(QRunnable):
    def __init__(self):
        super(Task, self).__init__()
        self._mediator = Mediator()

    def mySignal():
        def fget(self):
            return self._mediator.mySignal
        return locals()
    mySignal = property(**mySignal())

Using this approach any classes working with Task instances can connect directly to the mySignal signal as if it was bound directly to the Task class. So the property hides the fact that a mediator is in use.

Embedding fonts in Gnuplot generated PDFs

2010-01-27T13:40:00.004+01:00

I have recently submitted two papers to an IEEE conference, and in doing so I had to make sure that _all_ fonts were embedded within the PDFs I submitted. This was not the case with one of my papers, and after some detective work I found out that this pertained to my using Gnuplot's PDF terminal to generate PDF graphs for my paper. Apparently Gnuplot is incapable of embedding fonts into the generated PDFs...

I found this solution though. You need to pass the PDF that Gnuplot generated through ghostscript with the following command:

gs -q -dSAFER -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=tmp.pdf\
-dCompatibilityLevel=1.5 -dPDFSETTINGS=/prepress -c .setpdfwrite\
-f infile.pdf

Where 'infile.pdf' is your Gnuplot graph. If you follow the link to where I found this hack, you may notice that I have changed the commandline slightly, so that it now creates PDF 1.5 compatible output. This is a requirement of IEEE papers I think.

Avoiding "Size mismatch" Errors When Upgrading Maemo 5 Scratchbox

2010-01-14T11:18:00.004+01:00

I have just returned to developing Maemo application, and as usual when I come back to my Maemo scratchbox installation, I start off by upgrading the development environment. So I opened my scratchbox and issued the commands

fakeroot apt-get update
fakeroot apt-get dist-upgrade

Unfortunately this did not work for me this time because of some inconsistencies in the index files kept at the Maemo servers (I think). I kept getting these "Failed to fetch http://someurl Size mismatch" errors. An example:

...
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-application-manager-l10n/hildon-application-manager-l10n-plpl_6.0+r7389+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-application-manager-l10n/hildon-application-manager-l10n-ptpt_6.0+r7389+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-application-manager-l10n/hildon-application-manager-l10n-ruru_6.0+r7389+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-application-manager-l10n/hildon-application-manager-l10n-svse_6.0+r7389+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-theme-alpha/hildon-theme-alpha_9.1+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/free/m/maemo-launcher/maemo-launcher_0.35-7+0m5_i386.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/free/m/maemo-launcher/maemo-launcher-dev_0.35-7+0m5_all.deb  Size mismatch
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

To avoid these errors I manually downloaded the packages involved and installed them with dpkg. To make this process easier I simply copied the error messages into a text file, removed the "Failed to fetch" prefix and the "Size mismatch" postfix from all lines using search/replace, and saved this as files.txt. Then I invoked the following commands from within my scratchbox targets:

cat files.txt | xargs wget
dpkg -i *.deb

And voila, it works ;-)

Building Stackless Python on a PS3 (PPC Linux)

2009-12-17T10:32:00.006+01:00

I have recently gotten my hands on a PlayStation 3 that I am trying to use as a surrogate[1] in my Scavenger system. The first thing I did was to install Ubuntu Linux 9.10 on it, and the next step was then getting Stackless Python to compile... Which turned out to be a bit of a bitch ;-)

But after some heavy debugging and googling I found out how to do it:

Download the Stackless source distribution. I used version 2.6.4.

Unpack it somewhere and enter the newly created directory.

$ ./configure --prefix=/where/you/want/it

Edit the file called "Makefile" and change the variable SLPFLAGS to "SLPFLAGS=-fomit-frame-pointer -O2"

make

make install

It's very simple actually, just took me some time to find the solution :-) By the way, if you do not edit the Makefile you get the following error:

./Stackless/core/slp_transfer.c: In function 'slp_transfer':
./Stackless/core/slp_transfer.c:144: error: 31 cannot be used in asm here
./Stackless/core/slp_transfer.c:144: error: 31 cannot be used in asm here
make: *** [Stackless/core/slp_transfer.o] Error 1

Looking at that error the 31 hints at that the save registers code for PPC is trying to save register r31. This is reminiscent of the error you get when compiling Stackless under Mac OS X, but in this case adding the --enable-stacklessfewerregisters option does not work.

[1]: A surrogate in a cyber foraging system is a strong machine that performs heavy work on behalf of the weaker (often mobile) clients.

New Presence2 Beta

2009-12-14T12:42:00.002+01:00

I have just released the fourth Presence2 beta. In it I have fixed some serious bugs, so if you are using the older betas please upgrade now ;-)

Grab the files from the usual place.

Binding keyboard shortcuts to Subclipse commands in Eclipse

2009-11-27T13:01:00.003+01:00

I have just created a small screencast about how to bind keyboard shortcuts to Subclipse commands within Eclipse. Now I can add, commit, and diff files without having to use my mouse - and I'm using the same Emacs shortcuts that I know and love :-)

B.t.w., it is cool how products like iShowU HD makes it incredibly easy to create screencasts. I've probably spent more time writing this blog entry than actually creating and uploading the video ;-)

Presence2 beta 2 released

2009-11-24T09:41:00.003+01:00

I found some bugs in the Python lib for the new version of Presence. These have now been fixed and therefore a second beta has been released. Grab it at the Presence Google Code page.

TeXlipse build that actually works...

2009-11-20T09:50:00.005+01:00

I am currently writing an article - which under normal circumstances means that I am working in Emacs (Aquamacs f.t.w.!) using AUCTeX, RefTeX, flyspell etc. But this time around I wanted to try something new, so I installed the TeXlipse plugin for Eclipse and started writing.

I quickly faced a major problem though: If you, like any sane person, split up your document into smaller files and then \include or \input these files in some master file, then the auto completion of BiBTeX references does not work in TeXlipse. This is a known bug - it has been known for almost six months, and yet they haven't released a bugfix release!

I have therefore built TeXlipse from CVS instead, and that seems to be working so far. You can fetch this build here. It was built using the CVS head as of today...

Note: This build may have other bugs - it is _just_ a build of the TeXlipse CVS head.

Presence 2.0.0-b1 Released

2009-11-06T14:38:00.003+01:00

I have just released the first beta of Presence2 (or qPresence as I also call it because of its Qt dependencies). I have put up downloadables for the daemon, the Python API, and the "daemonless" Python API (the one that allows for listening in on Presence announcements without running the Presence daemon).

Because of the new Qt dependencies you need to install the Qt library and SDK in order to build the daemon. You can fetch these at Nokia's Qt page.

Grab 'em while they're hot!

Byte Order Conversion in Qt

2009-11-05T08:23:00.003+01:00

For my recent re-writing of Presence to use the Qt library, I needed to be able to do byte order swapping on two and four byte integers (short and int). This I used when serializing/marshalling/packing stuff up before transmitting them over the wire. I know that the "right" way of doing that in Qt would be to use QDataStream, which will then handle stuff like byte order conversion, but I needed to be able to pack my own data, so that a QDataStream-capable reader was not required on the other side of the socket connection.

Byte order conversion is done using the four functions (macros really) called htons, ntohs, htonl, and ntohl that may be found in inet/arpa.h on any Unix system. So of course I included that file and went happily on with my coding. But... when I tried to compile my new version of Presence under Windows I realized that inet/arpa.h does not exist on Windows. It probably resides in winsock.h I guess, but I _really_ do not want to have platform specific code in my project, so I decided to implement them myself instead :-)

Using Qt you can ask for your systems endianness by using the QSysInfo class, so a portable implementation of the needed functions is easy to do:

#define BYTE_SWAP4(x) \
    (((x & 0xFF000000) >> 24) | \
     ((x & 0x00FF0000) >> 8)  | \
     ((x & 0x0000FF00) << 8)  | \
     ((x & 0x000000FF) << 24))

#define BYTE_SWAP2(x) \
    (((x & 0xFF00) >> 8) | \
     ((x & 0x00FF) << 8))

quint16 _htons(quint16 x) {
    if (QSysInfo::ByteOrder == QSysInfo::BigEndian) {
        return x;
    }
    else {
        return BYTE_SWAP2(x);
    }
}

quint16 _ntohs(quint16 x) {
    if (QSysInfo::ByteOrder == QSysInfo::BigEndian) {
        return x;
    }
    else {
        return BYTE_SWAP2(x);
    }
}

quint32 _htonl(quint32 x) {
    if (QSysInfo::ByteOrder == QSysInfo::BigEndian) {
        return x;
    }
    else {
        return BYTE_SWAP4(x);
    }
}

quint32 _ntohl(quint32 x) {
    if (QSysInfo::ByteOrder == QSysInfo::BigEndian) {
        return x;
    }
    else {
        return BYTE_SWAP4(x);
    }
}

Note that the functions have a '_' prepended - otherwise some compilers may get confused.

Using these functions I do not need to have any platform specific code in my project. This comes at the cost of actually doing a function call when byte order conversion is needed, as opposed to simply using a macro. So if your code is very performance critical you should not use this code - but then again, if your code is that performance critical you should probably not be using C++ ;-)

Complete rebuild of Presence

2009-11-04T13:34:00.002+01:00

I have spent the past week re-writing Presence from scratch using Qt. "Why?" you may ask. The previous version of Presence has been working very well for a long time, but it had a bug that only appeared on 64-bit operating systems. I spent a small amount of time debugging the old code base to find that bug, and then finally decided that a complete re-write would be better. Not that the old code base was that horrible, it was just written in a mixture of C and C++ that made it a bit hard to maintain. E.g., the network related code was written in pure C using a large select() statement and multiplexing over a large number of sockets - it seems obvious that this could be coded _much_ nicer using an event based architecture.

So I decided to recode it in Qt, and let me tell you: Qt = happiness! I am no fan of C++, but when using Qt it is actually a nice language to work with. The new version of Presence is easily maintainable, because it is so simple to read and understand Qt code (at least compared to reading and understanding C). No more waiting for data on a bunch of sockets, finding out which socket has activity, forwarding the task of handling the activity to someone else, and then back to the listening... In Qt all socket activity can of course be subscribed to as callbacks, so now the Announcer class takes care of all announcement packets, the ControlConnection class talks to application(s), and so forth.

In the process of porting the daemon I have altered the protocol slightly, which means that I also had to update the Python libraries. In doing this these have also been redesigned, making them a bit easier to read as well. They both work now, though I must admit that I am not completely satisfied with the main Python lib - but seeing as it works I will probably not get to updating it any time soon.

Controlling PDF Minor Version in LaTeX

2009-09-28T09:37:00.004+02:00

I just had to submit a paper to a conference that was using some web based submission system. As part of that system, the PDF version of the paper I submitted was validated. The system refused my paper because it was seemingly a version 1.4 PDF (w.t.f. do I know about PDF versions?). I spent some time trying to update my LaTeX installation on my Mac (didn't work), compiling the paper on my university's Linux machines (also didn't work), and then I hit Google...

It seems that you can control the PDF minor version by using this magical LaTeX command

\pdfminorversion=5

This will make pdflatex create a version 1.5 PDF document (you can also create version 1.6 and 1.7 PDFs, but 1.5 was all I needed).

Native Mac PyGTK for Snow Leopard

2009-09-18T21:46:00.003+02:00

I have just made a fresh build of native PyGTK for Mac on my new Snow Leopard install. It seems to be working - I have only tested my simple hello_pygtk.py script, and that works at least ;-)

You can grab the new build here: MacPyGTK_SnowLeopard_091809.dmg

Remember to follow the three simple instructions in README.txt in order to make it work. B.t.w., one has to be running Python in 32-bit mode for it to work. This can be done in two ways:

By setting this environment variable VERSIONER_PYTHON_PREFER_32_BIT=yes
By issuing this command (once): defaults write com.apple.versioner.python Prefer-32-Bit -bool yes

I would prefer setting the environment variable in a script starting my PyGTK application. That way I am changing as little as possible in the standard setup of the machine...

Piping output of shell commands into the clipboard

2009-09-17T20:52:00.002+02:00

I love my terminal! The availability of a proper shell and set of *nix tools is one of the main reasons that I dared switch to Mac OS X two years ago. I have been used to having the full GNU tool chain on the Linux systems I have been using the last 10 years, and there is no way I will ever give that up :-)

Today I learned something new about my terminal on OS X; I learned about the pbcopy and pbpaste commands. These two shell commands copy shell output into the clipboard, and pastes the clipboard content into the shell, respectively. I.e., today I needed to grep+cut through some large log files, and the results of these commands I wanted to paste to a spreadsheet. Instead of having to 1) perform the command, and then 2) use the mouse to scroll up and select all of the results to copy-paste them into the spreadsheet, I found that I could use pbcopy to pipe stdout directly to the clipboard:

grep foo logfile.txt | cut -d " " -f1,2,4 | pbcopy

After running this I could just go to my spreadsheet application and insert the three selected rows.

This little command is a real time saver I can tell you! I must have saved at least an hours work today alone ;-)

Fresh PyGTK build for Leopard/Intel

2009-09-13T20:00:00.002+02:00

I have just built a fresh version of native Mac OS X PyGTK and all of its dependencies. This time I have only made an Intel build for Leopard, so no PPC version this time around.

You can grab the new dmg file here: MacPyGTK_Leopard_Intel_091309.dmg

The install procedure is the usual:

Download and mount the dmg
Drag the MacPyGTK folder to /Developer
Copy the contents of site-packages into the site packages of your Python installation (on 10.5 this should be /Library/Python/2.5/site-packages)
Test that it works by running the hello_pygtk.py file with your Python interpreter.

That's it - now you should have a fully functional PyGTK installation to play around with.

New Logo for Locusts?

2009-07-21T08:41:00.005+02:00

I have for some time now been searching for a nice picture of a grasshopper to use as a logo for the Locusts project that I am working on. I never found one that I was satisfied with though, but yesterday I got lucky. While I was out walking in Lappeenranta this little fellow was sitting on the side of the road - and he really liked being photographed.

B.t.w., I edited this image using Pixelmator to remove the background. This little app is really turning out to be an excellent image editor - who needs Photoshop anyway? :-)

Excited about Scavenger...

2009-07-17T12:43:00.002+02:00

Just wanted to let you know that I have posted something about Scavenger on my "work" blog. I have conducted some benchmarks of Scavengers parallel processing abilities and it is looking very promising I must say.

Making your OS X hostname stick

2009-07-15T08:58:00.009+02:00

If you are both a Mac user and a terminal junkie, as I am, you will probably have noticed that your computers hostname changes whenever you enter a new network environment[1]. More precisely, when you join a network through a DHCP server in some cases it also assigns you a new hostname. This has been a major annoyance to me for some time now, so I finally decided to find out how to make my hostname stick. And, as it turns out, that is of course quite simple to do:

1) Issue the command "sudo vi /etc/hostconfig"
2) Add the line HOSTNAME=my-hostname to this file (pressing i to enter edit mode).
3) Save and close (that's Esc to quit editing mode, and then :wq + Enter for you vi rookies)

The HOSTNAME line may already exist on your system in which case its value will be "-AUTOMATIC-". In that case you just replace that with your hostname. It was not there on my Leopard (client) machine.

Oh yeah, thanks to Robert Brewer for teaching me how to do this. Take a look at his article if you are interested in getting some more detailed information about the whole hostname determination business on OS X.

[1]: For those of you not used to using a terminal a new command-line in a terminal typically starts with "username@hostname:top-level-path$". E.g., my prompt right now shows "madsk@leela:Development$".

Scavenger tutorial available

2009-07-13T18:41:00.002+02:00

I have finally gotten around to writing some documentation of Scavenger, so that others may in fact start writing Scavenger-enabled applications. I have spent the entire day updating the wiki pages of Scavengers GoogleCode homepage so now you will find a tutorial there along with some other related information about scavenger. Enjoy!

Python in the Android Emulator

2009-07-10T10:08:00.002+02:00

I have recently started playing around with the Android platform; more specifically I have begun experimenting with Python development on Android through the Android Scripting Environment (or ASE, as its called).

In ASE you are able to run Python 2.6.2 scripts that, as far as I can tell so far, have full access to the underlying Python libraries. Furthermore, through the "android" module, you gain access to some Android specific things like location services, sensor data, sms messaging, and more.

One problem with ASE though is the fact that it is impossible to run a Python program within the Android Emulator. When you try to start a Python script all you get is a black screen. This is due to some sensor-thingy not being available in the emulator, and the suggested fix is to rebuild ASE with the sensor service disabled.

To spare others of the pain involved rebuilding ASE from source (it is really not that hard, just takes some valuable time away from coding in Python), I will share my build of ASE that actually works in the emulator. You can download it here: ASE Emulator build (do not use this on a real device!). This build is provided as is; I will not be held liable if it blows up your computer, burns down your house, or runs away with your wife ;-)

Scavenger demo application released!

2009-07-07T15:30:00.006+02:00

I have just released the first Scavenger demo app: AugIM - the Augmented Image Manager. AugIM is an old demonstrator that I made a long time ago, and now I have ported it to use Scavenger for its cyber foraging tasks.

AugIM is an image browser/editor for Maemo (running on Nokia's N800/N810 devices) that allows you to do simple image manipulation tasks such as sharpening, resizing, adjusting color/contrast/brightness, rotating, flipping etc. It also allows you to share your images with the world through email or Google Picasa Web. The N800 and N810 devices are capable of performing these services on their own, but it takes a long time for them to do so and it consumes a very large amount of energy. Using Scavenger, AugIM makes sure that heavy processing is offloaded to larger surrogates whenever possible, which means that the image manipulation tasks are performed faster and using less energy on the mobile device.

Yet another Scavenger build...

2009-07-07T15:21:00.003+02:00

I have just uploaded a new self-contained Scavenger daemon build to Scavengers Google Code page. This build is for Mac OS X 10.5 (Leopard) and it contains the same components as the Ubuntu Linux version I released yesterday.

From your Mac simply download the archive, unpack it, and start the start_daemon script from a terminal. Then a Scavenger daemon should be running on your machine, and you are ready to start playing with cyber foraging! :-)

Scavenger build available for Ubuntu 9.04

2009-07-06T14:06:00.003+02:00

I have finally gotten around to creating a Scavenger build that you can use to easily experiment with Scavenger, without having to install Stackless Python and other dependencies (e.g., Presence).

The build, which is created and tested to run on Ubuntu 9.04, is available here. All you need to do is 1) download the archive, 2) unpack it, and 3) run the "start_daemon" script.

The "bundle" contains Stackless Python 2.6.2, the Presence python lib and daemon, PIL (Python Imaging Library), Google Data API, and the Scavenger daemon.

Timelapse photography using CHDK

2009-06-30T23:26:00.003+02:00

I thought it could be fun to do some time-lapse photography now that I have a programmable camera. So I have quickly whipped up a small time-lapse script that works on my Ixus 750:

@title Timelapse
@param a Number of steps 
@default a 30
@param b Period (seconds)
@default b 60

rem Sanity check.
if a<1 then let a=1
if b<5 then let b=5
let b=b*1000

rem Take the pictures!
for s=1 to a+1
    rem Take a single picture.
    print "Shoot", s, "of", a
    press "shoot_half"
    sleep 2500
    press "shoot_full"
    release "shoot_full"
    release "shoot_half"
    print "shot done"

    rem Sleep until its time to take the next picture.
    sleep b
next s

At the standard setting it takes thirty pictures in roughly 30 minutes.

CHDK oddities

2009-06-30T23:22:00.004+02:00

The script I posted in my last post, the one that does exposure bracketing, used to work - I swear it did! But all of a sudden it decided to stop working, and I can't find an good explanation for that at all ;-)

But, instead of an explanation i offer you a fix. The following script _does_ work on my Ixus 750 camera (right now it does anyway):

@title EXP bracketing 
@param a Number of +/- steps 
@default a 2
@param b Step size (1/3EV)
@default b 3

if a<1 then let a=2
if b<1 then let b=3

print "Preparing..."
for n=1 to a*b
    click "left"
next n

for s=1 to a*2
    print "Shoot", s, "of", a*2+1
    press "shoot_half"
    sleep 2500
    press "shoot_full"
    release "shoot_full"
    release "shoot_half"
    print "shot done"
    sleep 5000
    for n=1 to b
        click "right"
    next n
next s

print "Shoot", a*2+1, "of", a*2+1
press "shoot_half"
sleep 2500
press "shoot_full"
release "shoot_full"
release "shoot_half"
sleep 5000

print "Finalizing..."
for n=1 to a*b
    click "left"
next n

CHDK on an Ixus 750

2009-06-09T13:52:00.009+02:00

While on paternity leave I have been playing around with my Canon Ixus 750 digital camera. Primarily I have been doing macro (close-up) shots of insects etc., but I also wanted to experiment with other kinds of photography, e.g., High Dynamic Range images (HDR). In HDR you take pictures of the same scene at different exposure levels, and after that a computer program stacks these images to create the HDR image. When taking such pictures it is vital that you do not move the camera, which can be quite hard even if you have a stand for it. I was therefore looking for a software solution to HDR, that would automatically take 3 to 5 pictures at different exposure levels for me, so that Idid not need to touch the camera in between shots.

For that I installed CHDK on my Ixus 750 and I must say that I am impressed. Using the CHDK firmware extension a horde of new features are unlocked (for example the possibility to take RAW images instead of JPEGs). And most importantly CHDK is scriptable, so you may write your own scripts to control your camera.

Based on an example in the tutorial I quickly wrote an HDR script that works on an Ixus 750. When I activate this script my camera snaps five pictures at -2, 1, 0, +1, +2 exposure.

@title EXP bracketing 
@param a Number of +/- steps 
@default a 2
@param b Step size (1/3EV)
@default b 3

if a<1 then let a=2
if b<1 then let b=3

sleep 1000

print "Preparing..."
rem click "erase"
for n=1 to a*b
    click "left"
next n

click "shoot_half"
sleep 3000
for s=1 to a*2
    print "Shoot", s, "of", a*2+1
    click "shoot_full"
    print "shot done"
    sleep 5000
    for n=1 to b
        click "right"
    next n
next s

print "Shoot", a*2+1, "of", a*2+1
click "shoot_full"
sleep 5000

print "Finalizing..."
for n=1 to a*b
    click "left"
next n
rem click "erase"

end

In order to use this script the camera must be in manual mode and the exposure menu must be opened...

Controlling the firewall on Mac OS X 10.4 (Tiger) from the command line

2009-05-19T11:05:00.005+02:00

I have just had the need to control (actually disable) the firewall on a Mac running OS X 10.4 (Tiger). This should be no problem; just start System Preferences go to Sharing... The problem was that the only connection I had to the machine was through SSH, so going to System Preferences was not an option.

Luckily the firewall can be controlled with the ipfw command on the command line. Take a look at this article at macdevcenter for an example of how to use ipfw.

And, to disable the firewall all that is needed is thus:

sudo ipfw disable firewall

Update: To disable the firewall on Mac OS X 10.5 (Leopard) you can issue the following command:

sudo defaults write /Library/Preferences/com.apple.alf globalstate -int 0

Scavenger - Now With History Based Profiling

2009-04-29T16:17:00.002+02:00

I have just added yet another scheduler to Scavenger (yes, there is three of them now). This scheduler is the "smartest" of the three as it uses history based profiling in order to be able to choose the right surrogate for the job. History based profiling in a cyber foraging system is nothing new - the Chroma system used that some 6-7 years ago. What is new is the way that the profiling information is used within the Scavenger system. In Scavenger the profiling information can be used not only to tell you something about how the task will perform on certain surrogates (i.e., the ones where you have earlier performed the task), it is also capable of making an informed estimate of the running time on an unknown surrogate... more on that later ;-)

Building the Python Imaging Library (PIL) for iPhone/iPod Touch

2009-04-27T07:13:00.006+02:00

As my last post hints at I have started experimenting with building PIL for my iPod Touch. Doing so is non-trivial and I have therefore decided to write a small tutorial here, so that others may be spared of wasting their time on this project ;-)

What I did was to compile the thing directly on the iPod, and to do that I went through the following steps:

Jailbreak: I used QuickPWN 2.2.5 under Windows (it did not work on my Mac for some reason) and that was quite easy.
Install SSH on the iPod using Cydia (there is an entry in the main menu showing you how to do that).
SSH in to your iPod as the root user with password "alpine".
Install Python, the gcc compiler, the code signing tool, and make by issuing this command: apt-get install python iphone-gcc ldid make (alternatively you could install it using Cydia it you like).
Download the SDK headers, you may find them along with a description of how to use them here.
[Optional] Download the libjpeg source if you want jpeg support. Untar the source bundle, enter the directory containing the source and issue the following commands:
```
export C_INCLUDE_PATH=/var/include/
./configure --prefix=/usr/
make
make install
make install-lib
```
Download the PIL source and unpack it.

Enter the source directory and issue the following commands:

python setup.py build
python setup.py install
ldid -S /usr/lib/python2.5/site-packages/PIL/*.dylib

After going through all of these steps I ended up with a working PIL library on my iPod. Oh, and thanks to Renaud Blanch for pointing out to me that I needed to sign the dynamic libraries for them to work.

Scavenger on iPhone/iPod Touch

2009-04-26T19:57:00.004+02:00

The first thing I did after implementing the new Presence lib was of course to install Scavenger on my iPod Touch. And I am happy to report that it works! :-)

The Python version installable on a jailbroken iPhone/iPod is version 2.5.1, and since Scavenger is developed using Python 2.6 in a "backwards compatible" way everything worked like a charm.

The next step is getting the Python Imaging Library working on the iPod...

New Presence Consumer Lib

2009-04-26T14:17:00.003+02:00

I have just created a new Presence library - one that does not depend on the Presence daemon. This lib may be used by Python applications that only act as consumers of Presence services, i.e., application that do not advertise services through Presence but merely uses services announced by other peers.

So why did I make this new consumer-only lib? Because it enables devices that are not capable of running the Presence daemon to use Presence anyway (as a client/consumer right now). This means for example, that I can discover and use Presence services from my jailbroken iPod.

The new lib can be found in /trunk/libs/python_consumer_lib.

More about the cmp method in Python

2009-04-22T15:32:00.010+02:00

I just stumbled upon this little something about overloading the __cmp__ method in a Python class. If you are not careful, you will end up having a class whose instances can not be compared to None (as I always do to see if some optional parameter has been passed to a method or function). Lets take a small example:

class Foo(object):

def __init__(self, x):

    self._x = x

def __cmp__(self, other):

    if type(other) == Foo:

        return cmp(self._x, other._x)

    else:

        raise TypeError("Foo cannot be compared to %s"%str(type(other))) f1 = Foo(1)

f2 = Foo(2)

print 'f1 is smaller than f2:', f1 < f2 # True

print 'f1 is larger than f2:', f1 > f2  # False

print 'f1 is equal to f2:', f1 == f2    # False

print 'f1 is None:', f1 == None         # Crash!

This code crashes in the final statement, because the comparison to None calls the __cmp__ method, and it has no way of comparing to None. This could of course be fixed by adding "if other == None: return -1" but the right way to fix this would be by overloading another method called __eq__ that is used to test for object equality.

class Foo(object):

def __init__(self, x):

    self._x = x

def __eq__(self, other):

    if type(other) == Foo:

        return cmp(self, other) == 0

    else:

        return False

def __cmp__(self, other):

    if type(other) == Foo:

        return cmp(self._x, other._x)

     else:

         raise TypeError("Foo cannot be compared to %s"%str(type(other)))



f1 = Foo(1)

f2 = Foo(2)

print 'f1 is smaller than f2:', f1 < f2 # True print 'f1 is larger than f2:', f1 > f2  # False

print 'f1 is equal to f2:', f1 == f2    # False

print 'f1 is None:', f1 == None         # False

Now the comparison to None (or in fact to any object that is not of type Foo) is caught by the __eq__ method, and comparisons to other type Foo objects are correctly forwarded to the __cmp__ method.

Emacs ido-mode (minor mode)

2009-04-02T08:27:00.003+02:00

I have just spent some time play around with the Emacs "InteractivelyDoThings"-mode; ido-mode. Ido-mode replaces your regular "find-file" and "switch-buffer" mini-buffer with a much more flexible version; a version where you are allowed to do fussy matching, tab-complete your way to opening files on remote servers, and much much more...

Check out this screencast by Stuart Halloway if you have the time, or just dive into this EmacsWiki-page for a short explanation of what ido-mode does for you.

The "with"-statement in Python

2009-04-01T11:15:00.003+02:00

In Python 2.5 and onwards a new "with" statement has been introduced. I recently read a really great article by Fredrik Lundh about the inner workings of this statement, so please read it if you are interested in all the dreary details. The syntax of the with statement is shown below:


with expression [as name]
    code block

What it boils down to is this: The with statement expects its expression to evaluate to an object that has two methods: __enter__ and __exit__. These will then be called before and after executing the code within the statement. To quote the article "Now, when the “with” statement is executed, Python evaluates the expression, calls the __enter__ method on the resulting value (which is called a “context guard”), and assigns whatever __enter__ returns to the variable given by as. Python will then execute the code body, and no matter what happens in that code, call the guard object’s __exit__ method. "

This means, that you can use the with statement for other things than merely locking (which is probably the mostly know use). Consider the code below:


with open('foo.txt', 'r') as infile:
    print infile.read()

This would open the file, assign it the name infile in the statement code block, read and print the file, and finally close the file again. This is all because a file objects __enter__ method returns the opened file, and its __exit__ method closes the file. Pretty neat, huh? :-)

Reading battery status on an N800/N810

2009-03-31T15:15:00.003+02:00

For one of my projects I need to be able to read out the energy levels of the battery on my Nokia Internet Tablet (N810) - and I need to be able to do so from within a Python program, so that I may continuously monitor the energy usage of my program. After reading this blog post and talking to my friend Jussi, who had spent some days implementing such functionality in a C++ program of his, I spent about five minutes porting it to Python (thanks Jussi!). The result is shown below:


import dbus

bus = dbus.SystemBus()
hal_obj = bus.get_object ('org.freedesktop.Hal', 
                          '/org/freedesktop/Hal/Manager')
hal = dbus.Interface (hal_obj, 'org.freedesktop.Hal.Manager')
uids = hal.FindDeviceByCapability('battery')
dev_obj = bus.get_object ('org.freedesktop.Hal', uids[0])

print 'charge level percentage',\
      dev_obj.GetProperty('battery.charge_level.percentage')
print 'charge current', dev_obj.GetProperty('battery.reporting.current')
print 'charge design', dev_obj.GetProperty('battery.reporting.design')
print 'charge last full',\
      dev_obj.GetProperty('battery.reporting.last_full')
print 'charge unit', dev_obj.GetProperty('battery.reporting.unit')
print 'voltage current', dev_obj.GetProperty('battery.voltage.current')
print 'voltage design', dev_obj.GetProperty('battery.voltage.design')
print 'voltage unit', dev_obj.GetProperty('battery.voltage.unit')

This little snippet of Python code talks to the HAL layer through D-Bus, and in that way is capable of coming up with quite a lot of information about the battery. Check the complete HAL battery API for more information about what can be read through these calls.

Excellent article about securing Python

2009-03-30T17:13:00.003+02:00

It seems that there is in fact people out there that are actively trying to create a secure Python; as in a Python that can be used to execute untrusted code within, without endangering your trusted code (or filesystem for that matter).

I was google "securing python" out of interest, to see if something has happened in this respect since the time when I was looking into it when designing my Scavenger system, and it seems that there have been some important progress since then. I found this great article by Tav (Vageesan Sivapathasundaram). It describes how a security system based on object capabilities can be built (and is in fact being built) within the Python interpreter. Some of the ideas are reminiscent of my approach - but what they are doing here is way cooler than what I was playing around with ;-)

By the way they are having a hack-attack, exactly like I did when I was testing Scavenger's security system :-)

Building Stackless Python on a Mac

2009-03-28T11:34:00.004+01:00

I use Stackless Python for some of my work (and also for my open source project Scavenger). Seeing as I do most of my development on a Mac I have a number of times had the need to build Stackless Python from source. Doing that there is one thing that you should know: you need to add the configure option called --enable-stacklessfewerregisters, otherwise you will get the following error when compiling:


./Stackless/platf/switch_ppc_macosx.h:51: error: PIC register 'r31' 
clobbered in 'asm'

YASnippet LaTeX bundle

2009-03-25T14:35:00.003+01:00

I have started using YASnippet for Emacs and one thing that I immediately noticed was the shocking lack of LaTeX snippets. Yes, I know that the auxtex package lets you do everything by using some keyboard shortcuts, but I find it much more intuitive working with tab completed snippets. I have therefore begun a project of creating a proper LaTeX bundle for YASnippet - you may find the snippets I create here [old link!].

Update: The snippets have been moved to a GitHub project - you may find it here http://github.com/madsdk/yasnippets-latex/

It is still a work in progress, but as soon as I find my bundle to be complete enough I will start releasing versioned "builds". As of right now you have to download the snippets individually from the download section of the homepage.

Great article about the Python GIL

2009-03-25T09:55:00.002+01:00

I just stumbled upon this article about the Global Interpreter Lock in Python by Jesse Noller.

I was benchmarking my Stackless Python execution environment to see whether it was actually able to utilize my multi-core CPU fully - and it wasn't... In it I spawn a thread per CPU core and thus maintain separate stackless schedulers so that I may schedule multiple tasklets for execution at the same time - or so I thought :-)

The article by Jesse Noller is very nice, and I suggest that anyone interested in using threading in Python to do heavy lifting should read it. If you are using threads solely to get more responsiveness when doing blocking I/O calls there is no need to read it; just continue doing what it is you are doing ;-)

More Python YASnippets

2009-03-20T13:33:00.001+01:00

I found that snippets were missing for try- and with-statements as well, so I created some:


# contributor: Mads D. Kristensen 
# name: try
# --
try:
    $0
except ${1:Exception}, ${2:e}:
    pass


# contributor: Mads D. Kristensen 
# name: with
# --
with ${1:lock}:
     $0

These are very simple snippets but I find them immensely useful and I hope you will too :-)

Python property snippet for YASnippet

2009-03-20T13:04:00.002+01:00

I have recently begun using YASnippet for Emacs (AquaMacs on Mac actually). YASnippet adds powerful snippet support to Emacs reminiscent of the snippet support in TextMate. Using YASnippet you may for example in Python mode type "class" and then hit the Tab-key which would expand into a class-definition snippet where all of the boiler-plate code has already been written for you.

While YASnippet is quite powerful is does lack a number of standard snippets (or templates) in its standard library. For example I was missing a proper Python property snippet, so I wrote one:


# contributor: Mads D. Kristensen 
# name: prop
# --
def ${1:foo}():
   doc = """${2:Doc string}"""
   def fget(self):
       return self._$1
   def fset(self, value):
       self._$1 = value
   def fdel(self):
       del self._$1
   return locals()
$1 = property(**$1())

$0

Using this snippet you can write "prop" and press the Tab-key. Now you only need to provide a name for the property and, if you like, a doc string for the property. The snippet then creates a getter, setter, and deleter method for the property.

New blog

2009-03-19T13:29:00.003+01:00

I have decided to move part of my blogging to this blogspot.com site instead of my original blog. My old blog will still hold all my research related blogging, but any tech/development/personal blogging will from now on reside on this blog.

The first thing I will do, when I get the time, is to migrate old (relevant) blog posts from my old blog to this new site.

By the way, I will be writing on this blog in both English and Danish. I have therefore created both an "english" and a "danish" category, and I will try to remember to apply one of these to all posts. This way you may e.g. create an English-only feed if you should so desire.

Comparing and the cmp method in Python

2009-03-06T16:46:00.000+01:00

Sometimes it would be nice to be able to ask questions such as “does this array hold an object that has its instance variable ‘foo’ set to 42?” in a simpler way than by iterating through the entire array whilst comparing each objects ‘foo’ attribute to 42. This can be done quite easily in Python by overloading the __cmp__ method in your classes. The following example shows how that works in practice:


class Foo(object):
    def __init__(self, a_string, an_integer):
        self.my_string = a_string
        self.my_integer = an_integer

    def __cmp__(self, other):
        if type(other) == type(self):
            return cmp(self.my_string, other.my_string) and\
            cmp(self.my_integer, other.my_integer)
        elif type(other) == int:
            return cmp(self.my_integer, other)
        elif type(other) == str:
            return cmp(self.my_string, other)
        else:
            raise TypeError('Unable to compare %s to %s'%(type(self), type(other)))

f1 = Foo('hello', 1)
f2 = Foo('world', 2)
f3 = Foo('!', 3)
my_list = [f1, f2]
print f1 in my_list # Prints "True"
print f3 in my_list # Prints "False"
print 1 in my_list  # Prints "True"
print 42 in my_list # Prints "False"
print 'hello' in my_list  # Prints "True"
print 'mojn' in my_list  # Prints "False"
try:
    (1+0j) in my_list # Raises a TypeError
except TypeError, e:
    print e.message

PyGTK natively on Mac

2008-12-19T16:54:00.000+01:00

I recently found out that the GTK+ GUI widget set had been ported to run natively (i.e. without the use of X11) on Mac OS X. Information about this project can be found on this page. Now, I like GTK+ but life is too short to be coding in C, so I imediately started looking into the possibility of using PyGTK together with this native Mac version of GTK+ and that is indeed possible. To get a fully functional version of GTK+ and PyGTK installed onto your Mac follow the instructions on Imendio's page and then after finishing the installation add the following command:

jhbuild build pygtk

This will build libglade, pygtk, pycairo and some other stuff and install the necessary files into your site-packages folder of your local Python.

Building all of this from source is a lengthy process, and I have therefore created a .dmg image (Intel or PPC) of a GTK+-with-PyGTK that you may use as a shortcut :-) Using this approach will give you a fully functional native PyGTK to play with in less than two minutes.

Building PythonCE from source

2008-12-05T08:09:00.000+01:00

I have recently had the need to compile PythonCE from source, which turned out to be a quite complex task. Therefore I have decided to write this blog post, so that others may avoid wasting the many hours that I wasted trying to get it to build.

My setup for building PythonCE was this: Windows XP SP2, Microsoft eMbedded Visual C++ 4 with SP4, Pocket PC 2003 SDK, and Python 2.5.

To actually build the thing you need to download the PythonCE source, unzip it, enter the PCBuild/WinCE directory, and typing “scons”. This failed on my machine though, which I later found was due to a missing registry key in my Pocket PC 2003 SDK install (or something of the likes). To get it to compile i edited the file PCbuild/WinCE/msevc.py and made a small correction to the get_msevc_paths function - this is shown below. The addition I made was in the except-clause, where I manually set the exe_path.


def get_msevc_paths(env):
    ...
    try:
        value, value_type = SCons.Util.RegQueryValueEx(hkey, 'Path Dirs')
        exe_path = str(value.decode('utf_16_le').rstrip(u'\x00'))
    except (SCons.Util.RegError, SCons.Errors.InternalError):
        exe_path="C:\\Program Files\\Microsoft eMbedded C++ 4.0\\"\
                 "EVC\\wce420\\bin;C:\\Program Files\\"\
                 "Microsoft eMbedded C++ 4.0\\Common\\EVC\\Bin"

Type checking in Python

2008-08-19T16:49:00.000+02:00

I love working with Python, and most of the time I love the fact that it is a dynamically typed language. But, sometimes type checking would be nice to have; especially when debugging larger projects. Therefore I decided to look into how function/method decorators could help me there, and I very quickly found that it is possible to implement simple type checking quite easily using such decorators. So I have written a small Python module that adds the possibility of doing run-time type checking on functions and methods. The code for this module is here:


def decorator_with_args(decorator):
    def new(*args, **kwargs):
        def new2(fn):
            return decorator(fn, *args, **kwargs)
        return new2
    return new

@decorator_with_args
def typecheck(fn, *decorator_args):
    def new(*args):
        if len(decorator_args) != len(args):
            raise Exception('Wrong number of arguments given to\
                             decorator.')
        for x in range(0, len(args)):
            if type(args[x]) != decorator_args[x]:
                raise TypeError('Argument %i is of wrong type.\
                                 %s expected, %s received.'%\
                               (x+1, str(decorator_args[x]),
                                str(type(args[x]))))
        return fn(*args)
    return new

The reason that two decorators are created in the above code is, that the typecheck decorator is decorated by the decorator_with_args decorator to enable the decorator to take arguments - which is not otherwise possible.

Using the typecheck decorator is as simple as:


@typecheck(int, int)
def add(x, y):
    return x+y

Now, whenever the add function is called, it will be checked whether the arguments x and y are indeed integers, and if not a TypeError exception will be thrown.

Feel free to copy the code into your own projects - consider it released under a BSD licence :-)