Building Stackless Python on Snow Leopard

I have had a hard time debugging this problem I have had with running Stackless Python on my Snow Leopard Mac - that is until I stumbled upon this post. It seems that the problem is with using Stackless on a 64-bit Mac OS, and the solution is thus to build Stackless in a 32-bit version. So, if you're having problems with Stackless Python crashing with a "Bus error", do this when you configure during the build process:

CC="gcc -arch i386" ./configure --enable-stacklessfewerregisters

That seems to have solved the problem for me anyway ;-)

My LaTeX snippets for YaSnippet/Emacs are still alive

I have very few words for you this time. All I want to do, is to point out that my LaTeX snippets for YaSnippet (an Emacs snippet plugin) is still alive. I have not updated it myself for quite some time, since I seem to be satisfied with the collection of snippets I already have, but others have been doing the updating for me! Oh the joys of open source ;-)

The snippet collection is therefore still growing, thanks to additions made by Bjorn Reese, Song Qiang, and Claudio Marforio. Thanks you guys!

If you are interested in adding snippets to the collection please do one of the following (listed in prioritized order):
1) Create your own branch on github and send me a pull request, or
2) Send me the snippets by email.

SQLite with stdev (standard deviation) and more!

At the moment I am collecting data - hoards of data - my test suite is a veritable cornucopia of boring data. Normally I would be logging such data into flat files, collecting some giga bytes of log files that I would then need to write a handful of Python, Perl, or Ruby scripts to make any sense of. I have done this a thousand times before ... and I'm getting tired of it!

In comes my new best friend - SQLite. This small relational-database-in-a-file is excruciatingly easy to use from almost any programming language you could think of. So instead of writing a lot of data handling scripts, this time I have written only one - a script that imports the log data into an SQLite database. All other questions about the data, and how the different test results relate to each other, can thus be formulated in simple SQL instead of through increasingly complex Python scripts.

"When are you coming to the stdev part!", you say? Calm down, I'm getting there. Working with my data in a relational database I found that there were only one piece of functionality that I was missing, and that was the possibility to select out both an average _and_ the standard deviation of a data set. Luckily, this is possible in SQLite through a simple extension.

This is what you do:

1. Fetch the source code for the extension here [direct link]


2. Compile it:

   Mac$ gcc -fno-common -dynamiclib extension-functions.c -o libsqlitefunctions.dylib
   Linux$ gcc -fPIC -lm -shared extension-functions.c -o libsqlitefunctions.so



3. Copy the resulting library to some location where you want to store it. In my case that would be $HOME/opt/lib


4. Before executing your select statements be sure to load the extension in SQLite by issuing the following command (within SQLite):

   select load_extension('/full/path/to/libsqlitefunctions.dylib');

And you're done! Now you can issue select statements with stdev as an aggregate function. E.g.:

SELECT name, stdev(seconds) 
FROM benchmarks 
GROUP BY name

Note: The SQLite version included in Mac OS X 10.6 does not allow extensions, so you will have to build it yourself. That is luckily easily done - fetch the source here.

On CPU profiling

CPUs are different ... no really, they are. And when working with task scheduling over heterogeneous peers, which is the metier of Scavenger, this heterogeneity of the peers makes it very hard to come up with precise running time estimates. When estimating a task's running time on a peer one needs two things: 1) the running time of the task (with the given input) on a known peer, and 2) a "strength" rating on each peer that says something about how strong the peer is w.r.t. processing work. Having this information an estimated running time can be derived simply by assuming that a peer of strength x can perform a given task twice as fast as a peer of strength x/2.

This is all fairly straight forward, but the tricky part is finding that magical "strength" value for each peer. In related systems, such as Spectra and Chroma, the CPUs clock speed and its Linux BogoMIPS rating has been used, respectively. I expected that these values were pretty useless at estimating peer strength, and my experiments up until now seem to confirm that suspicion. Because of that I in Scavenger chose to use a real CPU benchmark for the strength value, thinking that a CPU benchmark would be written specifically to exercise all the various parts of a CPU and therefore would be more apt at describing its relative strength. I have used the nbench benchmarking suite - primarily because it is available as source code, so that I can compile and run it on any platform that I would like to test. Nbench returns two scores, an integer and a floating point score, so I chose to use the average of these two scores as peer strength. My experiments show that this value does indeed model CPU strength much better than clock speed or BogoMIPS - but it is far from perfect!

Because of that I have started work on a new profiling approach - one that will either 1) increase the accuracy when estimating running time on hitherto unknown peers, or 2) show that the performance of different CPU architectures can not be scored on a linear scale... Of course I am hoping for 1 but I somehow expect to conclude 2 :-)

Getting EyeTV to Put Your Mac to Sleep ... Intelligently!

One feature that I _really_ miss in EyeTV is the ability to put my Mac to sleep after a recording. It is possible to write an AppleScript that reacts to the RecordingDone signal in EyeTV, but my previous foray into that area ended up with a simple script that said something like

on RecordingDone(recordingID)
   tell application "Finder" to sleep
end RecordingDone

This works to some extent but it does have some serious flaws. First and foremost it ruins your second recording if you have scheduled to record two programs back-to-back. In that case EyeTV records the first program, emits the RecordingDone signal, starts recording the second program, and about five seconds into that recording the computer goes to sleep ;-) Another problem with this approach is when you are recording something while you are working on the computer - in that case your computer suddenly goes to sleep while you are working.

This has annoyed me quite a bit, but seemingly not enough for me to actively search for a better solution - until today. Googling around a bit I found this script: http://forums.elgato.com/viewtopic.php?f=91&t=1045#p18834. I have altered it a tiny bit to change the timeouts and to put the computer to sleep instead of powering it off. The result is here:

on RecordingDone(recordingID)
   set isCurrentlyRecording to check_eyetv_is_recording()
   if isCurrentlyRecording is false then
       with timeout of 300 seconds
           tell me to activate
           display dialog "Warning: EyeTV has now finished recording and the system will automatically shut down in 5 minutes unless you click Stop!" with icon 2 buttons {"Stop!"} giving up after 300 default button "Stop!"
           set theresults to the result
           if gave up of theresults is true then
               tell application "System Events" to sleep
           end if
       end timeout
   else
       quit
   end if
end RecordingDone

to check_eyetv_is_recording()
   delay 30
   tell application "EyeTV"
       if is_recording then
           return true
       else
           return false
       end if
   end tell
end check_eyetv_is_recording

This script works perfectly! When the recording is done a dialog is displayed telling you that the computer will be put to sleep in five minutes, and if you press the stop button you can keep the computer awake.

Update: A guy called Michael asked we where to put the script to make it work. You are supposed to save the scripts as /Library/Application Support/EyeTV/Scripts/TriggeredScripts/RecordingDone.scpt

YASnippet LaTeX bundle GitHub Project

A long time ago I wrote about a LaTeX YASnippet bundle that I was building - but I never really got around to publishing it in a way so that anybody except me could get at them. I did create a Google Sites homepage for it, but that's a terrible way to share source code :-)

I have therefore just created a GitHub project called yasnippets-latex. Now my snippets are freely available - released under the GPL of course. So grab them, play with them, branch the project, create your own snippets, improve my snippets, etc.

Playing around with MacJournal

I have recently bought the MacHeist nanoBundle2 which included a license for MacJournal - yet another journaling software for Mac. I already have Evernote, which seems to be far better than MacJournal for journalling, but MacJournal has some blogging features that I may be interested in. So I guess I’ll give it a try. This entry is written in MacJournal b.t.w. ;-)

This image was just copied into the editor window - and then resized a bit to try that out as well. It’s a photo I took during my recent vacation in Sweden b.t.w.


Now lets try some pre-formatted text (I need that for displaying code examples). I did this by changing the font in MacJournal to Courier 12pt - I would have liked to edit HTML tags instead...:

def compare_us(x, y):
return true if x == y else false

And then some regular text again...

Edit: Well, that did not work at all! I guess MacJournal is not for me then :-) My initial experiences with the software have not been very good. The first time I added a blog the application crashed, and I had to try five times before the auto update function actually worked. When adding a blog entry it did not append any of the tags I chose, the editor did not allow me to edit the HTML source etc. Conclusion: it is a crappy blog editor.

XeLaTeX on a Mac

As I wrote a couple of hours ago, I am currently writing my thesis using XeLaTeX. In the process I have found that the MacPorts version of XeLaTeX seems to be broken when it comes to accepting/finding user-installed fonts. My LaTeX template needs a font called Fontin Sans that I have downloaded and installed into my "Font Book" on my Mac. After doing that the MacPorts version of XeLaTeX still couldn't find the font, so I uninstalled the texlive packages fro MacPorts and installed the MacTeX 2009 package instead. And now everything works :-)

XeLaTeX and Aquamacs Emacs

I have finally begun writing my thesis... or sort of anyway - I have definitely started playing around with a LaTeX skeleton for it :-P

My friend Clemens gave me the LaTeX template that he used for his thesis, and it creates really pretty documents so I definitely wanted to use it for my thesis as well. But, this template makes use of XeLaTeX, a LaTeX variant that supports use of Unicode and "modern font technologies". I found that I already had xelatex installed on my system; apparently it comes with the MacPorts LaTeX installation, but it did not work immediately in Aquamacs. There _was_ a XeLaTeX command in Aquamacs, but in the background it was calling pdflatex, which in turn went into some never ending loop using a full CPU core...

In order to fix that you need to do change the way the xelatex command is called in Aquamacs, as described here.

1. Open Aquamacs in LaTeX mode (by opening a .tex file).
2. Go the the LaTeX menu, choose "Customize AUCTeX", choose "TeX Command", choose "TeX Command List".
3. Navigate to the XeLaTeX command (just search for xelatex).
4. Alter the "Command" part of that section to "%`xelatex --synctex=1%(mode)%' %t"
5. Set the "How" part to "TeX-run-command".
6. Make sure that LaTeX, ConTeXt, and AmSTeX have been checked off in the bottom part of the section.
   
7. Save the settings (Ctrl+x, Ctrl+s as usual).

And now it works!

Scavenger Daemon - VMWare Fusion Image

I have just uploaded a VMWare Fusion 3 virtual machine image that you may use to tryout Scavenger. The image is a minimal Ubuntu 9.10 server install with Presence and Scavenger pre-installed - you can fetch it here.

Here's how you use it:

• Download and unpack the zip file.


• Open the VM in VMWare Fusion 3. When it prompts you the first time select "Take ownership", and the second time you must choose "I copied it".


• Login with username "scavenger" and password "scavenger".


• Go to $HOME/scavenger (i.e., "cd scavenger").


• Start the Scavenger daemon by issuing this command: "./start_daemon.sh"

Now you have a fully functional Scavenger daemon running. The Scavenger daemon is running from an SVN checkout, so you may update it to the most current version by issuing the command "svn up daemon scavenger_common" from within the $HOME/scavenger directory.

In order to make the daemon running within the VM discoverable you of course have to configure the networking options to fit your environment. I think it is set to use bridged networking at the moment, which should fit in most circumstances.

AugIm v. 2 released!

I have just released an entirely new version of my Scavenger demonstrator AugIm (The Augmented Image Manager). AugIm is now Maemo 5 only - and it is written in PyQt so you need to add the extras-devel repository in order to install it.

You can grab the demonstrator at Scavenger's Google Code page, or you could just use this direct link.

B.t.w., if you look in 'branches' in Scavenger's SVN you will find an alternative version of AugIm. This version does _not_ use Scavenger, so it may be used as a comparison to the fully-fledged, Scavenger-enabled version.

I have also created a VMWare Fusion image of a Scavenger daemon to make testing Scavenger easier, but I have yet to find out how to make such a VM distributable using Fusion... I'll probably link to the current Fusion-only image in the near future, and if I get the time I will try to create an OVF compliant version.

PyQt: Emitting events from non-QObject subclasses.

In order to emit events in PyQt the emitting class must inherit directly from QObject. This can be quite a nuisance, especially so because multiple inheritance is impossible when using PyQt. E.g., I recently had the problem that I had task running in a QThreadPool that I would like to emit a signal when progress had been made, so that I could update a progressbar in my UI. To make the task runnable in a QThreadPool the task class had to inherit from QRunnable, and as QRunnable is not a subclass of QObject it was impossible to emit signals from within the task class. I found a solution to this problem though - one that I find quite elegant, so I'd like to share it with you :-)

The trick is to use another object as the mediator for emitting signals, and then using Python properties to hide the fact that a mediator is being used. Consider the example below:

class Mediator(QObject):
    mySignal = pyqtSignal(int)
    def __init__(self):
        super(Mediator, self).__init__()

class Task(QRunnable):
    def __init__(self):
        super(Task, self).__init__()
        self._mediator = Mediator()

    def mySignal():
        def fget(self):
            return self._mediator.mySignal
        return locals()
    mySignal = property(**mySignal())

Using this approach any classes working with Task instances can connect directly to the mySignal signal as if it was bound directly to the Task class. So the property hides the fact that a mediator is in use.

Embedding fonts in Gnuplot generated PDFs

I have recently submitted two papers to an IEEE conference, and in doing so I had to make sure that _all_ fonts were embedded within the PDFs I submitted. This was not the case with one of my papers, and after some detective work I found out that this pertained to my using Gnuplot's PDF terminal to generate PDF graphs for my paper. Apparently Gnuplot is incapable of embedding fonts into the generated PDFs...

I found this solution though. You need to pass the PDF that Gnuplot generated through ghostscript with the following command:

gs -q -dSAFER -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=tmp.pdf\
-dCompatibilityLevel=1.5 -dPDFSETTINGS=/prepress -c .setpdfwrite\
-f infile.pdf

Where 'infile.pdf' is your Gnuplot graph. If you follow the link to where I found this hack, you may notice that I have changed the commandline slightly, so that it now creates PDF 1.5 compatible output. This is a requirement of IEEE papers I think.

Avoiding "Size mismatch" Errors When Upgrading Maemo 5 Scratchbox

I have just returned to developing Maemo application, and as usual when I come back to my Maemo scratchbox installation, I start off by upgrading the development environment. So I opened my scratchbox and issued the commands

fakeroot apt-get update
fakeroot apt-get dist-upgrade

Unfortunately this did not work for me this time because of some inconsistencies in the index files kept at the Maemo servers (I think). I kept getting these "Failed to fetch http://someurl Size mismatch" errors. An example:

...
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-application-manager-l10n/hildon-application-manager-l10n-plpl_6.0+r7389+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-application-manager-l10n/hildon-application-manager-l10n-ptpt_6.0+r7389+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-application-manager-l10n/hildon-application-manager-l10n-ruru_6.0+r7389+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-application-manager-l10n/hildon-application-manager-l10n-svse_6.0+r7389+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/nokia-binaries/4bc37c7c77ebe90177c050b805a8dc79/nokia-binaries/h/hildon-theme-alpha/hildon-theme-alpha_9.1+0m5_all.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/free/m/maemo-launcher/maemo-launcher_0.35-7+0m5_i386.deb  Size mismatch
Failed to fetch http://repository.maemo.org/pool/maemo5.0/free/m/maemo-launcher/maemo-launcher-dev_0.35-7+0m5_all.deb  Size mismatch
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

To avoid these errors I manually downloaded the packages involved and installed them with dpkg. To make this process easier I simply copied the error messages into a text file, removed the "Failed to fetch" prefix and the "Size mismatch" postfix from all lines using search/replace, and saved this as files.txt. Then I invoked the following commands from within my scratchbox targets:

cat files.txt | xargs wget
dpkg -i *.deb

And voila, it works ;-)