Testing Restlet webservices from telnet

For the upcoming P2P course at Aarhus University (which I have been co-teaching for the past four years) I am currently playing around with the Restlet REST API. We want the students to learn something about peer-to-peer algorithms, protocols, and techniques, and seeing as we like a hands-on approach we always ask them to code a small P2P system as an ongoing exercise throughout the course. In past years we have asked them to do this programming using regular sockets, but that is about to change... While learning about the socket abstraction may be instrumental in teaching a coming computer scientist about the underlying constructs of the network stack, it may be time to teach them this stuff only cursory and then skip ahead to a higher level of abstraction. I mean, who (besides myself) actually works with sockets anymore?

We have therefore decided to teach the students about REST and let them use the Restlet API to create their own P2P system. But before throwing a new API at my students, I like to experiment with it myself, and I therefore dived into a little bit of Restlet programming. In the process I needed to be able to test a HTTP PUT call to my Restlet, which can not (afaik) be done easily from a webbrowser. I therefore went to my old friend telnet for help. So, if you need to test a HTTP PUT call to a webserver from telnet do the following:

telnet localhost 8182
PUT /peerlist HTTP/1.1
Host: localhost
Content-Length: 13

Hello, World!

Substitute host, URL, and content to your hearts contend.

Scavenger and Presence are moving to GitHub

I am currently re-booting my work on Scavenger (and therefore also on the Presence service discovery framework) as I need it for some upcoming research. In the process I have decided to move the source code from the Google Code pages and onto GitHub instead. In the last couple of months I have become increasingly fond of git---especially as a collaborative tool where other developers can freely fork my repositories, play around with the source on their own, and then send me a pull request when they are satisfied with their changes. This way of working has worked out extremely well in my small LaTeX YaSnippet project. I hope that switching to GitHub means that more people will consider adding to both Scavenger and Presence.

While moving the code I have also done quite a bit of refactoing and "spring cleaning". This means that the code has now been split into six (more or less) independent projects. All of these can be found here https://github.com/madsdk/

Installing the Scavenger client library or the Scavenger daemon has also been simplified quite a bit. Both the lib and the daemon now have build scripts that work on Mac and Linux. If I e.g., would like a Scavenger daemon running om my laptop, I would simply download and run this script. The script must be placed in the folder where you would like to install the Scavenger daemon, and it will then download and build all dependencies (Stackless Python, Presence daemon, diverse Python modules). Notice that in order to build the Presence daemon you need to have the Qt development tools installed.

Happy hacking!

Building Stackless Python on Snow Leopard

I have had a hard time debugging this problem I have had with running Stackless Python on my Snow Leopard Mac - that is until I stumbled upon this post. It seems that the problem is with using Stackless on a 64-bit Mac OS, and the solution is thus to build Stackless in a 32-bit version. So, if you're having problems with Stackless Python crashing with a "Bus error", do this when you configure during the build process:

CC="gcc -arch i386" ./configure --enable-stacklessfewerregisters

That seems to have solved the problem for me anyway ;-)

My LaTeX snippets for YaSnippet/Emacs are still alive

I have very few words for you this time. All I want to do, is to point out that my LaTeX snippets for YaSnippet (an Emacs snippet plugin) is still alive. I have not updated it myself for quite some time, since I seem to be satisfied with the collection of snippets I already have, but others have been doing the updating for me! Oh the joys of open source ;-)

The snippet collection is therefore still growing, thanks to additions made by Bjorn Reese, Song Qiang, and Claudio Marforio. Thanks you guys!

If you are interested in adding snippets to the collection please do one of the following (listed in prioritized order):
1) Create your own branch on github and send me a pull request, or
2) Send me the snippets by email.

SQLite with stdev (standard deviation) and more!

At the moment I am collecting data - hoards of data - my test suite is a veritable cornucopia of boring data. Normally I would be logging such data into flat files, collecting some giga bytes of log files that I would then need to write a handful of Python, Perl, or Ruby scripts to make any sense of. I have done this a thousand times before ... and I'm getting tired of it!

In comes my new best friend - SQLite. This small relational-database-in-a-file is excruciatingly easy to use from almost any programming language you could think of. So instead of writing a lot of data handling scripts, this time I have written only one - a script that imports the log data into an SQLite database. All other questions about the data, and how the different test results relate to each other, can thus be formulated in simple SQL instead of through increasingly complex Python scripts.

"When are you coming to the stdev part!", you say? Calm down, I'm getting there. Working with my data in a relational database I found that there were only one piece of functionality that I was missing, and that was the possibility to select out both an average _and_ the standard deviation of a data set. Luckily, this is possible in SQLite through a simple extension.

This is what you do:

1. Fetch the source code for the extension here [direct link]


2. Compile it:

   Mac$ gcc -fno-common -dynamiclib extension-functions.c -o libsqlitefunctions.dylib
   Linux$ gcc -fPIC -lm -shared extension-functions.c -o libsqlitefunctions.so



3. Copy the resulting library to some location where you want to store it. In my case that would be $HOME/opt/lib


4. Before executing your select statements be sure to load the extension in SQLite by issuing the following command (within SQLite):

   select load_extension('/full/path/to/libsqlitefunctions.dylib');

And you're done! Now you can issue select statements with stdev as an aggregate function. E.g.:

SELECT name, stdev(seconds) 
FROM benchmarks 
GROUP BY name

Note: The SQLite version included in Mac OS X 10.6 does not allow extensions, so you will have to build it yourself. That is luckily easily done - fetch the source here.

On CPU profiling

CPUs are different ... no really, they are. And when working with task scheduling over heterogeneous peers, which is the metier of Scavenger, this heterogeneity of the peers makes it very hard to come up with precise running time estimates. When estimating a task's running time on a peer one needs two things: 1) the running time of the task (with the given input) on a known peer, and 2) a "strength" rating on each peer that says something about how strong the peer is w.r.t. processing work. Having this information an estimated running time can be derived simply by assuming that a peer of strength x can perform a given task twice as fast as a peer of strength x/2.

This is all fairly straight forward, but the tricky part is finding that magical "strength" value for each peer. In related systems, such as Spectra and Chroma, the CPUs clock speed and its Linux BogoMIPS rating has been used, respectively. I expected that these values were pretty useless at estimating peer strength, and my experiments up until now seem to confirm that suspicion. Because of that I in Scavenger chose to use a real CPU benchmark for the strength value, thinking that a CPU benchmark would be written specifically to exercise all the various parts of a CPU and therefore would be more apt at describing its relative strength. I have used the nbench benchmarking suite - primarily because it is available as source code, so that I can compile and run it on any platform that I would like to test. Nbench returns two scores, an integer and a floating point score, so I chose to use the average of these two scores as peer strength. My experiments show that this value does indeed model CPU strength much better than clock speed or BogoMIPS - but it is far from perfect!

Because of that I have started work on a new profiling approach - one that will either 1) increase the accuracy when estimating running time on hitherto unknown peers, or 2) show that the performance of different CPU architectures can not be scored on a linear scale... Of course I am hoping for 1 but I somehow expect to conclude 2 :-)

Getting EyeTV to Put Your Mac to Sleep ... Intelligently!

One feature that I _really_ miss in EyeTV is the ability to put my Mac to sleep after a recording. It is possible to write an AppleScript that reacts to the RecordingDone signal in EyeTV, but my previous foray into that area ended up with a simple script that said something like

on RecordingDone(recordingID)
   tell application "Finder" to sleep
end RecordingDone

This works to some extent but it does have some serious flaws. First and foremost it ruins your second recording if you have scheduled to record two programs back-to-back. In that case EyeTV records the first program, emits the RecordingDone signal, starts recording the second program, and about five seconds into that recording the computer goes to sleep ;-) Another problem with this approach is when you are recording something while you are working on the computer - in that case your computer suddenly goes to sleep while you are working.

This has annoyed me quite a bit, but seemingly not enough for me to actively search for a better solution - until today. Googling around a bit I found this script: http://forums.elgato.com/viewtopic.php?f=91&t=1045#p18834. I have altered it a tiny bit to change the timeouts and to put the computer to sleep instead of powering it off. The result is here:

on RecordingDone(recordingID)
   set isCurrentlyRecording to check_eyetv_is_recording()
   if isCurrentlyRecording is false then
       with timeout of 300 seconds
           tell me to activate
           display dialog "Warning: EyeTV has now finished recording and the system will automatically shut down in 5 minutes unless you click Stop!" with icon 2 buttons {"Stop!"} giving up after 300 default button "Stop!"
           set theresults to the result
           if gave up of theresults is true then
               tell application "System Events" to sleep
           end if
       end timeout
   else
       quit
   end if
end RecordingDone

to check_eyetv_is_recording()
   delay 30
   tell application "EyeTV"
       if is_recording then
           return true
       else
           return false
       end if
   end tell
end check_eyetv_is_recording

This script works perfectly! When the recording is done a dialog is displayed telling you that the computer will be put to sleep in five minutes, and if you press the stop button you can keep the computer awake.

Update: A guy called Michael asked we where to put the script to make it work. You are supposed to save the scripts as /Library/Application Support/EyeTV/Scripts/TriggeredScripts/RecordingDone.scpt