Building the Scavenger Python library

This is the second "building scavenger" post -- see the post immediately before this one for a description of how to build the Scavenger daemon. In this post I will describe how you can install the Scavenger Python library on a client device, so that you may start offloading tasks to your Scavenger daemon instance.

When installing the Scavenger library you designate a Python interpreter that you wish to install the Python modules into, so the first dependency is of course Python. Secondly, the script building the library uses git to fetch the source code from GitHub, so git is a dependency too.

Once you have these two dependencies installed, building and installing the Scavenger library (on a Mac or Linux machine) is as simple as 1) downloading this script, and 2) running the script with the path to the chosen Python interpreter as an argument.

$ wget https://github.com/madsdk/scavenger-library/raw/master/build/build_and_install.sh
$ sh build_and_install.sh /usr/bin/python

If you have any questions about getting Scavenger to run on your system, please don't hesitate to contact me. The best thing to do is to leave a comment on this page, but you are also welcome to send me a mail.

Building the Scavenger daemon

I recently moved the development of Scavenger onto GitHub. In that process I created two build scripts: One for building the Scavenger daemon on a Linux or Mac machine, and another for building and installing the Scavenger library on the same platforms. In this post I will describe the installation of the Scavenger daemon.

In order to build the Scavenger daemon, you need to install a couple of dependencies. You need git, the gcc compiler and, in order to compile the Presence daemon used for discovery, you also need to install the Qt SDK. If you are on a Mac, getting gcc installed means that you should install Xcode and getting git installed can e.g., be done using MacPorts.

After having installed these dependencies building the Scavenger daemon is as simple of downloading and running the Mac script or the Linux script. The script should be downloaded to and run from the folder in which you would like to build the Scavenger daemon. When run it will download and build Stackless Python, checkout and build the relevant GitHub projects, and configure a Scavenger daemon instance.

After having successfully run the build script, you can start a Scavenger daemon instance by entering the scavenger_daemon directory and running the start_daemon.sh script.

If you have any questions about getting Scavenger to run on your system, please don't hesitate to contact me. The best thing to do is to leave a comment on this page, but you are also welcome to send me a mail.

Scavenger and Presence are moving to GitHub

I am currently re-booting my work on Scavenger (and therefore also on the Presence service discovery framework) as I need it for some upcoming research. In the process I have decided to move the source code from the Google Code pages and onto GitHub instead. In the last couple of months I have become increasingly fond of git---especially as a collaborative tool where other developers can freely fork my repositories, play around with the source on their own, and then send me a pull request when they are satisfied with their changes. This way of working has worked out extremely well in my small LaTeX YaSnippet project. I hope that switching to GitHub means that more people will consider adding to both Scavenger and Presence.

While moving the code I have also done quite a bit of refactoing and "spring cleaning". This means that the code has now been split into six (more or less) independent projects. All of these can be found here https://github.com/madsdk/

Installing the Scavenger client library or the Scavenger daemon has also been simplified quite a bit. Both the lib and the daemon now have build scripts that work on Mac and Linux. If I e.g., would like a Scavenger daemon running om my laptop, I would simply download and run this script. The script must be placed in the folder where you would like to install the Scavenger daemon, and it will then download and build all dependencies (Stackless Python, Presence daemon, diverse Python modules). Notice that in order to build the Presence daemon you need to have the Qt development tools installed.

Happy hacking!

On CPU profiling

CPUs are different ... no really, they are. And when working with task scheduling over heterogeneous peers, which is the metier of Scavenger, this heterogeneity of the peers makes it very hard to come up with precise running time estimates. When estimating a task's running time on a peer one needs two things: 1) the running time of the task (with the given input) on a known peer, and 2) a "strength" rating on each peer that says something about how strong the peer is w.r.t. processing work. Having this information an estimated running time can be derived simply by assuming that a peer of strength x can perform a given task twice as fast as a peer of strength x/2.

This is all fairly straight forward, but the tricky part is finding that magical "strength" value for each peer. In related systems, such as Spectra and Chroma, the CPUs clock speed and its Linux BogoMIPS rating has been used, respectively. I expected that these values were pretty useless at estimating peer strength, and my experiments up until now seem to confirm that suspicion. Because of that I in Scavenger chose to use a real CPU benchmark for the strength value, thinking that a CPU benchmark would be written specifically to exercise all the various parts of a CPU and therefore would be more apt at describing its relative strength. I have used the nbench benchmarking suite - primarily because it is available as source code, so that I can compile and run it on any platform that I would like to test. Nbench returns two scores, an integer and a floating point score, so I chose to use the average of these two scores as peer strength. My experiments show that this value does indeed model CPU strength much better than clock speed or BogoMIPS - but it is far from perfect!

Because of that I have started work on a new profiling approach - one that will either 1) increase the accuracy when estimating running time on hitherto unknown peers, or 2) show that the performance of different CPU architectures can not be scored on a linear scale... Of course I am hoping for 1 but I somehow expect to conclude 2 :-)

Scavenger Daemon - VMWare Fusion Image

I have just uploaded a VMWare Fusion 3 virtual machine image that you may use to tryout Scavenger. The image is a minimal Ubuntu 9.10 server install with Presence and Scavenger pre-installed - you can fetch it here.

Here's how you use it:

• Download and unpack the zip file.


• Open the VM in VMWare Fusion 3. When it prompts you the first time select "Take ownership", and the second time you must choose "I copied it".


• Login with username "scavenger" and password "scavenger".


• Go to $HOME/scavenger (i.e., "cd scavenger").


• Start the Scavenger daemon by issuing this command: "./start_daemon.sh"

Now you have a fully functional Scavenger daemon running. The Scavenger daemon is running from an SVN checkout, so you may update it to the most current version by issuing the command "svn up daemon scavenger_common" from within the $HOME/scavenger directory.

In order to make the daemon running within the VM discoverable you of course have to configure the networking options to fit your environment. I think it is set to use bridged networking at the moment, which should fit in most circumstances.

AugIm v. 2 released!

I have just released an entirely new version of my Scavenger demonstrator AugIm (The Augmented Image Manager). AugIm is now Maemo 5 only - and it is written in PyQt so you need to add the extras-devel repository in order to install it.

You can grab the demonstrator at Scavenger's Google Code page, or you could just use this direct link.

B.t.w., if you look in 'branches' in Scavenger's SVN you will find an alternative version of AugIm. This version does _not_ use Scavenger, so it may be used as a comparison to the fully-fledged, Scavenger-enabled version.

I have also created a VMWare Fusion image of a Scavenger daemon to make testing Scavenger easier, but I have yet to find out how to make such a VM distributable using Fusion... I'll probably link to the current Fusion-only image in the near future, and if I get the time I will try to create an OVF compliant version.

Building Stackless Python on a PS3 (PPC Linux)

I have recently gotten my hands on a PlayStation 3 that I am trying to use as a surrogate[1] in my Scavenger system. The first thing I did was to install Ubuntu Linux 9.10 on it, and the next step was then getting Stackless Python to compile... Which turned out to be a bit of a bitch ;-)

But after some heavy debugging and googling I found out how to do it:

1. Download the Stackless source distribution. I used version 2.6.4.


2. Unpack it somewhere and enter the newly created directory.


3. $ ./configure --prefix=/where/you/want/it
   


4. Edit the file called "Makefile" and change the variable SLPFLAGS to "SLPFLAGS=-fomit-frame-pointer -O2"


5. make


6. make install

It's very simple actually, just took me some time to find the solution :-) By the way, if you do not edit the Makefile you get the following error:

./Stackless/core/slp_transfer.c: In function 'slp_transfer':
./Stackless/core/slp_transfer.c:144: error: 31 cannot be used in asm here
./Stackless/core/slp_transfer.c:144: error: 31 cannot be used in asm here
make: *** [Stackless/core/slp_transfer.o] Error 1

Looking at that error the 31 hints at that the save registers code for PPC is trying to save register r31. This is reminiscent of the error you get when compiling Stackless under Mac OS X, but in this case adding the --enable-stacklessfewerregisters option does not work.

[1]: A surrogate in a cyber foraging system is a strong machine that performs heavy work on behalf of the weaker (often mobile) clients.

Excited about Scavenger...

Just wanted to let you know that I have posted something about Scavenger on my "work" blog. I have conducted some benchmarks of Scavengers parallel processing abilities and it is looking very promising I must say.

Scavenger tutorial available

I have finally gotten around to writing some documentation of Scavenger, so that others may in fact start writing Scavenger-enabled applications. I have spent the entire day updating the wiki pages of Scavengers GoogleCode homepage so now you will find a tutorial there along with some other related information about scavenger. Enjoy!

Scavenger demo application released!

I have just released the first Scavenger demo app: AugIM - the Augmented Image Manager. AugIM is an old demonstrator that I made a long time ago, and now I have ported it to use Scavenger for its cyber foraging tasks.


AugIM is an image browser/editor for Maemo (running on Nokia's N800/N810 devices) that allows you to do simple image manipulation tasks such as sharpening, resizing, adjusting color/contrast/brightness, rotating, flipping etc. It also allows you to share your images with the world through email or Google Picasa Web. The N800 and N810 devices are capable of performing these services on their own, but it takes a long time for them to do so and it consumes a very large amount of energy. Using Scavenger, AugIM makes sure that heavy processing is offloaded to larger surrogates whenever possible, which means that the image manipulation tasks are performed faster and using less energy on the mobile device.

Yet another Scavenger build...

I have just uploaded a new self-contained Scavenger daemon build to Scavengers Google Code page. This build is for Mac OS X 10.5 (Leopard) and it contains the same components as the Ubuntu Linux version I released yesterday.

From your Mac simply download the archive, unpack it, and start the start_daemon script from a terminal. Then a Scavenger daemon should be running on your machine, and you are ready to start playing with cyber foraging! :-)

Scavenger build available for Ubuntu 9.04

I have finally gotten around to creating a Scavenger build that you can use to easily experiment with Scavenger, without having to install Stackless Python and other dependencies (e.g., Presence).

The build, which is created and tested to run on Ubuntu 9.04, is available here. All you need to do is 1) download the archive, 2) unpack it, and 3) run the "start_daemon" script.

The "bundle" contains Stackless Python 2.6.2, the Presence python lib and daemon, PIL (Python Imaging Library), Google Data API, and the Scavenger daemon.

Scavenger - Now With History Based Profiling

I have just added yet another scheduler to Scavenger (yes, there is three of them now). This scheduler is the "smartest" of the three as it uses history based profiling in order to be able to choose the right surrogate for the job. History based profiling in a cyber foraging system is nothing new - the Chroma system used that some 6-7 years ago. What is new is the way that the profiling information is used within the Scavenger system. In Scavenger the profiling information can be used not only to tell you something about how the task will perform on certain surrogates (i.e., the ones where you have earlier performed the task), it is also capable of making an informed estimate of the running time on an unknown surrogate... more on that later ;-)

Scavenger on iPhone/iPod Touch

The first thing I did after implementing the new Presence lib was of course to install Scavenger on my iPod Touch. And I am happy to report that it works! :-)

The Python version installable on a jailbroken iPhone/iPod is version 2.5.1, and since Scavenger is developed using Python 2.6 in a "backwards compatible" way everything worked like a charm.

The next step is getting the Python Imaging Library working on the iPod...