rpython and numpy

looks like at least a few people connected with pypy have an eye on numpy. there's a very small and limited demo in the svn right now. but other devs have said it is hard and not their main interest. i hope it at least remains possible, at some future date. maybe more promising is some work on ufuncs by one of the enthought people. it shows a good example of how to use rpython to compile a function. maybe i can use this to autogenerate c code for scipy.weave.inline. here's the most relevant example:

from pypy.translator.interactive import Translation
class compdec:
   def __init__(self, func):
       self.func = func
       self.argtypes = None
   def __call__(self, *args):
       argtypes = tuple(type(arg) for arg in args)
       if argtypes != self.argtypes:
           self.argtypes = argtypes
           t = Translation(self.func)
           t.annotate(argtypes)
           self.cfunc = t.compile_c()
       return self.cfunc(*args)
@compdec
def is_prime(n):
   if n < 2:
       return False
   for i in xrange(2, n):
       if n%i == 0:
           return False
   return True
print sum(is_prime(n) for n in xrange(100000))

i think that it would be easy to change the check/recompile to a dict lookup. that __call__ might also be compiled with rpython since args is always a tuple. that would add only 1 c function call overhead and preserve dynamic typing.

python guis with avc

avc looks like an interesting concept. it's a pure python library that connects gui toolkit variables to variables in the scope of your python script, with bidirectional communication. looks like a very clean and easy way to separate the view from the logic, and it has a standardized interface for gtk, qt3, qt4, tk, and wxwidgets. tk doesn't look as good as the others, but it's easy to get with python. gtk and wxwidgets look like the most feature-complete. all of the toolkits have their own optional external interface def files, and avc can use those. but you can also set up everything from the comfort of your python script. only problem: it's gpl, so i might need to use something else if i want to sell my stuff. EDIT: i think enthought's Traits might be usable in a similar way. and if i'm not mistaken, they license their open source stuff bsd.

python data storage

after a little bit of optimization, i'm finding the bottleneck now is reading in the data. i think i've found about all the ways to speed up cPickle (most recent protocol, Pickler().fast = True) and the next step will be to a real database. i'm not sure the python builtins will buy me much, though, and i think if i'm going to have to install something it might as well be pytables. pytables is an interface layer on top of hdf5, so it's probably best for large volumes of numerical data. it only requires hdf5 (which built without problems: configure && make install) and numpy. i had to set HDF5_DIR and add paths for LD_LIBRARY_PATH and PYTHONPATH, not being root. but overall a very painless install. the data structure is bound to be more complex than a simple pickle, but there are some good tutorials out there. also, the nmag project has some good real-life experience with using pytables for unstructured grid data. (see hdf5*py in nmag-0.1/nsim/interface/nfem) uiuc and cei (the people who make ensight) also defined an hdf5 mesh api, but it looks pretty krufty now. another plus with pytables is that you can use vitables to interact with the data. that's even easier than a pickle. EDIT: hers's a site that covers a lot of the issues with scientific data storage and refers to specific examples, including hdf. one problem that might arise with pytables is that numpy arrays can be memory mapped to a file on disk, but pytables can't do that if i'm not mistaken. am i? according to this email exchange, pytables doesn't do mmap but it can be as fast or faster if used properly. sounds like i can still use pytables without losing performance, but i will need to reread that and some of the refs therein to implement. here's an interesting conversation about large file i/o in python, with specific applications in finance.

python profiling and optimization

python's builtin cProfile/pstats are great, and it is really easy to get started with them. (just look at the top of the doc page for cProfile.) one problem, though, is that cProfile doesn't do line-by-line profiling. hotshot did, but it's deprecated and arguably harder to work with. a little poking around turned up line_profiler and kernprof. looks like they're the pet project of one of the enthought people. i haven't tried it yet but it looks pretty easy to use, even through the api as a part of a larger development tool. i'm thinking the best way to use it would be to run cProfile first and take the top 3 time hog functions (or 1 or 2 if they get over 90% total time) and auto run it again with line_profiler turned on for those functions. maybe i could build this into my doctester by first running the small and fast doctests to confirm correctness, then run some __profileTests__ module string that has more realistic usages. while i'm writing code, i can interrupt the profile tests to rerun the doctests, and i'll see the profile results any time i happen to let it run all the way through. kernprof is pure python and acts as a helper/wrapper for both/either cProfile and line_profiler. but line_profiler uses cython. speaking of cython, for optimization cython looks like an impressive way to generate c source and build it to run fast (and obfuscate, if that matters). i might look at it as a possible alternative to pypy/rpython, if i can't get that to work. one problem, though, is that it looks like so many changes and type annotations might be needed with numpy that i might as well just write the c. i've been very impressed lately with results i get using scipy.weave, with both the blitz converter and the default c-ish converter. i can narrow the hotspots down to one or two methods, and translating the python to c is not too hard then (especially with the default converter). scipy.weave.inline is not too shabby either, for speeding up numpy expressions. so pypy is another very interesting possibility for autogenerating c from python. someone has already shown an example of how to do this, including some initial work with numpy. EDIT: maybe using cython is not as bad as i had thought. apparently it will compile just about any valid python (with a couple of restrictions on closures) with a modest speed up. all the extra syntax is just for making it run faster, and even those annotations can be done in a way that preserves python compatibility of the original source. maybe i can even look at multiple-dispatch packages to make an auto-type discovery based on doctests and profile tests to make my augmenting .pxd files for me. probably the first thing i should do is try to replicate some of the speed-up results from the cython numpy tutorial. there's also another tutorial with a convolution example (similar to chapter 10 in the manual, but slightly updated including warning about None args and calling numpy c funcs from cython) that might be a better place to start.

chapter 11 in the user manual gives an example of how to profile with python's profiling tools, while the wiki page on profiling shows a (partial) example of low-overhead profiling with valgrind. debugging with ddd and gdb is also discussed in a wiki page at wiki.cython.org. also, since a discussion on their email list in april/may 2009, an --embed option has emerged that facilitates embedding a python interpreter with a python entry point. i had been looking at pyinstaller, which looks like a very nice cross-platform way to make standalone executables from python. maybe i won't even need it with cython --embed. details on the above + more cython tips on the wiki. okay, looks like the readme for cython_freeze says cython --embed can only put one module in with the interpreter, while cython_freeze can put any number of cython modules in there. also, it can put a normal interpreter shell in, not just a 'main' module so you can use it interactively. not sure if cython --embed can do that. also, i should look into using plipy for this sort of thing. it will allow the executable to run on any linux machine, from ancient redhat boxen to the latest ubuntu machine. 'twould be nice to preserve some of python's portability when i freeze it to binary. and there's a link there to portable python, which seems to do the same thing for windows. also, virtualenv is worth a look. lots of people endorse it.

https://translate.svn.sourceforge.net/svnroot/translate/src/trunk/virtaal/devsupport/profiling.py
saw a ref to this, which appears to be a way to profile python callables with KCacheGrind.

python webapp frameworks

been looking at python webapp frameworks lately, for some reason. spyce had caught my eye, and it looks relatively easy to get started. i have dabbled with cherrypy before, but not enough really to know the fundamental differences between them. django is one of the big ones, of course, and integration with google app engine gives it a boost. it seems to be built for a need to integrate with database backends, and generally looks like overkill for anything i would be doing. turbogears is also big and composed of a number of packages. also overkill for me. interestingly, they switched from cherrypy to pylons. i wonder why. pylons itself contains a collection of 3rd party packages, including paste, webob, etc. i generally applaud code reuse and non-wheel-reinvention, but if i just need something small and simple i don't want to have to climb 5 different learning curves. zope is another behemoth that is behemothy in an interesting, oo way: the content is represented as python objects in a persistent database. pyjamas looks interesting as a python to javascript converter, if i ever need to avoid writing javascript. (hmmm. i wonder if i could use this for adding features to embedded 3d objects in pdf files.) web2py is billed as a database-driven framework, but it also is supposed to be easy to learn as it was developed as a teaching tool. getting started looks pretty straightforward, as you can just run it and edit stuff through the web admin interface. it also has a built-in shell (with ajax)and error logs. i think i might need to check this one out more thoroughly, as it might be particularly convenient for remote admining my headless server. quixote and porcupine i only glanced at in passing, failing to see anything really compelling.

mlb gameday audio on linux

well, the free baseball audio i used to use doesn't seem to work anymore, so i've been looking for ways to use the mlb gameday audio service. the problem is, they assume i want to spend the whole day sitting in front of my windoze machine listening to my web browser. so i found the excellent mlbviewer project. the dependencies were mostly easy to resolve: pyxml and simplejson have gentoo packages, and suds just needed an easy_install suds. rtmpdump, however, has been declared an enemy to truth, justice, and the american way. almost every site that once hosted a copy seems to have taken it down after getting peed on by adobe's legal department. (fritz hollings, still keeping the country safe from technology.) i had to get a copy of rtmpdump-v1.5 from a bsd repo, since mlbviewer claims not to work on newer versions (although i seem to recall seeing a changelog entry noting that 1.6 would...). i had to add an include for stdint.h in rtmppacket.h to get it to compile on linux. make rtmpdump, make streams worked fine after that. don't forget to run Patch.sh in rtmpdump-patches/ that comes with mlbviewer. it also needs a very recent svn version of mplayer; the install file shows you how to check it out. ./configure --enable-dynamic-plugins, make, and you're done. i built mlbviewer-0.1alpha12 in its own directory, with a dedicated rtmpdump and mplayer. the install file says to install these to root, so they are found by mlbviewer. but it turns out mplayer and rtmpdump_x86 are only referenced once in the whole mlbviewer package (other than test/), so it's not hard to change the paths. set PYTHONPATH if you don't install to root python setup.py --prefix=/path/to/mlbviewer/installation install mkdir ~/.mlb cp MediaService.* ~/.mlb/ set user and pass in ~/.mlb/config i think that the -dumpstream -dumpfile options will work with the mplayer called from mlbviewer, so this could be a nice little audio tivo for yours truly. and mlbviewer comes with other handy tools for finding out which files are available, for some easy cron automation.

EDIT: managed to compile rtmpdump 2.2b, which claims to support everything (including rtmpe). had to point it to some includes with

export XCFLAGS='-I/home/epd-6.0.1-rh5-x86/include'

export XLDFLAGS='-L/home/epd-6.0.1-rh5-x86/lib'

seems to work just fine with mlbviewer, even without any of its patches for 1.5. mlbplayer warns that you must compile a svn version from the last 2 weeks (before its release 2009-06-11) for it to play stably. ubuntu's current version (2:1.0~rc3+svn20090426-1ubuntu10.1, SVN-r29237-4.4.1) of mplayer plays the flv (with a mp3 extension!?) file dumped out by test/gamedayaudio.py, though it does get the length wrong which might cause problems with seeking. otherwise, maybe the warnings apply more for video than audio.