Tuesday, July 29, 2008

couple of problems fixed

i managed to fix a couple of problems recently, and i figured i should record how i did it since i know this wasn't the first time i had to figure either of them out. my cisco 678 had been so reliable that i hadn't needed to log into it for years. i only needed to do it now to get the settings to put on the new modem, but it had been so long i forgot how to get minicom to connect. i finally figured out that i need to turn off both hardware and software flow control. i don't think any of the other settings is as critical, and defaultish settings (8n1) should work. another problem that i had once again was with getting my laptop to start the wireless interface when i plug in the card. i fixed the broken ld dependencies in pccardctl and pcmcia-socket-startup, but it still responded with 'eth0: no such interface'. i randomly tried running pcmcia-socket-startup manually, and that did it. according to the docs, this should run from the the pcmcia_socket.agent hotplug script whenever a new socket is registered with the pcmcia module. so i'm not sure why it didn't work automatically, but at least i can connect.

Monday, July 28, 2008

google app engine

looks like google is opening up an app engine that runs python. maybe they'll even take this in the direction of commodity computing. i might need to check that out later.

aspect oriented programming

every once in a while i think i might play around with aspect-oriented programming. wikipedia seems to have a good article to get started. i realized from reading about one python implementation (lightweight python aop: aspects.py) that i could use this to make a caching object database that automatically decides whether to compute store objects or recompute them. or maybe i should read the wikipedia page on programming paradigms first, so i don't jump from one rut to another.

python testing with nose

sometimes i wonder if i should use a utility like nose. i guess if i ever start using unittest, i will. so far, though, i've found ways to do everything i want with doctest. maybe an autodiscovery tool would be helpful, even for doctest. it would be even better if the test framework handled dependencies intelligently. i thought about doing this with bicyclerepairman, but i couldn't do it in under an hour so i quit. one of these days... someone else will do it and i'll copy and paste.

simple file transfer

this has got to be the most useful python one-liner ever: python -m SimpleHTTPServer it starts a directory browser on port 8000. for those argh-i-just-need-to-get-windows-to-transfer-this-one-file moments. 

Thursday, July 24, 2008

build a better pickle...

with all the things i do with python pickles, i might need to try cerealizer some time. it's small (~300 lines), secure, and pure python. but with psyco it can still compete with cPython (even beat it sometimes).

pics and sd/mmc

i found a few links describing how to use sd/mmc flash cards in spi mode with pics. i think this guy has the approach that is easiest to follow with his gps logger, though i haven't tried it yet myself. doesn't look too hard to write raw data, at least.

Wednesday, July 23, 2008

python testing and debugging

looks like some commercial debuggers are talking about offline debugging like it's a great new thing. maybe i should dig up the latest version of pycrash i can find and make it my own. it would be cool, especially to get better feedback from users and auto test suites. maybe something like this, that would allow the __main__ module to be pickled, would work as well. except that i would need the contexts for the whole stack down to where the exception happened. could i catch it with pdb and then pickle the debugger state instead? i don't see a really simple way to do it, since frames and tracebacks can't be pickled. i would have to walk through these data structures and pickle what is picklable and pprint the rest, i guess. also, i really should integrate cProfile into my testing utility. fuzzing and fault injection would be really nice, too. once i have a good debug dump working, maybe i could use an email process monitoring tool like gmailpm to send it to me automatically through gmail.
i guess all these problems would be solved, along with a number of others, if i had a checkpoint/pause/resume/load coredump/reification/mobile computing type of capability. i've thought about trying to use pyro and such things, but it would be a significant effort.
maybe something like dmtcp and/or urdb for checkpointing (maybe even if it's in the __exit__ method of a with context) are worth a look, especially since they specifically claim success with python, along with matlab, perl, and other binaries via gdb. (they even used it on ipython's parallel demo.) unlike other binary checkpointing packages, like blcr, there is no need for any violence against the kernel or binary. wow, they even claim it will work with files, pipes, sockets, etc., memmap, and x windows (minus extensions, gl and video). only on linux, but still... LGPL and pretty cool as long as the performance hit isn't too bad. ok, according to the paper, performance is virtually unaffected between checkpoints. for programs to control their own checkpointing, there is a c api. probably easy to wrap, maybe even ctypes it if there a shared object lib. section 1.1 of the paper, 'use cases', explicitly identifies save/restore, dump/undump, offline debugging, and bug report image as applications (as well as being robust to deadlock and race conditions by stepping back and retrying, though this is less interesting to me).
EDIT: ok, the ubuntu package only has a static lib for the api, but it also has the .c and .h files in /usr/lib/dmtcp/, so i just
gcc -fPIC -c dmtcpaware.c
gcc -shared -W1,-soname,libdmtcpaware.so -o libdmtcpaware.so dmtcpaware.o
and i had a .so that opened with
a = ctypes.cdll.LoadLibrary('/usr/lib/dmtcp/libdmtcpaware.so')
now i run dmtcp_coordinator in another terminal and
dmtcp_checkpoint python -c "import ctypes; a = ctypes.cdll.LoadLibrary('/usr/lib/dmtcp/libdmtcpaware.so'); print a.dmtcpIsEnabled(); a.dmtcpGetCoordinatorStatus()"
works. some symbols are not in the .so, so those things need some more tweaking.
but a.dmtcpCheckpoint() runs and returns DMTCP_AFTER_CHECKPOINT. an ipython session with 2 checkpoints and a small numpy array is about 12.5 MB, and each one took a few seconds to generate. the dmtcp_restart_script.sh script in the dir where dmtcp_coordinator was run starts the process up again, and everything is in there! puts me right back to after the call to a.dmtcpCheckpoint(), except now it has returned DMTCP_AFTER_RESTART. works great for a simple example, except that it segfaults when i finish the thread.
so for offline debugging, i could put a top level in the __main__:
class Wrapper(object):
def __enter__(self):
pass
def __exit__(self,t,v,tb):
if badness:
dump_checkpoint()
tarball_dump_and_send_it_to_me()
import pdb; pdb.set_trace()
with Wrapper() as w:
do_stuff()

Wednesday, July 16, 2008

visualregexp

visualregexp makes it easy to write and debug one of the hardest parts of my code: regular expressions. just put a re in the upper box and the text to search in the bottom box. to test how a re would operate on iterated lines of a file, check the 'line' box. then, any time you hit enter, it will color code matching groups with parts of the re.