build a better pickle...

with all the things i do with python pickles, i might need to try cerealizer some time. it's small (~300 lines), secure, and pure python. but with psyco it can still compete with cPython (even beat it sometimes).

pics and sd/mmc

i found a few links describing how to use sd/mmc flash cards in spi mode with pics. i think this guy has the approach that is easiest to follow with his gps logger, though i haven't tried it yet myself. doesn't look too hard to write raw data, at least.

python testing and debugging

looks like some commercial debuggers are talking about offline debugging like it's a great new thing. maybe i should dig up the latest version of pycrash i can find and make it my own. it would be cool, especially to get better feedback from users and auto test suites. maybe something like this, that would allow the __main__ module to be pickled, would work as well. except that i would need the contexts for the whole stack down to where the exception happened. could i catch it with pdb and then pickle the debugger state instead? i don't see a really simple way to do it, since frames and tracebacks can't be pickled. i would have to walk through these data structures and pickle what is picklable and pprint the rest, i guess. also, i really should integrate cProfile into my testing utility. fuzzing and fault injection would be really nice, too. once i have a good debug dump working, maybe i could use an email process monitoring tool like gmailpm to send it to me automatically through gmail.

i guess all these problems would be solved, along with a number of others, if i had a checkpoint/pause/resume/load coredump/reification/mobile computing type of capability. i've thought about trying to use pyro and such things, but it would be a significant effort.

maybe something like dmtcp and/or urdb for checkpointing (maybe even if it's in the __exit__ method of a with context) are worth a look, especially since they specifically claim success with python, along with matlab, perl, and other binaries via gdb. (they even used it on ipython's parallel demo.) unlike other binary checkpointing packages, like blcr, there is no need for any violence against the kernel or binary. wow, they even claim it will work with files, pipes, sockets, etc., memmap, and x windows (minus extensions, gl and video). only on linux, but still... LGPL and pretty cool as long as the performance hit isn't too bad. ok, according to the paper, performance is virtually unaffected between checkpoints. for programs to control their own checkpointing, there is a c api. probably easy to wrap, maybe even ctypes it if there a shared object lib. section 1.1 of the paper, 'use cases', explicitly identifies save/restore, dump/undump, offline debugging, and bug report image as applications (as well as being robust to deadlock and race conditions by stepping back and retrying, though this is less interesting to me).

EDIT: ok, the ubuntu package only has a static lib for the api, but it also has the .c and .h files in /usr/lib/dmtcp/, so i just

gcc -fPIC -c dmtcpaware.c

gcc -shared -W1,-soname,libdmtcpaware.so -o libdmtcpaware.so dmtcpaware.o

and i had a .so that opened with

a = ctypes.cdll.LoadLibrary('/usr/lib/dmtcp/libdmtcpaware.so')

now i run dmtcp_coordinator in another terminal and

dmtcp_checkpoint python -c "import ctypes; a = ctypes.cdll.LoadLibrary('/usr/lib/dmtcp/libdmtcpaware.so'); print a.dmtcpIsEnabled(); a.dmtcpGetCoordinatorStatus()"

works. some symbols are not in the .so, so those things need some more tweaking.

but a.dmtcpCheckpoint() runs and returns DMTCP_AFTER_CHECKPOINT. an ipython session with 2 checkpoints and a small numpy array is about 12.5 MB, and each one took a few seconds to generate. the dmtcp_restart_script.sh script in the dir where dmtcp_coordinator was run starts the process up again, and everything is in there! puts me right back to after the call to a.dmtcpCheckpoint(), except now it has returned DMTCP_AFTER_RESTART. works great for a simple example, except that it segfaults when i finish the thread.

so for offline debugging, i could put a top level in the __main__:

class Wrapper(object):

def __enter__(self):

pass

def __exit__(self,t,v,tb):

if badness:

dump_checkpoint()

tarball_dump_and_send_it_to_me()

import pdb; pdb.set_trace()

with Wrapper() as w:

do_stuff()

visualregexp

visualregexp makes it easy to write and debug one of the hardest parts of my code: regular expressions. just put a re in the upper box and the text to search in the bottom box. to test how a re would operate on iterated lines of a file, check the 'line' box. then, any time you hit enter, it will color code matching groups with parts of the re.

new dsl modem

qwest is now refusing to check my dsl line until i replace my venerable cisco 678. the recommended replacement is the gt701-wg. it actually looks like a nice piece of hardware; i'm amazed that qwest made a good decision on something. they couldn't fool me into ordering one from them, though. $100? i found a pile on ebay for $30-40. i'll have to set it up myself, but i wouldn't trust a qwest tech to do it right, anyway. besides, when i called their tech support about it while setting up the ol' 678, i always got the same thing: 'i need to know the settings to put in so it will connect to your hardware on the other end.' 'what operating system are you using?' 'ah, i don't see what that has to do with anything. it's an external modem.' 'well, i need to ask what operating system you're using.' 'linux.' 'we don't support linux. goodbye.' on a tech support call, i can find people reading a flipchart. on the web, i can find people who have done it.

using picprog

i used picprog successfully to read (on both laptop and desktop) and write (on desktop only; wimpy laptop!) hexes to/from pics. i found that sometimes i needed to use --erase with --burn to get it to start when a totally different hex was already on the pic, and --slow helped with the bad idea of trying to burn a pic in a breadboard (picprog --erase --burn -i code.hex --slow). always make sure to read the pic (picprog -o save.hex) before writing in case you screw up the calibrations. sometimes the auto device recognition didn't work, and i had to use --device pic18f877a (for example) the first time i ran it after plugging in the programmer. after the first time, though, it seemed to remember what i had on there. when i first tried to burn a hex to my pics, i found that picprog 1.7-r1 didn't work with my jdm2 programmer (worked for reading, though). i had to upgrade to 1.8.3, which worked just fine. i haven't encountered this yet, but i might need to upgrade to 1.9.0 to use the --reboot option if i use the internal oscillator. jdm programmers have a well-known problem with pics in this configuration because they will start running as soon as they're turned on.

usb oscilloscope

wow, i've gotta try this. it gives you about 60hHz with almost nothing but a pic 18f and a usb port. i think i could do it without the serial bootloader, which makes it even simpler. sure, it's not the most powerful dso, but it's so simple and would help me make a better one if i wanted to. the biggest challenge would probably be software. the author wrote his scope code in visual basic. (!!?! well, i guess i can't complain.) so i'd have to reimplement. but it might be a good way for me to learn how to do usb data xfer in linux, anyway. and maybe i can make an inteface to comedi. i think that would let me use it with xoscope. neither looks like it's released a new version in a while, but i've tried xoscope before (with the soundcard oscope, heh) and it worked okay. i've also been thinking about how to do a cheap and easy high-speed dso. i think video capture adcs are the way to go, with data fed directly to 5-10ns sram. the TVP7001PZP, for example, can do up to 165MHz with a pll internally-multiplied clk for 3 syched channels. built-in scaling and offset, with i2c interface would make it easy to control with a usb pic. for triggers, maybe i could set voltages on dacs and use an opamp comparator with a latch so that the leisurely pic could stop the adc a given time after triggering. the position of the trigger time within the data window wouldn't be exact, but the trigger wouldn't need to be checked at the same speed as the adc. i think it could actually work, and i'm surprised i can't find anyone who's already done it. hmm, maybe that should tell me something.... well, i can't find any reports of failure, either. EDIT: there is an updated firmware version here that provides 5 analog and 8 digital inputs. i tried messing around with gpdasm and gpasm on the hexes, and the non-bootloader hexes produced by gpasm for the 18f458, 18f2550, and 18f4550 are identical. i think i will do the same thing with the superprobe and put a diode instead of a resistor on mclr. rb6 and rb7 will already have some leads for the digital io, so i just need to plug in to the +5 and gnd on the usb to program this thing with my serial programmer.