open source maps

stumbled across of open source map codes: roadnav and roadmap. both use census data for generating street maps, and both can use gps input to show your location. roadnav can compute paths for you, though neither knows about one-ways (it's not in the data). a fork of roadmap is working to possibly remedy that problem, however. might be interesting to put them on my laptop, for when i travel.

list.it information scrap organizer

i'm a big user of google notebooks and i'm always on the lookout for better ways to keep quick, short bit of info organized (and off of the paper that seem to surround me anyway). list.it sounds like an interesting concept, and the folks at csail always seem to know what they're doing.

investment entities

why do tax laws have to be so complicated? i found a site that gives some info on the various types of entities to use as vehicles for active trading, and the tax pros and cons for each. i'm afraid i'll need to deal with this at some point. i guess it might be a choice between hiring a lawyer before or an accountant after. or both.

some books on ica

an information-theoretic approach to neural computing, qa76.87.d47 1996 has intros for both information theory and anns, shows connection between pca and ica. has ref to mdl with supervised learning, but it's the old (not stochastic complexity) mdl. independent component analysis: theory and applications, tk5102.9.l44 1998 time-delayed decorrelation, nonlinear ica, some historical info in preface. separated discussions of methods for sub- and super-gaussian (kurtosis < or > gaussian kurtosis) good illustrations of how ica can solve some problems of pca. looks like some deep math, but intelligible advances in independent component analysis, qa76.87.a378 2000 temporal effects: multivariate time series, ref to financial time series; time-varying mixtures, particle filters nonlinear mapping, analyzing independence assumption, ica on noisy data the time-varying stuff really looks interesting independent component analysis: principles and practice, qa76.87.i516 2000 non-stationary sources, including particle filters nonlinear, though not many equations or examples a lot of material seems very similar to the previous book independent component analysis, qa278.h98 2001 lots of background, justification of ica approach preprocessing and time filtering, comparison of algorithms good overall tutorial financial applications independent component analysis: a tutorial introduction, qa76.87.s78 2004 shorter, more intuitive explanations probably better for getting the basic idea kind of dumbed-down in some ways (i know what lambda looks like!) skip the first 6 chapters if you have a decent math background a lot of the ica books seem connected not only with information theory, which i would expect, but also with other stuff on neural nets, which i didn't necessarily expect. this one might be worth checking out. refers to some generalizations of the ica/pca ideas, with good presentation. more stuff on the group's website.

ubuntu, sabayon, etc.

i've been thinking about trying ubuntu (or xubuntu) on my next machine. it seems to be the biggest fad to hit linux in a while, and i have to admit that my more geriatric hardware has a hard time keeping up with gentoo updates. and packages like checkinstall give me hope that i will still be able to pick up scruffy-looking hitchhiker software along the way. the best of both worlds, of course, would be a meta-distro built on top of gentoo. that way i could let somebody else build binaries for most of my packages and still grunge around when i want. i've read a little debate about how this could be accomplished, especially given the exponential combinations of useflag combinations. i lean toward the p2p approach, but i don't see anyone doing it yet. but apparently some people are working on distros based on gentoo, like the stupidly named sabayon project. maybe one of these will eventually be worth the trouble.

ica and svd

been poking around some more with svds, and came across this presentation that refers to independent component analysis. the wikipedia article is decent, with good refs. one way to view the ica is as a nonlinear generalization of pca; the components are not necessarily orthogonal but they minimize the mutual information and all the higher-order cross cumulants, rather than just the second like pca. the conditions of uncorrelated and independent are equivalent only for gaussion random vars, but ica has determinacy problems of more than one of the components are gaussian. this ebook has good info on it in chapter 6, if you can get access.

3d pdf objects

i tried for way too much time to compile the u3d reference library from sourceforge, with something tantalizingly close to success. the only way i could actually use it, though, was to use wine on the precompiled windows binary on an idtf file. the only way i have successfully created an idtf is by exporting an imported stl file from meshlab. i did manage to shove the resulting u3d into a pdf, and it looked pretty cool. but i need to find an easier way to make idtf files. EDIT: apparently vrml 97 can be converted to u3d with acrobat 3d. s2plot site gives some instructions, including some basic frame-based animation. also, some russian dude made a vtk->idtf exporter. not sure if he published anywhere. also also, mathgl supposedly has idtf export capability, so things like udav that use it should, too.

so many time wasters, so little time...

there are some neat os and free games out there. if only i were idle rich. planeshift looks very well done for a completely volunteer project. i think their idea for npc ai is really cool and opens up a lot of possibilities, basically allowing an interactive fiction game integrated into an mmorpg. i tried it once and couldn't see much in the way of npcs and other players. i hope they keep working on it. the spring project has an interesting engine for a variety of rts. tremulous is a fps with some rts elements. vega strike is a sci-fi themed mmorpg with some stunning graphics and very involved back story. just like eve online, if only players could disembark and walk around, this could be a killer game.

time-varying svd

recently i've been trying to see if anyone else has worked on time-varying singular value decompositions. there's surprisingly little out there on it, at least that i can find. i've tried thinking of the svd eigenmodes in terms of rotating coordinate systems. i remembered from dynamics classes once upon a time that some ways of representing rotation have a gimbal lock problem, but quaternions do not. and (i think) they also have linear propagation equations. the wikipedia article on it looks pretty good. i should probably read that and the 'see also' articles at the bottom.

random thought fragments on multiresolution analysis

someday i'd like to write a textbook on multiresolution techniques in surrogate modeling. in the intro i would present a unified approach to orthogonal decompositions like eigensystems, svd, and ffts with wavelets, linear model sensitivity analysis, and mdl. i would go on to answer questions like, 'how much information is gained in terms of the modes and singular values if i update an svd representation of a surrogate model?' and how do i do an svd surrogate model in the first place? will it capture features on different scales, or can i force it to? how do i use that predicted information gain to guide adaptive sampling?

mp3 streaming with SqueezeCenter

i've been using ampache for a while, though i've recently had trouble streaming through a squid firewall. but i'm always on the lookout for a better way. SqueezeCenter (formerly slimserver) might be the answer. haven't tried it yet, but it's on my radar.

quantlib

the quantlib project has seemed interesting ever since i first found out about it. looks like many of the most of the lead developers are at banca imi, an italian investment bank. some others are at statpro, which provides data and analysis services. so it looks like at least some of the people involved are real pros in the business. i should take a more in-depth look some time. and there's a python binding! check out luigi's book as a great ref.

parallel python

i just read through the examples for parallel python (pp package on gentoo). some day i'll have to try it. it basically makes a queuing service to which you can submit function calls as jobs. so you submit the function calls to run and then block for the result later in the script. very handy and easy parallelism. maybe i can hook it up to an object caching/generating database as an aop aspect. now _that_ would be cool.

apache upgrade

i recently had to update my apache server from 2.0 to 2.2. they made some major changes to the default http.conf file, so in order to get ampache working again i had to add this: Alias /ampache/ "/usr/share/webapps/ampache/3.3.2.1/htdocs/" <Directory "/usr/share/webapps/ampache/3.3.2.1/htdocs/"> Options Indexes AllowOverride None Order allow,deny Allow from all </Directory> previously, i think i only needed the alias. but apparently some security settings were changed because with 2.2 i would get 'client denied by server configuration' in the error log. also, i had to add -D PHP5 to APACHE2_OPTS in /etc/conf.d/apache2 just like before.

couple of problems fixed

i managed to fix a couple of problems recently, and i figured i should record how i did it since i know this wasn't the first time i had to figure either of them out. my cisco 678 had been so reliable that i hadn't needed to log into it for years. i only needed to do it now to get the settings to put on the new modem, but it had been so long i forgot how to get minicom to connect. i finally figured out that i need to turn off both hardware and software flow control. i don't think any of the other settings is as critical, and defaultish settings (8n1) should work. another problem that i had once again was with getting my laptop to start the wireless interface when i plug in the card. i fixed the broken ld dependencies in pccardctl and pcmcia-socket-startup, but it still responded with 'eth0: no such interface'. i randomly tried running pcmcia-socket-startup manually, and that did it. according to the docs, this should run from the the pcmcia_socket.agent hotplug script whenever a new socket is registered with the pcmcia module. so i'm not sure why it didn't work automatically, but at least i can connect.

google app engine

looks like google is opening up an app engine that runs python. maybe they'll even take this in the direction of commodity computing. i might need to check that out later.

aspect oriented programming

every once in a while i think i might play around with aspect-oriented programming. wikipedia seems to have a good article to get started. i realized from reading about one python implementation (lightweight python aop: aspects.py) that i could use this to make a caching object database that automatically decides whether to compute store objects or recompute them. or maybe i should read the wikipedia page on programming paradigms first, so i don't jump from one rut to another.

python testing with nose

sometimes i wonder if i should use a utility like nose. i guess if i ever start using unittest, i will. so far, though, i've found ways to do everything i want with doctest. maybe an autodiscovery tool would be helpful, even for doctest. it would be even better if the test framework handled dependencies intelligently. i thought about doing this with bicyclerepairman, but i couldn't do it in under an hour so i quit. one of these days... someone else will do it and i'll copy and paste.

simple file transfer

this has got to be the most useful python one-liner ever: python -m SimpleHTTPServer it starts a directory browser on port 8000. for those argh-i-just-need-to-get-windows-to-transfer-this-one-file moments. 

build a better pickle...

with all the things i do with python pickles, i might need to try cerealizer some time. it's small (~300 lines), secure, and pure python. but with psyco it can still compete with cPython (even beat it sometimes).

pics and sd/mmc

i found a few links describing how to use sd/mmc flash cards in spi mode with pics. i think this guy has the approach that is easiest to follow with his gps logger, though i haven't tried it yet myself. doesn't look too hard to write raw data, at least.

python testing and debugging

looks like some commercial debuggers are talking about offline debugging like it's a great new thing. maybe i should dig up the latest version of pycrash i can find and make it my own. it would be cool, especially to get better feedback from users and auto test suites. maybe something like this, that would allow the __main__ module to be pickled, would work as well. except that i would need the contexts for the whole stack down to where the exception happened. could i catch it with pdb and then pickle the debugger state instead? i don't see a really simple way to do it, since frames and tracebacks can't be pickled. i would have to walk through these data structures and pickle what is picklable and pprint the rest, i guess. also, i really should integrate cProfile into my testing utility. fuzzing and fault injection would be really nice, too. once i have a good debug dump working, maybe i could use an email process monitoring tool like gmailpm to send it to me automatically through gmail.

i guess all these problems would be solved, along with a number of others, if i had a checkpoint/pause/resume/load coredump/reification/mobile computing type of capability. i've thought about trying to use pyro and such things, but it would be a significant effort.

maybe something like dmtcp and/or urdb for checkpointing (maybe even if it's in the __exit__ method of a with context) are worth a look, especially since they specifically claim success with python, along with matlab, perl, and other binaries via gdb. (they even used it on ipython's parallel demo.) unlike other binary checkpointing packages, like blcr, there is no need for any violence against the kernel or binary. wow, they even claim it will work with files, pipes, sockets, etc., memmap, and x windows (minus extensions, gl and video). only on linux, but still... LGPL and pretty cool as long as the performance hit isn't too bad. ok, according to the paper, performance is virtually unaffected between checkpoints. for programs to control their own checkpointing, there is a c api. probably easy to wrap, maybe even ctypes it if there a shared object lib. section 1.1 of the paper, 'use cases', explicitly identifies save/restore, dump/undump, offline debugging, and bug report image as applications (as well as being robust to deadlock and race conditions by stepping back and retrying, though this is less interesting to me).

EDIT: ok, the ubuntu package only has a static lib for the api, but it also has the .c and .h files in /usr/lib/dmtcp/, so i just

gcc -fPIC -c dmtcpaware.c

gcc -shared -W1,-soname,libdmtcpaware.so -o libdmtcpaware.so dmtcpaware.o

and i had a .so that opened with

a = ctypes.cdll.LoadLibrary('/usr/lib/dmtcp/libdmtcpaware.so')

now i run dmtcp_coordinator in another terminal and

dmtcp_checkpoint python -c "import ctypes; a = ctypes.cdll.LoadLibrary('/usr/lib/dmtcp/libdmtcpaware.so'); print a.dmtcpIsEnabled(); a.dmtcpGetCoordinatorStatus()"

works. some symbols are not in the .so, so those things need some more tweaking.

but a.dmtcpCheckpoint() runs and returns DMTCP_AFTER_CHECKPOINT. an ipython session with 2 checkpoints and a small numpy array is about 12.5 MB, and each one took a few seconds to generate. the dmtcp_restart_script.sh script in the dir where dmtcp_coordinator was run starts the process up again, and everything is in there! puts me right back to after the call to a.dmtcpCheckpoint(), except now it has returned DMTCP_AFTER_RESTART. works great for a simple example, except that it segfaults when i finish the thread.

so for offline debugging, i could put a top level in the __main__:

class Wrapper(object):

def __enter__(self):

pass

def __exit__(self,t,v,tb):

if badness:

dump_checkpoint()

tarball_dump_and_send_it_to_me()

import pdb; pdb.set_trace()

with Wrapper() as w:

do_stuff()

visualregexp

visualregexp makes it easy to write and debug one of the hardest parts of my code: regular expressions. just put a re in the upper box and the text to search in the bottom box. to test how a re would operate on iterated lines of a file, check the 'line' box. then, any time you hit enter, it will color code matching groups with parts of the re.

new dsl modem

qwest is now refusing to check my dsl line until i replace my venerable cisco 678. the recommended replacement is the gt701-wg. it actually looks like a nice piece of hardware; i'm amazed that qwest made a good decision on something. they couldn't fool me into ordering one from them, though. $100? i found a pile on ebay for $30-40. i'll have to set it up myself, but i wouldn't trust a qwest tech to do it right, anyway. besides, when i called their tech support about it while setting up the ol' 678, i always got the same thing: 'i need to know the settings to put in so it will connect to your hardware on the other end.' 'what operating system are you using?' 'ah, i don't see what that has to do with anything. it's an external modem.' 'well, i need to ask what operating system you're using.' 'linux.' 'we don't support linux. goodbye.' on a tech support call, i can find people reading a flipchart. on the web, i can find people who have done it.

using picprog

i used picprog successfully to read (on both laptop and desktop) and write (on desktop only; wimpy laptop!) hexes to/from pics. i found that sometimes i needed to use --erase with --burn to get it to start when a totally different hex was already on the pic, and --slow helped with the bad idea of trying to burn a pic in a breadboard (picprog --erase --burn -i code.hex --slow). always make sure to read the pic (picprog -o save.hex) before writing in case you screw up the calibrations. sometimes the auto device recognition didn't work, and i had to use --device pic18f877a (for example) the first time i ran it after plugging in the programmer. after the first time, though, it seemed to remember what i had on there. when i first tried to burn a hex to my pics, i found that picprog 1.7-r1 didn't work with my jdm2 programmer (worked for reading, though). i had to upgrade to 1.8.3, which worked just fine. i haven't encountered this yet, but i might need to upgrade to 1.9.0 to use the --reboot option if i use the internal oscillator. jdm programmers have a well-known problem with pics in this configuration because they will start running as soon as they're turned on.

usb oscilloscope

wow, i've gotta try this. it gives you about 60hHz with almost nothing but a pic 18f and a usb port. i think i could do it without the serial bootloader, which makes it even simpler. sure, it's not the most powerful dso, but it's so simple and would help me make a better one if i wanted to. the biggest challenge would probably be software. the author wrote his scope code in visual basic. (!!?! well, i guess i can't complain.) so i'd have to reimplement. but it might be a good way for me to learn how to do usb data xfer in linux, anyway. and maybe i can make an inteface to comedi. i think that would let me use it with xoscope. neither looks like it's released a new version in a while, but i've tried xoscope before (with the soundcard oscope, heh) and it worked okay. i've also been thinking about how to do a cheap and easy high-speed dso. i think video capture adcs are the way to go, with data fed directly to 5-10ns sram. the TVP7001PZP, for example, can do up to 165MHz with a pll internally-multiplied clk for 3 syched channels. built-in scaling and offset, with i2c interface would make it easy to control with a usb pic. for triggers, maybe i could set voltages on dacs and use an opamp comparator with a latch so that the leisurely pic could stop the adc a given time after triggering. the position of the trigger time within the data window wouldn't be exact, but the trigger wouldn't need to be checked at the same speed as the adc. i think it could actually work, and i'm surprised i can't find anyone who's already done it. hmm, maybe that should tell me something.... well, i can't find any reports of failure, either. EDIT: there is an updated firmware version here that provides 5 analog and 8 digital inputs. i tried messing around with gpdasm and gpasm on the hexes, and the non-bootloader hexes produced by gpasm for the 18f458, 18f2550, and 18f4550 are identical. i think i will do the same thing with the superprobe and put a diode instead of a resistor on mclr. rb6 and rb7 will already have some leads for the digital io, so i just need to plug in to the +5 and gnd on the usb to program this thing with my serial programmer.

icd2 clone?

now that i've finished a pic icsp, i wonder how hard it would be to make an icd2 clone for debugging. i've seen a few versions out there, but they all look somewhat complicated. this one is probably the simplest i've seen, but according to the author, it worked with mplab 6.5, didn't work with 7.11, and hasn't been tested with other versions. i guess i'll wait until someone smarter than i (and with more free time) comes out with something that does more in software.

finished my superprobe!

i finally finished my superprobe. i even managed to squeeze it into an empty habanero shake-a-salsa bottle for a stylish homebrew aesthetic. i made just a few changes from the mondo-technology design: 1. i put a diode on /mclr and resistors on rb6 and rb7, so i could program the pic in-circuit without frying everything. 2. i used a 16f877a instead of a 16f870. i changed two lines in supmp.asm: <       list      p=16f870               ; list directive to define processor <       #include          ; processor specific variable definitions --- >       list      p=16f877a >       #include but the two parts are similar enough that when i compile both of these with gpasm, the resulting hexes are identical. oddly, these hexes are significantly different from the SUPER.HEX provided by the mondo-technology page, which i assume was compiled from super.src with an unknown compiler. i used the gpasm version and it seems to work just fine. 3. instead of a passive crystal, i used a ECS-2200AX-200 on osc1 and left osc2 flapping in the breeze. so i didn't need the pair of 22pF caps, but i did find that a tiny bit capacitance (<1nF) from clk to gnd helped the osc get started. 4. instead of using power from the circuit, i decided it was much more convenient and a little safer to use a 9v battery. i have so many of those things from changing smoke alarms (and the superprobe takes so little power) that i have practically a limitless supply. it works great, and it has an amazing number of features for such a simple project. there is one glaring omission, however; there is no ohmmeter. were it not for this, i could almost entirely replace my old multimeter with this superprobe. i wonder if i could add such a feature... microchip shows a circuit and code in an512 for an ohmmeter that uses a capacitor for measuring time constants. the problem is, the resistor to be measured needs to be between the cap and gnd, and the superprobe puts the neg probe electrode at gnd. could i just use a simple voltage divider with the existing resistors on rax? the resistance would then be v/(1-v)*r, where v is the ratio of adc voltage to supply and r is the resistance of the feed resistor. by putting ra1-ra5 and rc6-rc7 high, the feed resistance can be as low as 60 ohm, though this probably isn't much better that just using the 100ohm resistor (puts out up to 85mA, too). on the high end, i could use a 100kohm feed resistor which might let me measure up to 750kohm or so. i guess the big question is whether or not there is room on the pic for this. EDIT: maybe an algorithm like this would work. and i could use the 100, 10k, and 100k ohm resistors. that way i could set the decimal location (in kohms) by the resistor that was used. 10.00 to 99.99 for the 100kohm 1.000 to 9.999 for the 10kohm .0000 to .9999 for the 100ohm and i could get up to 1Mohm by moving the decimal for voltage ratios >.5 with the 100kohm 100.0 to 999.9 for the 100kohm, v > .5 i don't think i would get good precision for much higher than 2Mohm, anyway. the max power dissipation would be with a 0omh load, at .25 watt in the 100ohm resistor and 50mA current, which is no problem. the max power dissipation in the measured device would be for a 100ohm load, at .125 watt and 25mA. this could be a problem for smd parts. typically 1206 and bigger resistors are rated at 1/8 watt, but 0805 and smaller are rated lower. maybe i could get away with it if i just used pulses and delayed between measurements, so that the average power is lower. i read somewhere that smd resistors can have ratings as low as 1/20 watt; if a measurement cycles through the three source resistors with the voltage turned off during the computation, the average power should be less than 1/24 watt. so maybe power dissipation is nothing to worry about. hmm. actually it would probably make the code simpler if i used these ranges: 100.0 to 999.9 for the 100kohm 01.00 to 99.99 for the 10kohm .0000 to .9999 for the 100ohm i would lose a digit of precision for the 1k to 9.99k range. but i wouldn't have to do anything to account for the source resistance.

finished my pic programmer!

i finally finished my high-voltage pic programmer and tested it out. i used ian's icsp jdm2-style design, with a few modifications: 1. i made a one-sided board with 3 jumpers. (ian mentions this variant; i think he designed it to be easy to do one-sided.) 2. i use alligator clips instead of header pins. 3. most significantly, i used a stack of diodes in place of the 8v2 zener. radio smack didn't have an 8v2, and i didn't want to pay s&h for a $.30 part. so i used a 5v1 zener, an led, an ordinary silicon diode, and a 100ohm resistor in series. that puts the programming voltage right in the 12-13v sweet spot, and the led gives me a nice programming activity indicator. i tried it out on a 16f877a and a 18f458 with picprog. my dell laptop couldn't muster the 12v for burning, but it can read them okay. the desktop could also read just fine, but i had problems burning the chips while they were in a breadboard. i could barely limp with a little blink-an-led test program by repeatedly burning without erasing. each time it would write a few more bytes before failing to verify. when i tried it on a 16f877a that i had soldered into a perf board, though, it worked perfectly. first time, every time. lesson learned: breadboards are evil. when i tried out the blink-an-led on the breadboard, i couldn't get a stable clock. that's not too surprising, considering i was using a 20MHz crystal. but i didn't think that nasty capacitance was so bad that couldn't even use the relatively low-speed programmer. oh, well. now that i'm used to soldering, i don't really see an advantage to the breadboard anymore. i think the easiest way to do it is to solder the pic in first, program it, then solder other components and test along the way.

pic bootloaders

the tiny pic bootloader page lists a number of other bootloaders. i haven't looked at all of them in detail, but it sounds like zpl might be the best one for me. it's not much bigger than the tiny bl, and it requires no modification of the bootloaded hex. i think it can even use hexes written to use other bootloaders, but i'd have to try that to be sure. the interface hardware is really simple (3 resistors, one transistor, and an optional led). other than vdd and vss, it only connects to /mclr. so it doesn't use _any_ i/o pins and can be left in while the pic is running. the front end is in python (with pyserial) so it's multiplatform. i just had to change 'import serial' to 'import serial.serialposix as serial' in zpl.py and delete the non-functional serial.py that comes with it so it would use my built-in pyserial. one downside is that the communication is one-way; there is no way to verify the reprogramming. there is an option to rewrite the hex a given number of times, as a quick fix in a noisy situation. given the way it works, it might also be quite a bit slower but i'll have to check that. the bootloader won't overwrite itself, nor will it change the fuses (even though the pic could do either). so fuse configuration changes and changes in clock frequency will need the old jdm2 with the vpp sledgehammer, but at least zpl will never kill itself. for quick dev/debug iterations without big changes to the circuit, this bootloader should be ideal.

3d pdf objects

apparently there a couple of ways to embed 3d objects in pdf files. viewers can pan, rotate, and zoom on a quite smooth 3d model *drool*. i haven't seen much on the really really new format, and it's only supported in new versions of acroread. but the older u3d format is out there and already seems to be near-abandonware. there's a spec def on sourceforge that is the only free implementation i've found so far, though this page has some links i should check out. (click fast, it's tripod!) maybe one of these days i'll write something to convert meshes to u3d so i can embed them in docs.

project management software

in my never-ending quest for project management software that will organize me, i've come across a few that might be worth a try. ganttproject seems to be focused on gantt charts, and can import and export ms project files. gnome planner (formerly from imenio) looks nice and does gantt/resource management, but i've read reviews saying it lacks too many features that ms project has and can't really replace it. opensched sounds very interesting, as it will do the scheduling for you, given tasks and resources. it produces latex reports, too. but it looks like it hasn't been updated since 1999. projman sounds like it has scheduling capability and import/export compatibility with gnome planner, but i can't find out much more about it. xplan looks ancient and crappy. (developed in an all-sun environment? come on, i don't think sun develops in an all-sun environment.) taskjuggler looks like one of the most feature-rich os pm packages out there. i'm particularly intrigued by the mention of 'risk analysis'. i've also had my interest piqued by descriptions of chandler. from their docs, it sounds like a dream come true. but i've also read bad things about their own project management. doesn't bode well when an organization software project seems disorganized.... but i'll give it a look.

EDIT: they finally kicked 1.0 out the door, just before daddy warbucks kicked them out the door. but then the blog abruptly stops in mid 2008. wonder if it will keep going in the foss wild.

free space optics

i just stumbled across RONJA. sounds pretty cool. they claim to get almost a mile with just a high-brightness led.  i wonder if it would work better than my cantenna...

pdftotext

recently i needed to rip the text out of a pdf file so i could read it on my pda. (it doesn't have a pdf reader right now, and even if i got one it would take forever to render all the images.) here's a command that works pretty well, even preserving the layout with pretty good accuracy: pdftotext -layout -enc ASCII7 -nopgbrk file.pdf file.txt

gmplayer with .m3u

i found a good way to use gmplayer with .m3u files from my streaming music server (using ampache). i made a little bash script: #!/bin/bash # cache up to 16MB, so it won't get cut off if paused # since the cache is so big, only cache 1% before starting #echo Opening "$1" # for firefox #gmplayer -cache 16384 -cache-min 1 -playlist $@ # for opera (it makes filenames like "playlist (3).m3u", and it waits for this script to finish) gmplayer -cache 16384 -cache-min 1 -playlist "$1" & then i set up my browser to run this with the downloaded .m3u as the argument.

plplot

i just tried out plplot, in my eternal saga to find a good publication-quality 3d plotting package for python. looks pretty good and the interface isn't too bad. it looks like it will only spit out hardcopies; i can't find an interactive mode. fine for publication, a little inconvenient for debugging/prototyping. also, the only really standard vector formats are b+w and color postscript. but i could ps2pdf with no problems and tremendously improved compression. i tried the /usr/share/plplot/examples/python/xw08.py example from the admittedly old 5.5.2 for a surface contour. i did need to put in plinit() after the import plplot line for it to work. i also used plscolbg(255,255,255) to make a white background, with decent results.

rubber and sam2p

i'm always on the lookout for packages that combine my two loves: python and tex. looks like rubber will allow me to drive latex from python (even though it was made to have a cli). i'm not sure if it's still maintained, but it has a great way to add functionality based on modules. i've also just tried out sam2p for converting rasters to eps and pdf. it does indeed result in _much_ smaller files than imagemagick's convert, especially for eps. i did have to download, build, and install tif22pnm for sam2p to work with pngs. the link was buried in a huge readme, which was less than convenient, but it works now. EDIT: this might be a smaller, simpler alternative to rubber. might be pre-alpha, though. here's another one that is small and simple and probably will work.

and plastex converts latex to html.

installing tex packages

it had been a while since i'd installed latex packages, but it was easy once i remembered how. just put the stuff in a searched directory (such as /usr/share/texmf/tex/latex/) and run mktexlsr. some packages come as .dtx files; run tex (not latex) on these first to generate the files you need.

interesting books

i ran across a couple of books that look interesting. the first is 'Impossible?: Surprising Solutions to Counterintuitive Conundrums' by julian havil. a great way to challenge students and keep the puzzler sharp. the second book is 'How Round Is Your Circle?: Where Engineering and Mathematics Meet'. i was thinking this might help people who struggle with the traditional approach to mathematics instruction to develop a more physical intuition.

kindle

amazon is selling a device called the kindle. it's an ebook reader that looks like it uses electronic ink. looks like a very high-quality display and has a long battery life. built-in qwerty, too. i wonder how hackable? if there's a way for me to break off amazon's manacles, this might be my zaurus replacement.... UPDATE: apparently the kindle runs linux and a number of other os packages. amazon has released source of their changes, as they are obligated to do. maybe this thing is hackable, after all. i'm guessing amazon will not be forthcoming with hardware specs, though.

latex images

found a neat little tool for producing pngs from latex: pytexipng. fast, simple, and python. can't beat that. great for making equations for presentations. try to mangle that, powerpoint!

pycrypto

i found a recipe that gives a simple example of how to use pycrypto. Here's the important part:

import os, sys
from random import randrange
from Crypto.Cipher import Blowfish
from getpass import getpass
import getopt

class BFCipher:
    def __init__(self, pword):
        self.__cipher = Blowfish.new(pword)
    def encrypt(self, file_buffer):
        ciphertext = self.__cipher.encrypt(self.__pad_file(file_buffer))
        return ciphertext
    def decrypt(self, file_buffer):
        cleartext = self.__depad_file(self.__cipher.decrypt(file_buffer))
        return cleartext
    # Blowfish cipher needs 8 byte blocks to work with
    def __pad_file(self, file_buffer):
        pad_bytes = 8 - (len(file_buffer) % 8)                                
        for i in range(pad_bytes - 1): file_buffer += chr(randrange(0, 256))
        # final padding byte; % by 8 to get the number of padding bytes
        bflag = randrange(6, 248); bflag -= bflag % 8 - pad_bytes
        file_buffer += chr(bflag)
        return file_buffer
    def __depad_file(self, file_buffer):
        pad_bytes = ord(file_buffer[-1]) % 8
        if not pad_bytes: pad_bytes = 8
        return file_buffer[:-pad_bytes]

wireless measurements

right now, with one of the default whip monopoles, i get about 30 yards range outside my house for my wap. i'd like to go about 10 times that in one direction. so that would be 10 dbi gain needed for an antenna. that should be doable with a homebrew.

grandcentral

so, google bought grandcentral.com and is now letting people into a free beta. but they don't have any numbers available in my area. i wonder if that's because other people here already took them or if they haven't put any in yet. i guess i'll check it later, just in case.

homebrew wifi antennae, aircrack

i found a few good links on homebrew directional wifi antennae. this one gives a comparison between different designs and shows that soup can wave guides beat pringles cans. bottom line: nalley 40oz 'big chunk' beef stew wins, with a 16 dbi gain. here's a design for a helical, also advertised at 16 dbi. maybe i could make it better by increasing the length, but if it's the same gain as the soup can, i might as well make the easier one (which costs practically nothing considering i can eat the soup). some dish designs might do even better, but that means i have to buy a usb wifi and set up network bridging with the server. either of the other two can connect directly to the wap, after i remove one of the built-ins. speaking of which, that means i'll need to pick up a RP-TNC connect at radio smack. on a related subject, here's a tutorial on using aircrack. no, i don't plan to use these together, and neither should you.

linux blogging... ugh

i am sorely disappointed by the state of linux blogging clients for blogger 2 (atom). the only one that even claims to support it is drivel, and this in svn only (not in any stable release). checking out was easy enough: svn checkout http://svn.gnome.org/svn/drivel/trunk then, autogen.sh, configure, make, make install. but it crashes when i try to connect to http://www.blogger.com/api. maybe i'll try drivel again if it gets more work. but it looks like i'll be using the !@#$% browser for the foreseeable future.

pure python ib i am eager to try out the pure puthon interface to interactive brokers. ibpy on google code has almost everything i need. profitpy provides a login_helper script to send x events to the java tws window, which i'm currently doing with the klunky external perl guitest. login_helper is meant to be a command-line script, but i could use it as a module by putting together an options object and calling login_helper.main(). login_helper needs the guitest module from enthought. the link in the docstring seems to be old, but it's not hard to find on the enthought site. the ib_tws script (also in the bin dir) shows a good way to start the java tws, though for some reason it's a bash script instead of python. shouldn't be hard to translate.

just started using BloGTK. we'll see how it goes.

mach64 on gentoo

i think i finally found a way to get accelerated video on my ancient laptop with an ati rage mobility p/m (mach64). i'll give it a try and see if i can use that thing for some kind of multimedia use.

blog client apps

looks like the big names in non-browser blog editors for linux are gnome-browser and BloGTK. gnome-browser is the only one that specifically claims to work with blogspot, and they both appear to be primarily python. i'm having problems with both of them right now because it seems to be linking some stuff with python 2.5, even though i'm using 2.4.

homebrew eeg

i came across this project today. looks pretty cool, especially when combined with brainwave synchronization with binaural beats. i would need to make some electrodes, but they have designs for both passive (saltwater) electrodes and fairly simple active ones. i could probably get silver wire from a jewelry supplier. EDIT: i think active electrodes are the way to go. the links here to joe's, jim's, and pedro's (the other 2 links to bioera.net are dead) show schematics and construction tips. jim and pedro use joerg's modification of jarek's original v2 prototype. looks like joerg made the following changes. 1. added a 200 ohm decoupling resistor between the opamp and the shield 2. changed the power bypass cap from 100nF to 10nF 3. added what looks like a lowpass with 2 100pF caps and a 10k ohm resistor 4. added 100 ohm power protection resistor to power supply 5. added a 2v 'vgnd' for bias current electrode

led circuits

i've seen a number of circuits online lately for squeezing the last joule from dead batteries and using it to power an led. this page seems to have the most info.

homebrew scopes

i found instructions for building a scope from a dspic and a few other components. looks easy enough for a dilettante like me. the author says the next version might allow for upload to a computer. i would be all over that. here's another one that might hold promise. also has some info on usb interfacing for homebrew electronics. here's one that is more capable but _way_ more complicated.

compressing text images

i was playing around with potrace and found it to be ideal for compressing images of text. (i tried autotrace, too, but i couldn't get it working well within 2 minutes and gave up.) based on settings from the example page, here are some good defaults to try: mkbitmap -s 2 -t 0.48 test.eps if the image is blurry or has too many colors, you might want to add '-f 2' to the mkbitmap options to make it do a high-pass filter. only problem with that is, it will mess up graphics. the above options worked really well for an image that was about 2000 pixels across a page of text. the eps file was only about 1/10 the size of the pbm, which was only about 1/5 the size of the original ppm. and it still looks great -- very readable. apparently inkscape uses potrace and allows color instead of just b&w. maybe i can figure out how to pipe through that...

teddy roosevelt

"It is not the critic who counts, not the man who points out how the strong man stumbles or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly, who errs and comes up short again and again. Because there is no effort without error and shortcomings, he who knows the great devotion, who spends himself in a worthy cause, who at the best knows in the end the high achievement of triumph and who at worst, if he fails while daring greatly, knows his place shall never be with those timid and cold souls who know neither victory nor defeat" -theodore roosevelt i find this quote very inspiring. here's one i'm working on: "the greatest failure is fear to try." -duhctaep

knowledge representation

i've been thinking about planning and organization lately, especially with respect to innovation and creativity. i was reminded of this when i saw gentoo update freemind, a mind mapping program. five minutes of poking around on wikipedia turned up articles on concept mapping, semantic networks, and other things that look similar to the directed graphs i've started to use. i've been trying to find a way to represent dependencies and alternatives so that i can identify critical paths and distractions. maybe i should look into concept mapping and semantic networks for this. the knowledge representation category has a number of related articles.

cython and sympy

looks like cython is the new coolness for compiling python-ishness to c. they forked from pyrex last year to meet the needs of the sage project, and they seem to have a more active development. plus, they intend to get picked up into the main python distro. and speaking of sage, i remember giving it a spin a while ago. i was impressed at the scope, but i was disappointed at how so many kitchen sinks could bog me down. i think it took more than a day to compile. there's one reason to give sympy a chance. looks like they have some serious support and heavyweight devs (including the author of the original pyginac and one of the authors of the more recent swiginac), so i'm hopeful about where that goes. and they're integrating with sage, too, so it shouldn't be hard to migrate to sage if necessary. sage, meanwhile, looks to be a contender with maple and mathematica. o joyful day! also, a couple of projects use sage-style packages but don't make you install all of them at once: http://code.google.com/p/spdproject/ http://femhub.org/ ...and i found out that matlab now uses mupad instead of maple for its symbolic stuff. interesting.

kino for zaurus

found a neat little frontend for zauri. same page also has a package for switching a zaurus to usb storage. i'll have to try it some time. might also be able to get a console-based frontend like mplay working. pymplayer is another one i might check out.

fixed scratchy mplayer

mplayer was sounding really scratchy when playing mp3s. i figured out that it was using -ao openal by default, so i recompiled it to use alsa instead (USE='alsa' on gentoo). now it sounds great.

embed files in pdf

looks like there are a couple of ways to embed files in a pdf document. attachfile2 (see docs for attachfile for usage) attaches a file with a tag in the pdf. embedfile puts the file in with no association to anything else in the file. it also makes a table at the end of the pdf, which i will have to see if i can turn off.

embed movies in pdf

just noticed the movie15 tex package. wow, this could make a killer combo with my beamer setup. i'll definitely need to try this. are there any other tex packages for embedding multimedia? the 'related packages' lists 'none known'.

sqa and cyclomatic complexity

read a very interesting post about measuring code complexity and risk using mccabe cyclomatic complexity. i should have this in my dev process, right alongside my doctests and coverage analysis (probably integrated into test coverage somehow). be sure to check out the link near the bottom to the python wrapper.

auto python checking with emacs

there's a description of how to use pyflakes with emacs flymode here: http://www.plope.com/Members/chrism/flymake-mode. apparently pyflakes works by walking the ast rather than executing the code, so it's fast and has no side effects. there are also some neat customizations in the comments at the bottom. pychecker can execute the code, so i might be able to set things up to autorun my doctests. according to the emacswiki, there's a way to plug pylint into the default python mode (look at the bottom of the page.) looks like it's not too hard to customize, too, so maybe i could get my doctest-running script in there. EDIT: auto-compile looks like the kind of hook i want -- it runs 'make' automatically when a file is saved. i would need to modify it, though...

cvoicecontrol

i just rediscovered a cool package: cvoicecontrol. i've got to play with it some more, once i have some free time.... i remember i once got it working on my zaurus (old school 5500), which made me feel like a really macho geek for a day. not maintained anymore, apparently, but should still work.

google plans to store my data. do it, google! and they can eat my entropy when i mirror an encrypted fs. can't wait to see which content-related ads that will churn up. not much to go on yet, but the internal codename is 'my stuff'.

more reasons to put off buying a cell phone

looks like the neo1973 open cell phone is coming out, and its hardware is comparable to the iphone. runs openmoko, for double geek drool. google is also telling us that their android phone os will run on many different phones, according to wikipedia. i want it. all. but i will definitely wait until things stabilize, prices drop a little, and some 3rd party development gets released. i'll keep an eye on it.

google apps on thin clients

i just discovered this google service for simplifying and reformatting web pages for mobile devices: http://www.google.com/gwt/n looks like it would be very handy for pdas, phones, and ancient laptops on life support that i can't throw away. i also hope to get the official google mobile stuff (http://mobile.google.com/) working on my zaurus 5500. it's just java, and someone apparently made the .jar work with an ibm jvm for a pocket pc. (http://www.downloadsquad.com/2006/11/06/how-to-run-gmail-locally-on-a-pocket-pc/) maybe i can do the same thing with the built-in jvm on my zaurus. in case i can't get that working, i might be able to use this: http://sourceforge.net/projects/gmail-mobile/ i would need to get php for the z, though. [EDIT] looks like the gmail-mobile project is for html<->wml translation/filtering, not for offline reading and editing. too bad. i did manage to snag the mobile gmail java app with wget. (google puts up a bit of a fight because it wants to get your system info, i guess to send you the right version.) looks like it needs some java microedition libs for handling midlets. these posts look like they might be helpful: http://www.oesf.org/forum/index.php?showtopic=17019&st=0 http://www.oesf.org/forum/index.php?showtopic=15610 according to google support, the gmail mobile app requres j2me with midp2. the me4se.jar refered to in the oesf posts has incomplete support for midp2, but one of the posters indicates that classes can be extracted from midp-2_0-src-linux-i686.zip in the midp developer kit. or get everything from here. i'm still not sure, though, if the gmail mobile app will let me have some offline capability or if it's just replacing a web browser with a jvm. maybe google desktop instead? or google gears?

xming for gnucash

i set up xming to use gnucash remotely from a windows machine. the xlaunch file linked by the desktop icon has <xlaunch xmlns="http://www.straightrunning.com/XmingNotes" xsi="http://www.w3.org/2001/XMLSchema-instance" schemalocation="http://www.straightrunning.com/XmingNotes XLaunch.xsd" windowmode="MultiWindow" clientmode="StartProgram" program="gnucash" clientstart="PuTTY" passwordstart="true" remotehost="10.0.0.6" remoteuser="linuxuser" display="0" clipboard="true"> the user is prompted for the password (to ssh) and it starts gnucash on the server. substitute gnucash for any other server program you frequently want to use on a windows client.

fixing the wireless access point

my linksys wrt54g hasn't been working well lately, thanks to neighbors who just pop up their own networks without checking which channels are already being used. my problem is that i'm not using the gateway/dhcp features of the linksys, since i already had a lan on 10.0.0.x. so i couldn't log in to the linksys' web admin page. here's how i fixed it: with kismet, i determined that channels 1, 6, 9, and 11 were being used within range of my house. a little scanning the neighborhood picked up channel 3, too. channels 1-11 are legal in north america, according to /etc/conf.d/wireless.example. so i'm going to use channel 4. i got into the default linksys network by booting the wap unplugged from ethernet starting my eth1 on the same channel, with dhcp. my wireless ip was then 192.168.1.101. http://192.168.1.1/ got me to the admin page for the linksys. default username and password are 'admin' and 'admin'. once i was in, i turned off the dhcp server. (apparently things got turned back on from a reset.) pretty simple menu navigation to change the channel settings and get security back. i used a wep passphrase. (i seem to recall it was too much hassle to use wpa with linux before, and i don't think any of my neighbors are very 1337, given their ignorant clobbering of my network in the first place.) 128 bit, 26 hex digits conveniently generated by linksys. had to reboot linksys and restart eth1 (with manual ip now that dhcp is off) after changing the channel and turning on wep. reset the password under the administration tab, changed the ip to 10.0.0.101, and rebooted linksys. after a couple of minutes, i could log into web admin again. no more messing around with 192.168! hooray! something odd that i noticed was that the wireless card kept wanting to use the old channel 6, even though i set channel_eth1="4" in /etc/conf.d/net. the only workaround i could find was to take out the channel_eth1 line and let it choose the channel automatically by the essid. this worked as long as i was connecting only to the linksys, without dhcp. once i plugged the linksys into the lan and used the dhcp server on my dsl modem, it would still report using channel 6. but it worked. weird.