amazon aws/ec2, picloud

looks like amazon gives you up to a year to try out some free cpu time on their cloud computing nodes.
http://aws.amazon.com/free/terms/

also, their spot instances allow you to bid for time, rather than paying the fixed on-demand rates. looks like the discount is significant, if you can handle the unpredictability. nice example of running a jenkins build slave on spot, too. also refs princeton consultants and their optispotter, which helps smallish ($50mil) hedge funds find hft opportunities.
http://aws.amazon.com/ec2/spot-instances/
http://www.youtube.com/watch?v=-vAAuTs9iu4

picloud still looks like a good way to get started. can do fractional hours, and prices are comparable to ec2 on-demand. they allow you to create an environment on a virtual ubuntu, so you can install whatever you need as if you had a local filesystem.
http://aws.typepad.com/aws/2012/12/picloud-and-princeton-consultants-win-the-first-amazon-ec2-spotathon.html

continuous integration

recently decided it was time to stop putting off trying continuous integration for software development. (i'm only a decade behind the times; not bad.)

since i mostly use python, i had to look at buildbot. apache gump and cruisecontrol also seemed like possibilities. but in the end i tried hudson since i'd read it was easy to set up and use, and it really was. all i had to do was download the war file and run

java -jar .\hudson.war *>output.txt

(i had to redir output so the blocking to console wouldn't make it wait for me to scroll or press a key.)

here are some motivational/informative quotes on ci:
wikipedia:
continuous integration -- the practice of frequently integrating one's new or changed code with the existing code repository -- should occur frequently enough that no intervening window remains between commit and build, and such that no errors can arise without developers noticing them and correcting them immediately.

martin fowler:
continuous integration doesn't get rid of bugs, but it does make them dramatically easier to find and remove. in this respect it's rather like self-testing code. if you introduce a bug and detect it quickly it's far easier to get rid of. since you've only changed a small bit of the system, you don't have far to look. since that bit of the system is the bit you just worked with, it's fresh in your memory -- again making it easier to find the bug. you can also use diff debugging -- comparing the current version of the system to an earlier one that didn't have the bug.

bugs are also cumulative. the more bugs you have, the harder it is to remove each one. this is partly because you get bug interactions, where failures show as the result of multiple faults -- making each fault harder to find. It's also psychological -- people have less energy to find and get rid of bugs when there are many of them...

if you have continuous integration, it removes one of the biggest barriers to frequent deployment. frequent deployment is valuable because it allows your users to get new features more rapidly, to give more rapid feedback on those features, and generally become more collaborative in the development cycle. this helps break down the barriers between customers and development -- barriers which i believe are the biggest barriers to successful software development.

paul duvall, cto, stelligent incorporated:
6 anti patters
infrequent checkins, which lead to delayed integration
broken builds, whcih prevent teams from moving on to other tasks
minimal feedback, which prevents action from occurring
receiving spam feedback, wich causes people to ignore messages
possessing a slow machine, which delays feedback
relying on a bloated build, which reduces rapid feedback

python/c++ with microsoft visual c++

finally got a 64 bit pyd python extension working, compiled with ms visual c++ and visual studio for epd on windows. figured out that the version string in python (MSC v.1500 64 bit (AMD64)) was for visual c++ 2008 == v9.0, and the express edition only builds 32 bit. so i had to get the sdk (version 7 works with 2008) and make sure the amd64 stuff got installed with it. some notes say to install the service pack before the 64 bit stuff, but i didn't find this necessary. i ran the 'Windows SDK Configuration Tool' from the start menu, since it sounded logical, and ticked the box to link the sdk with VC 2008. not sure if that was necessary or not. one change i had to kludge manually was changing the references in C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat under the amd64 label. originally they were pointing to "%~dp0bin\amd64\vcvarsamd64.bat"; they need to be "%~dp0bin\vcvars64.bat". at that point, running vcvarsall.bat amd64 should work and a simple use of weave passes: import scipy.weave as w c = w.inline(r'printf("hi.");',verbose=2) now that the compiler, etc., are set up i can use swig and distutils to build bigger extensions (like in the swig docs), and it Just Works!

wireless on suspend/resume

had a problem with some ubuntu upgrades hosing my ath5k wifi after resuming from a sleep. something was messed up with the power getting turned back on, i think, and it seemed to be something that would get fixed and rebroken judging from the dmesg errors and bugposts on kernel.org. downgrading my kernel didn't fix the problem. fortunately, i found that turning it off and on with rfkill avoided the problems, so i put it into a script to do it automatically. /etc/pm/sleep.d/50_wireless:

#!/bin/sh case "${1}" in suspend) #date >> /tmp/pm.log #echo ' suspend' >> /tmp/pm.log /usr/sbin/rfkill block wlan ;; resume) #date >> /tmp/pm.log #echo ' resume' >> /tmp/pm.log /usr/sbin/rfkill unblock wlan ;; hibernate) # nothing ;; thaw) # nothing ;; esac

cygwin-ports

cygwinports seems to be just what i was looking for to get packages that don't come with the standard set. they have rtmpdump and mplayer, and i'll check back there before trying to compile stuff again.

android sdk, avd, !*&$!$%^

tried to get android emulator working on my admittedly underpowered netbook. it works... sort of. the 3.1 platform was _really_ slow. i tried the snapshot method to make it at least boot faster; no dice. maybe 2.2 would be better, but i think what i'll really try (if i do again) is the android-x86 iso on a flash drive. or just wait -- google and/or intel seems to be coming out with x86 native android later this year.

kindle on linux

followed these instructions to get kindle for pc running on my ubuntu netbook with wine:

sudo add-apt-repository ppa:ubuntu-wine/ppa

sudo apt-get update

sudo apt-get install wine1.3

after installation, just run wine on the installed Kindle.exe in ~/.wine/drive_c/ etc.

audio + video

figured out how to get an alternate audio into a video file: # take mp3 out of audio flv ffmpeg -i altAudio.flv -acodec copy altAudio.mp3 # take the video only out of the video mencoder -ni -ovc copy -nosound -o noAudio.mp4 origVideo.mp4 # put the audio and video together mencoder -ovc copy -oac copy -audiofile altAudio.mp3 -o out.avi noAudio.mp4 might not need both the ffmpeg step and -ni on the video extraction, but at least one of them is necessary. i tried without both and the video played at double speed while the audio was normal speed in the out.avi. might also be able to use a different format out with -of. not sure if avi transcodes it. EDIT: tried without -ni, still works fine. might need to adjust delay with - and +. also, i found that rtmpdump has very handy -e and -o options. i dropped the -A option with -e and it seems to work (kind of), not sure if that is absolutely necessary. now i'm trying to get mencoder on a file to let me playback faster. tried -vf filmdint and filmdint=io=2:1, and it doesn't hurt but doesn't seem to be essential. so far, best results are with just -fps 30000/1001 -ofps 20000/1001 and speed it up with ']'. -oac copy doesn't work; have to use something else like pcm (uncompressed) or mp3lame. the original audio is 48000Hz aac, which i can encode with -oac lavc -lavcopts acodec=aac, but that makes encoding really slow and messes up a/v sync, even when i try to compensate by changing -srate. so far this version looks promising, but the avi has no idx: mencoder -ovc copy -oac mp3lame -fps 30000/1001 -ofps 20000/1001 -o out.avi infile.mp4 could try -of mpeg or lavf, with a lavf format=mp4 for example, since the original is mp4 and plays pretty well (except when i speed it up too much). ok, this one seems to work pretty well: mencoder -ovc copy -oac mp3lame -fps 30000/1001 -ofps 20000/1001 -of lavf -lavfopts format=mp4 -o out.mp4 infile.mp4

windows python in cygwin

finally solved a problem (or found a workaround, at least) for something that had bothered me for a while: when i tried to use windows python (not cygwin python, which worked fine) in a xterm, it seemed not to be connected to stdout, stderr, and stdin. neither the interpreter nor the debugger prompt would show up, and nothing happened when i used print or sys.stdout.write. the mysterious thing was it would work from a non-x cygwin shell. but i needed mouse action on the desktop and screen (which uses a text-based x windows server) remotely. turns out the problem is how cygwin interfaces a non-cygwin console app from the terminal. it talks to it through pipes rather than with a real pty, and the issues there are deep and woolly. so all these windows programs are buffering in the pipe, not realizing how impatient i'm getting on the other end. fortunately, python has an easy workaround. the -i option makes it assume interactivity, skipping the tty check. i can use it on the cli or #! shebang, and now it's working. only problem is it drops me into an interpreter when the script finishes, so i have to type quit() (c-d, c-z, c-c are all ignored). ref here

scraping off microsoft's look and feel

visually impaired people have at least one thing going for them: the high contrast windows theme is much better than windows standard. and the opticwhite theme for google chrome helps as well. now i don't feel quite so out of place and my eyeballs won't melt.

virtualbox

holy grail: windows 7 and ubuntu dual boot, with either also running under a vm in the other. i've been looking around for a good virtual machine; some of the old ones seem kinda dead or just emulate (read: way too slow) (bochs, plex86) or are too slow except on linux host or maybe forked off somewhere else (qemu/kqemu or whatever). xen looks pretty good, especially with its capability to run a guest os off a partition. but it only runs on a linux host, so that's only half the answer, and i'm not sure if anyone has gotten it to use an existing windows install. virtual box can run on windows or linux host, and it can run either guest at virtualized native speed. and maybe i can get it to run an existing windows or ubuntu from the other. one problem with booting the oem windows partition is that setting up virtualbox from linux will require bootrec.exe from an install (not recovery) dvd. (colinux might be able to boot an existing install, but colinux is limited to one processor atm.)

getting spyder and python to work on windows 7

had to struggle to get my python install working with numpy and spyder, probably because i copied them over from another install.

with spyder, i had to move the 2.0.0beta5 egg and give the --prefix option to setup.py in order to install and use 2.0.3.

with numpy, i was getting 'ImportError: DLL load failed: The specified module could not be found.' when i tried to import numpy, _unless_ i was in the Python26_64\Scripts dir. i think it's because of the mkl lapack dlls in there. but it all worked once i added that to my PATH (not PYTHONPATH) with the help of cygwin -- just using export in bash allows me to peek at what it would be in windowsese with os.environ['PATH'] so i could put it into the env editor in spyder. viola!

sshd on cygwin

got a new machine with windows 7, and i think i'll have to actually use the windows partition. ugh.

fortunately, cygwin comes with a kajillion unix packages that make microsoft bearable. and i just got my openssh server up and running, so i can still log in from home or elsewhere. here's how:

first, i followed the steps here to get rid of any old failed-attempt kruft. then, even though i did 'run as administrator' on the cygwin bash startup bat, it still gave me warnings when i ran ssh-host-config and tried to use my windows user for running sshd. so i went back and did all the editrights lines. rerunning ssh-host-config (probably unnecessary) gave no warnings, so i started it up with 'cygrunsrv -S sshd' as suggested here (also used the 'tty ntsec' for CYGWIN, as he suggests). and it Just Works.

ath5k woes

okay, is everyone else out there as bugged as i am about the ath5k driver? it seems like every ubuntu maintenance update i've done over the last couple of months on my acer aspire one has made it worse. i just downgraded back to 2.6.32-21-generic, which works pretty well (though i did get a bunch of 'ath5k phy0: failed to wake up the MAC Chip' and 'ath5k phy0: can't reset hardware (-5)' after i let it run for a while). i've been have other hardware problems with the ssd drive and the web cam, but i think this one is a driver issue given the chatter i've found on mail lists and bug reports. i know it's oss, so i can't really complain, but i'm really looking forward to this being resolved. i wonder if i should lag my kernel updates and/or switch to madwifi or something.... EDIT: don't think madwifi is the answer, since it looks like their dev effort got folded back into the ath5k. at least, i couldn't find a madwifi version that would both compile with my kernel and work with my pci-e wifi. but i seem to recall looking at the L0s, L1 ASPM stuff before with the unsupported jumbo problems, and i noticed that now all aspm support is turned off by default on my card. (dmesg even says the pci driver explicitly decides to do it.) this change was apparently made because junky old pcie wireless cards, like i have in my acer aspire, get the unsupported jumbos in L0s. but L1 is supposed to be a mandatory part of the standard (L0s is optional). so maybe something is trying to shove it into L1 without checking that that's enabled. i tried turning L1 back on with the enable-aspm script (with root complex 00:1c.2 and endpoint 03:00.0) but it didn't seem to do anything. so i put the pcie_aspm=force kernel option into /boot/grub/grub.cfg, and that worked. in fact, it only enabled aspm for the atheros and its pci-e port, and only L1 for that. i guess the ath5k devs or somebody actually put code in to check if it can do it. so that's good; shouldn't get any of the jumbo problems, even with the =force. so now we'll just see if this helps with the 'ath5k phy0: failed to wakeup the MAC Chip' and 'ath5k phy0:can't reset hardware (-5)' problems.

booting from sdhc on acer aspire one

finally got my aa1 to boot with root on the sdhc card. these instructions are really good, and this reference is good, too. (though he is of the opinion that the nand flash write cycle limit is nothing to worry about, and i used to think that, too, but.... i wonder if my swap and ext4 fs had anything to do with the built-in ssd drive crashing so completely and painfully. this time i'll be going ext2 retro, just to be safe.) had to boot from a usb drive (yeah, ssd is that hosed... can't even spare a few mb for a boot partition. heck, i can't even alter the partition table.) but it seems to be booting. now i wonder if i can hack this case open and put the usb drive in the unused bay...

flashed my aa1 bios

my acer aspire one seemed to be having hardware issues, so i flashed the bios from 0.3310 10/06/2008, InsydeH2O Rev. 3.5, vga bios IntelV1585 to... the same thing. it's the latest version on acer's website, but i reflashed it anyway in case it had been munged somehow. i used the FLASHIT.EXE and zg5ia32.fs method that doesn't require any freedos junk. but it doesn't seem to make any difference.

using googlecl with blogger from bash

i think i finally found a good way to post from the cli.

google blogger post --title 'blog post title' --tags 'tag1,tag2' $'multiline content...'

the $ in front of the hard (single) quote lets bash escape single quotes inside that last string, but nothing else. so i can put all kinds of crazy chars in my multiline post, including a backslash-escaped single quote, with impunity. only problem is editing a previous line.... well, now i have an excuse for leaving the typos in.

tahoe-lafs

decentralized secure data store. if i ever need cloud storage, this might be a good alternative to cryptoloop.

a bit less temporary /tmp

hat-tip to pycuda, i got my ubuntu machine to relax about wiping my temp dir on every reboot:

On Debian (and possibly Ubuntu?), edit the file /etc/default/rcS and change

TMPTIME=0

to the number of days that you'd like to keep files in /tmp around. "30" works for me.

now i won't be so paranoid about installing kernel updates right away.

mplayer dump

tried to record a lecture in realmedia format with mplayer, but it keeps dying in the middle. cache seems to help it get farther, but it still chokes. maybe i just need to give it a _big_ cache, or use the -cache-min option:

mplayer -audiofile-cache 8192 -cache 8192 -dumpstream -dumpfile out.rm rtsp://etc