Friday, June 22, 2012

hudson

i figured out how to use a number of hudson's features for greater awesomeness in keeping my code base under control.

for svn+ssh:
on svn server: (make sure it's openssh)
ssh-keygen -t dsa -f newkey
append newkey.pub to ~/.ssh/authorized_keys
on Hudson: upload newkey as private key file

select 'trigger builds remotely'
put this script into the svn server, in hooks/post-commit
# force a build on the Continuous Integration server
echo "calling url to build r$2 in $1" >> /tmp/svnBuild
/usr/bin/wget -O - http://hudsonserver:8080/job/JobName/build?token=tokenString&cause=SVN_commit > /dev/null

to do the build, i put everything into a bash script at the root of my svn repo. so in hudson, i put this into the 'execute shell' 'command':
#!C:\cygwin\bin\bash.exe
. hudsonBuild.sh

(i also had to make a windows link from one directory to another at one point -- i didn't know how to do this before, but from cygwin you can use this:)
cmd.exe /c mklink /D testdata "c:\otherTestdata"

i also tried a few other things that i didn't end up using, mostly because their outputs didn't jive with hudson.
pylint -f parseable python > pylint.txt # i used pep8 instead
sloccount --wide --details python > sloccount.sc # i used pynocle instead

under post-build actions,
'publish junit test result report' with **/nosetests.xml as the filename
'publish covertura coverage report' with **/coverage.xml
'publish html report' with directory 'complexity' and name metrics.html, directory pynocle and name index.html
'report violations' with **/clonedigger.xml for cpd and **/pep8.txt for pylint

the hudsonBuild.sh script is like this:

echo "PATH="
echo $PATH

echo "building extension"
cd python/module/extension
/cygdrive/c/Python27/python -c 'import __init__'
cd ../../

echo "starting doctest/coverage"
#cd python
which python
echo "PYTHONPATH="
echo $PYTHONPATH
coverage run 'c:\Python27\Scripts\nosetests-script.py' --with-doctest --with-xunit --with-coverage --with-profile --profile-stats-file=nosetests.hotshot --verbose
coverage xml
mv coverage.xml coverage_nopath.xml
sed 's//D:\\Hudson\\.hudson\\jobs\\JobName\\workspace\\python<\/source><\/sources>/g' coverage_nopath.xml > coverage.xml
# put hotshot output into the html report dir, so it will get saved for each build
mv nosetests.hotshot ../complexity/
cd ..

echo "starting pep8"
pep8 --repeat python | perl -ple 's/:\d+: ([WE]\d+)/: [$1]/' > pep8.txt

echo "starting clonedigger"
clonedigger --cpd-output -o clonedigger.xml python

echo "starting pymetrics"
pymetrics `/usr/bin/find python -iname "*.py"` > $COMPLEXITY_DIR/complexity.txt

echo "starting pycabehtml"
pycabehtml.py -i $COMPLEXITY_DIR/complexity.txt -o $COMPLEXITY_DIR/metrics.html -a $ACC -g $GRAPH

echo "starting pynocle"
cd python
./pynocleGenerate.py
cd ..

echo "build script finished"

continuous integration

recently decided it was time to stop putting off trying continuous integration for software development. (i'm only a decade behind the times; not bad.)

since i mostly use python, i had to look at buildbot. apache gump and cruisecontrol also seemed like possibilities. but in the end i tried hudson since i'd read it was easy to set up and use, and it really was. all i had to do was download the war file and run

java -jar .\hudson.war *>output.txt

(i had to redir output so the blocking to console wouldn't make it wait for me to scroll or press a key.)

here are some motivational/informative quotes on ci:
wikipedia:
continuous integration -- the practice of frequently integrating one's new or changed code with the existing code repository -- should occur frequently enough that no intervening window remains between commit and build, and such that no errors can arise without developers noticing them and correcting them immediately.

martin fowler:
continuous integration doesn't get rid of bugs, but it does make them dramatically easier to find and remove. in this respect it's rather like self-testing code. if you introduce a bug and detect it quickly it's far easier to get rid of. since you've only changed a small bit of the system, you don't have far to look. since that bit of the system is the bit you just worked with, it's fresh in your memory -- again making it easier to find the bug. you can also use diff debugging -- comparing the current version of the system to an earlier one that didn't have the bug.

bugs are also cumulative. the more bugs you have, the harder it is to remove each one. this is partly because you get bug interactions, where failures show as the result of multiple faults -- making each fault harder to find. It's also psychological -- people have less energy to find and get rid of bugs when there are many of them...

if you have continuous integration, it removes one of the biggest barriers to frequent deployment. frequent deployment is valuable because it allows your users to get new features more rapidly, to give more rapid feedback on those features, and generally become more collaborative in the development cycle. this helps break down the barriers between customers and development -- barriers which i believe are the biggest barriers to successful software development.

paul duvall, cto, stelligent incorporated:
6 anti patters
infrequent checkins, which lead to delayed integration
broken builds, whcih prevent teams from moving on to other tasks
minimal feedback, which prevents action from occurring
receiving spam feedback, wich causes people to ignore messages
possessing a slow machine, which delays feedback
relying on a bloated build, which reduces rapid feedback

Tuesday, June 12, 2012

finding methods from parent classes in python

instanceName.methodName.im_func.func_code
or
instanceName.methodName.im_func.__code__

i always forget how i can find out where a method is defined if it comes from somewhere up the inheritance tree. maybe there's a better way, but the special attrs above will at least give me the file and line number.

guitar patches

http://www.jameslimborg.com/boss-gt-10-patches-download.html

cool. lots of van halen and boston sounds.

the guy mentions wanting another 64-band spectrum analyzer. i wonder if he is aware of wavelet transforms, or empirical mode decomposition/other hilbert-huang based transforms that could make his life easier by splitting time and frequency more optimally than stft.