A Pragmatic Look at Exception Specifications

http://www.gotw.ca/publications/mill22.htm

nice ref on how to handle exception specifications in c++, especially useful with inheritance, etc.

debugging c++ extensions to python

trying to debug a python extension module written in c++ (wrapped with swig). i think this would be so much easier if i were using gcc, but python is built with msvc... setup.py wants the debug versions of python libs, but i don't have them and don't really want to try to build python from scratch right now. these refs seem relevant: http://www.velocityreviews.com/forums/t677466-please-include-python26_d-lib-in-the-installer.html http://vtk.org/gitweb?p=VTK.git;a=blob;f=Wrapping/Python/vtkPython.h;h=9d01ac21bafae0a24252398f268b6b3563df62cd

design patterns

comment from duffy on most useful from gof (gang of four) book of object-oriented design patterns: "GOF is useful but should not become an objective in itself. The most useful ones in general (dependent on the domain of course) are Visitor, Strategy, Facade and Template Method pattern. Singleton and Observer can best be avoided."

thinking in c++

lots of versions out there, but this one seems to be the best one of volume 1 i can find. (get the non-landscape one.) don't get that version of volume 2. it is old and incomplete. the 27 dec 2003 version of volume 2 is the print version; i can't find any electronic version other than the html. i think the quality and depth is bettern than vol 1, but it's too long for me to go through right now. c++ is definitely a language that has a vast landscape to explore. i think i would need to read the 'design and evolution' book to understand the rationale and really get at how things work, and i will eventually need to read the templates book and the gang of four design patterns book to know how to use it properly. maybe the 'effective' and 'exceptional' books, too. for now i will prioritize the parts of vol 2 that i haven't read yet. 1: Exceptions: p. 53-75 9: Multiple inheritance 10: Design patterns 6: Generic algorithms 7: Generic containers 8: Runtime type identification 2: Defensive Programming 5: Templates: template programming idioms - end (p. 252 in html imported to word doc) 11: Concurrency (might skip, focuses on specific library) a few tips to remember: -- return value optimization: return a constructor call; it saves a copy-ctor and dtor -- it's ok to throw in a ctor, as long as you clean up resource allocations first -- never throw in a dtor, since the dtor could have been called due to a previous exception and c++ requires 2 unhandled exceptions to crash the program -- throw by value, catch by reference: allow the exception handling to deal with the memory management -- deal with errors and catch exceptions at the nearest handler that has enough information to deal with them

parallel, numpy, shared memory,...

trying to figure out how to do parallel processing efficiently with python, and numpy in particular. i want something simple, closely related to the original serial code (sorry, mpi, you're not welcome here). parallelpython holds some promise, dodging the gil by starting separate interpreters and piping pickles back and forth. similar to pyro, and it looks pretty seemless between smp vs. cluster. unfortunately, pp does not provide for any shared mem so big data (even read only) must be copied (and pickled!) on smp. multiprocessing is now built in to 2.6 and backported as far as 2.4 or 2.3. doesn't handle remote processes, though the pp/pyro-type pickle server (manager) interfaces with inet ports. i think it basically forks the process to make the worker processes, so you get less overhead (os service vs. cranking up a new python). and there's no need to feed it modules or any other globals; these get copied on the fork. it has some capability to share memory, though i think these are only kinda raw ctype buffers. (i think all of this is similar to the approach posh used, though more generally for user-defined types -- high quality hackery but unmaintained since 2003.) apparently some people have coaxed numpy into using these ctype arrays to make np arrays sit in shared memory land, with views available to the children. (maybe using this sort of thing.) the approach got an attaboy from the big man himself, travis oliphant, but (in the same dir) sturla has a sharedmem module written later (cleaned up and posted here) that looks like it makes lower level sys calls to create shared memory space manually. does that mean the multiprocessing shm is unsatisfactory? the paper does warn that it's a moving target, and the scipy cookbook indicates the same thing: 'this page was obsolete as multiprocessing's internals have changed.' epd has a webinar coming up promising to demo multiprocessing with large arrays, so maybe i should see what they do. anyway, if i do use this for parallel stuff, this blog post might be useful. here's another page that looks very useful for multiprocessing. fabric looks interesting, too, though more geared toward sysadmin stuff. maybe similar to posh in some ways.

coroutines in python

wow, am i behind the times or what? i didn't realize python 2.5 added real coroutine support by making yield an expression. now i can call the send method on a generator and it will pop in as the evaluated result of the yield.

also the webbrowser module can pop up a browser window (or tab). handy for showing docs.

i'd only just started using with. i need to stay up to speed on these things.

test coverage in python

i think it's about time for me to abandon the half-home-baked test coverage tool i have been using in favor of coverage.py by ned batchelder. nice html reports (easy to lauch view, now that i have google-chrome) and i'm already using html reports generated by cython -a. another possibility is canopy and instrumental. uses ast tools to analyse code; maybe useful for other things, too.

memory profiling with python

tough to find good memory profiling for python. heapy-pe and the other (didn't bother to remember the name; pysizer?) turned out to be no good to me with numpy arrays which (surprise!) tend to be the biggest data structures i deal with. here a couple of others to try some time:

meliae is new and more cli-oriented, but looks easy enough to try (and script).

dowser spawned off of cherrypy, but i think it works for any python code with the web server as sort of a gui (i think unlike dozer, which targets wsgi apps. or maybe dozer is just a wsgi version of the 'gui'?).

here's an example of objgraph to analyse memory usage.

i think these are more garbage collector approaches, rather than hook-and-trace, so maybe more likely to work with libs like numpy.

memory_profiler also comes recommended and looks interesting. pure python, so portable and hackable.

__get__ method for fun and profit

just learned (or maybe relearned) something cool about python: the __get__ special method gets called when an instance is accessed as an attribute of another instance. not only are there potential uses for this, it also holds the key to understanding the 'self' and 'class' special arg in methods. this is something that confused me a couple of times before, such as passing references to instance vs. class methods from outside the class to be used inside the instance.

so, for example, i could allow instances of one of my classes to know how and where it's getting passed around, and something about the context when something is asked of it. maybe a quick and dirty memory leak tracker, when i know beforehand which objects are the big boys but i don't know who's pointing at them.

or maybe a little internal usage auditor, when i'm considering the impact of a refactor.

hudson-ci

web interface on a continuous integration system for building, testing, revision tracking, and distributing code. if i ever have any users for my code, and especially if i have other people writing code, i need to check it out. too bad it's all java.

buildbot is apparently more python-oriented, and bitten (for trac) is in python. but hudson seems to get the vote for easy install and customization. here's a useful summary:

We use both Buildbot and Hudson for Jython development. Both are useful, but have different strengths and weaknesses.

Buildbot's configuration is pure Python and quite simple once you get the hang of it (look at the epydoc-generated API docs for the most current info). Buildbot makes it easier to define non-testing tasks and distribute the testers. However, it really has no concept of individual tests, just textual, HTML, and summary output, so if you want to have multi-level browsable test output and so forth you'll have to build it yourself, or just use Hudson.

Hudson has terrific support for drilling down from overall results into test suites and individual tests; it also is great for comparing test output between builds, but the distributed (master/slave) stuff is comparatively more complicated because you need a Java environment on the slaves too; also, Hudson is less tolerant of flaky network links between the master and slaves.

So, to get the benefits of both tools, we run a single instance of Hudson, which catches the common test failures, then we do multi-platform regression with Buildbot.

file readers

i've spent way too much time already writing a file reader for slurping up binary data from an experimental setup. i've been burned in so many different ways by incorrect or nonsensical data in these files, i've decided on a rule i should follow any time i need to do this again: read each atomic unit of data with as few context assumptions as possible; i.e., loop over units in the stream, do not loop over any assumed structure for the units. both missing and duplicate data have wreaked havoc on my pretty little reader, each requiring a new refactoring. next time i will just start out iterating on the stream and plan on dealing with nonsensical structure, even when (!!) there is metadata that could turn out to be wrong. for now i'll just build the most convenient structure in memory for all the data that makes sense and throw junk into an extra array that i can check later if i need to. i won't be sure if that's the best way until i get a chance to use it.

maybe i should use something like python-hachoir for this.

math-like programming

i couldn't prove it at the time, but i had a feeling that i knew what i was doing....

i think i better understand now why it helps to keep a piece of code running correctly with unit tests while modifying it. it's like a mathematical equation. you can use substitutions and properties of operators and functions to modify the expressions, but only if you maintain equality at every step. maintaining that equality requires an understanding of the underlying math objects, and knowing which way to transform the expressions requires skill and creativity. but the hardest part is to lay down the governing equations to begin with -- to state the paradoxically precise abstraction of reality.

inasmuch as a computer language is an alternate grammar for discrete math, it is to be expected that to manipulate an existing expression while maintaining correctness is easier than to derive an algorithm from scratch (or from incorrect code). sometimes i have correct code that i still want to modify, for example, to generalize or to optimize. i need to think like a compiler and make my modifications in smaller steps, each maintaining correctness with respect to the unit tests, rather than try to leap at once to a large rewrite. faster, easier, and clearer thinking, often with solutions that present themselves along the way.

for example, can i move that assignment from the beginning to the end of the loop? i want to eliminate that variable; first i'll make it redundant. i think these two expressions are equivalent, so i'll put in an assert to test that before replacing the old one. now i have a better way to refactor and i know why to do it.

python coverage testing

coverage.py: actively developed, simple command line execution for html output. integration into my test framework would require some work with the api, though it already has a nose plugin i could look at. figleaf is based on coverage.py but runs faster because it ignores python builtins by default. better separation between code analysis and reporting, so you can more easily combine results from multiple runs. maybe not quite as much spit and polish. the coverage langlet module (?) takes a different approach to coverage monitoring by inserting sensor objects into block entry points of the compiled code. the module seems to be more alpha quality than ned batchelder's, but perhaps an interesting alternative if that one doesn't work out for some reason. i think i'll try coverage and/or figleaf to see if they are helpful beyond the builtin trace capability.

automatic debugging

saw some interesting work on automatic debugging. the idea is to use tests (passing tests with one failing tests) to evaluate random code changes and find one that works. i don't really expect my computer to debug my code any time soon. but one interesting thought was that the ast nodes to change were weighted by positive and negative test coverage. maybe i could use coverage in this one to localize a bug. (more likely is places covered by multiple negative tests, less likely in places covered by positive tests.) that would help me find the bug, which is almost always the hardest part. refs to the spike black-box fuzzer from immunitysec.com, strata dynamic binary transformation (from virginia).