delta debugging, automated debugging

i was thinking about unit testing and delta debugging, or differential debugging?, and i thought python's introspection tools could make it possible (and not too hard) to do this automatically. i could make a tool that takes 2 adjacent versions from a repo, runs a test synchronously on both of them (that fails on only one of them), and drops into 2 debuggers at the point where a callable is about to return with 2 different values. that would make it really easy to debug the failure, right at the point where it happens. i could make it smart enough to distinguish between private, internal interfaces (starting with an underscore) which are allowed to change functionality between revisions and external interfaces which should be invariant (except possibly on marked, backwards-compatibility-breaking revisions). i could speed things up by only trapping callables that contain modified lines in the revision diff and only running the tests with coverage on those lines. i could even do interesting things like adding in only some of the diff blocks to preserve correctness and walk backward or forward through revisions -- truly automated debugging! long term vision: maybe combining selective modification with an auto-commit-on-branch with each file save could allow me to tinker and spew out code in an entirely creative mode. the computer then assembles that into something correct and only asks me to clarify when it gets too confused. and all those microrevisions could provide a cleanroom-like estimate of bug introduction and elimination rates, as well as functionality addition rate based on number of paths through external interfaces.

that could also mean that i wouldn't need to write as many unit tests, since the same tool could catch calls and returns to any callable and compare results when they return. all i need to do is have my top level/integration tests and enough unit tests to fill in the coverage gaps. i could detect any changes on the internal interfaces if i wanted to, without necessarily breaking the build each time. any 'unit test' that was derived from the top level would be updated automatically. and if my coverage/diff analysis is reliable enough, i could speed up my continuous integration build a lot without losing any information. (i would have to be very careful about this, though, and track all kinds of external dependencies, etc.)

all of this would be a huge productivity boost since debugging is one of the biggest and most unpredictable time sinks i have. having unit tests is important and writing them often helps the development process, but updating lots of test results due to a small change in a core bit of code is a real pain. also, i don't like choosing between long-running tests or tag-along intermediate result pickle files.

came across some sites that should be useful for debugging, especially automatic debugging:

http://root.cern.ch/drupal/content/cint
http://root.cern.ch/drupal/content/cling
http://root.cern.ch/drupal/content/reflex
http://clang.llvm.org

cffi is the foreign function interface for python, used by pypy.
http://cffi.readthedocs.org/en/release-0.6/

some work people have done on delta debugging
http://delta.tigris.org
http://www.st.cs.uni-saarland.de/dd/ddusage.php3

useful to auto debug python
git bisect run http://lwn.net/Articles/317154
inspect module
http://pymotw.com/2/trace/index.html
http://pymotw.com/2/profile/index.html

also i should use vbench in my continuous integration. i think it was written by the same guy who started pandas. maybe there's a way to get it or multi-mechanize.py to output jmeter format so jenkins/hudson can read and plot it over time.