Tuesday, November 24, 2009

execnet

execnet is another python remote execution package. looks kind of like a pythonic mpi with minimal setup overhead, since the sends and receives are manual. can give modules to remote instances, but you have to resolve import dependencies manually.

support vector machines with pyml

there are a number of svm codes out there, but the only non-swiged pure python one for which i've seen a third-party recommendation is pyml. decent general svm intro doc, too.

Information Theory, Inference, and Learning Algorithms

another book available online, including latex and octave, perl, etc, source. might be a good ref on IT basics, neural nets, coding, compression, and monte carlo.

data clustering for screen scraping

i just had an idea for an application of unsupervised data clustering. a quick google popped up python-cluster. looks like it's been abandoned for a couple of years, but it has a hierarchical algorithm and (maybe) k-means. i might try it. also, scipy.cluster has kmeans and vector quantization, with self organized feature maps and other methods promised later. the app is screen scraping web pages and trying to get the main content (an article, for example) without the ads, links, and other junk around the edges. i think it might be possible to look at each line (after tossing everything inside script tags) and separate the lines based on their length and the percent of the line inside html markup. the reasoning is that real content usually has long lines in the source and a small fraction of html taggage. i probably want to throw in line index as a third variable, since the lines i want will probably be close together. another thing i could do is grab multiple pages from the same site: either multiple articles that should have the same format or multiple copies from different days for a frequently updated page. that would allow me to do two things. first, i could combine data points from multiple pages to get higher point density for the cluster detection. (might need to test if an each individual page is sampled from the same distribution as the others to throw out outliers.) second, i could detect identical lines to throw away as nonunique boilerplate.
the ubuntu python-mvpa package looks like it might fit the bill.

Wednesday, November 18, 2009

Software for Hidden Markov Models and Dynamical Systems

all the source and the whole build system for a book by andrew fraser is online. the code is written in python, and the typesetting uses rubber, the python wrapper for all things tex. not only is the subject matter interesting (an application chapter on obstructive sleep apnea), but i think it has some good examples of how to write a book or long report with python and latex. took a _long_ time to put everything together, but it generates the whole book. if i actually use it, i think i'll have to buy the book in gratitude. (had to comment out a \printnomenclature line to get latex to run. guess i have a bad nomencl.)

Tuesday, November 17, 2009

Flexible Algorithms for Image Registration

book by jan modersitzki that has an associated website with matlab (meh) software and other ref docs.

Scientific Computation

book from cambridge u press by gaston gonnet. i'm not sure the methods are extremely advanced, but it gives an interesting spread of applications, including protein structure and stock price prediction.