trading optimization

i've been thinking about how to optimize trades for some algorithmic trading. i need to define objective functions, obviously, but how exactly? risk vs. return, but risk (for a liquid asset) depends on a portfolio state between transactions, and returns are realized upon the transaction. it just occurred to me that i can reduce it to simple shift and scaling operations on a (somewhat unknown) pdf. if i have a certain unhedged amount, x, of a risky asset, and i buy or sell to end up with a*x, then that scales the pdf f(x) to f(a*x)/a. the trade should give me a return, r, so the pdf of my total return after the trade would be something like f(a*x-r)/a. so i know what the shift and the scale are, even though i don't exactly know what f(x) is. now it's a question of whether f(a*x-r)/a is better than f(x), and i trade if it is (i.e., maximize whatever metrics to find the optimal r,a). does this answer the question of how to place limit orders away from the market prices? i'm not sure yet.

psuade

another uncertainty/optimization/sensitivity analysis package, this time from llnl. looks like they want to make their own built in environment (why?! why?!) but at least it's gpl.

amalgam and dream

recently came across some interesting work by jasper vrugt. AMALGAM is a self-adaptive global optimization algorithm. basically you start with a latin hypercube to get some initial sampling and feed those points to generate a couple of new points from each of a set of different global optimizers such as genetic, swarm, etc. after you sample that second generation of points, you can evaluate the quality of those optimizers on how well they pick new points for that type of problem. for each successive generation, you can sample more points from the optimizers that have done well. strikes me as similar to cover's optimal portfolio theory in the way it weights the winners. DREAM (differential evolution adaptive metropolis) is a sampling algorithm that has some advantages over markov chain monte carlo (mcmc) methods while maintaining desirable statistical properties that i don't really understand yet. useful for optimization and parameter fitting for high-dimensional inverse problems.

python design optimization

this article looks like it might be interesting. multidisciplinary design optimization, with a significant section on software issues and python.

python profiling and optimization

python's builtin cProfile/pstats are great, and it is really easy to get started with them. (just look at the top of the doc page for cProfile.) one problem, though, is that cProfile doesn't do line-by-line profiling. hotshot did, but it's deprecated and arguably harder to work with. a little poking around turned up line_profiler and kernprof. looks like they're the pet project of one of the enthought people. i haven't tried it yet but it looks pretty easy to use, even through the api as a part of a larger development tool. i'm thinking the best way to use it would be to run cProfile first and take the top 3 time hog functions (or 1 or 2 if they get over 90% total time) and auto run it again with line_profiler turned on for those functions. maybe i could build this into my doctester by first running the small and fast doctests to confirm correctness, then run some __profileTests__ module string that has more realistic usages. while i'm writing code, i can interrupt the profile tests to rerun the doctests, and i'll see the profile results any time i happen to let it run all the way through. kernprof is pure python and acts as a helper/wrapper for both/either cProfile and line_profiler. but line_profiler uses cython. speaking of cython, for optimization cython looks like an impressive way to generate c source and build it to run fast (and obfuscate, if that matters). i might look at it as a possible alternative to pypy/rpython, if i can't get that to work. one problem, though, is that it looks like so many changes and type annotations might be needed with numpy that i might as well just write the c. i've been very impressed lately with results i get using scipy.weave, with both the blitz converter and the default c-ish converter. i can narrow the hotspots down to one or two methods, and translating the python to c is not too hard then (especially with the default converter). scipy.weave.inline is not too shabby either, for speeding up numpy expressions. so pypy is another very interesting possibility for autogenerating c from python. someone has already shown an example of how to do this, including some initial work with numpy. EDIT: maybe using cython is not as bad as i had thought. apparently it will compile just about any valid python (with a couple of restrictions on closures) with a modest speed up. all the extra syntax is just for making it run faster, and even those annotations can be done in a way that preserves python compatibility of the original source. maybe i can even look at multiple-dispatch packages to make an auto-type discovery based on doctests and profile tests to make my augmenting .pxd files for me. probably the first thing i should do is try to replicate some of the speed-up results from the cython numpy tutorial. there's also another tutorial with a convolution example (similar to chapter 10 in the manual, but slightly updated including warning about None args and calling numpy c funcs from cython) that might be a better place to start.

chapter 11 in the user manual gives an example of how to profile with python's profiling tools, while the wiki page on profiling shows a (partial) example of low-overhead profiling with valgrind. debugging with ddd and gdb is also discussed in a wiki page at wiki.cython.org. also, since a discussion on their email list in april/may 2009, an --embed option has emerged that facilitates embedding a python interpreter with a python entry point. i had been looking at pyinstaller, which looks like a very nice cross-platform way to make standalone executables from python. maybe i won't even need it with cython --embed. details on the above + more cython tips on the wiki. okay, looks like the readme for cython_freeze says cython --embed can only put one module in with the interpreter, while cython_freeze can put any number of cython modules in there. also, it can put a normal interpreter shell in, not just a 'main' module so you can use it interactively. not sure if cython --embed can do that. also, i should look into using plipy for this sort of thing. it will allow the executable to run on any linux machine, from ancient redhat boxen to the latest ubuntu machine. 'twould be nice to preserve some of python's portability when i freeze it to binary. and there's a link there to portable python, which seems to do the same thing for windows. also, virtualenv is worth a look. lots of people endorse it.

https://translate.svn.sourceforge.net/svnroot/translate/src/trunk/virtaal/devsupport/profiling.py
saw a ref to this, which appears to be a way to profile python callables with KCacheGrind.

random thought fragments on multiresolution analysis

someday i'd like to write a textbook on multiresolution techniques in surrogate modeling. in the intro i would present a unified approach to orthogonal decompositions like eigensystems, svd, and ffts with wavelets, linear model sensitivity analysis, and mdl. i would go on to answer questions like, 'how much information is gained in terms of the modes and singular values if i update an svd representation of a surrogate model?' and how do i do an svd surrogate model in the first place? will it capture features on different scales, or can i force it to? how do i use that predicted information gain to guide adaptive sampling?