memory profiling with python

tough to find good memory profiling for python. heapy-pe and the other (didn't bother to remember the name; pysizer?) turned out to be no good to me with numpy arrays which (surprise!) tend to be the biggest data structures i deal with. here a couple of others to try some time:

meliae is new and more cli-oriented, but looks easy enough to try (and script).

dowser spawned off of cherrypy, but i think it works for any python code with the web server as sort of a gui (i think unlike dozer, which targets wsgi apps. or maybe dozer is just a wsgi version of the 'gui'?).

here's an example of objgraph to analyse memory usage.

i think these are more garbage collector approaches, rather than hook-and-trace, so maybe more likely to work with libs like numpy.

memory_profiler also comes recommended and looks interesting. pure python, so portable and hackable.

r-cran-fimport

ubuntu has a package for downloading free econometrics data: provides import function to access (free) data from Economagic, the US Federal Reserve, Forecasts.Org, Yahoo and other web sources. worth a look to see some sources that would be good to look at. the group of people who wrote this also have a link to a brief discussion of portfolio risk surfaces over the convex hull of achievable sets. interesting... i was thinking something along the same lines, and it's a little gratifying that working pros seem already to be doing something similar.

__get__ method for fun and profit

just learned (or maybe relearned) something cool about python: the __get__ special method gets called when an instance is accessed as an attribute of another instance. not only are there potential uses for this, it also holds the key to understanding the 'self' and 'class' special arg in methods. this is something that confused me a couple of times before, such as passing references to instance vs. class methods from outside the class to be used inside the instance.

so, for example, i could allow instances of one of my classes to know how and where it's getting passed around, and something about the context when something is asked of it. maybe a quick and dirty memory leak tracker, when i know beforehand which objects are the big boys but i don't know who's pointing at them.

or maybe a little internal usage auditor, when i'm considering the impact of a refactor.

valuation books

couple of books recommended by people in the valuation business. one comment about duffie is that he was kind of a disappointment as a consultant, since he likes to stay more in the theoretical than the practical. not sure if this is the best book from him, but it's fairly recent. (search for 'dynamic')

Investment valuation : tools and techniques for determining the value of any asset

Damodaran, Aswath.

interesting that he says most analysis/justification for valuation is on discounted cash flows (as it seems to be in the book), but most valuation in practice is with ratios in relative valuation. contingent claim valuation is a more recent perspective, looking at opportunities available to a firm and pricing them like options. i was disappointed at how little there is on bond and commodity valuation, especially given the promise in the subtitle. interesting chapter on evidence of market efficiency.

Credit risk : pricing, management and measurement

Duffie, Darrell.

financial statement analysis and security valuation, 3rd ed

stephen h. penman

658.15

more on valuation, including slightly less than simple forecasting and detecting financial statement manipulation

dynamics of markets: econophysics and finance

joseph l. mccauley

658.15:519.217

empirical refutation of common modeling assumptions

value at risk: the new benchmark for managing financial risk, 3rd ed

philippe jorion

658.155

different types of risk, some 'industry-standard' real-life practical-experience rules of thumb

mplayer dump

tried to record a lecture in realmedia format with mplayer, but it keeps dying in the middle. cache seems to help it get farther, but it still chokes. maybe i just need to give it a _big_ cache, or use the -cache-min option:

mplayer -audiofile-cache 8192 -cache 8192 -dumpstream -dumpfile out.rm rtsp://etc

encrypted pdf files

managed to get a pdf file with restricted permissions using pdftk (pdf toolkit). the man page was a bit unclear on this, since if you just put in an owner password it will not actually restrict the file and no passwd is requested (or needed) to open it. this will do the trick, with all restrictions:

pdftk in.pdf output out.pdf user_pw foo

or, to make 2 levels of access:

pdftk in.pdf output out.pdf owner_pw baz user_pw foo

now the restrictions will be in place if you put in foo as the passwd, but they will not if you put in baz.

redirecting stdin, stdout, stderr from python

saw a nice comment in a post on python daemon forks that explains how to redirect std* pipes, even when they are accessed from c. i've been frustrated trying to figure that out before.

More reliable i/o stream redirection. Just reassigning to the sys streams is not 100% effective if you are importing modules that write to stdin and stdout from C code. Perhaps the modules shouldn't do that, but this code will make sure that all stdin and stdout will go where you expect it to.

import os, sys

out_log = file('/out/log/file/name', 'a+')
err_log = file('/err/log/file/name', 'a+', 0)
dev_null = file('/dev/null', 'r')
sys.stdout.flush()

sys.stderr.flush()

os.dup2(out_log.fileno(), sys.stdout.fileno())

os.dup2(err_log.fileno(), sys.stderr.fileno())
os.dup2(dev_null.fileno(), sys.stdin.fileno())

(and another poster suggests closing sys.std* before duping.) cool. i need to remember this next time i wrap somebody's code that thinks it's a good idea to barf on the terminal without a --quiet option.

also, the demon implementation looks pretty clean, with extra tidbits sprinkled into the comments, and the author explains the reasons for doing things. i don't think i will need the double fork for my udev script, but the first fork will be necessary.