Friday, July 16, 2010

notchup

here's a recruiting/headhunter marketplace jobs site that i should get on my radar (or, rather, on theirs). they use independent talent scouts that scour sites like linkedin and personal contacts to find candidates that might already be employed ('passive' candidates) under the assumption that really good people are very rarely out of a job or trolling the job websites, looking for work. hang out my shingle and let them race to me with offers... now that's the way to get a job.

Wednesday, July 14, 2010

using googlecl with blogger from bash

i think i finally found a good way to post from the cli.

google blogger post --title 'blog post title' --tags 'tag1,tag2' $'multiline content...'

the $ in front of the hard (single) quote lets bash escape single quotes inside that last string, but nothing else. so i can put all kinds of crazy chars in my multiline post, including a backslash-escaped single quote, with impunity. only problem is editing a previous line.... well, now i have an excuse for leaving the typos in.

stock index futures and price forecasting

trying to figure out exactly what numbers financial commentators use to make headlines like, "futures point to higher open..." often they point to specific futures contracts, like SPc1, NDc1, or DJc1. the cme group equity futures quotes seem to be what people are referring to (or at least have equivalent price percent changes). unfortunately, the javascript in those pages that handle the quote updates prevents convenient scraping. options pages like these, otoh, are exceedingly easy to scrape as whitespace-delimited plain text. and with the eom (end of month, european-style) options, the price/probability is fairly simple. if i really need to get at the futures contract prices, i could use bloomberg, which just has static html tables (with major us and world indices on one page). back on the cme group site, i couldn't find quotes for options for oil, and even the time and price transactions list is another inaccessible js job. however, if i only care about end-of-day data, there are a lot in the ftp server (see the settle/ dir, for example) and other links from the volume:volume & open interest tab of one of the futures pages. daily settlement prices (near the bottom) has links for interest rates, equities, (agricultural) commodities, precious and industrial metals (in comex), oil and energies (in nymex and cme clearport clearing), and fx. the volume by price data on that page might be particularly useful for building distributions. it combines total volume for the day at a each price, and the historical data available on the ftp server goes back a year and a half. the format is a bit obfuscated, but there are links to one-page format descriptions next to the data links.
one other thought about using derivatives for price forecasting/asset valuation: the price of the derivative depends not only on what people believe about the future price, but also on what everyone knows about the current price. so, if i had intraday data for the derivative price, i should subtract out the effect of the underlying price to get the future price. i wonder if ica on the log derivatives would achieve this... obvious maybe, but worth noting so i don't forget.

more python libs

rthread exposes a threading interface on remote processes, if you are into that kind of thing. \
\
struct (standard library) interprets a string as a c structure.\
\
timeit (standard library) for quick and dirty timing of a command string, like the ipython magic and tic,toc in matlab.

Tuesday, July 13, 2010

python from excel

xlrd, xlwt, and xlutils are fine for accessing excel files from python, but what about accessing python from excel (like calling out to vba hooks to define user functions)? pyxll seems to do what i would want (even claims numpy, ctypes, and cython support!), but the license seems very restrictive. if there is nothing better, it is probably the way to go. i might also check out the ironpython hook (http://www.ironpython.info/index.php/Interacting_with_Excel) and discoveryscript (http://www.xefion.com/discoveryscript.html). discoveryscript is free, but i think its license will also restrict redistribution (and the other products on the website are in the $500/licence range), so that might be problematic.

Monday, July 5, 2010

clyther

tool for generating opencl code directly from python, as an alternative to cuda and pycuda. might be a little alpha at the moment, but i should keep tabs....

ubuntu one

can't believe i haven't signed up for this yet. is there any downside? i guess i could use cryptoloop on all my machines that link to it if i'm paranoid about privacy. but that would prevent me from accessing files on the web.

googlecl

google released command-line access. now i can post (like this one) from my tilda window.

Friday, July 2, 2010

tripadvisor for cheap hotels

tripadvisor.co.uk has a nice feature -- it searches a number of other sites for cheap rates on the same hotel. helps prevent non-refundable clicker's remorse.

coroutines in python

wow, am i behind the times or what? i didn't realize python 2.5 added real coroutine support by making yield an expression. now i can call the send method on a generator and it will pop in as the evaluated result of the yield.
also the webbrowser module can pop up a browser window (or tab). handy for showing docs.
i'd only just started using with. i need to stay up to speed on these things.

test coverage in python

i think it's about time for me to abandon the half-home-baked test coverage tool i have been using in favor of coverage.py by ned batchelder. nice html reports (easy to lauch view, now that i have google-chrome) and i'm already using html reports generated by cython -a. another possibility is canopy and instrumental. uses ast tools to analyse code; maybe useful for other things, too.

tahoe-lafs

decentralized secure data store. if i ever need cloud storage, this might be a good alternative to cryptoloop.

Thursday, July 1, 2010

3d pdf animation

was looking into adding animation to my 3d pdf objects again. it's really hard to find any useful info on the newer prc format (the best i could find was a low level adobe tech support guy whining about the iso application taking a long time). u3d has at least some skin and bone based animation, beyond the rigid body + scale part-level animation that is useless to me (every reference to key frame animation, afaik, refers to this). but i'm not sure if acrobat or the adobe reader support that part of the u3d ecma standard. this page (google cache) from right hemisphere seems to say that it doesn't, but that might refer just to their deep exploration code. at any rate, i think they were one of the founding parties to the standard, so it's not promising if they never bothered. somewhere i found an unreferenced statement that everything supports ecma 1st edition, but acroread 7 doesn't support ecma 3rd edition.
however, the JavaScript for Acrobat 3D Annotations API Reference doc on the adobe website describes a Bone object and specifically says it 'is usually moved over time to create animated characters.' Mesh objects contain geometry, but nothing in the api doc looks like it could be actual vertex positions, unless they're lurking in the Node.metadataString. i can find no additional info on Bone.
the U3D Supported Elements doc from the 3d dev center states that skeleton descriptions are 'parsed and unused (that is, no skeletal animation, skinning, or bones)'
also, opacity and material color animation are not supported, and i'm pretty sure that's a basic u3d limitation.
it's hard to find much info about the newer prc format. i don't see anything in the adobe spec docs that obviously provides for it (though not much anything obvious in the specs... quite sparse). i think the best i can do right now is look at the asymptote code (there was a passing reference to animating the prc in one of their journal articles as a todo item; alas, the only animation they show in that article is an ordinary movie). rumor has it that adobe turned the prc implementation over to a third party, bailing again on a drive into the 3d market, so i'm not sure prc has a future any brighter than u3d, which they have said (fwiw) they will always support.
so the only way out i can see right now is to create multiple meshes, one for each time step, and use javascript to cycle the visibility toggles. i guess the only real wasted space comes from the redundant connectivity, and that might be manageable if i continue to use only the exterior surfaces. and it would allow me to circumvent the u3d material color animation limitation. a rough estimate is 6.5 bytes/exterior node for u3d file size. actually, maybe that's not too bad... a 5 second, 20 fps animation with 8000 exterior nodes would fit in under 5MB. not too big for a short report or presentation. (maybe ~10 fps is a more realistic limit on what acroread can achieve.)
a little playing around with the quality options: diffuse color quality moved down from 300 to 100, -dcq 100, make practically no difference in file size. specular color quality, -spq 100, and animation quality 1000 -> 100, -aq 100, made zero difference since i'm not using either. geometry default quality down from 300 to 100, -gq 100, makes no difference, as does texture quality 75 -> 50, -tq 50. texture coordinate quality 300 to 100, -tcq 100, and normal quality 300 to 100, -nq 100, make zero difference. the one thing that does change the file size significantly is position quality. i had been using -pq 500, and -pq 100 decreases the size by about 7-8%. and maybe decreasing the palette from 256 to, say, 16 colors would help, though it would surely be less than a 10% difference. so i don't think there's any way i could squeeze it down more than 20-25%.

pypy 1.3

pypy now has stackless, jit, sandbox, and low-memory versions out in 1.3. and support for c extension modules is coming along. and, as part of one of this year's gsoc projects, they are working on adding numpy to pypy. cool!