Wednesday, October 16, 2013

trading with python

http://www.tradingwithpython.com/
looks like it might be interesting. found it via his blog; he also has a for-pay instructional course.

Tuesday, October 15, 2013

Wednesday, September 25, 2013

tox

http://testrun.org/tox/latest/
generic virtualenv management and test command line tool.

used by coverage.py, looks pretty interesting for testing against multiple versions and environments.

Thursday, September 5, 2013

scm-latexdiff

http://www.numbertheory.nl/2012/02/09/scm-latexdiff-a-python-script-to-calculate-diffs-for-tex-files-in-git-or-mercurial-repositories/

works for git and hg repositories.

inspect-shell

https://github.com/amoffat/Inspect-Shell

looks like a pretty cool way to open a manhole into a running python script. it is not a debugger and it doesn't lock or interrupt. allows you to call functions, inspect or (dangerously) modify globals, etc.

Tuesday, September 3, 2013

gitflow

http://nvie.com/posts/a-successful-git-branching-model/
procedure for working with a sane graph of git branches, from Vincent Driessen

more info and tools from
https://github.com/nvie/gitflow

gource

http://code.google.com/p/gource/

version control visualization. makes really cool movies showing the people making revisions over time.

Tuesday, August 27, 2013

windows python path bug

had a weird bug trying to import numpy:
...
from numpy.linalg import lapack_lite
ImportError: DLL load failed: The specified module could not be found.

looks like it's actually because the python Scripts dir needs to be in the path
http://bugs.activestate.com/show_bug.cgi?id=89474

export PATH=$PATH:/cygdrive/c/Python27/Scripts/
made it work.

Monday, August 19, 2013

git - merging a merge from a clone

still trying to figure git out.... i had a clone in which i had forked off a previous commit and made some changes, then did a merge to bring both branches together. now i need to bring that merged update (along with the intermediate forked-off commit, if possible) to the origin. 'git fetch file:///remote' didn't seem to do anything, and even after 'git merge' i didn't see those other commits in my original repo.

finally, the thing that seemed to do the trick was 'git merge FETCH_HEAD'. so i guess git fetch grabbed the info, but it just didn't make it visible. the clue was the return from 'git fetch':
* branch HEAD -> FETCH_HEAD

now my repos match.

git baby steps

http://www.youtube.com/watch?v=ZDR433b0HJY

gitref.org
gitpro.org
whygitisbetterthanx.com
git svn

git config --global user.name "Trevor Tippetts"
git config --global user.email "trevorbtippetts@gmail.com"
git config --global color.ui true
git init
git clone git://github.com/schaco/ticgit.git

git add (file) # snapshots; commit will take files as they were at time of 'git add'
git status # -s gives one-liner summaries
git diff # patch for unstaged changes
git diff --cached # patch for staged but uncommitted changes
git diff HEAD # patch for staged and unchanged (everything since commit, ignoring staging)
git diff --stat # summary of changes

git commit -m 'my comment' # commits changes present at the time of git add
git commit -a # svn style commit snapshot at present time; auto stages everything
git add -p # stage patches; it will ask about each diff block

git branch # create branch or view local branches
git branch -r # view remote branches
git checkout # switch between branches
git checkout -b newBranchName # create a branch and switch to it immediately
git checkout master; git merge newBranchName
git status # will show files needing merge
git log [file] # show commits on 2 sides that created the conflicts in conflicted file
git mergetool # external: kdiff3 tkdiff xxdiff meld gvimdiff opendiff emerge vimdiff
git add [merged file] # will take this snapshot as the new (merged) file on next commit
git commit

git branch -d newBranchName # safely delete (only delete if it is reachable in the history of HEAD)
git branch -D branch # unsafely delete (changes on that branch might not be reachable by any other branch)

git push origin master # move remote master branch to local master branch
git fetch # get remote changes and new remote origin master; then you can do normal merge and re-push
git push origin iss53 # make/move remote branch pointer iss53, not master

git pull # wrapper to fetch and merge a branch; can be confusing if not working on master or not familiar with tracking branches; easier to use fetch+merge manually
git remote add somebody git://github.com/etc # to pull from collaborator
get fetch somebody; git merge somebody/master

git log # show all commits reachable from HEAD
git log --oneline branchName # shorter one-line for each commit reachable from branchName
git log --oneline --graph # ascii graph going backwards from HEAD
git log --oneline --graph --all --decorate # all commits, show where branch pointers are pointing
git config --global alias.lol "log --oneline --graph --decorate" # git lol; git lol --all
git config --global alias.ff "merge --ff-only"

# log subsets with graph operations
git log branchA ^branchB # all commits reachable by branchA but not by branchB
git log branchA ^master # what will merge if i merge to master? what will i lose if i delete this branch?
git log origin/master ^master # commits that came in from remote; no output after merging
git log master ^origin/master # commits that will get pushed to remote; will still have output after merging but before pushing
git log -p # show patches for each commit
git log --stat # show files that changed, with indication of extent of changes

# these 12 above commands are 99% of what you need to do with git

# more stuff
git svn

git blame -C # find where snippets were copied from

git bisect start # initialize bisection search
git bisect bad # current commit (HEAD) is broken
git bisect good goodCommit # identify last known good branch; checkouts a revision in the middle
git bisect good OR git bisect bad # after testing
...
git bisect reset # go back to the revision where you started

# or make your script return 0 on success, nonzero otherwise
git bisect start HEAD goodCommit
git bisect run test-error.sh

git reset HEAD -- fileName # unstage a file (reset the hash for the file in the index to its value after last commit)
git config --global alias.unstage "reset HEAD"
git reset --soft HEAD~ # undo last commit (go to parent of HEAD) and restage last committed files; basically same as git commit --amend
git reset --hard # reset --soft, then undo changes in working dir (make working dir like the index)
git rm # remove from staging and delete; can also just delete it and git commit -a; --cached will leave file in working dir
git mv # alias for git rm --cached orig; mv orig new; git add new

git stash # save work-in-progress state and checkout last commit
git stash list # show saved states
git stash apply # reapply stash to working dir
git stash drop # remove last stash from the stack
git stash pop # apply and drop
git stash clear # drop the whole stack



lol = log --graph --decorate --pretty=oneline --abbrev-commit

So, just copy the following into ~/.gitconfig for your full color git lola action:

[alias]
lol = log --graph --decorate --pretty=oneline --abbrev-commit
lola = log --graph --decorate --pretty=oneline --abbrev-commit --all
[color]
branch = auto
diff = auto
interactive = auto
status = auto


git update-index --assume-unchanged filename
might be better than
git rm --cached filename
when working with others, since it will stop updating but won't take it out of the repo. and you can always bring it back with
git update-index --no-assume-unchanged filename

Wednesday, July 31, 2013

A Pragmatic Look at Exception Specifications

http://www.gotw.ca/publications/mill22.htm

nice ref on how to handle exception specifications in c++, especially useful with inheritance, etc.

Thursday, July 18, 2013

articles

Geostatistical interpolation using copulas. András Bárdossy, Jing Li

Wednesday, July 17, 2013

Hypervolume-based Multiobjective Optimization: Theoretical Foundations and Practical Implications

Anne Auger, Johannes Bader, Dimo Brockhoff, Eckart Zitzler

http://hal.archives-ouvertes.fr/docs/00/63/89/89/PDF/tcspaper.pdf
see page 19 for good overview of test problems and their hype densities

networking resources

"how to work a room" susan roane
"the ultimate guide to successful networking" carol stone
general confidence building: "feel the fear and do it anyway" susan jeffers

tips
keep communicating
  2 or 3 times/year, before key events, things of interest to them, my personal news
  remind them: "story so far", for for lunch, etc., always do what you say you will
  contact them before you want something!
  thank you notes/emails
  refer to previous contact, re-introduce yourself

building credibility
http://changingminds.org/explanations/theories/credibility.htm

jobhunting, but also more generally:
http://www.realworldmagazine.com/networking-its-one-of-the-most-efficient-job-hunting-strategies-but-also-one-of-the-most-misunderstood/

Wednesday, June 26, 2013

tastytrade.com

nice video podcast with a number of shows that discuss practical aspects of trading.

Monday, June 24, 2013

latex, tex package install

figured out a good way to install tex packages for a non root user.

grab what you need from a tds.zip file from here:
http://www.ctan.org/tex-archive/install/macros/latex/contrib

then unzip it in, eg, ~/texmf/
then i think you need to set an env var like TEXMF or something, and maybe run texhash or something, or maybe you don't need to do any of that (should have posted this right after i did it!) but at least i know this is the file type to use and where to get them.

this might help:
http://tex.stackexchange.com/questions/30307/how-to-install-latex-zip-package-from-ctan-using-texhash-on-a-nix-system

ipad for writing equations, etc

possibly handy info for drawing, math, latex, etc on an ipad:

http://math.stackexchange.com/questions/193072/tablet-for-reading-textbooks-and-writing-math-by-hand/204393#204393

folding bluetooth keyboard

http://www.amazon.com/Perixx-PERIBOARD-805L-Bluetooth-Folding-Keyboard/dp/B00BU4OV50/ref=sr_1_1?ie=UTF8&qid=1371726428&sr=8-1&keywords=periboard+805l
http://www.amazon.com/Perixx-PERIBOARD-806-Bluetooth-Folding-Keyboard/dp/B00C6Y0ODG/ref=sr_1_3?s=pc&ie=UTF8&qid=1371715310&sr=1-3&keywords=folding+keyboard+-silicone
http://www.amazon.com/Universal-Folding-Bluetooth-Keyboard-Backlight/dp/B00AFRBF8O/ref=pd_sim_sbs_pc_5
http://www.amazon.com/QQ-Tech%C2%AE-Foldable-Wireless-Bluetooth-Keyboard/dp/B006XAGB1S/ref=pd_sim_sbs_pc_4
http://www.amazon.com/Matias-Bluetooth-Folding-Keyboard-MAC/dp/B001Q2COXC/ref=sr_1_6?s=pc&ie=UTF8&qid=1371715310&sr=1-6&keywords=folding+keyboard+-silicone
http://www.amazon.com/HDE%C2%AE-Wireless-Bluetooth-Folding-Keyboard/dp/B007FS994S/ref=pd_sim_sbs_pc_3
http://www.amazon.com/QQ-Tech%C2%AE-Foldable-Wireless-Bluetooth-Keyboard/dp/B006XAGB1S/ref=pd_sim_sbs_pc_2
http://www.cellphoneshop.net/foldable.html?CA_6C15C=1648900766
http://www.amazon.com/Wireless-Foldable-Keyboard-Laptops-Bluetooth/dp/B005F2VQ8I/ref=sr_1_5?s=pc&ie=UTF8&qid=1371715310&sr=1-5&keywords=folding+keyboard+-silicone
http://eshop.macsales.com/item/Matias/FK204F/?utm_source=amazon&utm_medium=shoppingengine&utm_campaign=productads
http://www.amazon.com/Stowaway-Ultra-Slim-Bluetooth-Blackberry-Handhelds/dp/B0002OKCXE/ref=sr_1_1?ie=UTF8&qid=1371720733&sr=8-1&keywords=thinkoutside+stowaway
http://www.amazon.com/s/ref=nb_sb_ss_i_2_7?url=search-alias%3Daps&field-keywords=matias+folding+keyboard&sprefix=matias+%2Caps%2C276&rh=i%3Aaps%2Ck%3Amatias+folding+keyboard
http://www.amazon.com/Wireless-Bluetooth-Keyboard-GT-I9500-Foldable/dp/B00B0L6TG2/ref=sr_1_10?s=pc&ie=UTF8&qid=1371715310&sr=1-10&keywords=folding+keyboard+-silicone

http://www.amazon.co.uk/BestDealUK-Foldable-Bluetooth-Wireless-Smartphone/dp/B00A1YUJHO/ref=sr_1_35?ie=UTF8&qid=1371718772&sr=8-35&keywords=folding+keyboard
http://www.amazon.co.uk/Perixx-PERIBOARD-805-Bluetooth-Foldable-Keyboard/dp/B0083JE5Q0/ref=sr_1_1?ie=UTF8&qid=1371718246&sr=8-1&keywords=folding+keyboard
http://www.amazon.co.uk/Perixx-PERIBOARD-805L-Bluetooth-Folding-Keyboard/dp/B00BU4OV50/ref=sr_1_23?ie=UTF8&qid=1371718567&sr=8-23&keywords=folding+keyboard
http://www.amazon.co.uk/Freedom-i-Connex-Folding-Bluetooth-Keyboard/dp/B003LO2FR8/ref=sr_1_4?ie=UTF8&qid=1371718246&sr=8-4&keywords=folding+keyboard
http://www.amazon.co.uk/Folding-Wireless-Bluetooth-Keyboard-Notebook/dp/B009ZP0Z24/ref=sr_1_8?ie=UTF8&qid=1371718246&sr=8-8&keywords=folding+keyboard
http://www.amazon.co.uk/Matias-Wireless-Folding-Keyboard-iPhone/dp/B005EG4880/ref=sr_1_41?ie=UTF8&qid=1371718772&sr=8-41&keywords=folding+keyboard

http://www.amazon.co.uk/DUSIEC-Bluetooth-Wireless-keyboard-Keyboard/dp/B005M36BNK/ref=sr_1_2?ie=UTF8&qid=1371718246&sr=8-2&keywords=folding+keyboard
http://uk.cellphoneshop.net/foldable.html?CA_6C15C=1810944409
http://www.amazon.co.uk/Bluetooth-Wireless-Portable-bluetooth-compatible/dp/B006RMJ3NA/ref=pd_sim_sbs_computers_3
http://www.amazon.co.uk/Logitech-Fold-Up-Keyboard-iPad/dp/B005UKH922/ref=sr_1_7?ie=UTF8&qid=1371718246&sr=8-7&keywords=folding+keyboard
http://www.amazon.co.uk/HDE%C2%AE-Wireless-Bluetooth-Folding-Keyboard/dp/B007FS994S/ref=sr_1_19?ie=UTF8&qid=1371718567&sr=8-19&keywords=folding+keyboard
http://www.amazon.co.uk/Lerway-Portable-Backlight-Bluetooth-Keyboard/dp/B00CAKPFVC/ref=sr_1_29?ie=UTF8&qid=1371718567&sr=8-29&keywords=folding+keyboard
http://www.amazon.co.uk/Skque-Foldable-Wireless-Bluetooth-Keyboard/dp/B008KEQ78C/ref=sr_1_67?ie=UTF8&qid=1371719265&sr=8-67&keywords=folding+keyboard
http://www.amazon.co.uk/Think-Outside-Bluetooth-Stowaway-Keyboard/dp/B0002OKCXE/ref=tag_dpp_lp_edpp_ttl_in

Friday, June 14, 2013

mutual information and signal processing

some interesting articles:

Massoud Babaie-Zadeh, Christian Jutten, and Kambiz Nayebi. Differential of the Mutual Information
Kenneth E. Hild, II, Deniz Erdogmus, and José Príncipe. Blind Source Separation Using Renyi’s Mutual Information
George Atia and Venkatesh Saligrama. A Mutual Information Characterization for Sparse Signal Processing
Liam Paninski. Estimation of Entropy andMutual Information (recommended by others)
Janett Walters-Williams and Yan Li. Estimation of Mutual Information: A Survey


Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy (2005) by Hanchuan Peng , Fuhui Long , Chris Ding
lots of good definitions, results comparison

http://vserver1.cscs.lsa.umich.edu/~crshalizi/notabene/information-theory.html
lots of information theory links


Modelling time series using information theory
L Diambra, A Plastino
L. Zunino, M. C. SorianoI. Fischer, O. A. Rosso, and C. R. Mirasso. Permutation-information-theory approach to unveil delay dynamics from time-series analysis

Monday, June 3, 2013

running away...

how many times?

FYF!

i want to count how many views i get...

Friday, May 31, 2013

writing

need to do content generation writing in a separate mode from editing. 'write with the door closed. edit with the door open.' it's too easy to overthink things if i'm trying to argue with a room full of people while i'm putting down my first draft. just get it all out and pretend no one will read it when i write the first draft. then come back to edit while imagining explaining it to someone else.

have faith in your own editing. yes, the first draft will be awful. but, yes, it get better with more work.

liverebel

http://zeroturnaround.com/
automatic continuous deployment, with a plugin hook available for jenkins.

http://www.youtube.com/watch?v=3HI7mv_791k

amazon aws/ec2, picloud

looks like amazon gives you up to a year to try out some free cpu time on their cloud computing nodes.
http://aws.amazon.com/free/terms/

also, their spot instances allow you to bid for time, rather than paying the fixed on-demand rates. looks like the discount is significant, if you can handle the unpredictability. nice example of running a jenkins build slave on spot, too. also refs princeton consultants and their optispotter, which helps smallish ($50mil) hedge funds find hft opportunities.
http://aws.amazon.com/ec2/spot-instances/
http://www.youtube.com/watch?v=-vAAuTs9iu4

picloud still looks like a good way to get started. can do fractional hours, and prices are comparable to ec2 on-demand. they allow you to create an environment on a virtual ubuntu, so you can install whatever you need as if you had a local filesystem.
http://aws.typepad.com/aws/2012/12/picloud-and-princeton-consultants-win-the-first-amazon-ec2-spotathon.html

Thursday, May 30, 2013

hotel wifi rip-offs

wow, cool idea i need to try next time i have to: socks proxy through ssh tunnel that persists after your web login wifi connection is terminated.

http://stockrt.github.io/p/navegando-com-seguran%C3%A7a-e-sem-tarifa%C3%A7%C3%A3o-em-caf%C3%A9s-e-hot%C3%A9is/

Wednesday, May 29, 2013

learning git

sooner or later (probably later) i need to learn something about git. here are some links that i don't have time to read right now:

http://www.youtube.com/watch?v=ZDR433b0HJY
http://betterexplained.com/articles/aha-moments-when-learning-git/
http://kentnguyen.com/development/visualized-git-practices-for-team/
http://haacked.com/archive/2012/03/15/visualize-git-with-seegit.aspx
http://danlynn.org/home/2012/4/30/git-branch-visualization-on-command-line.html
http://think-like-a-git.net/sections/graphs-and-git.html
https://github.com/esc/git-big-picture
http://blog.kfish.org/2010/04/git-lola.html
http://www.syntevo.com/smartgithg/index.html
http://www.sourcetreeapp.com/

git vs. mercurial
http://stackoverflow.com/questions/35837/what-is-the-difference-between-mercurial-and-git
http://mercurial.selenic.com/wiki/GitConcepts
http://jhw.dreamwidth.org/1868.html
http://www.atlassian.com/dvcs/overview/dvcs-options-git-or-mercurial
http://www.wikivs.com/wiki/Git_vs_Mercurial
http://www.sqlalchemy.org/blog/2013/05/25/sqlalchemy-migrated-to-git/

apparently bitbucket hosts free private repos, and the hg-git plugin for mercurial allows you to use a hg client on a git repo.
http://hg-git.github.io/

hg is in python, which is very nice for extensibility, but dulwich is a pure python implementation of git. can't tell for sure, though it does look like it's a little low level.
https://github.com/jelmer/dulwich

Monday, May 27, 2013

delta debugging, automated debugging

i was thinking about unit testing and delta debugging, or differential debugging?, and i thought python's introspection tools could make it possible (and not too hard) to do this automatically. i could make a tool that takes 2 adjacent versions from a repo, runs a test synchronously on both of them (that fails on only one of them), and drops into 2 debuggers at the point where a callable is about to return with 2 different values. that would make it really easy to debug the failure, right at the point where it happens. i could make it smart enough to distinguish between private, internal interfaces (starting with an underscore) which are allowed to change functionality between revisions and external interfaces which should be invariant (except possibly on marked, backwards-compatibility-breaking revisions). i could speed things up by only trapping callables that contain modified lines in the revision diff and only running the tests with coverage on those lines. i could even do interesting things like adding in only some of the diff blocks to preserve correctness and walk backward or forward through revisions -- truly automated debugging! long term vision: maybe combining selective modification with an auto-commit-on-branch with each file save could allow me to tinker and spew out code in an entirely creative mode. the computer then assembles that into something correct and only asks me to clarify when it gets too confused. and all those microrevisions could provide a cleanroom-like estimate of bug introduction and elimination rates, as well as functionality addition rate based on number of paths through external interfaces.

that could also mean that i wouldn't need to write as many unit tests, since the same tool could catch calls and returns to any callable and compare results when they return. all i need to do is have my top level/integration tests and enough unit tests to fill in the coverage gaps. i could detect any changes on the internal interfaces if i wanted to, without necessarily breaking the build each time. any 'unit test' that was derived from the top level would be updated automatically. and if my coverage/diff analysis is reliable enough, i could speed up my continuous integration build a lot without losing any information. (i would have to be very careful about this, though, and track all kinds of external dependencies, etc.)

all of this would be a huge productivity boost since debugging is one of the biggest and most unpredictable time sinks i have. having unit tests is important and writing them often helps the development process, but updating lots of test results due to a small change in a core bit of code is a real pain. also, i don't like choosing between long-running tests or tag-along intermediate result pickle files.

came across some sites that should be useful for debugging, especially automatic debugging:

http://root.cern.ch/drupal/content/cint
http://root.cern.ch/drupal/content/cling
http://root.cern.ch/drupal/content/reflex
http://clang.llvm.org

cffi is the foreign function interface for python, used by pypy.
http://cffi.readthedocs.org/en/release-0.6/

some work people have done on delta debugging
http://delta.tigris.org
http://www.st.cs.uni-saarland.de/dd/ddusage.php3

useful to auto debug python
git bisect run http://lwn.net/Articles/317154
inspect module
http://pymotw.com/2/trace/index.html
http://pymotw.com/2/profile/index.html

also i should use vbench in my continuous integration. i think it was written by the same guy who started pandas. maybe there's a way to get it or multi-mechanize.py to output jmeter format so jenkins/hudson can read and plot it over time.

Thursday, May 23, 2013

nadex.com

nadex exchange based in chicago. binary and bull spread options trading, legal for u.s. residents.

Friday, May 17, 2013

spyder and matplotlib

been getting some angry red and blue barf in my spyder console when a python script exits with undisplayed matplotlib figures hanging around. for example,

Traceback (most recent call last):
File "C:\Python27\lib\site-packages\spyderlib\mpl_patch.py", line 79, in
# lambda: self._widgetclosed())
File "C:\Python27\lib\site-packages\matplotlib\backends\backend_qt4.py", line 340, in _widgetclosed
if self.window._destroying: return
AttributeError: FigureManagerQT instance has no attribute 'window'
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\spyderlib\mpl_patch.py", line 79, in
# lambda: self._widgetclosed())
File "C:\Python27\lib\site-packages\matplotlib\backends\backend_qt4.py", line 340, in _widgetclosed
if self.window._destroying: return
AttributeError: FigureManagerQT instance has no attribute 'window'


so i changed lines 78-79 in spyderlib.mpl_patch from

QObject.connect(self.window, SIGNAL('destroyed()'),
lambda: self._widgetclosed())

to

QObject.connect(self.window, SIGNAL('destroyed()'),
lambda: self._widgetclosed() if hasattr(self, 'window') else None)

and now it dies silently, just the way i like it.

Sunday, April 21, 2013

travis-ci

https://travis-ci.org/

very cool public build server for open source projects. you can set it up to run the test suite on commits, and it takes its config out of the repo itself.

there's a travis pro for continuous integration on private repos, but it looks like it's not live yet. (?)

Saturday, April 20, 2013

pythonanywhere

https://www.pythonanywhere.com/

hosted python environments, for a webapp for your site, an on-the-go interpreter session, or... whatever else you can think of. i think they also give you bash and latex, maybe other goodies. it does have numpy, scipy, sympy, matplotlib, cython. it doen't have enthought, numba. the (very impressive) list of packages it has is here:
https://www.pythonanywhere.com/batteries_included/

free for basic account, $5/mo for 'hacker' account, and 12$/mo for more serious hosting. sounds very reasonable, considering you get access to their hardware and no wasting time on sysadmin maintenance.

demo console here:
http://blog.pythonanywhere.com/4/

run selenium test example here:
http://blog.pythonanywhere.com/3/

logging

http://vimeo.com/couchmode/pydata/videos/sort:date/63377197

lots of good advice on writing reliable, maintainable infrastructure code from a guy who now works at disney supporting social media games. in particular, some points on logging:

appropriate levels:
LOG for start and end of each major section of work
WARN for input data errors and environment errors
ERROR for code problems
DIE for unrecoverable errors, after a short delay

say what happened, what was expected, and what to do/whom to contact

vagrant

http://www.vagrantup.com/

vagrant vm looks like very interesting open source (MIT) tool for setting up clean, uniform development environments. looks like it's written in ruby.

evernote

http://evernote.com/

here's something i need to check out when i have more time: evernote. looks like you can put notes, copies of web pages, scanned documents, audio and photos, etc. in one place and have it synced across your devices. recommended by one of the developers of google glass.

lots of video tutorials with more info:
https://evernote.com/video/

Wednesday, April 17, 2013

rapidquant

http://www.rapidquant.com/

new startup by some former quants and developers of pandas. looks like it might have some overlap with quantopian, but i'm not sure what their business model is. part open source, part proprietary.

jake vanderplas blog

http://jakevdp.github.io/

blog with lots of interesting python examples and demos. not just code, though; the guy seems plugged in to big-picture developments.

Saturday, April 13, 2013

python presentation videos

http://vimeo.com/pydata/videos/page:1/sort:date
http://pyvideo.org/category

no need to attend pycon or pydata, or no need to take notes if you do.

time management quotes

don't lose interest when it's easy; don't lose hope when it's hard. just the right amount of stress and difficulty is what makes us improve.

http://www.youtube.com/watch?v=oTugjssqOT0
randy pausch

most important point: in 7 habits 4 task quadrants, do 1. important/urgent, 2. important/not urgent, 3. unimportant/urgent, 4. unimportant/not urgent. that way, 2s don't become 1s and you don't get stressed.
many people do unimportant/urgent second, but that's wrong.

being successful doesn't make you manage your time well.
managing your time well makes you successful.

(me) time are money are related; so are time and stress.

don't do the wrong things beautifully; do the right things adequately.

why am i doing this? what will happen if i don't?

good judgement comes from experience.
experience comes from bad judgement.
there are no shortcuts to good judgement.

you can't change the plan unless you have one.

do the ugliest thing first.
if you have to eat a frog, don't look at it first.
if you have to eat 3 frogs, start with the big one.

only have 1 document on your desk at a time; that way you fight them one-on-one
touch each piece of paper (or email) only once.

use a speaker phone when you're, and tell them politely how long you've been on hold.
stand while talking on the phone.
start with giving an agenda for the call.
have something on your desk that you'd like to get back to.
'i'd love to keep talking but i have ___ waiting.'
group your calls and make them right before lunch and quitting time. then they will have something to get back to.
use a phone headset.

keep kleenex and thank you notes on your desk.
people will remember you if you send them a thank you note.

keep a paper recycle bin near your desk that is emptied infrequently. then you will not hesitate to throw it out since you have a grace period to get it back.

make your office comfortable for you, and optionally comfortable for others.
keep folding chairs to the side.

you don't 'find' time for important things. you 'make' time by neglecting something else. consider opportunity cost and learn to say gentle 'no's.

gentle no:
'i'm strapped, but i don't want you to be in a bind. i'll be your fallback if you can't find someone else.' then you will see how that person treats you, based on whether or not they keep looking.

find your creative time. defend it ruthlessly and spend it alone.
find your dead time. schedule meetings, phone calls, and exercise (when you don't need to be at your mental best.)

an interruption of 6-9 minutes takes 4-5 minutes to recover from.
when people interrupt you, say 'i'm in the middle of something' or 'i only have 5 minutes'. then they will want to handle it quickly and you can leave gracefully later.

if someone won't leave your office, go to the door, complement and thank them, and shake their hand. if they still won't leave, leave.

time journal
monitor yourself in 15min intervals, update every 30min, for 3days to 2weeks.
what am i doing that doesn't need to be done, what can i delegate to someone else, where am i wasting time, and where am i wasting other people's time?
time management is collaborative, not selfish; help other people be more efficient
remember the point is not to do more, it's to go home on time.

effective vs. efficient
do what will work better, even if it takes more time.

we don't procrastinate from laziness. identify why you are not enthusiastic. are you afraid of failure? are you avoiding the decision to prioritize?
doing things at the last minute is much more expensive than just before the last minute.
make up a fake deadline and pretend it's real.
sometimes you just have to ask and wonderful things happen.

delegation
don't treat it like dumping.
give responsibility and authority.
keep the ugliest part of the job yourself, so you and they know you are not dumping.
treat your people well.
you cannot be vague if you want something to get done. specific thing to do, specific date _and_ time. specific penalty or reward (for them).
challenge people; they usually yearn for an opportunity.
communication has to be clear. get it in writing, even if you've spoken.
give objectives, not procedures. let them surprise you with their solutions. you want to get people smarter than you, so let them prove it.
let them know which tasks are most important.
beware upward delegation; don't take it back.
praise and thank people. it's the best reward.
don't email more than one person to get something done, unless you are ccing their boss.
it's okay to nag if they haven't responded within 48 hours.

meetings
book: one minute manager
don't go to a meeting without an agenda.
don't let people be half-there. it's either worth the time or not.
only rarely have meetings > 1 hour.
1 minute for the minutes: who is responsible for what by when and email to everybody.

use technology that changes the workflow.

manage from beneath
when do we next meet?
what should i have done by then?
where else can i get help?
your boss wants results, not an excuse.

time off
2 options: alternate contact, or contact me when i get back. that way you don't have work piling up when you're gone.

kill your television.
turn money into time, especially with young children.
you always have time to sleep.

never break a promise, but renegotiate if necessary.
if you don't have time to do it right, you don't have time to do it wrong.
most things are pass/fail.
get feedback sleuths, and ask in confidence. the most valuable thing in the world is someone will tell you the truth about what you're doing right and wrong.

Wednesday, April 10, 2013

mobiunpack.py

http://wiki.mobileread.com/wiki/Mobi_unpack

nice little script for yanking the epub out of an amazon mobi files: azw, tpz, azw4, or kf8.

exporting word doc to html

http://www.mobileread.com/forums/showthread.php?t=142530

microsoft word puts a lot of junk into html documents when you export them from word. this macro might help clean it up.

aveotsd

i did some looking around for a tongue-retaining device to reduce the incidence of sleep apnea events. looks like the best one is from aveosleep. clinical trial info is here:
http://www.aveosleep.co.uk/clinical-trials.php

journal article here:
http://www.aveosleep.co.uk/documents/Tongue%20Stabilizing%20Device%20-%20Pilot%20Study.pdf

quandl

http://www.quandl.com/

amazing amount of economic indicators, financial, and social data for free, in convenient formats. if i need any time series in the future, i'll check to make sure it's not there already.

quantopian already has hooks to grab data from there.

Friday, April 5, 2013

quantopian and zipline

quantopian allows you to write algorithmic trading (black box trading algorithms) in python and then backtest them easily with their data. really cool. i need to try this out.

i think i heard them say they have 1 min data for 10 years for all equities. once they have their features in place, they will add futures and then options.

zipline is their backtest system, which they have opensourced. eventually they will support real trading with their algo system.

https://www.quantopian.com/

When cointegration of a pair breaks down

interesting article with more info on multivariate cointagration and pairs trading (or other multiples trading). comments contain lots of other links.
http://epchan.blogspot.ca/2011/06/when-cointegration-of-pair-breaks-down.html

get important things done

completion of an unimportant task is not a success. prevent or punish it as the failure that it is.

there is a world of difference between staying busy and getting things done.

why is working on a task so much more comfortable than defining and prioritizing tasks?

when writing or presenting research: "the truth, the whole truth, and nothing but the truth." the truth, only the important parts that guide the audience through the present narrative, and including simplifications that facilitate comprehension.

"you can waste a lot of time trying to do things quickly." i've seen this happen, but why does it happen? maybe because i try to build complex processes that will run very quickly once they finally start. but the facts on the ground change before my system is in place, and i can't adapt my process or the goal looses relevance and importance by the time i'm ready to use it. better to do something less efficient but starting sooner.

your home is not your head. remember to interact.

ipython parallel and acyclic graphs

heard an interesting tidbit from a talk by brian granger. apparently the ipython parallel kernel has the ability to take acyclic graph dependencies and intelligently distribute the computation. i need to look into that.

python garbage collection

read a couple of interesting articles on how the garbage collector works in python. this post points out performance issues when you create lots of new objects without deleting any:

http://dsvensson.wordpress.com/2010/07/23/the-garbage-garbage-collector-of-python/

pymotw is very good, very informative:

http://pymotw.com/2/gc/

thomson reuters qa direct

continuum.io is doing some interesting things these days. wakari is something i don't really understand yet, but things like blaze, numba, and anaconda look really neat. i need to check them out for my next project.

an interesting example of wakari usage is in a data service from thomson reuters. they have some demo data available if you sign up.

http://continuum.io/blog/wakari-and-big-finance

uncertainty in global warming data

here's an interesting paper on some of the uncertainty present in temperature data and in results that use it. similar issues of measurement error and computational uncertainty come up everywhere you look.

https://www.ncdc.noaa.gov/oa/climate/research/ersst/papers/SEA.temps08.pdf

Friday, March 22, 2013

work and planning quotes

don't try to learn too much at once -- you'll get overwhelmed and discouraged.

you are paid to perform, so make sure you face the audience.

don't do any work that no one else will find out about.

--duhctaep

Wednesday, March 13, 2013

city a.m.

city a.m. is _the_ free financial newspaper read by commuters to the city of london. it is also available free online, at cityam.com . the full paper layout (with the exception of the market data page full of numbers) is available as a pdf, and each section of the paper has its own rss feed.

news, the-forum, the-capitalist (maybe), business-features, and wealth-management feeds are worth reading. the main feed has is probably also worth it for the news. and there are a lot of articles i can't find in the feeds, maybe because they are in the main feed and stuff falls off after just a couple of hours. but news has the first couple of pages from the print edition, and the-forum has the opinion page. a lot of the wealth management articles are missing from wealth-management, though it does have articles from the 'city dashboard' market report.

if and when i loose access to the financial times, this would make a nice replacement as a uk finance news source.

Thursday, February 28, 2013

investor sentiment metrics

interesting quick overview of ways to measure investor sentiment as a market indicator. following is a snippet with 6 examples.

http://www.ft.com/cms/s/0/9200dbf4-7b6f-11e2-8eed-00144feabdc0.html#axzz2MBYXC97A

Panic or euphoria: six ways to measure the market
● AAII bull-bear ratio

Nothing illustrates the flakiness of some sentiment measures more than the weekly survey by the American Association of Individual Investors. Widely used as a guide to the proportion of bulls and bears in the market, it involves sometimes fewer than 100 self-selected investors reporting their mood. Yet its long history and broad accuracy – super-bullish at market peaks, uber-bearish when the market bottoms out – has made it a favourite.
● Investors Intelligence

There are some very shrewd writers of investment newsletters (Jim Grant of Grant’s Interest Rate Observer is an example). Taken as a whole, newsletters capture the feeling in the market. Investors Intelligence categorises each newsletter as bullish or bearish; the spread between the two shows when writers are becoming emotionally attached to the market.
● Futures market positioning

Many investors try to copy what the “smart money” is up to. They would do better to watch it as a contrary indicator, preparing to do the opposite. Positions in S&P 500 derivatives (not the e-mini) offer a handy guide to how sophisticated traders feel about the market. The net positioning shown in the chart can be used to test whether they are too optimistic or pessimistic.

● Equity put/call ratio

The options market offers investors the chance to buy insurance for their portfolios or speculate on future gains. The ratio between put options (which make money if the market falls) and calls (which profit from rising markets) is an immediately-available guide to how relatively sophisticated investors feel. When it becomes very high, investors are extremely cautious, when very low, they feel no need for insurance.
● Combined measures

Lots of consultancies and investment banks produce combined measures, all constructed somewhat differently. Shown here is one, from Absolute Strategy Research, which combines the different investor surveys into a single poll of polls. At the moment it suggests investors feel dangerously sanguine.
● Investment bank equity weighting

The model contrarian would only invest in things that made them feel physically sick and only sell when convinced they should buy more. Merrill Lynch strategists measure recommendations from the rest of Wall Street’s strategists, and suggest doing the opposite. At the moment the Street’s strategists have a low weighting in equities. Merrill sees this as a bullish signal for shares.

Saturday, February 23, 2013

oculus vr

http://www.oculusvr.com/
i want one.
$300 for a dev kit. maybe i'll wait for next gen. probly cheaper and better. already looks pretty good, and pretty cheap.

Tuesday, January 29, 2013

python file encryption

just ripped off a stackexchange post to make a quick file encrypt/decrypt tool.

from pdb import set_trace as dbg

from Crypto.Cipher import AES
from Crypto import Random
import base64
import hashlib
BS = 16 # couldn't get it to decrypt properly with 32
pad = lambda s: s + (BS - len(s) % BS) * chr(BS - len(s) % BS)
unpad = lambda s : s[0:-ord(s[-1])]
def passwdToKey(passwd):
return hashlib.sha256(passwd).digest()[:BS]

class AESCipher:
def __init__( self, key ):
self.key = key

def encrypt( self, raw ):
raw = pad(raw)
iv = Random.new().read( AES.block_size )
cipher = AES.new( self.key, AES.MODE_CBC, iv )
return base64.b64encode( iv + cipher.encrypt( raw ) )

def decrypt( self, enc ):
enc = base64.b64decode(enc)
iv = enc[:BS]
cipher = AES.new(self.key, AES.MODE_CBC, iv )
return unpad(cipher.decrypt( enc[BS:] ))

def executeAction(action, fileIn, fileOut, pw):
aesc = AESCipher(passwdToKey(pw))
if action == 'encrypt':
unencrypted = file(fileIn, 'rb').read()
encrypted = aesc.encrypt(unencrypted)
# verify encryption
assert aesc.decrypt(encrypted) == unencrypted, 'invalid encryption'
file(fileOut, 'wb').write(encrypted)
elif action == 'decrypt':
file(fileOut, 'wb').write(aesc.decrypt(file(fileIn, 'rb').read()))
else:
raise Exception('Unknown command: ' + action)
def main():
import sys
args = sys.argv[1:]
action, fileIn, fileOut, pw = args[-4:]
executeAction(action, fileIn, fileOut, pw)

if __name__ == '__main__':
main()

Thursday, January 10, 2013

scan to pdf

some scanners use jpgs for each page when scanning to pdf. i've found it works better to scan to a mtiff and then covert that with
tiff2pdf -o output.pdf input.tif

i got pretty good results in terms of size and readability with 300dpi, medium quality, black and white.

i've tried encrypting the pdf, and it all works, except that on the kindle it won't let me do any annotations on an encrypted file, even when I try to allow it explicitly:
pdftk in.pdf output out.pdf user_pw foo owner_pw bar allow AllFeatures

all annotations, etc, work fine on the desktop so i think it's an issue with the kindle acroread. at least i can work with it after decrypting, kindle or elsewhere
pdftk encrypted.pdf input_pw foobar output decrypted.pdf

Monday, January 7, 2013

quant finance links

http://www.sierrachart.com/index.php?l=doc/developers.php
came across sierrachart while trying to research restrictions on the use of google finance data (based on http://www.google.com/intl/en/googlefinance/disclaimer/?ei=v2zjUPjvIeOZwQPGhAE ). sierrachart looks like it's written by a loose coalition of general nerds often for their own use, and sold for a fee to non-nerds. they advertise the google finance data importer openly, so i guess google doesn't mind? written in c++, so maybe easy to integrate with other quant libs?

http://www.derivitec.com/
started by a couple of equity derivative quants about a year ago, they sell risk models that go into spreadsheets and compute on the cloud with microsoft azure. head dude wrote a book ( http://www.amazon.com/The-Value-Uncertainty-Dealing-Derivatives/dp/1848167725/ref=sr_1_1?ie=UTF8&qid=1339681713&sr=8-1 ) (not out yet) about how to do risk quantification, including practical considerations. one of the few refs i've seen to model error risk and model parameter uncertainty.

i wanted to see if google would allow people to share the results of computation that used their finance data. the disclaimer statement seems ridiculously restrictive (can't even 'download or save' it? erm, then why is your server giving it to me?) for-fee data services like thomson-reuters ( http://thomsonreuters.com/products_services/financial/financial_products/a-z/datascope_select/#tab1 ) and xignite ( http://www.xignite.com/Product/XigniteBondsRealTime/ ) are very pricey. www.kibot.com is cheaper, but still hundreds of $ for each data type. sierrachart and ninjatrader both advertise their capability to download and extract data from finance.google.com and both show up on the first page of a google search. hmmm.

incidentally, i found (maybe re-found) possibly useful indicator historical data available for free from the world bank (eg, http://data.worldbank.org/data-catalog/world-development-indicators?cid=GPD_WDI ).

also, openquant might be interesting:
http://www.smartquant.com/openquant.php
okay, apparently that one is a few hundred bucks/month. this one is foss:
http://code.google.com/p/openquant/

otc derivatives data

the cftc is now in charge of collecting data from off-exchange trades of swaps, etc., thanks to dodd-frank. right now they are doing cdo and rates derivative; equity, forex, and commodities will come later.

i can't find any data on the cftc site except highly aggregated volume data. apparently anyone can apply to become a repository, they can meet the requirements. so far, the dtcc is the best source i can find. they offer rss and csv downloads:
https://rtdata.dtcc.com/gtr/dashboard.do

this could get very interesting, and could change the way investment banks run their business for good.

Tuesday, January 1, 2013

amazon book indie publishing links

salesrankexpress sends search requests over to http://sre.novelrank.com/
http://www.salesrankexpress.com/
looks promising. we'll see how well it works.

http://www.novelrank.com/
see above.

http://rankforest.com/
data limited to last 30 days unless you upgrade.

TitleZ is currently free for beta testing. might charge later.
http://www.titlez.com/
looks like it has the best features, but it's not working atm and the docs are really old (2006!).

i looked at tictap.com but wasn't very impressed.

info on correlating amazon sales rank with book sales per day:
http://www.fonerbooks.com/surfing.htm

he mentions using salesrankexpress.com to track ranking.

amazon's affiliate program has some data available through their api:
https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html