When cointegration of a pair breaks down

interesting article with more info on multivariate cointagration and pairs trading (or other multiples trading). comments contain lots of other links.
http://epchan.blogspot.ca/2011/06/when-cointegration-of-pair-breaks-down.html

investor sentiment metrics

interesting quick overview of ways to measure investor sentiment as a market indicator. following is a snippet with 6 examples.

http://www.ft.com/cms/s/0/9200dbf4-7b6f-11e2-8eed-00144feabdc0.html#axzz2MBYXC97A

Panic or euphoria: six ways to measure the market
● AAII bull-bear ratio

Nothing illustrates the flakiness of some sentiment measures more than the weekly survey by the American Association of Individual Investors. Widely used as a guide to the proportion of bulls and bears in the market, it involves sometimes fewer than 100 self-selected investors reporting their mood. Yet its long history and broad accuracy – super-bullish at market peaks, uber-bearish when the market bottoms out – has made it a favourite.
● Investors Intelligence

There are some very shrewd writers of investment newsletters (Jim Grant of Grant’s Interest Rate Observer is an example). Taken as a whole, newsletters capture the feeling in the market. Investors Intelligence categorises each newsletter as bullish or bearish; the spread between the two shows when writers are becoming emotionally attached to the market.
● Futures market positioning

Many investors try to copy what the “smart money” is up to. They would do better to watch it as a contrary indicator, preparing to do the opposite. Positions in S&P 500 derivatives (not the e-mini) offer a handy guide to how sophisticated traders feel about the market. The net positioning shown in the chart can be used to test whether they are too optimistic or pessimistic.

● Equity put/call ratio

The options market offers investors the chance to buy insurance for their portfolios or speculate on future gains. The ratio between put options (which make money if the market falls) and calls (which profit from rising markets) is an immediately-available guide to how relatively sophisticated investors feel. When it becomes very high, investors are extremely cautious, when very low, they feel no need for insurance.
● Combined measures

Lots of consultancies and investment banks produce combined measures, all constructed somewhat differently. Shown here is one, from Absolute Strategy Research, which combines the different investor surveys into a single poll of polls. At the moment it suggests investors feel dangerously sanguine.
● Investment bank equity weighting

The model contrarian would only invest in things that made them feel physically sick and only sell when convinced they should buy more. Merrill Lynch strategists measure recommendations from the rest of Wall Street’s strategists, and suggest doing the opposite. At the moment the Street’s strategists have a low weighting in equities. Merrill sees this as a bullish signal for shares.

factor shares

ib is offering its customers commision-free trades on factorshares spread etfs. this looks really interesting: diy hedge fund with asset classes. trying to figure out how the funds operate, based on the holdings info from their website... each of the 5 has basically $3m in treasuries, $2m in cash, and a thousand in a treasury fund. each one then takes a position in nearest future contract (long and short) for the two spread assets, for 2x the nav/share * 100k shares/unit * 2 units (currently). the treasuries are probably to offset the time discount on the futures contracts, and that and the cash are for margin, obviously. the fund is probably for a little bit of liquidity during the daily rebalance. i'm guessing the nav comes from the $5m risk free + net market value of the futures. the nav/share doesn't exactly match the price, probably given the constraints of the contract sizes and maybe the lower liquidity of the etf at the moment. the futures holdings are updated daily, to restore dollar neutrality (same forward contract dollar amounts). for the current number of shares and risk-free holdings, the funds basically started at $25/share. the s&p e minis are apparently for lots of 50. the 30yr tbill futures are for lots of 100. the 5 etfs are s&p/tbill, tbill/s&p, s&p/usd, oil/s&p, gold/s&p, where each is bull/bear. 2x leverage on each leg, for a 4x total leverage (but still just 2x on the spread). right now the volume is pretty light, around 10k/day, since they just lauched a few weeks ago. i wonder if there is any arbitrage opportunity for these, knowing their methodology. especially for s&p/tbill vs. tbill/s&p. does the rebalance accumulate anything based on independent underlying price movements? no, because all the money gained from selling an advancing future is put into the one that declined, and the same thing will happen at the end of the next day. but if i rebalance between these, it will. i'm just not sure if it's any better than just rebalancing among 2x single asset funds, unless you think they're anticorrelated instead of just uncorrelated. maybe it has the advantage of not needing to rebalance with cash, like single funds would. one thing maybe i can try is to arb the s&p/tbill vs. tbill/s&p like this: right now fse has 139 emini and -71 tbill, and fsa has -177 emini and 91 tbill. closing for fse was $22.57 and fsa was 28.18. if i could buy 177/139 fse it would cost 28.74, or 91/71 would cost 28.92. so if i had some of each, i could have sold fse and bought fsa near the end of the day, eg sell 9 fse for 203.13 and buy 7 fsa for 197.26. that would be selling a share of 1251 emini, -639 tbill and buying a share of -1239, 637. so the net would be 42, -2. nav calculation time is based on the first of the futures contracts to settle: s&b/tbond 3pm (ET) tbond/s&p 3pm s&p/usd 3pm oil/s&p 2:30pm gold/s&p 1:30pm the nav must depend on the price of the futures and what exposure they can be rebalanced to. futures prices might not be as easy to come by, so maybe i could compare to other leveraged etfs like sso, sds, dgl (or iau/gld (not leveraged) or dgp/dzz (monthly, not daily)), dbo (or dig?, uso, oil), tlt (not leveraged), tbt (or pst? no), udn (not leveraged) (or uup, not leveraged and bull instead of bear but much higher volume) (these only use dx contracts, not front month). this wouldn't necessarily be perfect since nav can deviate from price, but it should be close for the heavily traded ones. here's a good ref list for leveraged etfs

financial times rss

FT offers a dizzying array of separate rss feeds for its articles, but i don't think all of them end up in the print newspaper. i'd like to get the articles that the editors have deemed most important, without having to pay the £2/paper to get them. so... i bought a copy and tried to find where the articles in the paper show up in rss. and here they are, in roughly the order of importance given at the end of 'all you need to know about the city': lex column lex main 2nd (companies and markets) section companies: companies main, uk (companies|uk, though there's a lot that's not in the paper here), uk smaller companies (companies|uk) markets: markets main (markets section, last page) 1st section management: management main (business life) comment: comment main, opinion, analysis (comment & analysis) world: world main, europe, asia-pacific, africa, us, uk business, uk economy looks like google reader might be a convenient way to combine all the streams into one and maybe even keep track of which have been read already. and i was hoping the mobile version (http://www.google.com/reader/m) would simplify the scraping to whittle it down around the body text. but it doesn't always work; sometimes it only takes the first paragraph or two and scraps the rest, probably because it catches a break before a table or image. so i think i'll have to navigate through to the original page and scrape from there. EDIT: i guess the people at ft are smart enough to make it easy for me. they post links for their print edition (and us, europe, middle east, and asia editions). only thing i didn't find on that page was the 'money' special pullout from the weekend edition. i think most of those articles were in the 'personal finance' section of the website. and i discovered that each html page for a section has an rss icon link in the upper right, so it's easy to snag stuff once i know where they are on the website. going through the sections, i found they went roughly in order with pretty close, though not exact, correspondence to the articles in print. here are the sections on the website and the page numbers of articles listed under them, to give you an idea of the density: front page: 1,1 must read national news: 2,2,2,3,3,4,4,4,4,4,4 skip 1/2 to 2/3 world: 5,5,5,5,6,6,6,6,6,6,6,6,7,7,7,7,8,8,8 good read comment & analysis: ,9,10,10,10,10,11,11,11,11 skip some of these, though the latter ones are really good letters: 10,10,10,10,10 skip all of these! life & arts: (pull-out) 1,2,2,2,2,3,4,4,5,5,5,6,7,6,6,7,8,9,19,10,10,11,11,11,12,12,13,13,14,14,20,20,17,17,17,17,17,17,17,17,17,17,17 a lot of things skipped between 14 ad 17, but i would skip this whole section. ft magazine: (pull-out) 15,54,12,7,10,8,44,44,47,43,43,46,46,52,53,51,50,49,48 i would skip almost all of this house & home: (pull-out) 1,2,2,3,6,7,7,8 i would skip practically all of this section lex: 24,24,24,24 must read companies: 12,12,?,13,13,12,14,14,14,14,14,15,15,15,14,15,16,16,16,16,?,17,17,?,17 good read, especially toward the end markets: 22,22,?,23,23,23,23,24 must read

interactive brokers news pages

ib has news pages on interest rates in various countries, fx markets (including an rss feed), and futures and options (also with a commentary rss feed on a few individual tickers). the futures and options page includes a summary explanation at the top that explains what things like implied vs. historical volatility and futures arbitrage can tell you about what derivatives traders believe will happen in the market short-term.

python data storage

after a little bit of optimization, i'm finding the bottleneck now is reading in the data. i think i've found about all the ways to speed up cPickle (most recent protocol, Pickler().fast = True) and the next step will be to a real database. i'm not sure the python builtins will buy me much, though, and i think if i'm going to have to install something it might as well be pytables. pytables is an interface layer on top of hdf5, so it's probably best for large volumes of numerical data. it only requires hdf5 (which built without problems: configure && make install) and numpy. i had to set HDF5_DIR and add paths for LD_LIBRARY_PATH and PYTHONPATH, not being root. but overall a very painless install. the data structure is bound to be more complex than a simple pickle, but there are some good tutorials out there. also, the nmag project has some good real-life experience with using pytables for unstructured grid data. (see hdf5*py in nmag-0.1/nsim/interface/nfem) uiuc and cei (the people who make ensight) also defined an hdf5 mesh api, but it looks pretty krufty now. another plus with pytables is that you can use vitables to interact with the data. that's even easier than a pickle. EDIT: hers's a site that covers a lot of the issues with scientific data storage and refers to specific examples, including hdf. one problem that might arise with pytables is that numpy arrays can be memory mapped to a file on disk, but pytables can't do that if i'm not mistaken. am i? according to this email exchange, pytables doesn't do mmap but it can be as fast or faster if used properly. sounds like i can still use pytables without losing performance, but i will need to reread that and some of the refs therein to implement. here's an interesting conversation about large file i/o in python, with specific applications in finance.

ica and quantitative finance

this site has some interesting papers on quantitative finance. in particular, i think the report on quant education would be an interesting read, even though it is a bit old now. the other paper, 'a first application of independent component analysis to extracting structure from stock returns' is the earliest reference i've seen on ica on financial data. now a number of people have been doing it, with mixed results imho. but there is a good point to be made here in that, if you are assuming independence, why just look at correlation? why choose an orthogonal basis orientation based on reprojection error L_2 minimization? why not look at mutual infomation or higher order moments and cumulants? if there are components that are uninterpretable, it is self-deceptive to force them to be small artificially and it will probably lead to overly optimistic estimates of risk. truth is, i have two goals for modeling log price relative time series: classification and time-windowed average estimation. for classification i want independence, and for the time averages i want to minimize time-averaged error (not necessarily time-averaged error^2). not only is amplitude significant; autocorrelation of the error time series is, too.

investment optimization functions

i already know that an objective function for an investment optimization needs to have more than just the expected value of the rate of return (though i think this should be the o.f. if constraints are applied as parameter boundaries). otherwise, i would be saying that i would tolerate an unlimited increase in any risk metric for a small increase in expected return, and that's not very smart. so, what else do i throw into the o.f.? the sortino ratio looks interesting. i think it makes a good critique of the sharpe ratio: upside variability should not be punished like downside variability. maybe a variant of it that, like the information ratio, uses a benchmark rather than a risk-free asset for active return. the wikipedia articles on these list others, and the upside potential ratio article refers to an article comparing it to sharpe and sortino ratios. value-at-risk and expected shortfall are both easy to compute, given a pdf on returns. i know value-at-risk has taken a beating. but even though expected shortfall has nice mathematical properties as a coherent, spectral risk measure, it is sensitive to errors in the distribution in the tail. i'm nervous about estimating distributions on something that provides few data by definition. hmmm, that makes me wonder... are there financial risk metrics based on extreme value theory? one thing they all have in common, though, is that they are functionals of the rate of return. the rate of return is a function of the time to reach a limit order, the price relative, and the transaction cost. so, given a pdf for time as a function of price relative, and the function defining rate of return, it should be easy to wrap an optimizer around any of the objective functions above. EDIT: i think the upside potential ratio is the same thing as the sortino ratio, differing perhaps only in the context. the information ratio is the same as the sharpe ratio, except that it uses a benchmark rather than a risk-free return for comparison. looks like the sharpe ratio, sortino ratio, value-at-risk, and expected shortfall (and probably many/most others) are not only funtionals of the rate of return pdf, they are functions of first and second partial moments. if i can boil those partial moments down to a small set, it would be possible to define a multi-objective space in which constant-value contours of the various financial objective functions are simple manifolds. the advantage of this is that i could find a pareto-optimal front in that partial moment space, and it would be easy to see not only the trade-offs but also how sensitive the o.f.s are at any point. otoh, maybe it would be just as easy to look at an overlay of all the o.f.s plotted in the control parameter space, and see values/sensitivities that way. at the very least, splitting up the moments into partials would facilitate simultaneous computation.

investment entities

why do tax laws have to be so complicated? i found a site that gives some info on the various types of entities to use as vehicles for active trading, and the tax pros and cons for each. i'm afraid i'll need to deal with this at some point. i guess it might be a choice between hiring a lawyer before or an accountant after. or both.

quantlib

the quantlib project has seemed interesting ever since i first found out about it. looks like many of the most of the lead developers are at banca imi, an italian investment bank. some others are at statpro, which provides data and analysis services. so it looks like at least some of the people involved are real pros in the business. i should take a more in-depth look some time. and there's a python binding! check out luigi's book as a great ref.