financial times rss

FT offers a dizzying array of separate rss feeds for its articles, but i don't think all of them end up in the print newspaper. i'd like to get the articles that the editors have deemed most important, without having to pay the £2/paper to get them. so... i bought a copy and tried to find where the articles in the paper show up in rss. and here they are, in roughly the order of importance given at the end of 'all you need to know about the city': lex column lex main 2nd (companies and markets) section companies: companies main, uk (companies|uk, though there's a lot that's not in the paper here), uk smaller companies (companies|uk) markets: markets main (markets section, last page) 1st section management: management main (business life) comment: comment main, opinion, analysis (comment & analysis) world: world main, europe, asia-pacific, africa, us, uk business, uk economy looks like google reader might be a convenient way to combine all the streams into one and maybe even keep track of which have been read already. and i was hoping the mobile version (http://www.google.com/reader/m) would simplify the scraping to whittle it down around the body text. but it doesn't always work; sometimes it only takes the first paragraph or two and scraps the rest, probably because it catches a break before a table or image. so i think i'll have to navigate through to the original page and scrape from there. EDIT: i guess the people at ft are smart enough to make it easy for me. they post links for their print edition (and us, europe, middle east, and asia editions). only thing i didn't find on that page was the 'money' special pullout from the weekend edition. i think most of those articles were in the 'personal finance' section of the website. and i discovered that each html page for a section has an rss icon link in the upper right, so it's easy to snag stuff once i know where they are on the website. going through the sections, i found they went roughly in order with pretty close, though not exact, correspondence to the articles in print. here are the sections on the website and the page numbers of articles listed under them, to give you an idea of the density: front page: 1,1 must read national news: 2,2,2,3,3,4,4,4,4,4,4 skip 1/2 to 2/3 world: 5,5,5,5,6,6,6,6,6,6,6,6,7,7,7,7,8,8,8 good read comment & analysis: ,9,10,10,10,10,11,11,11,11 skip some of these, though the latter ones are really good letters: 10,10,10,10,10 skip all of these! life & arts: (pull-out) 1,2,2,2,2,3,4,4,5,5,5,6,7,6,6,7,8,9,19,10,10,11,11,11,12,12,13,13,14,14,20,20,17,17,17,17,17,17,17,17,17,17,17 a lot of things skipped between 14 ad 17, but i would skip this whole section. ft magazine: (pull-out) 15,54,12,7,10,8,44,44,47,43,43,46,46,52,53,51,50,49,48 i would skip almost all of this house & home: (pull-out) 1,2,2,3,6,7,7,8 i would skip practically all of this section lex: 24,24,24,24 must read companies: 12,12,?,13,13,12,14,14,14,14,14,15,15,15,14,15,16,16,16,16,?,17,17,?,17 good read, especially toward the end markets: 22,22,?,23,23,23,23,24 must read

dashboard and screen scraping

one thing that's been on my low-priority radar is a way to scrape through the complex flaming hoops that banks, credit cards, and investment brokerages put up so i can have an auto dashboard, showing me account balances and net worth at a glance. mechanize looks like a nice package for performing many browser functions, including form interaction; probably the best of its kind i've found (and nice faq). however, it does mean writing a browsing session from scratch (read: lots of online debugging) and i'm not sure how well it can handle javascript, frames/windows, and all the other eye candy screen junk these sites like to throw at you. someone out there recommended pyxpcom (combined with pydom in pythonext) as a way to do anything mozilla can. i think that must be true, since it seems to be just the pieces that mozilla-esque browers are made of. as powerful and difficult to use as a build-your-own-ferrari kit. i think the most promising option seems to be selenium, which is apparently merging with webdriver for version 2.0. basically drives a real browser, but can record and play back scripts in a variety of languages (including python). the webdriver type of interface seems to be the future of selenium, and it has the advantages of better navigation and less to install. written in java, but i think it can do python (though the docs are behind if so). so i'm not sure if i should just wait for an official release of 2.0, but it does look like selenium is what i'm after. here's the doc on using ide. EDIT: did some more looking around with selenium, and wow! i love the ide/rc combo. i think i need to look at this blog post to get the most out of locators (css vs. xpath). some of the extra plugins for selenium-ide are worth getting, and the selenium.py module can apparently just be copied into the python path to use selenium-rc. 1.0.11 has firefox 4 support in the ide, but it's very recent (2011-04-12). they have put out a number of rcs for v2; apparently the v2 release is coming summer 2011. no remote control javascript server is necessary for version 2 since it's integrated with webdriver. i need to know if the ide and python export will still work. right now i think python will work, but no ide yet (though 2.0 is probably backwards compatible so might run the code generated by the version 1 ide). more selenium links: command locators, xpath/css/dom rosetta, css locators are faster than xpath, good info, stay up to date,good example, managed to get selenium python bindings installed on a windows machine (not surprisingly, a bit more involved than on linux) with my epd python. had to manually download tar ball, python setup.py install, and manually create the test dir structure that it would then complain about. maybe there's an option to make it skip tests, but the kludge was faster than looking that up. now i have selenium 2 with the webdriver interface, much better than rc! and btw, my experiments confirm what others have said about locators: css is much faster than xpath, even on firefox. i've also found that, while the selenium ide is really good for getting started with the locators, it's often possible to find shorter, more informative, and likely more stable tags and ids by poking around in the html just a little rather than using the first thing that pops up in the ide table. so i'm not going to try to keep a drop-in interface to call into the ide-generated scripts; cut-n-paste of one-liners will be good enough for both dev and maintenance. still, there is tremendous value in starting with something that works, and that alone makes the ide worth the install. some other things i've learned: the 'andWait' stuff is only relevant from the java interface. in python, there's no way to keep running asynch while stuff is still loading. click, get, etc. only return to the python script once it's fully loaded, so that can be a latency bottleneck. i did poke around and find a possible place to change that, but i'll see if i really need to.

data clustering for screen scraping

i just had an idea for an application of unsupervised data clustering. a quick google popped up python-cluster. looks like it's been abandoned for a couple of years, but it has a hierarchical algorithm and (maybe) k-means. i might try it. also, scipy.cluster has kmeans and vector quantization, with self organized feature maps and other methods promised later. the app is screen scraping web pages and trying to get the main content (an article, for example) without the ads, links, and other junk around the edges. i think it might be possible to look at each line (after tossing everything inside script tags) and separate the lines based on their length and the percent of the line inside html markup. the reasoning is that real content usually has long lines in the source and a small fraction of html taggage. i probably want to throw in line index as a third variable, since the lines i want will probably be close together. another thing i could do is grab multiple pages from the same site: either multiple articles that should have the same format or multiple copies from different days for a frequently updated page. that would allow me to do two things. first, i could combine data points from multiple pages to get higher point density for the cluster detection. (might need to test if an each individual page is sampled from the same distribution as the others to throw out outliers.) second, i could detect identical lines to throw away as nonunique boilerplate.

the ubuntu python-mvpa package looks like it might fit the bill.