stock index futures and price forecasting

trying to figure out exactly what numbers financial commentators use to make headlines like, "futures point to higher open..." often they point to specific futures contracts, like SPc1, NDc1, or DJc1. the cme group equity futures quotes seem to be what people are referring to (or at least have equivalent price percent changes). unfortunately, the javascript in those pages that handle the quote updates prevents convenient scraping. options pages like these, otoh, are exceedingly easy to scrape as whitespace-delimited plain text. and with the eom (end of month, european-style) options, the price/probability is fairly simple. if i really need to get at the futures contract prices, i could use bloomberg, which just has static html tables (with major us and world indices on one page). back on the cme group site, i couldn't find quotes for options for oil, and even the time and price transactions list is another inaccessible js job. however, if i only care about end-of-day data, there are a lot in the ftp server (see the settle/ dir, for example) and other links from the volume:volume & open interest tab of one of the futures pages. daily settlement prices (near the bottom) has links for interest rates, equities, (agricultural) commodities, precious and industrial metals (in comex), oil and energies (in nymex and cme clearport clearing), and fx. the volume by price data on that page might be particularly useful for building distributions. it combines total volume for the day at a each price, and the historical data available on the ftp server goes back a year and a half. the format is a bit obfuscated, but there are links to one-page format descriptions next to the data links.

one other thought about using derivatives for price forecasting/asset valuation: the price of the derivative depends not only on what people believe about the future price, but also on what everyone knows about the current price. so, if i had intraday data for the derivative price, i should subtract out the effect of the underlying price to get the future price. i wonder if ica on the log derivatives would achieve this... obvious maybe, but worth noting so i don't forget.