mutual information and signal processing

some interesting articles:

Massoud Babaie-Zadeh, Christian Jutten, and Kambiz Nayebi. Differential of the Mutual Information
Kenneth E. Hild, II, Deniz Erdogmus, and José Príncipe. Blind Source Separation Using Renyi’s Mutual Information
George Atia and Venkatesh Saligrama. A Mutual Information Characterization for Sparse Signal Processing
Liam Paninski. Estimation of Entropy andMutual Information (recommended by others)
Janett Walters-Williams and Yan Li. Estimation of Mutual Information: A Survey


Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy (2005) by Hanchuan Peng , Fuhui Long , Chris Ding
lots of good definitions, results comparison

http://vserver1.cscs.lsa.umich.edu/~crshalizi/notabene/information-theory.html
lots of information theory links


Modelling time series using information theory
L Diambra, A Plastino

L. Zunino, M. C. Soriano, I. Fischer, O. A. Rosso, and C. R. Mirasso. Permutation-information-theory approach to unveil delay dynamics from time-series analysis

compressive sampling

just came across a new take on an old problem: compressed sensing. here's a gentle introduction to the subject. looks like youtube has some interesting lectures on this, too, especially linking compressive sensing or sampling to information theory. one thing that occurred to me is that the sparseness object expressed by the '0-norm' requires the part of the domain where the function goes to zero to do so faster than the norm parameter. otherwise, the 0^0 would be 1, just like x^0==1 for x!=0. so, something like lim x->0 exp(-1/x)^x. maybe i can think of it as an epsilon-norm, where epsilon is infinitessimal for all non-zero quantities but not exactly zero for the norm of zero.

Python for Information Theoretic Analysis of Neural Data

an excellent article comparing mi bias correction methods also discusses reasons for switching from matlab to python. one of the interesting ideas mentioned is the shuffled information estimator, which can be used along with any of the other techniques. i just read through it quickly, but i think there is some kind of connection there to the bootstrap or other resampling methods to estimate, and thereby reduce, conditional entropy bias. more on that in 'tight data-robust bounds to mutual information combining shuffling and model selection techniques', which claims robust upper and lower bounds and precise estimates even for high correlation.

the python code used in the paper for estimation of entropies and mi is on google code. i installed it from the svn downloaded source by going into ~/checkinstall/pyentropy/pyentropy-read-only/pyentropy and

checkinstall python setup.py install

the section, 'A python library for information theoretic estimates' gives a description of the library and how to use it. only problem for me is that it assumes a finite alphabet. i think some smart histogram binning should allow me to work with it, though.

reasons for migrating their code from matlab to python are interspersed throughout the article (for example, they say cython is easier to use than mex), and they mention using mlabwrap to facilitate.

estimating mutual information

just read a very interesting article on estimating mutual information from random variable samples. pretty much everything else i've seen on this subject is based on either histograms or kdes. so improving the algorithms comes down to improving histogram or kernel parameters. 'estimating mutual information' by a. kraskov, h. st\:ogbauer, and p. grassberger takes a unique approach, based on nearest neighbors. seems to do pretty well on few data points and nearly independent sets. the paper also shows an interesting application to ica. certainly worth a try, especially if i'm comparing a number of approaches to mi estimation. it points out that the norms need not be the same or even have the same space, so i could use the rank or any other transform (log is a popular one) to spread out some data or otherwise emphasize some parts to reduce estimation error without changing the theoretical result. the main results to implement are equations 8 and 9. be careful that the definitions of n_x and n_y are different in 8 and 9, though they could be counted simultaneously (on the same pass). compare fig 4 to 13. the exact value for I in the caption comes from eq 11. ref 34 goes into the uniqueness and robustness of the components that drop out of ica analysis. the paragraph under fig 17 has computed values from web-accessible data that could be used for testing code. it would be interesting to see if component 1 in fig 19 reflects phase differences in component 2 due to propagation delay. the second term of the second line in eq a4 equals 0, which confirms the fact that reparameterization does not change mutual information. using small values for k increases statistical errors while large k increases systematic errors. probably best to try multiple ks and compare trends to fig 4. the digamma function is implemented in scipy.special as psi.