mutual information and signal processing

some interesting articles:

Massoud Babaie-Zadeh, Christian Jutten, and Kambiz Nayebi. Differential of the Mutual Information
Kenneth E. Hild, II, Deniz Erdogmus, and José Príncipe. Blind Source Separation Using Renyi’s Mutual Information
George Atia and Venkatesh Saligrama. A Mutual Information Characterization for Sparse Signal Processing
Liam Paninski. Estimation of Entropy andMutual Information (recommended by others)
Janett Walters-Williams and Yan Li. Estimation of Mutual Information: A Survey


Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy (2005) by Hanchuan Peng , Fuhui Long , Chris Ding
lots of good definitions, results comparison

http://vserver1.cscs.lsa.umich.edu/~crshalizi/notabene/information-theory.html
lots of information theory links


Modelling time series using information theory
L Diambra, A Plastino

L. Zunino, M. C. Soriano, I. Fischer, O. A. Rosso, and C. R. Mirasso. Permutation-information-theory approach to unveil delay dynamics from time-series analysis

Malliavin calculus

saw a reference to Malliavin calculus. used in financial math to take derivatives of stochastic processes. looks interesting, might be useful to learn some day.

rotation, quaternions, etc.

i always have to look this stuff up when i need it, so i found a great ref here. from the abstract: We present the three main mathematical constructs used to represent the attitude of a rigid body in three- dimensional space. These are (1) the rotation matrix, (2) a triple of Euler angles, and (3) the unit quaternion. To these we add a fourth, the rotation vector, which has many of the bene¯ts of both Euler angles and quaternions, but neither the singularities of the former, nor the quadratic constraint of the latter. There are several other subsidiary representations, such as Cayley-Klein parameters and the axis-angle representation, whose relations to the three main representations are also described. Our exposition is catered to those who seek a thorough and uni¯ed reference on the whole subject; detailed derivations of some results are not presented. Keywords{Euler angles, quaternion, Euler-Rodrigues parameters, Cayley-Klein parameters, rotation matrix, di- rection cosine matrix, transformation matrix, Cardan angles, Tait-Bryan angles, nautical angles, rotation vector, orientation, attitude, roll, pitch, yaw, bank, heading, spin, nutation, precession, Slerp 1

mutual information for linearly dependent rvs

i can't remember if i had recorded this anywhere, but i want to make sure i have it because it took me a little while to derive. if x1 and x2 are both uniform random variables on [0,1] and y = a*x1 +(1-a)*x2, then the mutual information between x1 and y is ln(a/(1-a)) + (1-a)/(2*a) nats, where ln is the natural log. this is a useful result for testing mutual information estimators because it's on a bounded domain so might converge more quickly than functions of exponents. the key to deriving this result is to realise that the marginal pdf for y is a trapezoid, and be careful about the log base when using the chain rule/integration by parts.

beyond deterministic optimization

i've come to realize that my multiobjective optimization problem is far from simple, with a lot of functions with noise-induced multimodality (FNIM) (numerical noise creating local minima). so i've been looking for previous work on how to deal with it. the fnim literature i've found seems to be very navel-gazing in that they look at how global minima bifurcate at the noise level is increased. but not as much on how to overcome the problem.

the term 'robust optimization' seems to be dominated by some guys from stanford, mit/singapore, and israel who have a particular framework that is not so useful to me right now. they assume the data going into the objective function are from a distribution that is an unknown member of a set of possible distros, and they want to protect against the worst-case from that set while strictly obeying the constraints. i think it's more of a mini-max problem, and what i need is to account for the uncertainty without needing to be so conservative. also, most, though not all, of the applications have a linear programming bend.

i'm not sure if stochastic optimization is what i need, but i'll check it out.

math for people

blog post in response to 'math for programmers'. for those of you with an interest in math education.

factorization of matrices with unknown elements

been thinking about a problem that can be represented as a factorization (like svd or ica) of a matrix when only some of the elements are known. clearly not a simple problem, but one that apparently comes up in image recognition. google turned up an interesting report from oxford from 5 years ago that gives a good review of the problem. good descriptions of a few algorithms, observations on optimization methods (that may or may not carry over to other applications), and some synthetic examples that show the effect of the distribution of known elements. the residual function uses a hadamard (elementwise) product with a mask matrix to represent the partial knowledge, although sec 4.3 points out that other forms might better incorporate prior knowledge.