### Abstract

The estimation of relevant information theoretical quantities, such as entropy, mutual information, and various divergences is computationally expensive in high dimensions. However, for this task, one may apply pairwise Euclidean distances of sample points, which suits random projection (RP) based low dimensional embeddings. The Johnson-Lindenstrauss (JL) lemma gives theoretical bound on the dimension of the low dimensional embedding. We adapt the RP technique for the estimation of information theoretical quantities. Intriguingly, we find that embeddings into extremely small dimensions, far below the bounds of the JL lemma, provide satisfactory estimates for the original task. We illustrate this in the Independent Subspace Analysis (ISA) task; we combine RP dimension reduction with a simple ensemble method. We gain considerable speed-up with the potential of real-time parallel estimation of high dimensional information theoretical quantities.

Original language | English |
---|---|

Pages (from-to) | 146-153 |

Number of pages | 8 |

Journal | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |

Volume | 5441 |

DOIs | |

Publication status | Published - 2009 |

### Fingerprint

### Keywords

- Independent subspace analysis
- Information theoretical estimations
- Pairwise distances
- Random projection

### ASJC Scopus subject areas

- Computer Science(all)
- Theoretical Computer Science

### Cite this

*Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*,

*5441*, 146-153. https://doi.org/10.1007/978-3-642-00599-2_19

**Fast parallel estimation of high dimensional information theoretical quantities with low dimensional random projection ensembles.** / Szabo, Zoltan; Lőrincz, A.

Research output: Contribution to journal › Article

*Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*, vol. 5441, pp. 146-153. https://doi.org/10.1007/978-3-642-00599-2_19

}

TY - JOUR

T1 - Fast parallel estimation of high dimensional information theoretical quantities with low dimensional random projection ensembles

AU - Szabo, Zoltan

AU - Lőrincz, A.

PY - 2009

Y1 - 2009

N2 - The estimation of relevant information theoretical quantities, such as entropy, mutual information, and various divergences is computationally expensive in high dimensions. However, for this task, one may apply pairwise Euclidean distances of sample points, which suits random projection (RP) based low dimensional embeddings. The Johnson-Lindenstrauss (JL) lemma gives theoretical bound on the dimension of the low dimensional embedding. We adapt the RP technique for the estimation of information theoretical quantities. Intriguingly, we find that embeddings into extremely small dimensions, far below the bounds of the JL lemma, provide satisfactory estimates for the original task. We illustrate this in the Independent Subspace Analysis (ISA) task; we combine RP dimension reduction with a simple ensemble method. We gain considerable speed-up with the potential of real-time parallel estimation of high dimensional information theoretical quantities.

AB - The estimation of relevant information theoretical quantities, such as entropy, mutual information, and various divergences is computationally expensive in high dimensions. However, for this task, one may apply pairwise Euclidean distances of sample points, which suits random projection (RP) based low dimensional embeddings. The Johnson-Lindenstrauss (JL) lemma gives theoretical bound on the dimension of the low dimensional embedding. We adapt the RP technique for the estimation of information theoretical quantities. Intriguingly, we find that embeddings into extremely small dimensions, far below the bounds of the JL lemma, provide satisfactory estimates for the original task. We illustrate this in the Independent Subspace Analysis (ISA) task; we combine RP dimension reduction with a simple ensemble method. We gain considerable speed-up with the potential of real-time parallel estimation of high dimensional information theoretical quantities.

KW - Independent subspace analysis

KW - Information theoretical estimations

KW - Pairwise distances

KW - Random projection

UR - http://www.scopus.com/inward/record.url?scp=67149130044&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=67149130044&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-00599-2_19

DO - 10.1007/978-3-642-00599-2_19

M3 - Article

AN - SCOPUS:67149130044

VL - 5441

SP - 146

EP - 153

JO - Lecture Notes in Computer Science

JF - Lecture Notes in Computer Science

SN - 0302-9743

ER -