Information sources with different cost scales and the principle of conservation of entropy

I. Csiszár, G. Katona, G. Tusnády

Research output: Contribution to journalArticle

5 Citations (Scopus)


The aim of this paper is to provide a mathematically rigorous and sufficiently general treatment of the basic information-theoretic problems concerning sources with symbols of different costs and noiseless coding in a general sense. The main new concepts defined in this paper are the entropy rate (entropy per unit cost) of a source with respect to a stochastic cost scale and the encoding (in particular decodable encoding) of a source in a general sense. On the basis of these concepts, we prove some general theorems on the relation of entropy rates with respect to different cost scales and on the effect of encoding to the entropy rate. In particular, the "principle of conservation of entropy" and the "noiseless coding theorem" are proved under very general conditions.

Original languageEnglish
Pages (from-to)185-222
Number of pages38
JournalZeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete
Issue number3
Publication statusPublished - Sep 1 1969


ASJC Scopus subject areas

  • Analysis
  • Statistics and Probability
  • Mathematics(all)

Cite this