A linear memory algorithm for Baum-Welch training

I. Miklós, Irmtraud M. Meyer

Research output: Contribution to journalArticle

29 Citations (Scopus)

Abstract

Background: Baum-Welch training is an expectation-maximisation algorithm for training the emission and transition probabilities of hidden Markov models in a fully automated way. It can be employed as long as a training set of annotated sequences is known, and provides a rigorous way to derive parameter values which are guaranteed to be at least locally optimal. For complex hidden Markov models such as pair hidden Markov models and very long training sequences, even the most efficient algorithms for Baum-Welch training are currently too memory-consuming. This has so far effectively prevented the automatic parameter training of hidden Markov models that are currently used for biological sequence analyses. Results: We introduce the first linear space algorithm for Baum-Welch training. For a hidden Markov model with M states, T free transition and E free emission parameters, and an input sequence of length L, our new algorithm requires O(M) memory and O(LMTmax (T + E)) time for one Baum-Welch iteration, where Tmax is the maximum number of states that any state is connected to. The most memory efficient algorithm until now was the checkpointing algorithm with O(log(L)M) memory and O(log(L)LMTmax) time requirement. Our novel algorithm thus renders the memory requirement completely independent of the length of the training sequences. More generally, for an n-hidden Markov model and n input sequences of length L, the memory requirement of O(log(L)Ln-1M) is reduced to O(Ln-1M) memory while the running time is changed from O(log(L)LnMTmax + Ln(T + E)) to O(LnMTmax (T + E)). An added advantage of our new algorithm is that a reduced time requirement can be traded for an increased memory requirement and vice versa, such that for any c ε {1,...,(T + E)}, a time requirement of LnMTmax c incurs a memory requirement of Ln-1M(T + E - c). Conclusions: For the large class of hidden Markov models used for example in gene prediction, whose number of states does not scale with the length of the input sequence, our novel algorithm can thus be both faster and more memory-efficient than any of the existing algorithms.

Original languageEnglish
Article number231
JournalBMC Bioinformatics
Volume6
DOIs
Publication statusPublished - Sep 19 2005

Fingerprint

Hidden Markov models
Data storage equipment
Markov Model
Requirements
Efficient Algorithms
Training
Checkpointing
Levenberg-Marquardt
Expectation-maximization Algorithm
Transition Probability
Linear Space
Sequence Analysis
Genes
Gene
Iteration
Prediction

ASJC Scopus subject areas

  • Medicine(all)
  • Structural Biology
  • Applied Mathematics

Cite this

A linear memory algorithm for Baum-Welch training. / Miklós, I.; Meyer, Irmtraud M.

In: BMC Bioinformatics, Vol. 6, 231, 19.09.2005.

Research output: Contribution to journalArticle

@article{d1b944563983435c9dbb551a9fba01ab,
title = "A linear memory algorithm for Baum-Welch training",
abstract = "Background: Baum-Welch training is an expectation-maximisation algorithm for training the emission and transition probabilities of hidden Markov models in a fully automated way. It can be employed as long as a training set of annotated sequences is known, and provides a rigorous way to derive parameter values which are guaranteed to be at least locally optimal. For complex hidden Markov models such as pair hidden Markov models and very long training sequences, even the most efficient algorithms for Baum-Welch training are currently too memory-consuming. This has so far effectively prevented the automatic parameter training of hidden Markov models that are currently used for biological sequence analyses. Results: We introduce the first linear space algorithm for Baum-Welch training. For a hidden Markov model with M states, T free transition and E free emission parameters, and an input sequence of length L, our new algorithm requires O(M) memory and O(LMTmax (T + E)) time for one Baum-Welch iteration, where Tmax is the maximum number of states that any state is connected to. The most memory efficient algorithm until now was the checkpointing algorithm with O(log(L)M) memory and O(log(L)LMTmax) time requirement. Our novel algorithm thus renders the memory requirement completely independent of the length of the training sequences. More generally, for an n-hidden Markov model and n input sequences of length L, the memory requirement of O(log(L)Ln-1M) is reduced to O(Ln-1M) memory while the running time is changed from O(log(L)LnMTmax + Ln(T + E)) to O(LnMTmax (T + E)). An added advantage of our new algorithm is that a reduced time requirement can be traded for an increased memory requirement and vice versa, such that for any c ε {1,...,(T + E)}, a time requirement of LnMTmax c incurs a memory requirement of Ln-1M(T + E - c). Conclusions: For the large class of hidden Markov models used for example in gene prediction, whose number of states does not scale with the length of the input sequence, our novel algorithm can thus be both faster and more memory-efficient than any of the existing algorithms.",
author = "I. Mikl{\'o}s and Meyer, {Irmtraud M.}",
year = "2005",
month = "9",
day = "19",
doi = "10.1186/1471-2105-6-231",
language = "English",
volume = "6",
journal = "BMC Bioinformatics",
issn = "1471-2105",
publisher = "BioMed Central",

}

TY - JOUR

T1 - A linear memory algorithm for Baum-Welch training

AU - Miklós, I.

AU - Meyer, Irmtraud M.

PY - 2005/9/19

Y1 - 2005/9/19

N2 - Background: Baum-Welch training is an expectation-maximisation algorithm for training the emission and transition probabilities of hidden Markov models in a fully automated way. It can be employed as long as a training set of annotated sequences is known, and provides a rigorous way to derive parameter values which are guaranteed to be at least locally optimal. For complex hidden Markov models such as pair hidden Markov models and very long training sequences, even the most efficient algorithms for Baum-Welch training are currently too memory-consuming. This has so far effectively prevented the automatic parameter training of hidden Markov models that are currently used for biological sequence analyses. Results: We introduce the first linear space algorithm for Baum-Welch training. For a hidden Markov model with M states, T free transition and E free emission parameters, and an input sequence of length L, our new algorithm requires O(M) memory and O(LMTmax (T + E)) time for one Baum-Welch iteration, where Tmax is the maximum number of states that any state is connected to. The most memory efficient algorithm until now was the checkpointing algorithm with O(log(L)M) memory and O(log(L)LMTmax) time requirement. Our novel algorithm thus renders the memory requirement completely independent of the length of the training sequences. More generally, for an n-hidden Markov model and n input sequences of length L, the memory requirement of O(log(L)Ln-1M) is reduced to O(Ln-1M) memory while the running time is changed from O(log(L)LnMTmax + Ln(T + E)) to O(LnMTmax (T + E)). An added advantage of our new algorithm is that a reduced time requirement can be traded for an increased memory requirement and vice versa, such that for any c ε {1,...,(T + E)}, a time requirement of LnMTmax c incurs a memory requirement of Ln-1M(T + E - c). Conclusions: For the large class of hidden Markov models used for example in gene prediction, whose number of states does not scale with the length of the input sequence, our novel algorithm can thus be both faster and more memory-efficient than any of the existing algorithms.

AB - Background: Baum-Welch training is an expectation-maximisation algorithm for training the emission and transition probabilities of hidden Markov models in a fully automated way. It can be employed as long as a training set of annotated sequences is known, and provides a rigorous way to derive parameter values which are guaranteed to be at least locally optimal. For complex hidden Markov models such as pair hidden Markov models and very long training sequences, even the most efficient algorithms for Baum-Welch training are currently too memory-consuming. This has so far effectively prevented the automatic parameter training of hidden Markov models that are currently used for biological sequence analyses. Results: We introduce the first linear space algorithm for Baum-Welch training. For a hidden Markov model with M states, T free transition and E free emission parameters, and an input sequence of length L, our new algorithm requires O(M) memory and O(LMTmax (T + E)) time for one Baum-Welch iteration, where Tmax is the maximum number of states that any state is connected to. The most memory efficient algorithm until now was the checkpointing algorithm with O(log(L)M) memory and O(log(L)LMTmax) time requirement. Our novel algorithm thus renders the memory requirement completely independent of the length of the training sequences. More generally, for an n-hidden Markov model and n input sequences of length L, the memory requirement of O(log(L)Ln-1M) is reduced to O(Ln-1M) memory while the running time is changed from O(log(L)LnMTmax + Ln(T + E)) to O(LnMTmax (T + E)). An added advantage of our new algorithm is that a reduced time requirement can be traded for an increased memory requirement and vice versa, such that for any c ε {1,...,(T + E)}, a time requirement of LnMTmax c incurs a memory requirement of Ln-1M(T + E - c). Conclusions: For the large class of hidden Markov models used for example in gene prediction, whose number of states does not scale with the length of the input sequence, our novel algorithm can thus be both faster and more memory-efficient than any of the existing algorithms.

UR - http://www.scopus.com/inward/record.url?scp=25444496966&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=25444496966&partnerID=8YFLogxK

U2 - 10.1186/1471-2105-6-231

DO - 10.1186/1471-2105-6-231

M3 - Article

C2 - 16171529

AN - SCOPUS:25444496966

VL - 6

JO - BMC Bioinformatics

JF - BMC Bioinformatics

SN - 1471-2105

M1 - 231

ER -