Modular reinforcement learning: An application to a real robot task

Zsolt Kalmár, Csaba Szepesvári, András Lőrincz

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

The behaviour of reinforcement learning (RL) algorithms is best understood in completely observable, finite state- and action-space, discrete-time controlled Markov-chains. Robot-learning domains, on the other hand, are inherently infinite both in time and space, and moreover they are only partially observable. In this article we suggest a systematic design method whose motivation comes from the desire to transform the task-to-be-solved into a finite-state, discrete-time, "approximately" Markovian task, which is completely observable too. The key idea is to break up the problem into subtasks and design controllers for each of the subtasks. Then operating conditions are attached to the controllers (together the controllers and their operating conditions which are called modules) and possible additional features are designed to facilitate observability. A new discrete time-counter is introduced at the "module-level" that clicks only when a change in the value of one of the features is observed. The approach was tried out on a real-life robot. Several RL algorithms were compared and it was found that a model-based approach worked best. The learnt switching strategy performed equally well as a handcrafted version. Moreover, the learnt strategy seemed to exploit certain properties of the environment which could not have been seen in advance, which predicted the promising possibility that a learnt controller might overperform a handcrafted switching strategy in the future.

Original languageEnglish
Title of host publicationLearning Robots - 6th European Workshop, EWLR 1997, Proceedings
EditorsJohn Demiris, Andreas Birk
PublisherSpringer Verlag
Pages29-45
Number of pages17
ISBN (Print)3540654801, 9783540654803
Publication statusPublished - Jan 1 1998
Event6th European Workshop on Learning Robots - EWLR 1997 - Brighton, United Kingdom
Duration: Aug 1 1997Aug 2 1997

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume1545
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other6th European Workshop on Learning Robots - EWLR 1997
CountryUnited Kingdom
CityBrighton
Period8/1/978/2/97

    Fingerprint

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Kalmár, Z., Szepesvári, C., & Lőrincz, A. (1998). Modular reinforcement learning: An application to a real robot task. In J. Demiris, & A. Birk (Eds.), Learning Robots - 6th European Workshop, EWLR 1997, Proceedings (pp. 29-45). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 1545). Springer Verlag.