Towards a smart wearable tool to enable people with SSPI to communicate by sentence fragments

Gyula Vörös, Anita Verő, BaláZs Pintér, Brigitta Miksztai-réthey, Takumi Toyama, A. Lőrincz, Daniel Sonntag

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)

Abstract

The ability to communicate with others is of paramount importance for mental well-being. In this paper, we describe an interaction system to reduce communication barriers for people with severe speech and physical impairments (SSPI) such as cerebral palsy. The system consists of two main components: (i) the head-mounted humancomputer interaction (HCI) part consisting of smart glasses with gaze trackers and text-to-speech functionality (which implement a communication board and the selection tool), and (ii) a natural language processing pipeline in the backend in order to generate complete sentences from the symbols on the board. We developed the components to provide a smooth interaction between the user and the system thereby including gaze tracking, symbol selection, symbol recognition, and sentence generation. Our results suggest that such systems can dramatically increase communication efficiency of people with SSPI.

Original languageEnglish
Title of host publicationPervasive Computing Paradigms for Mental Health - 4th International Symposium, MindCare 2014, Revised Selected Papers
PublisherSpringer Verlag
Pages90-99
Number of pages10
Volume100
ISBN (Print)9783319115634
DOIs
Publication statusPublished - 2014
Event4th International Symposium on Pervasive Computing Paradigms for Mental Health, MindCare 2014 - Tokyo, Japan
Duration: May 8 2014May 9 2014

Publication series

NameLecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST
Volume100
ISSN (Print)18678211

Other

Other4th International Symposium on Pervasive Computing Paradigms for Mental Health, MindCare 2014
CountryJapan
CityTokyo
Period5/8/145/9/14

Fingerprint

Communication
Pipelines
Glass
Processing

Keywords

  • Augmentative and alternative communication
  • Eye tracking
  • Head-mounted display
  • Language models
  • Natural language processing
  • Smart glasses
  • Speech synthesis

ASJC Scopus subject areas

  • Computer Networks and Communications

Cite this

Vörös, G., Verő, A., Pintér, B., Miksztai-réthey, B., Toyama, T., Lőrincz, A., & Sonntag, D. (2014). Towards a smart wearable tool to enable people with SSPI to communicate by sentence fragments. In Pervasive Computing Paradigms for Mental Health - 4th International Symposium, MindCare 2014, Revised Selected Papers (Vol. 100, pp. 90-99). (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Vol. 100). Springer Verlag. https://doi.org/10.1007/978-3-319-11564-1_10

Towards a smart wearable tool to enable people with SSPI to communicate by sentence fragments. / Vörös, Gyula; Verő, Anita; Pintér, BaláZs; Miksztai-réthey, Brigitta; Toyama, Takumi; Lőrincz, A.; Sonntag, Daniel.

Pervasive Computing Paradigms for Mental Health - 4th International Symposium, MindCare 2014, Revised Selected Papers. Vol. 100 Springer Verlag, 2014. p. 90-99 (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Vol. 100).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Vörös, G, Verő, A, Pintér, B, Miksztai-réthey, B, Toyama, T, Lőrincz, A & Sonntag, D 2014, Towards a smart wearable tool to enable people with SSPI to communicate by sentence fragments. in Pervasive Computing Paradigms for Mental Health - 4th International Symposium, MindCare 2014, Revised Selected Papers. vol. 100, Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, vol. 100, Springer Verlag, pp. 90-99, 4th International Symposium on Pervasive Computing Paradigms for Mental Health, MindCare 2014, Tokyo, Japan, 5/8/14. https://doi.org/10.1007/978-3-319-11564-1_10
Vörös G, Verő A, Pintér B, Miksztai-réthey B, Toyama T, Lőrincz A et al. Towards a smart wearable tool to enable people with SSPI to communicate by sentence fragments. In Pervasive Computing Paradigms for Mental Health - 4th International Symposium, MindCare 2014, Revised Selected Papers. Vol. 100. Springer Verlag. 2014. p. 90-99. (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST). https://doi.org/10.1007/978-3-319-11564-1_10
Vörös, Gyula ; Verő, Anita ; Pintér, BaláZs ; Miksztai-réthey, Brigitta ; Toyama, Takumi ; Lőrincz, A. ; Sonntag, Daniel. / Towards a smart wearable tool to enable people with SSPI to communicate by sentence fragments. Pervasive Computing Paradigms for Mental Health - 4th International Symposium, MindCare 2014, Revised Selected Papers. Vol. 100 Springer Verlag, 2014. pp. 90-99 (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST).
@inproceedings{e9a57addc788407f96034fb966e5d468,
title = "Towards a smart wearable tool to enable people with SSPI to communicate by sentence fragments",
abstract = "The ability to communicate with others is of paramount importance for mental well-being. In this paper, we describe an interaction system to reduce communication barriers for people with severe speech and physical impairments (SSPI) such as cerebral palsy. The system consists of two main components: (i) the head-mounted humancomputer interaction (HCI) part consisting of smart glasses with gaze trackers and text-to-speech functionality (which implement a communication board and the selection tool), and (ii) a natural language processing pipeline in the backend in order to generate complete sentences from the symbols on the board. We developed the components to provide a smooth interaction between the user and the system thereby including gaze tracking, symbol selection, symbol recognition, and sentence generation. Our results suggest that such systems can dramatically increase communication efficiency of people with SSPI.",
keywords = "Augmentative and alternative communication, Eye tracking, Head-mounted display, Language models, Natural language processing, Smart glasses, Speech synthesis",
author = "Gyula V{\"o}r{\"o}s and Anita Verő and Bal{\'a}Zs Pint{\'e}r and Brigitta Miksztai-r{\'e}they and Takumi Toyama and A. Lőrincz and Daniel Sonntag",
year = "2014",
doi = "10.1007/978-3-319-11564-1_10",
language = "English",
isbn = "9783319115634",
volume = "100",
series = "Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST",
publisher = "Springer Verlag",
pages = "90--99",
booktitle = "Pervasive Computing Paradigms for Mental Health - 4th International Symposium, MindCare 2014, Revised Selected Papers",

}

TY - GEN

T1 - Towards a smart wearable tool to enable people with SSPI to communicate by sentence fragments

AU - Vörös, Gyula

AU - Verő, Anita

AU - Pintér, BaláZs

AU - Miksztai-réthey, Brigitta

AU - Toyama, Takumi

AU - Lőrincz, A.

AU - Sonntag, Daniel

PY - 2014

Y1 - 2014

N2 - The ability to communicate with others is of paramount importance for mental well-being. In this paper, we describe an interaction system to reduce communication barriers for people with severe speech and physical impairments (SSPI) such as cerebral palsy. The system consists of two main components: (i) the head-mounted humancomputer interaction (HCI) part consisting of smart glasses with gaze trackers and text-to-speech functionality (which implement a communication board and the selection tool), and (ii) a natural language processing pipeline in the backend in order to generate complete sentences from the symbols on the board. We developed the components to provide a smooth interaction between the user and the system thereby including gaze tracking, symbol selection, symbol recognition, and sentence generation. Our results suggest that such systems can dramatically increase communication efficiency of people with SSPI.

AB - The ability to communicate with others is of paramount importance for mental well-being. In this paper, we describe an interaction system to reduce communication barriers for people with severe speech and physical impairments (SSPI) such as cerebral palsy. The system consists of two main components: (i) the head-mounted humancomputer interaction (HCI) part consisting of smart glasses with gaze trackers and text-to-speech functionality (which implement a communication board and the selection tool), and (ii) a natural language processing pipeline in the backend in order to generate complete sentences from the symbols on the board. We developed the components to provide a smooth interaction between the user and the system thereby including gaze tracking, symbol selection, symbol recognition, and sentence generation. Our results suggest that such systems can dramatically increase communication efficiency of people with SSPI.

KW - Augmentative and alternative communication

KW - Eye tracking

KW - Head-mounted display

KW - Language models

KW - Natural language processing

KW - Smart glasses

KW - Speech synthesis

UR - http://www.scopus.com/inward/record.url?scp=84910663347&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84910663347&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-11564-1_10

DO - 10.1007/978-3-319-11564-1_10

M3 - Conference contribution

AN - SCOPUS:84910663347

SN - 9783319115634

VL - 100

T3 - Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST

SP - 90

EP - 99

BT - Pervasive Computing Paradigms for Mental Health - 4th International Symposium, MindCare 2014, Revised Selected Papers

PB - Springer Verlag

ER -