Towards Detecting Trigger-Based Behavior in Binaries: Uncovering the Correct Environment

Dorottya Papp, Thorsten Tarrach, L. Buttyán

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we present our first results towards detecting trigger-based behavior in binary programs. A program exhibits trigger-based behavior if it contains undocumented, often malicious functionality that is executed only under specific circumstances. In order to determine the inputs and environment required to trigger such behavior, we use directed symbolic execution and present techniques to overcome some of its practical limitations. Specifically, we propose techniques to overcome the environment problem and the path selection problem. We implemented our techniques and evaluated their performance on a real malware sample that launches denial-of-service attacks upon receiving specific remote commands. Thanks to our techniques, our implementation was able to determine those specific commands and all other requirements needed to trigger the malicious behavior in reasonable time.

Original languageEnglish
Title of host publicationSoftware Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings
EditorsPeter Csaba Ölveczky, Gwen Salaün
PublisherSpringer Verlag
Pages491-509
Number of pages19
ISBN (Print)9783030304454
DOIs
Publication statusPublished - Jan 1 2019
Event17th International Conference on Software Engineering and Formal Methods, SEFM 2019 - Oslo, Norway
Duration: Sep 18 2019Sep 20 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11724 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference17th International Conference on Software Engineering and Formal Methods, SEFM 2019
CountryNorway
CityOslo
Period9/18/199/20/19

Fingerprint

Trigger
Binary
Symbolic Execution
Malware
Denial of Service
Attack
Path
Denial-of-service attack
Requirements

Keywords

  • Directed symbolic execution
  • Software verification
  • Trigger-based behavior

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Papp, D., Tarrach, T., & Buttyán, L. (2019). Towards Detecting Trigger-Based Behavior in Binaries: Uncovering the Correct Environment. In P. C. Ölveczky, & G. Salaün (Eds.), Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings (pp. 491-509). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11724 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-30446-1_26

Towards Detecting Trigger-Based Behavior in Binaries : Uncovering the Correct Environment. / Papp, Dorottya; Tarrach, Thorsten; Buttyán, L.

Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings. ed. / Peter Csaba Ölveczky; Gwen Salaün. Springer Verlag, 2019. p. 491-509 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11724 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Papp, D, Tarrach, T & Buttyán, L 2019, Towards Detecting Trigger-Based Behavior in Binaries: Uncovering the Correct Environment. in PC Ölveczky & G Salaün (eds), Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11724 LNCS, Springer Verlag, pp. 491-509, 17th International Conference on Software Engineering and Formal Methods, SEFM 2019, Oslo, Norway, 9/18/19. https://doi.org/10.1007/978-3-030-30446-1_26
Papp D, Tarrach T, Buttyán L. Towards Detecting Trigger-Based Behavior in Binaries: Uncovering the Correct Environment. In Ölveczky PC, Salaün G, editors, Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings. Springer Verlag. 2019. p. 491-509. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-30446-1_26
Papp, Dorottya ; Tarrach, Thorsten ; Buttyán, L. / Towards Detecting Trigger-Based Behavior in Binaries : Uncovering the Correct Environment. Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings. editor / Peter Csaba Ölveczky ; Gwen Salaün. Springer Verlag, 2019. pp. 491-509 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{981c82d9a94b4774b9ed2a4c7260d40e,
title = "Towards Detecting Trigger-Based Behavior in Binaries: Uncovering the Correct Environment",
abstract = "In this paper, we present our first results towards detecting trigger-based behavior in binary programs. A program exhibits trigger-based behavior if it contains undocumented, often malicious functionality that is executed only under specific circumstances. In order to determine the inputs and environment required to trigger such behavior, we use directed symbolic execution and present techniques to overcome some of its practical limitations. Specifically, we propose techniques to overcome the environment problem and the path selection problem. We implemented our techniques and evaluated their performance on a real malware sample that launches denial-of-service attacks upon receiving specific remote commands. Thanks to our techniques, our implementation was able to determine those specific commands and all other requirements needed to trigger the malicious behavior in reasonable time.",
keywords = "Directed symbolic execution, Software verification, Trigger-based behavior",
author = "Dorottya Papp and Thorsten Tarrach and L. Butty{\'a}n",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/978-3-030-30446-1_26",
language = "English",
isbn = "9783030304454",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "491--509",
editor = "{\"O}lveczky, {Peter Csaba} and Gwen Sala{\"u}n",
booktitle = "Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings",

}

TY - GEN

T1 - Towards Detecting Trigger-Based Behavior in Binaries

T2 - Uncovering the Correct Environment

AU - Papp, Dorottya

AU - Tarrach, Thorsten

AU - Buttyán, L.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - In this paper, we present our first results towards detecting trigger-based behavior in binary programs. A program exhibits trigger-based behavior if it contains undocumented, often malicious functionality that is executed only under specific circumstances. In order to determine the inputs and environment required to trigger such behavior, we use directed symbolic execution and present techniques to overcome some of its practical limitations. Specifically, we propose techniques to overcome the environment problem and the path selection problem. We implemented our techniques and evaluated their performance on a real malware sample that launches denial-of-service attacks upon receiving specific remote commands. Thanks to our techniques, our implementation was able to determine those specific commands and all other requirements needed to trigger the malicious behavior in reasonable time.

AB - In this paper, we present our first results towards detecting trigger-based behavior in binary programs. A program exhibits trigger-based behavior if it contains undocumented, often malicious functionality that is executed only under specific circumstances. In order to determine the inputs and environment required to trigger such behavior, we use directed symbolic execution and present techniques to overcome some of its practical limitations. Specifically, we propose techniques to overcome the environment problem and the path selection problem. We implemented our techniques and evaluated their performance on a real malware sample that launches denial-of-service attacks upon receiving specific remote commands. Thanks to our techniques, our implementation was able to determine those specific commands and all other requirements needed to trigger the malicious behavior in reasonable time.

KW - Directed symbolic execution

KW - Software verification

KW - Trigger-based behavior

UR - http://www.scopus.com/inward/record.url?scp=85072869915&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072869915&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-30446-1_26

DO - 10.1007/978-3-030-30446-1_26

M3 - Conference contribution

AN - SCOPUS:85072869915

SN - 9783030304454

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 491

EP - 509

BT - Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings

A2 - Ölveczky, Peter Csaba

A2 - Salaün, Gwen

PB - Springer Verlag

ER -