Towards Detecting Trigger-Based Behavior in Binaries: Uncovering the Correct Environment

Dorottya Papp, Thorsten Tarrach, L. Buttyán

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we present our first results towards detecting trigger-based behavior in binary programs. A program exhibits trigger-based behavior if it contains undocumented, often malicious functionality that is executed only under specific circumstances. In order to determine the inputs and environment required to trigger such behavior, we use directed symbolic execution and present techniques to overcome some of its practical limitations. Specifically, we propose techniques to overcome the environment problem and the path selection problem. We implemented our techniques and evaluated their performance on a real malware sample that launches denial-of-service attacks upon receiving specific remote commands. Thanks to our techniques, our implementation was able to determine those specific commands and all other requirements needed to trigger the malicious behavior in reasonable time.

Original languageEnglish
Title of host publicationSoftware Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings
EditorsPeter Csaba Ölveczky, Gwen Salaün
PublisherSpringer Verlag
Pages491-509
Number of pages19
ISBN (Print)9783030304454
DOIs
Publication statusPublished - Jan 1 2019
Event17th International Conference on Software Engineering and Formal Methods, SEFM 2019 - Oslo, Norway
Duration: Sep 18 2019Sep 20 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11724 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference17th International Conference on Software Engineering and Formal Methods, SEFM 2019
CountryNorway
CityOslo
Period9/18/199/20/19

Keywords

  • Directed symbolic execution
  • Software verification
  • Trigger-based behavior

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Towards Detecting Trigger-Based Behavior in Binaries: Uncovering the Correct Environment'. Together they form a unique fingerprint.

  • Cite this

    Papp, D., Tarrach, T., & Buttyán, L. (2019). Towards Detecting Trigger-Based Behavior in Binaries: Uncovering the Correct Environment. In P. C. Ölveczky, & G. Salaün (Eds.), Software Engineering and Formal Methods - 17th International Conference, SEFM 2019, Proceedings (pp. 491-509). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11724 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-30446-1_26