Scientific workflow makespan reduction through cloud augmented Desktop Grids

Christopher J. Reynolds, Stephen Winter, Gabor Z. Terstyanszky, Tamas Kiss, Pamela Greenwell, Sandor Acs, Peter Kacsuk

Research output: Chapter in Book/Report/Conference proceedingConference contribution

14 Citations (Scopus)

Abstract

Scientific workflows are common in biomedical research, particularly for molecular docking simulations such as those used in drug discovery. Such workflows typically involve data distribution between computationally demanding stages which are usually mapped onto large scale compute resources. Volunteer or Desktop Grid (DG) computing can provide such infrastructure but has limitations resulting from the heterogeneous nature of the compute nodes. These constraints mean that reducing the makespan of a given workflow stage submitted to a DG becomes problematic. Late jobs can significantly affect the makespan, often completing long after the bulk of the computation has finished. In this paper we present a system capable of significantly reducing the makespan of a scientific workflow. Our system comprises a DG which is dynamically augmented with an infrastructure as a service (IaaS) Cloud. Using this solution, the Cloud resources are used to process replicated late jobs. Our system comprises a core component termed the scheduler, which implements an algorithm to perform late job detection, Cloud resource management (instantiation and reuse), and job monitoring. We offer a formal definition of this algorithm, whilst we also provide an evaluation of our prototype using a production scientific workflow.

Original languageEnglish
Title of host publicationProceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011
Pages18-23
Number of pages6
DOIs
Publication statusPublished - 2011
Event2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011 - Athens, Greece
Duration: Nov 29 2011Dec 1 2011

Other

Other2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011
CountryGreece
CityAthens
Period11/29/1112/1/11

Fingerprint

Grid computing
Monitoring
Drug Discovery

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Computer Networks and Communications

Cite this

Reynolds, C. J., Winter, S., Terstyanszky, G. Z., Kiss, T., Greenwell, P., Acs, S., & Kacsuk, P. (2011). Scientific workflow makespan reduction through cloud augmented Desktop Grids. In Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011 (pp. 18-23). [6133122] https://doi.org/10.1109/CloudCom.2011.13

Scientific workflow makespan reduction through cloud augmented Desktop Grids. / Reynolds, Christopher J.; Winter, Stephen; Terstyanszky, Gabor Z.; Kiss, Tamas; Greenwell, Pamela; Acs, Sandor; Kacsuk, Peter.

Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011. 2011. p. 18-23 6133122.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Reynolds, CJ, Winter, S, Terstyanszky, GZ, Kiss, T, Greenwell, P, Acs, S & Kacsuk, P 2011, Scientific workflow makespan reduction through cloud augmented Desktop Grids. in Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011., 6133122, pp. 18-23, 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011, Athens, Greece, 11/29/11. https://doi.org/10.1109/CloudCom.2011.13
Reynolds CJ, Winter S, Terstyanszky GZ, Kiss T, Greenwell P, Acs S et al. Scientific workflow makespan reduction through cloud augmented Desktop Grids. In Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011. 2011. p. 18-23. 6133122 https://doi.org/10.1109/CloudCom.2011.13
Reynolds, Christopher J. ; Winter, Stephen ; Terstyanszky, Gabor Z. ; Kiss, Tamas ; Greenwell, Pamela ; Acs, Sandor ; Kacsuk, Peter. / Scientific workflow makespan reduction through cloud augmented Desktop Grids. Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011. 2011. pp. 18-23
@inproceedings{86208b0954cc47989053ca94df5280de,
title = "Scientific workflow makespan reduction through cloud augmented Desktop Grids",
abstract = "Scientific workflows are common in biomedical research, particularly for molecular docking simulations such as those used in drug discovery. Such workflows typically involve data distribution between computationally demanding stages which are usually mapped onto large scale compute resources. Volunteer or Desktop Grid (DG) computing can provide such infrastructure but has limitations resulting from the heterogeneous nature of the compute nodes. These constraints mean that reducing the makespan of a given workflow stage submitted to a DG becomes problematic. Late jobs can significantly affect the makespan, often completing long after the bulk of the computation has finished. In this paper we present a system capable of significantly reducing the makespan of a scientific workflow. Our system comprises a DG which is dynamically augmented with an infrastructure as a service (IaaS) Cloud. Using this solution, the Cloud resources are used to process replicated late jobs. Our system comprises a core component termed the scheduler, which implements an algorithm to perform late job detection, Cloud resource management (instantiation and reuse), and job monitoring. We offer a formal definition of this algorithm, whilst we also provide an evaluation of our prototype using a production scientific workflow.",
author = "Reynolds, {Christopher J.} and Stephen Winter and Terstyanszky, {Gabor Z.} and Tamas Kiss and Pamela Greenwell and Sandor Acs and Peter Kacsuk",
year = "2011",
doi = "10.1109/CloudCom.2011.13",
language = "English",
isbn = "9780769546223",
pages = "18--23",
booktitle = "Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011",

}

TY - GEN

T1 - Scientific workflow makespan reduction through cloud augmented Desktop Grids

AU - Reynolds, Christopher J.

AU - Winter, Stephen

AU - Terstyanszky, Gabor Z.

AU - Kiss, Tamas

AU - Greenwell, Pamela

AU - Acs, Sandor

AU - Kacsuk, Peter

PY - 2011

Y1 - 2011

N2 - Scientific workflows are common in biomedical research, particularly for molecular docking simulations such as those used in drug discovery. Such workflows typically involve data distribution between computationally demanding stages which are usually mapped onto large scale compute resources. Volunteer or Desktop Grid (DG) computing can provide such infrastructure but has limitations resulting from the heterogeneous nature of the compute nodes. These constraints mean that reducing the makespan of a given workflow stage submitted to a DG becomes problematic. Late jobs can significantly affect the makespan, often completing long after the bulk of the computation has finished. In this paper we present a system capable of significantly reducing the makespan of a scientific workflow. Our system comprises a DG which is dynamically augmented with an infrastructure as a service (IaaS) Cloud. Using this solution, the Cloud resources are used to process replicated late jobs. Our system comprises a core component termed the scheduler, which implements an algorithm to perform late job detection, Cloud resource management (instantiation and reuse), and job monitoring. We offer a formal definition of this algorithm, whilst we also provide an evaluation of our prototype using a production scientific workflow.

AB - Scientific workflows are common in biomedical research, particularly for molecular docking simulations such as those used in drug discovery. Such workflows typically involve data distribution between computationally demanding stages which are usually mapped onto large scale compute resources. Volunteer or Desktop Grid (DG) computing can provide such infrastructure but has limitations resulting from the heterogeneous nature of the compute nodes. These constraints mean that reducing the makespan of a given workflow stage submitted to a DG becomes problematic. Late jobs can significantly affect the makespan, often completing long after the bulk of the computation has finished. In this paper we present a system capable of significantly reducing the makespan of a scientific workflow. Our system comprises a DG which is dynamically augmented with an infrastructure as a service (IaaS) Cloud. Using this solution, the Cloud resources are used to process replicated late jobs. Our system comprises a core component termed the scheduler, which implements an algorithm to perform late job detection, Cloud resource management (instantiation and reuse), and job monitoring. We offer a formal definition of this algorithm, whilst we also provide an evaluation of our prototype using a production scientific workflow.

UR - http://www.scopus.com/inward/record.url?scp=84857175973&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84857175973&partnerID=8YFLogxK

U2 - 10.1109/CloudCom.2011.13

DO - 10.1109/CloudCom.2011.13

M3 - Conference contribution

AN - SCOPUS:84857175973

SN - 9780769546223

SP - 18

EP - 23

BT - Proceedings - 2011 3rd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2011

ER -