Buttressing volatile desktop grids with cloud resources within a reconfigurable Environment Service for Workflow Orchestration

Stephen C. Winter, Christopher J. Reynolds, Tamas Kiss, Gabor Z. Terstyanszky, Pamela Greenwell, Sharron McEldowney, Sandor Acs, P. Kacsuk

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Cloud technology has the potential for widening access to high-performance computational resources for e-science research, but barriers to engagement with the technology remain high for many scientists. Workflows help overcome barriers by hiding details of underlying computational infrastructure and are portable between various platforms including cloud; they are also increasingly accepted within e-science research communities. Issues arising from the range of workflow systems available and the complexity of workflow development have been addressed by focusing on workflow interoperability, and providing customised support for different science communities. However, the deployments of such environments can be challenging, even where user requirements are comparatively modest. RESWO (Reconfigurable Environment Service for Workflow Orchestration) is a virtual platform-as-aservice cloud model that allows leaner customised environments to be assembled and deployed within a cloud. Suitable distributed computation resources are not always easily affordable and can present a further barrier to engagement by scientists. Desktop grids that use the spare CPU cycles available within an organisation are an attractively inexpensive type of infrastructure for many, and have been effectively virtualised as a cloud-based resource. However, hosts in this environment are volatile: leading to the tail problem, where some tasks become randomly delayed, affecting overall performance. To solve this problem, new algorithms have been developed to implement a cloudbursting scheduler in which durable cloud-based CPU resources may execute replicas of jobs that have become delayed. This paper describes experiences in the development of a RESWO instance in which a desktop grid is buttressed with CPU resources in the cloud to support the aspirations of bioscience researchers. A core component of the architecture, the cloudbursting scheduler, implements an algorithm to perform late job detection, cloud resource management and job monitoring. The experimental results obtained demonstrate significant performance improvements and benefits illustrated by use cases in bioscience research.

Original languageEnglish
JournalJournal of Cloud Computing
Volume3
Issue number1
DOIs
Publication statusPublished - 2014

Fingerprint

Program processors
Interoperability
Monitoring

Keywords

  • Cloudbursting
  • Hybrid cloud
  • Workflow

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications

Cite this

Buttressing volatile desktop grids with cloud resources within a reconfigurable Environment Service for Workflow Orchestration. / Winter, Stephen C.; Reynolds, Christopher J.; Kiss, Tamas; Terstyanszky, Gabor Z.; Greenwell, Pamela; McEldowney, Sharron; Acs, Sandor; Kacsuk, P.

In: Journal of Cloud Computing, Vol. 3, No. 1, 2014.

Research output: Contribution to journalArticle

Winter, Stephen C. ; Reynolds, Christopher J. ; Kiss, Tamas ; Terstyanszky, Gabor Z. ; Greenwell, Pamela ; McEldowney, Sharron ; Acs, Sandor ; Kacsuk, P. / Buttressing volatile desktop grids with cloud resources within a reconfigurable Environment Service for Workflow Orchestration. In: Journal of Cloud Computing. 2014 ; Vol. 3, No. 1.
@article{e4fd46f00ab04749b36670612d3dd18a,
title = "Buttressing volatile desktop grids with cloud resources within a reconfigurable Environment Service for Workflow Orchestration",
abstract = "Cloud technology has the potential for widening access to high-performance computational resources for e-science research, but barriers to engagement with the technology remain high for many scientists. Workflows help overcome barriers by hiding details of underlying computational infrastructure and are portable between various platforms including cloud; they are also increasingly accepted within e-science research communities. Issues arising from the range of workflow systems available and the complexity of workflow development have been addressed by focusing on workflow interoperability, and providing customised support for different science communities. However, the deployments of such environments can be challenging, even where user requirements are comparatively modest. RESWO (Reconfigurable Environment Service for Workflow Orchestration) is a virtual platform-as-aservice cloud model that allows leaner customised environments to be assembled and deployed within a cloud. Suitable distributed computation resources are not always easily affordable and can present a further barrier to engagement by scientists. Desktop grids that use the spare CPU cycles available within an organisation are an attractively inexpensive type of infrastructure for many, and have been effectively virtualised as a cloud-based resource. However, hosts in this environment are volatile: leading to the tail problem, where some tasks become randomly delayed, affecting overall performance. To solve this problem, new algorithms have been developed to implement a cloudbursting scheduler in which durable cloud-based CPU resources may execute replicas of jobs that have become delayed. This paper describes experiences in the development of a RESWO instance in which a desktop grid is buttressed with CPU resources in the cloud to support the aspirations of bioscience researchers. A core component of the architecture, the cloudbursting scheduler, implements an algorithm to perform late job detection, cloud resource management and job monitoring. The experimental results obtained demonstrate significant performance improvements and benefits illustrated by use cases in bioscience research.",
keywords = "Cloudbursting, Hybrid cloud, Workflow",
author = "Winter, {Stephen C.} and Reynolds, {Christopher J.} and Tamas Kiss and Terstyanszky, {Gabor Z.} and Pamela Greenwell and Sharron McEldowney and Sandor Acs and P. Kacsuk",
year = "2014",
doi = "10.1186/2192-113X-3-1",
language = "English",
volume = "3",
journal = "Journal of Cloud Computing",
issn = "2192-113X",
publisher = "Springer Science + Business Media",
number = "1",

}

TY - JOUR

T1 - Buttressing volatile desktop grids with cloud resources within a reconfigurable Environment Service for Workflow Orchestration

AU - Winter, Stephen C.

AU - Reynolds, Christopher J.

AU - Kiss, Tamas

AU - Terstyanszky, Gabor Z.

AU - Greenwell, Pamela

AU - McEldowney, Sharron

AU - Acs, Sandor

AU - Kacsuk, P.

PY - 2014

Y1 - 2014

N2 - Cloud technology has the potential for widening access to high-performance computational resources for e-science research, but barriers to engagement with the technology remain high for many scientists. Workflows help overcome barriers by hiding details of underlying computational infrastructure and are portable between various platforms including cloud; they are also increasingly accepted within e-science research communities. Issues arising from the range of workflow systems available and the complexity of workflow development have been addressed by focusing on workflow interoperability, and providing customised support for different science communities. However, the deployments of such environments can be challenging, even where user requirements are comparatively modest. RESWO (Reconfigurable Environment Service for Workflow Orchestration) is a virtual platform-as-aservice cloud model that allows leaner customised environments to be assembled and deployed within a cloud. Suitable distributed computation resources are not always easily affordable and can present a further barrier to engagement by scientists. Desktop grids that use the spare CPU cycles available within an organisation are an attractively inexpensive type of infrastructure for many, and have been effectively virtualised as a cloud-based resource. However, hosts in this environment are volatile: leading to the tail problem, where some tasks become randomly delayed, affecting overall performance. To solve this problem, new algorithms have been developed to implement a cloudbursting scheduler in which durable cloud-based CPU resources may execute replicas of jobs that have become delayed. This paper describes experiences in the development of a RESWO instance in which a desktop grid is buttressed with CPU resources in the cloud to support the aspirations of bioscience researchers. A core component of the architecture, the cloudbursting scheduler, implements an algorithm to perform late job detection, cloud resource management and job monitoring. The experimental results obtained demonstrate significant performance improvements and benefits illustrated by use cases in bioscience research.

AB - Cloud technology has the potential for widening access to high-performance computational resources for e-science research, but barriers to engagement with the technology remain high for many scientists. Workflows help overcome barriers by hiding details of underlying computational infrastructure and are portable between various platforms including cloud; they are also increasingly accepted within e-science research communities. Issues arising from the range of workflow systems available and the complexity of workflow development have been addressed by focusing on workflow interoperability, and providing customised support for different science communities. However, the deployments of such environments can be challenging, even where user requirements are comparatively modest. RESWO (Reconfigurable Environment Service for Workflow Orchestration) is a virtual platform-as-aservice cloud model that allows leaner customised environments to be assembled and deployed within a cloud. Suitable distributed computation resources are not always easily affordable and can present a further barrier to engagement by scientists. Desktop grids that use the spare CPU cycles available within an organisation are an attractively inexpensive type of infrastructure for many, and have been effectively virtualised as a cloud-based resource. However, hosts in this environment are volatile: leading to the tail problem, where some tasks become randomly delayed, affecting overall performance. To solve this problem, new algorithms have been developed to implement a cloudbursting scheduler in which durable cloud-based CPU resources may execute replicas of jobs that have become delayed. This paper describes experiences in the development of a RESWO instance in which a desktop grid is buttressed with CPU resources in the cloud to support the aspirations of bioscience researchers. A core component of the architecture, the cloudbursting scheduler, implements an algorithm to perform late job detection, cloud resource management and job monitoring. The experimental results obtained demonstrate significant performance improvements and benefits illustrated by use cases in bioscience research.

KW - Cloudbursting

KW - Hybrid cloud

KW - Workflow

UR - http://www.scopus.com/inward/record.url?scp=84926427505&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84926427505&partnerID=8YFLogxK

U2 - 10.1186/2192-113X-3-1

DO - 10.1186/2192-113X-3-1

M3 - Article

AN - SCOPUS:84926427505

VL - 3

JO - Journal of Cloud Computing

JF - Journal of Cloud Computing

SN - 2192-113X

IS - 1

ER -