Distributed Differentially Private Stochastic Gradient Descent: An Empirical Study

Istvan Hegedus, M. Jelasity

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

In fault-prone large-scale distributed environments stochastic gradient descent (SGD) is a popular approach to implement machine learning algorithms. Data privacy is a key concern in such environments, which is often addressed within the framework of differential privacy. The output quality of differentially private SGD implementations as a function of design choices has not yet been thoroughly evaluated. In this study, we examine this problem experimentally. We assume that every data record is stored by an independent node, which is a typical setup in networks of mobile devices or Internet of things (IoT) applications. In this model we identify a set of possible distributed differentially private SGD implementations. In these implementations all the sensitive computations are strictly local, and any public information is protected by differentially private mechanisms. This means that personal information can leak only if the corresponding node is directly compromised. We then perform a set of experiments to evaluate these implementations over several machine learning problems with both logistic regression and support vector machine (SVM) loss functions. Depending on the parameter setting and the choice of the algorithm, the performance of the noise-free algorithm can be closely approximated by differentially private variants.

Original languageEnglish
Title of host publicationProceedings - 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages566-573
Number of pages8
ISBN (Print)9781467387750
DOIs
Publication statusPublished - Mar 31 2016
Event24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016 - Heraklion, Crete, Greece
Duration: Feb 17 2016Feb 19 2016

Other

Other24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016
CountryGreece
CityHeraklion, Crete
Period2/17/162/19/16

Fingerprint

Stochastic Gradient
Gradient Descent
Empirical Study
Learning systems
Data privacy
Mobile devices
Learning algorithms
Privacy
Support vector machines
Logistics
Machine Learning
Internet of Things
Distributed Environment
Loss Function
Logistic Regression
Vertex of a graph
Mobile Devices
Learning Algorithm
Support Vector Machine
Fault

Keywords

  • distributed differential privacy
  • machine learning
  • stochastic gradient descent

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Software
  • Control and Optimization

Cite this

Hegedus, I., & Jelasity, M. (2016). Distributed Differentially Private Stochastic Gradient Descent: An Empirical Study. In Proceedings - 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016 (pp. 566-573). [7445391] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/PDP.2016.19

Distributed Differentially Private Stochastic Gradient Descent : An Empirical Study. / Hegedus, Istvan; Jelasity, M.

Proceedings - 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 566-573 7445391.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Hegedus, I & Jelasity, M 2016, Distributed Differentially Private Stochastic Gradient Descent: An Empirical Study. in Proceedings - 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016., 7445391, Institute of Electrical and Electronics Engineers Inc., pp. 566-573, 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016, Heraklion, Crete, Greece, 2/17/16. https://doi.org/10.1109/PDP.2016.19
Hegedus I, Jelasity M. Distributed Differentially Private Stochastic Gradient Descent: An Empirical Study. In Proceedings - 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016. Institute of Electrical and Electronics Engineers Inc. 2016. p. 566-573. 7445391 https://doi.org/10.1109/PDP.2016.19
Hegedus, Istvan ; Jelasity, M. / Distributed Differentially Private Stochastic Gradient Descent : An Empirical Study. Proceedings - 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 566-573
@inproceedings{92ce3a1d5d734445a69a3fa1dca500ad,
title = "Distributed Differentially Private Stochastic Gradient Descent: An Empirical Study",
abstract = "In fault-prone large-scale distributed environments stochastic gradient descent (SGD) is a popular approach to implement machine learning algorithms. Data privacy is a key concern in such environments, which is often addressed within the framework of differential privacy. The output quality of differentially private SGD implementations as a function of design choices has not yet been thoroughly evaluated. In this study, we examine this problem experimentally. We assume that every data record is stored by an independent node, which is a typical setup in networks of mobile devices or Internet of things (IoT) applications. In this model we identify a set of possible distributed differentially private SGD implementations. In these implementations all the sensitive computations are strictly local, and any public information is protected by differentially private mechanisms. This means that personal information can leak only if the corresponding node is directly compromised. We then perform a set of experiments to evaluate these implementations over several machine learning problems with both logistic regression and support vector machine (SVM) loss functions. Depending on the parameter setting and the choice of the algorithm, the performance of the noise-free algorithm can be closely approximated by differentially private variants.",
keywords = "distributed differential privacy, machine learning, stochastic gradient descent",
author = "Istvan Hegedus and M. Jelasity",
year = "2016",
month = "3",
day = "31",
doi = "10.1109/PDP.2016.19",
language = "English",
isbn = "9781467387750",
pages = "566--573",
booktitle = "Proceedings - 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Distributed Differentially Private Stochastic Gradient Descent

T2 - An Empirical Study

AU - Hegedus, Istvan

AU - Jelasity, M.

PY - 2016/3/31

Y1 - 2016/3/31

N2 - In fault-prone large-scale distributed environments stochastic gradient descent (SGD) is a popular approach to implement machine learning algorithms. Data privacy is a key concern in such environments, which is often addressed within the framework of differential privacy. The output quality of differentially private SGD implementations as a function of design choices has not yet been thoroughly evaluated. In this study, we examine this problem experimentally. We assume that every data record is stored by an independent node, which is a typical setup in networks of mobile devices or Internet of things (IoT) applications. In this model we identify a set of possible distributed differentially private SGD implementations. In these implementations all the sensitive computations are strictly local, and any public information is protected by differentially private mechanisms. This means that personal information can leak only if the corresponding node is directly compromised. We then perform a set of experiments to evaluate these implementations over several machine learning problems with both logistic regression and support vector machine (SVM) loss functions. Depending on the parameter setting and the choice of the algorithm, the performance of the noise-free algorithm can be closely approximated by differentially private variants.

AB - In fault-prone large-scale distributed environments stochastic gradient descent (SGD) is a popular approach to implement machine learning algorithms. Data privacy is a key concern in such environments, which is often addressed within the framework of differential privacy. The output quality of differentially private SGD implementations as a function of design choices has not yet been thoroughly evaluated. In this study, we examine this problem experimentally. We assume that every data record is stored by an independent node, which is a typical setup in networks of mobile devices or Internet of things (IoT) applications. In this model we identify a set of possible distributed differentially private SGD implementations. In these implementations all the sensitive computations are strictly local, and any public information is protected by differentially private mechanisms. This means that personal information can leak only if the corresponding node is directly compromised. We then perform a set of experiments to evaluate these implementations over several machine learning problems with both logistic regression and support vector machine (SVM) loss functions. Depending on the parameter setting and the choice of the algorithm, the performance of the noise-free algorithm can be closely approximated by differentially private variants.

KW - distributed differential privacy

KW - machine learning

KW - stochastic gradient descent

UR - http://www.scopus.com/inward/record.url?scp=84968866064&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84968866064&partnerID=8YFLogxK

U2 - 10.1109/PDP.2016.19

DO - 10.1109/PDP.2016.19

M3 - Conference contribution

AN - SCOPUS:84968866064

SN - 9781467387750

SP - 566

EP - 573

BT - Proceedings - 24th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, PDP 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -