This piece is part of a series on 2019 trends in the AI and Machine Learning industry. You can read my full thoughts on the past year in this summary I wrote for the Helixa blog, which also includes links to the other in-depth pieces in this series.
“Federated Learning” is a new term for many of us, and it looks like the dawn of a new AI epoch.
Federated Learning refers to machine learning (ML) techniques that can train algorithms across multiple decentralized machines holding different local data samples — all without exchanging data. This new approach is different from traditional centralized or distributed training because there is no assumption that the local data samples are identically distributed.
Federated Learning aims to mitigate the problem of Data Gravity, defined by Dave McCrory as “the ability of bodies of data to attract applications, services, and other data.”
In order to scale with the radical increase in devices, we need to move computation closer to the data generation.
Federated Learning is about centralizing models on decentralized data.
The major use case is edge computing, where issues of data privacy, security, and network traffic make it expensive and difficult to quickly collect and process data in the cloud. This is especially relevant to IoT devices and smartphones.
The TensorFlow team has recently released TFF, an extension to support Federated Learning natively into TensorFlow.
Alex Ingerman, Product Manager at Google, presented an interesting use case of TFF: Federated Learning for Mobile Keyboard Prediction. GoogleBoard was trained this way, with typing data from millions of smartphone devices, in order to improve keyboard prediction without infringing on the privacy consumers expect with their personal conversations.
For an easy to understand visualization of how Federated Learning works in practice, you can read this online comic illustration by Google AI.
The main privacy principles to respect are:
- Only access aggregated anonymized reports from devices (e.g. model updates), no raw data
- Use focused collection, which only reports the minimum needed
- Never persist per-device report
- Utilize federated model averaging to generate a global model
- Don’t memorize individual reports during training (overfitting on a single data sample)
Here is where Differential Privacy (DP) comes in place. DP relates to those systems for publicly sharing information about a dataset, describing patterns of groups within the dataset while withholding information about individuals. An algorithm is differentially private if it is robust to Membership Inference Attacks, which means that it is impossible for an external observer to tell from the output of the model whether a particular individual’s information was used in the computation.
In the specific case of training machine learning models on different data sets, DP mostly consists of two injections:
- Adding noise to the model parameters
- Clipping the maximum model parameters updates (helpful for privacy principle #5)
In addition to the primary use case of edge devices, Federated Learning and Differential Privacy can boost model performances for the case where different clients want to collaborate without sharing raw data with each other. An example is a framework developed by Georgian Partners and Bluecore Technologies leveraging the findings explained in Bolt-on differential privacy for scalable stochastic gradient descent-based analytics.
When you consider recent privacy regulations like GDPR and increased public awareness of data privacy issues, federated learning has the potential to sidestep one of the largest problems in the industry: to help society to securely cooperate for the common good.
Read this use case of Federated Learning applied to Cancer Research.
For curious, AI-focused professionals who want to innovate responsibly, Helixa derives complex human insights through ethical and intentional machine learning. We do this more effectively than anyone else by aggregating multiple data sources, prioritizing consumer privacy, and returning results in seconds. Visit www.helixa.ai to learn more.