Monday, 22 October 2018

How Spotify migrated everything from on-premise to Google Cloud Platform

How Spotify migrated everything from on-premise to Google Cloud Platform
01 Aug

Spotify announced that it was going all in on Google Cloud Platform (GCP) back in 2016, committing a reported US$450 million over three years. In Spotify, Google got itself an anchor customer, not just because of its brand and scale, but also its reputation as a data driven, engineering-centric company.

Spotify has since shut down both of its US data centres and will be free of on-premise infrastructure by the end of the year following a complex migration.


During the Google Cloud Next conference in San Francisco we heard from members of the Spotify and Google Cloud teams who were involved in the migration about how they went about it and some key lessons learned.

Why migrate?

Director of engineering at Spotify Ramon van Alteren started by explaining why Spotify decided to go all in on cloud infrastructure in the first place.

“If you think about the amount of effort it takes to maintain compute, storage and network capacity for a global company that serves more than 170 million users, that is a sizeable amount of work,” he said.

“If I’m really honest, what we really want to do at Spotify is be the best music service in the world, none of that work on data centres actually contributes directly to that,” he added.

As well as freeing up developers from worrying about provisioning and maintaining infrastructure, van Alteren said the company also wanted to start taking advantage of some of the innovation coming out of Google Cloud, specifically the BigQuery cloud data warehouse, Pub/Sub for messaging and DataFlow for batch and streaming processing.

Van Alteren also said that the driving force behind the move to cloud came, somewhat surprisingly, from the engineers tasked with maintaining those data centres. “A big part of that was asking what their job looks like once we move to cloud,” he said. “The net result was that group of engineers, some of the most deeply respected at Spotify, ended up being advocates for our cloud strategy.”

Services migration

The actual migration plan was formulated back in 2015 and was broadly split up into two parts: services and data.

The services migration focused on moving nearly 1,200 microservices from on-premise data centres to the Google Cloud Platform.

The three main goals during migration, according to van Alteren, were to minimise disruption to product development, finish as quickly as possible to avoid the cost and complexity of running in a hybrid environment, and cleanup, ensuring Spotify didn’t have any lingering services running in its data centres.

One of the first things Google and Spotify did was to build a small migration team of Spotify engineers and Googlers, and built a live visualisation of the entire migration state so that engineers could self-serve to see the progress of the project.

That visualisation looks like a set of red (data centre) and green (Google Cloud) bubbles, with each bubble representing a system, and the size of the bubble representing the number of machines involved.

“This had a number of interesting side effects, one that saved me a lot of time as a programme manager to save doing status updates,” van Alteren said. “Next it created a real sense of accomplishment for teams that were migrating and they could see the impact they were making.”

« »


Related Articles