Pokemon GO attracted massive traffic. The hit game shattered all prior estimates of player traffic with 50 times higher than expected spikes and ten times higher than the worst-case estimate. The game developer Niantic and Google had to work together for a solution. In a Google Cloud Platform Blog, engineering director Luke Stone has offered a detailed description of the enormous Pokemon GO Traffic.
Usually, user adoption happens gradually over several months, with new features and architectural changes scheduled over relatively long periods of time. However, this wasn’t the case with the launch of Pokemon GO. The game’s popularity quickly surged player traffic to 50 times the initial target. Google CRE had to arrange extra capacity on behalf of Niantic to stay well ahead of their record-setting growth.
Google Customer Reliability Engineering (CRE) is a new engagement model in which technical staff from Google works with customer teams to share the responsibility of critical cloud applications. Niantic was Google CRE’s first customer with its first assignment being the launch of Pokemon GO.
Within 15 minutes of launching in Australia and New Zealand, the player traffic surged well past Niantic’s expectations. The company collaborated with Google CRE for reinforcements, in anticipation of the US launch planned the next day. Niantic and Google Cloud [including CRE, SRE, development, product, support and executive teams] worked together to address the huge challenge offered by Pokemon GO.
The Pokemon GO mobile application uses many services across Google Cloud. The Cloud Datastore played an important role as the game’s primary database for capturing the Pokemon game world. As seen in the graph above, the teams targeted 1X player traffic, with a worst-case estimate of roughly 5X this target. However, this wasn’t the case. Google CRE had to arrange extra capacity on behalf of Niantic to stay ahead of their record-setting growth. Several issues emerged around the game’s stability. The teams had to work across the game architecture while new players were joining in.
The Pokemon GO uses a container-based development. The application logic for the game runs on Google Container Engine (GKE) powered by the open-source Kubernetes project. GKE can orchestrate container cluster at planetary-scale, which allowed Niantic to focus on installing live changes for their players.
The Niantic and the Google CRE team also upgraded to a newer version of GKE before Japan launch, which allowed more than a thousand additional nodes to be added to its container cluster. Careful measures were taken to avoid disrupting existing players. The teams cut over to the new version while millions of new players signed up and joined the Pokemon game world.
Additionally, the Niantic and Google engineers replaced the Network Load Balancer with the more sophisticated HTTP/S Load Balancer in its place. The HTTP/S Load Balancer is a global system tailored for HTTPS traffic, offering far more control, faster connections to users and higher throughput overall. This was done, keeping in mind the amount and types of traffic Pokemon GO was getting. All the lessons-learned from the US launch paid off as the Japan launch was carried out without any incident.
Stone says that Pokemon GO’s game world uses over a dozen Google Cloud service and the game was the largest Kubernetes (open-source deployment for apps) project to date. You can read the whole blog here.
Stone also include some fun facts in the blog. Here’s an excerpt:
The Pokemon GO game world was brought to life using over a dozen services across Google Cloud.
Pokemon GO was the largest Kubernetes deployment on Google Container Engine ever. Due to the scale of the cluster and accompanying throughput, a multitude of bugs were identified, fixed and merged into the open source project.
To support Pokemon GO’s massive player base, Google provisioned many tens of thousands of cores for Niantic’s Container Engine cluster.
Google’s global network helped reduce the overall latency for Pokemon Trainers inhabiting the game’s shared world. Game traffic travels Google’s private fiber network through most of its transit, delivering reliable, low-latency experiences for players worldwide. Even under the sea!