Playrix

Challenges that Playrix was encountering before moving to the cloud

Before moving to the cloud, the TeamCity server was installed in one of the offices while the build agents were also running on-premises in the same office. At the same time, engineers were spread across different locations, connecting to the single TeamCity server.

This setup used to be sufficient to meet the company’s needs. However, at one point, the company outgrew this setup.

Availability

The downside of having the server located in one particular office was the fact that, if anything happened to the office (e.g. electrical or network outage), developers couldn’t access their builds. The Playrix team is distributed across 40 countries and includes 500 TeamCity users. So having a reliable, responsive, and fast CI/CD server that is always available was crucial for the business.

Queues and long build times

As projects evolved, it became harder to scale the on-premises CI/CD infrastructure. At Playrix, builds are executed on each commit. As a result, during peak hours, builds would queue and take hours to run. The company would need hundreds of agents, while only 20 were available.

Provisioning additional infrastructure for CI/CD could solve this problem, but it’s cumbersome and results in an additional configuration management burden.

Solving the challenges with the help of TeamCity and AWS

Making CI/CD more available and reliable

The team started enhancing the availability of their CI/CD by moving the TeamCity server and artifact storage to the cloud while running build agents across multiple on-premises locations. “It works more reliably when the server is in the cloud. Normally, this way, the connection is more predictable than when the server is stored somewhere in a remote office location”, says Yuri Trufanov.

The team put the TeamCity server in an autoscaling group inside a Docker container and set up a Nomad orchestrator. On top of the orchestrator, the team set up a task with TeamCity. In case of any hardware fault, jobs move to another instance, and everything continues running smoothly.

The team also moved build artifact storage to Amazon S3 Artifact Storage while the TeamCity database was stored in Amazon Aurora MySQL.

The end result was a much more reliable and stable setup. Build artifacts are stored in AWS that offers backup. If any one of the offices falls out, the build queue just gets a little longer. But in the end, everything still works.

Autoscaling build agents in the cloud

Another problem that the Playrix team encountered was the fact that during the day, engineers needed a lot of build power to run their builds. During the night, however, all the build agents were sitting idle. The team wanted to allocate resources more efficiently and optimize costs.

The solution came in the form of the TeamCity Cloud Profiles. TeamCity allows you to create a cloud profile where you can specify all the configurations necessary for TeamCity to launch build agents in the cloud: what type of instance is needed, which resources, in which cloud to run, and which image to use. TeamCity offers out-of-the-box support for the cloud providers that the Playrix team needed, including AWS.

Playrix ended up using a hybrid solution. They would handle the macOS builds on the on-premises computers and run Linux and Windows build agents in the cloud, where they would be scaled on demand. That helped the company to solve the problem of long queues and extended build times.

“We ended up with a hybrid cloud solution that included TeamCity Cloud Profiles and AWS. In addition to that, we had on-premises computers for build agents. This combination allowed us to accommodate any number of builds throughout the day, while also providing a baseline agent count for the off-hours. So we could run whatever we wanted, wherever we wanted”, says Yuri Trufanov, Playrix’ Executive Technical Director.

Looking for ways to optimize costs

Having such flexibility came at a cost though. Running build agents in the cloud is expensive, and the Playrix team started looking for ways to make this solution more cost-efficient.

This is when Playrix looked at utilizing spot instances on AWS. TeamCity Cloud Profiles allow spot instances to be used seamlessly. If a spot instance gets revoked while the build agent is running a build, TeamCity will automatically launch a new instance and restart this build, without any additional input needed from the user.

Yuri confirmed that running build agents on spot instances turned out to be almost 10 times more cost-efficient for Playrix. So they made a decision to switch a major portion of their cloud-based Linux build agents to spot instances.

End result

With moving the TeamCity server and the majority of builds agents to AWS, the Playrix team created a highly available CI/CD solution.

Thanks to TeamCity Cloud Profiles, the team was also able to autoscale its build agents, scaling up during the rush hours and scaling down during the night. This helped the team to speed up builds and use the resources more efficiently.

The use of Spot Instances allowed Playrix to run their build agent farm in a more cost-efficient way, cutting the costs almost tenfold.