How Gearbox Uses TeamCity to Streamline Their CI/CD Process

Company information

Gearbox is an American video game company headquartered in Frisco, Texas, near Dallas. Founded in 1999, Gearbox has produced some of the most iconic video games in history, including Half-Life, Brothers in Arms, and Borderlands.

We sat down with Gearbox’s Steve Fortier, Lead Release Engineer, and Phillip Peterson, Senior Release Engineer, to talk about how the company was able to standardize and enhance their CI/CD processes with the help of JetBrains TeamCity.

About the team

Steve and Phillip are part of the core release engineering team at Gearbox. The team is spread across Quebec City, Montreal, and Frisco, and is tasked with servicing all projects at the company, setting up automation to meet each one’s needs. Whenever a new project is started, the team immediately sets up a depot in Perforce and ensures there is a project in TeamCity bound to it.

The team tries to dedicate one person to each project. Every single release engineer is part of both the release team and also a game project. That allows the team to be responsive to what each project needs in terms of automation.

It also allows release engineers to share knowledge about TeamCity best practices and jump in if another project needs help. Thanks to standardized CI/CD practices and reusable project templates, they can quickly react to any issues.

Gearbox’s technology stack

The team works with Perforce as their version control system and Unreal as their game engine. They have built C# scripts that sit on top of Unreal and are what TeamCity interacts with. This layer is CI-agnostic and allows them to work with Unity and Unreal projects.

The team also uses AWS for build machines and storage through Artifactory, for which they use the TeamCity JFrog plugin.

Why the team decided to look for a new CI/CD solution

The previous CI/CD solution that Gearbox used had a few limitations. Namely, the solution was poorly integrated with Perforce. The CI/CD tool the team was using also didn’t offer personal builds, which are available in TeamCity, and passing information between the build steps proved to be a challenge.

“We did have a product that we had used in-house for a long time. We looked to switch to a different competitor, and neither one of those worked out. So then some colleagues of ours who came from another game company said, ‘We used to use this thing called TeamCity.’ We looked into it, and understood that TeamCity solved a lot of our problems.”

— Phillip Peterson, Senior Release Engineer

Simplifying build step connections with TeamCity

In one of the competitors that Gearbox was using, the process of setting up a connection between two build steps was quite cumbersome. For instance, they’d have one build step that would produce an artifact. They would then want to pass the name of that artifact on to a subsequent job. It was quite a hurdle just to get that conversation between two build steps going. By contrast, this seemed extremely easy to do in TeamCity.

Another thing that the team was struggling with was passing build results between build steps, and TeamCity allowed them to accomplish this with a dependency.

The selection criteria for a new CI/CD solution

Applying bulk changes across projects

In a new CI/CD solution, the team was looking for two main things. One was the ability to apply bulk changes across projects easily. Previously, they’d needed to start each project from scratch. As the number of projects grew, the team started looking for templates that would allow them to copy-paste the project settings, change the name, and run the project.

Friendly UI

The second requirement was a friendly UI that would make it easy for end users and administrators to work with the new CI/CD tool.

“One of the competitors had a very unfriendly UI. The feeling that people should have when they come into a CI/CD system is that it is very sturdy, it won’t break when I try to do anything in it. I think TeamCity has a very well-polished UI. You have confidence that it is a well-working system when you navigate it.”

— Steve Fortier, Lead Release Engineer

TeamCity’s polished UI made it a very easy sell. Once the team proved the concept would work, showing it to people quickly convinced them to switch to the new CI/CD solution.

Access management

Another factor in Gearbox’s choice of a new CI/CD solution was access management and how it works in TeamCity. The team was looking for a way to edit user access by project.

In TeamCity, you can create hierarchies in which people are granted access to a project and then are able to access all of its subprojects. This made a big difference for Gearbox.

Choosing between TeamCity Cloud and On-Premises

When the team heard about TeamCity, they initially tried the on-premises version. However, the option to go with TeamCity Cloud made the Gearbox IT department extremely happy, knowing that all they had to do was to set up authentication. It also made the transition significantly faster than their estimates for if they had set it up on premises.

TeamCity Cloud users can choose between JetBrains-hosted and self-hosted build agents. The Gearbox team utilizes self-hosted agents, which allows them to fully customize the environment where they run their builds.

The transition process

When transitioning to TeamCity, the team started setting up all their projects from scratch. However, they had deliberately made their old build scripts CI-system agnostic. This meant that even though they started setting up many new projects in TeamCity, they were basically copy-pasting existing commands from the old system into them.

With a couple of workshops to discuss how to organize the projects, the team was able to shift from the old CI solution to TeamCity in just 6 weeks.

CI/CD metrics

Currently, Gearbox has 340 committers and 138 projects in TeamCity Cloud.

The team uses agents hosted in their own AWS accounts. They also make use of TeamCity’s cloud profiles. Depending on a build’s needs, TeamCity automatically selects from the “base”, “high”, “mega”, or “GPU” instances that the team uses.

During the transition period, Gearbox took the Amazon Machine Image (AMI) that they were spinning for their previous CI/CD solution and installed TeamCity right next to it. This way, Gearbox only had to maintain a single AMI, as the same one was used by both the old system and the new one. This made the migration process even easier.

Build chains in TeamCity

Gearbox uses build chains extensively throughout their CI/CD process. The Unreal process goes through 5 stages: compiling, cooking, staging, packaging, and publishing.

If a game is due to be released in less than 12 hours, and somewhere in that chain a volume fills up and halts the process, it’s impossible to expand the volume and start over. There’s simply not enough time to do that.

With the help of TeamCity, the Gearbox team was able to break every one of those pieces into its own build. This way, if the volume fills up and has to be extended during the build process, the build fails, but the team can handle that quickly. They can restart the process where it left off, remount that persistent volume, and continue from where it was. This is possible thanks to TeamCity’s built-in build chain optimizations, which are facilitated by the reuse of builds when employing snapshot dependencies.

Gearbox’s testing process: from local to centralized with dynamic orchestration

At Gearbox, once developers run local tests, TeamCity can then do much more, running the whole suite of tests centrally. The CI/CD system can also dynamically orchestrate these test runs across different types of machines, spinning up as many machines as needed at a given moment and shutting them down afterward to optimize resources.

Streamlining project setup and resource sharing with TeamCity at Gearbox

Knowledge sharing made easy

One of the biggest challenges that TeamCity helped Gearbox solve was spinning up new projects. With the alternatives they tried, every project had to be set up from scratch. As the projects grew over time, they would diverge from each other, becoming different enough that the company needed one subject matter expert per project. This made it difficult to share knowledge between teams, causing bottlenecks in the development process and increasing the risk of errors and inconsistencies.

Increased productivity

Since the team adopted TeamCity as their CI/CD solution, things have become significantly easier. Thanks to the templating that the team was able to do, they can share resources across projects. Once you’re familiar with one project, you’re familiar with all of them. This helped the team increase productivity and focus on efficient development.

Significant savings in terms of EC2 instances

In Gearbox’s previous CI solution, each build would build the editor, which could take up to an hour. Now, the team can make that step into its own build, which is then reused everywhere. This has helped the team make significant savings in terms of EC2 instances because they don’t have to compile the same thing several times.

Configuration as code at Gearbox

When Gearbox started using Kotlin for project configuration, the team was very excited about it. Even those with little experience with Kotlin were able to figure it out and start using it quickly enough. “There’s a learning curve, but generally, the vibe was positive,” said Steve Fortier.

As things stand now, when the team needs to change one thing in one project, they use the UI. If they need to change the way a particular build configuration works across multiple projects, they’ll use Kotlin. The ability to flexibly use this hybrid approach to project configurations is another one of TeamCity’s highlights for Gearbox.

Gearbox’s plans for the future

Gearbox aims to enhance their use of TeamCity by implementing personal builds to increase confidence in commits and providing working builds for QA testing. The team also seeks to improve build speed, shorten build chains, and minimize failures.

They currently track server performance, build duration, and build failures, but plan to include more granular metrics such as artifact size and queue duration. Their goal is to leverage TeamCity’s range of customizable features to establish a more efficient CI process that saves time and ensures the reliability of their builds.

Conclusion

Switching to TeamCity Cloud has improved Gearbox’s CI/CD practices in several ways.

• The ease of adoption allowed the team to migrate from their previous CI solution to TeamCity in just 6 weeks.
• TeamCity’s built-in Perforce integration helped the team streamline their whole CI/CD process and saved them time they’d have otherwise spent creating integration workarounds.
• With TeamCity’s project templating, the team can now launch new projects much faster and with less effort, following a standardized practice.
• Thanks to TeamCity’s cloud solution and friendly UI, the team finds maintaining projects is much easier than before.

TeamCity also helps Gearbox achieve significant savings in terms of EC2 instances, since the ability to use dependencies makes it possible to avoid compiling the same thing several times over.