Industry: Software Development

JetBrains products used: TeamCity

Organization Size: 200

Country: United States

Gradle

Gradle Build Tool is a popular open-source build automation tool that is used by millions of developers for building, testing, and deploying software projects. Gradle is at the heart of the Continuous Delivery pipeline at many of the world’s most popular SaaS companies, including LinkedIn and Netflix.

How Gradle Build Tool Uses TeamCity to Run 30,000 Green Builds Per Day

Gradle Build Tool is a popular open-source build automation tool used by millions of developers for building, testing, and deploying software. TeamCity is a general-purpose CI/CD solution that allows for flexible workflows, collaboration, and development practices.

In this case study, we’ll take a deep look into how Gradle Build Tool uses TeamCity to run tens of thousands of builds a day, while keeping the failure rate under control.

“We’ve relied on TeamCity as our CI system of choice for over a decade. It provides all the features we need out of the box. We also appreciate its reliability and like Kotlin DSL for configuring our build pipelines.”

— Piotr Jagielski, VP of Engineering, Gradle Build Tool

About Gradle Inc.

Gradle, Inc. is the company behind the Gradle Build Tool, which is one of the most popular open-source build automation tools, and Gradle Enterprise is the leading solution platform for improving developer productivity. Headquartered in San Francisco, California, Gradle employs ~200 people in 30 countries to build the open-source tool and enterprise platform that serves millions of developers worldwide.

Gradle has over 100 engineers working on two main codebases – Gradle Build Tool and Gradle Enterprise. Each one includes millions of lines of code. The company uses TeamCity for both code bases, but this case study will focus on the Gradle Build Tool development.


TeamCity at Gradle Inc.

For the past 10 years, the Gradle Build Tool team has been relying on TeamCity for its CI/CD process. During that time, the team has never missed a TeamCity update. Regular updates allow the team to always have the newest, most feature-rich version of the product.


Gradle’s technology stack

The Gradle Build Tool team uses Git and GitHub as their version control system. They write code in Java, Kotlin, and Groovy. Naturally, the team uses their own products, Gradle Build Tool for build automation and Gradle Enterprise for build acceleration and failure analytics. They rely on TeamCity as their CI solution to run anything they don’t want to run locally: build, deploy, cron jobs, agent provisioning, and more.


Gradle’s CI pipeline

Gradle Build Tool has a comprehensive test suite that verifies the product works correctly across different operating systems, versions of Java, and other components. The complete “release-ready” build encompasses over 150k tests. Additionally, changes must successfully pass the performance suite before they can be integrated into the main branch. This requires a complex CI setup with hundreds of build agents.

Build agent types

Gradle uses Windows, Linux, and macOS build agents. About half of the agents are bare metal machines. The team also makes use of Amazon EC2 spot agents. This helps Gradle maintain build speed even when demand is high.

Thanks to TeamCity’s built-in spot instance support, the team never struggles with agents disconnecting. TeamCity automatically restarts builds if a spot instance is gone.

Key CI/CD metrics

The TeamCity setup for Gradle Build Tool is publicly available and visible to anyone. The main (master) branch consists of around 500 build configurations set up in a very sophisticated chain. As Piotr notes, “At Gradle, we use build chains quite extensively. In fact, we cannot live without them.”

Due to the chain complexity, Gradle relies on the Kotlin DSL for configuring their pipelines.

Part of Gradle Build Tool’s build chain. Access the full version here. You can log in as a guest to see it.
Part of Gradle Build Tool’s build chain. Access the full version here. You can log in as a guest to see it.

Gradle has 200 fixed build agents and 200 additional elastic EC2 agents to manage spikes connected via cloud profiles. The total time to run builds per week stands at 283 days (or 6,792 hours).

At the same time, about 1,636 days per month are being optimized by TeamCity. Smart algorithms allow TeamCity to decide if the build should not be executed due to another build having already run on the same repository commit.

Gradle uses build chains, and TeamCity helps the team save a significant amount of build time by reusing builds.

The number of used agents
The number of used agents

Thanks to TeamCity’s ability to provide Prometheus-compatible server metrics, the team can automatically export the build data to Grafana for further visualization and analysis. The team tracks such metrics as queued build waiting time, TeamCity agent usage, queue length, and the number of connected agents.

Based on this, the team can decide whether they need to buy more build agents if the usage gets too high.

TeamCity agent usage at Gradle
TeamCity agent usage at Gradle

Monitoring feedback time

Providing feedback as fast as possible is at the core of any CI solution. Monitoring the feedback time helps Gradle’s developer productivity team understand how long engineers need to wait before their pull requests get merged.

Queued builds average waiting time at Gradle
Queued builds average waiting time at Gradle

To track that, Gradle Build Tool keeps an eye on the Trigger build – a composite build that aggregates results from several other builds combined by snapshot dependencies and presents them in a single place.

The feedback time varies depending on the complexity of the pull request. Simpler PRs get feedback in 10 minutes, whereas in more complex cases, it might take up to an hour. Since Gradle is open source, all the pull requests coming from the GitHub community go through the same build process.

Gradle Build Tool’s master branch projects in TeamCity
Gradle Build Tool’s master branch projects in TeamCity

Build Trigger

Due to cost considerations, not every commit triggers builds. Instead, developers can easily trigger a build by commenting on the related pull request, and a bot will call the TeamCity API to trigger it.

A few branches, like master and release, are treated differently. Any push on these branches triggers a TeamCity build.

The Kotlin DSL: A game changer

The Kotlin DSL makes a huge difference in helping the Gradle team manage the complexity of build dependencies. By storing project configuration as code, the team can efficiently replicate build logic across projects, consistently apply updates to multiple configurations, and systematically manage its pipelines.

With the help of the Kotlin DSL, the team can generate and test build dependencies with much less effort. The team uses the Kotlin DSL with the web UI editing disabled for almost all its projects.


Gradle Build Tool + TeamCity: The greatest achievement

TeamCity accelerates the whole CI/CD process for Gradle. With TeamCity’s help, Gradle Build Tool successfully runs 30,000 green builds per day while keeping the failure rate under control.

TeamCity is a one-stop shop for the Gradle team because of its rich functionality. It has all the features other tools have, plus many of the ones they don’t have.

The Kotlin DSL gives Gradle Build Tool an unprecedented level of flexibility, which allows the team to quickly create a complex build chain and test it. The team also runs tests on the Kotlin DSL configuration files. Gradle’s TeamCity configuration is open-source and available on GitHub.

Finally, the Gradle Build Tool team values TeamCity’s reliability. The team upgrades to each new version of TeamCity as soon as it becomes available and rarely encounters any issues. TeamCity manages over 400 build agents, yet the team almost never has to troubleshoot any connection or build agent issues.


Scaling the infrastructure moving forward

The Gradle Build Tool team is committed to ensuring their infrastructure scales as the team grows. Even if the team size doubles in the future, Gradle aims to maintain the current feedback time or reduce it yet further.

The Gradle Build Tool team plans to achieve that ambitious goal by fully embracing Developer Productivity Engineering practices, increasing automation to eliminate human intervention, and adding more build agents into the whole CI/CD process.

Similar Customer Stories

Picnic

Ivan Babiankou, Staff Software Engineer, Picnic

We were looking for a managed solution for all our CI use cases. Additionally to that, we needed self-hosted agents to control which software we’re running, and which exact tooling is in use. TeamCity Cloud with self-hosted agents provided a tailor-made solution that our team of more than 300 engineers happily uses and that pushes our productivity to the next level.

Gearbox

Phillip Peterson, Senior Release Engineer

We did have a product that we had used in-house for a long time. We looked to switch to a different competitor, and neither one of those worked out. So then some colleagues of ours who came from another game company said, ‘We used to use this thing called TeamCity.’ We looked into it, and understood that TeamCity solved a lot of our problems.

Playrix

Yuri Trufanov, Executive Technical Director of Technology Platform

We ended up with a hybrid cloud solution that included TeamCity Cloud Profiles and AWS. In addition to that, we had on-premises computers for build agents. This combination allowed us to accommodate any number of builds throughout the day, while also providing a baseline agent count for the off-hours. So we could run whatever we wanted wherever we wanted.

More customer stories