Industry: Software Development
JetBrains products used: TeamCity
Organization Size: 200
Country: United States
Gradle Build Tool is a popular open-source build automation tool that is used by millions of developers for building, testing, and deploying software projects. Gradle is at the heart of the Continuous Delivery pipeline at many of the world’s most popular SaaS companies, including LinkedIn and Netflix.
Gradle Build Tool is a popular open-source build automation tool used by millions of developers for building, testing, and deploying software. TeamCity is a general-purpose CI/CD solution that allows for flexible workflows, collaboration, and development practices.
In this case study, we’ll take a deep look into how Gradle Build Tool uses TeamCity to run tens of thousands of builds a day, while keeping the failure rate under control.
“We’ve relied on TeamCity as our CI system of choice for over a decade. It provides all the features we need out of the box. We also appreciate its reliability and like Kotlin DSL for configuring our build pipelines.”
— Piotr Jagielski, VP of Engineering, Gradle Build Tool
Gradle, Inc. is the company behind the Gradle Build Tool, which is one of the most popular open-source build automation tools, and Gradle Enterprise is the leading solution platform for improving developer productivity. Headquartered in San Francisco, California, Gradle employs ~200 people in 30 countries to build the open-source tool and enterprise platform that serves millions of developers worldwide.
Gradle has over 100 engineers working on two main codebases – Gradle Build Tool and Gradle Enterprise. Each one includes millions of lines of code. The company uses TeamCity for both code bases, but this case study will focus on the Gradle Build Tool development.
For the past 10 years, the Gradle Build Tool team has been relying on TeamCity for its CI/CD process. During that time, the team has never missed a TeamCity update. Regular updates allow the team to always have the newest, most feature-rich version of the product.
The Gradle Build Tool team uses Git and GitHub as their version control system. They write code in Java, Kotlin, and Groovy. Naturally, the team uses their own products, Gradle Build Tool for build automation and Gradle Enterprise for build acceleration and failure analytics. They rely on TeamCity as their CI solution to run anything they don’t want to run locally: build, deploy, cron jobs, agent provisioning, and more.
Gradle Build Tool has a comprehensive test suite that verifies the product works correctly across different operating systems, versions of Java, and other components. The complete “release-ready” build encompasses over 150k tests. Additionally, changes must successfully pass the performance suite before they can be integrated into the main branch. This requires a complex CI setup with hundreds of build agents.
Gradle uses Windows, Linux, and macOS build agents. About half of the agents are bare metal machines. The team also makes use of Amazon EC2 spot agents. This helps Gradle maintain build speed even when demand is high.
Thanks to TeamCity’s built-in spot instance support, the team never struggles with agents disconnecting. TeamCity automatically restarts builds if a spot instance is gone.
The TeamCity setup for Gradle Build Tool is publicly available and visible to anyone. The main (master
) branch consists of around 500 build configurations set up in a very sophisticated chain. As Piotr notes, “At Gradle, we use build chains quite extensively. In fact, we cannot live without them.”
Due to the chain complexity, Gradle relies on the Kotlin DSL for configuring their pipelines.
Gradle has 200 fixed build agents and 200 additional elastic EC2 agents to manage spikes connected via cloud profiles. The total time to run builds per week stands at 283 days (or 6,792 hours).
At the same time, about 1,636 days per month are being optimized by TeamCity. Smart algorithms allow TeamCity to decide if the build should not be executed due to another build having already run on the same repository commit.
Gradle uses build chains, and TeamCity helps the team save a significant amount of build time by reusing builds.
Thanks to TeamCity’s ability to provide Prometheus-compatible server metrics, the team can automatically export the build data to Grafana for further visualization and analysis. The team tracks such metrics as queued build waiting time, TeamCity agent usage, queue length, and the number of connected agents.
Based on this, the team can decide whether they need to buy more build agents if the usage gets too high.
Providing feedback as fast as possible is at the core of any CI solution. Monitoring the feedback time helps Gradle’s developer productivity team understand how long engineers need to wait before their pull requests get merged.
To track that, Gradle Build Tool keeps an eye on the Trigger
build – a composite build that aggregates results from several other builds combined by snapshot dependencies and presents them in a single place.
The feedback time varies depending on the complexity of the pull request. Simpler PRs get feedback in 10 minutes, whereas in more complex cases, it might take up to an hour. Since Gradle is open source, all the pull requests coming from the GitHub community go through the same build process.
Due to cost considerations, not every commit triggers builds. Instead, developers can easily trigger a build by commenting on the related pull request, and a bot will call the TeamCity API to trigger it.
A few branches, like master
and release
, are treated differently. Any push on these branches triggers a TeamCity build.
The Kotlin DSL makes a huge difference in helping the Gradle team manage the complexity of build dependencies. By storing project configuration as code, the team can efficiently replicate build logic across projects, consistently apply updates to multiple configurations, and systematically manage its pipelines.
With the help of the Kotlin DSL, the team can generate and test build dependencies with much less effort. The team uses the Kotlin DSL with the web UI editing disabled for almost all its projects.
TeamCity accelerates the whole CI/CD process for Gradle. With TeamCity’s help, Gradle Build Tool successfully runs 30,000 green builds per day while keeping the failure rate under control.
TeamCity is a one-stop shop for the Gradle team because of its rich functionality. It has all the features other tools have, plus many of the ones they don’t have.
The Kotlin DSL gives Gradle Build Tool an unprecedented level of flexibility, which allows the team to quickly create a complex build chain and test it. The team also runs tests on the Kotlin DSL configuration files. Gradle’s TeamCity configuration is open-source and available on GitHub.
Finally, the Gradle Build Tool team values TeamCity’s reliability. The team upgrades to each new version of TeamCity as soon as it becomes available and rarely encounters any issues. TeamCity manages over 400 build agents, yet the team almost never has to troubleshoot any connection or build agent issues.
The Gradle Build Tool team is committed to ensuring their infrastructure scales as the team grows. Even if the team size doubles in the future, Gradle aims to maintain the current feedback time or reduce it yet further.
The Gradle Build Tool team plans to achieve that ambitious goal by fully embracing Developer Productivity Engineering practices, increasing automation to eliminate human intervention, and adding more build agents into the whole CI/CD process.
Ivan Babiankou, Staff Software Engineer, Picnic
We were looking for a managed solution for all our CI use cases. Additionally to that, we needed self-hosted agents to control which software we’re running, and which exact tooling is in use. TeamCity Cloud with self-hosted agents provided a tailor-made solution that our team of more than 300 engineers happily uses and that pushes our productivity to the next level.
Phillip Peterson, Senior Release Engineer
We did have a product that we had used in-house for a long time. We looked to switch to a different competitor, and neither one of those worked out. So then some colleagues of ours who came from another game company said, ‘We used to use this thing called TeamCity.’ We looked into it, and understood that TeamCity solved a lot of our problems.
Yuri Trufanov, Executive Technical Director of Technology Platform
We ended up with a hybrid cloud solution that included TeamCity Cloud Profiles and AWS. In addition to that, we had on-premises computers for build agents. This combination allowed us to accommodate any number of builds throughout the day, while also providing a baseline agent count for the off-hours. So we could run whatever we wanted wherever we wanted.