Industry: Grocery delivery

JetBrains products used: TeamCity

Organization Size: 10,000

Country: Netherlands

Picnic

Picnic is a grocery delivery app founded in the Netherlands in 2015. Thanks to its cutting-edge technology and talented team, Picnic is among the fastest growing European technology companies.

“We were looking for a managed solution for all our CI use cases. Additionally to that, we needed self-hosted agents to control which software we’re running, and which exact tooling is in use. TeamCity Cloud with self-hosted agents provided a tailor-made solution that our team of more than 300 engineers happily uses and that pushes our productivity to the next level.”

— Ivan Babiankou, Staff Software Engineer, Picnic

Company information

Please introduce yourself, including your name and job title.

My name is Ivan Babiankou, I’m a Staff Software Engineer at Picnic. I’m part of the Java Platform team. I was involved in choosing the successor of our previous CI solution and implementing the migration.

What does your company do? What are its main offerings?

Picnic revolutionizes the way people do grocery shopping. Having started from just one Dutch city, Picnic now serves more than 200 cities in three countries – Germany, France, and the Netherlands. Without using physical stores, Picnic relies on an app-only approach to make grocery shopping fast and simple for millions of users.

Picnic iconic electric vehicle
Delivered in iconic electric vehicles

In addition to the user-facing mobile application, we build the majority of other systems that are critical for the business, including an in-house solution for warehouse management. We also build all the tools to plan the routes for drivers, help them drive safely, optimize the routes for picking shopping orders in the warehouse, etc.

Currently, Picnic employs over 300 developers. The company’s tech stack includes GitHub, Java, Python, Swift, Kotlin, TypeScript, Angular, React, Android, iOS, MongoDB, and Spring Boot, among others.


Challenges before adopting TeamCity

Which CI/CD products did you use before and what led you to look for a new one?

We started looking at JetBrains TeamCity at the end of 2021 as we were outgrowing our previous CI solution. We experienced two major pain points: slow builds and long build queues. At peak times, some builds would stay in the queue for up to an hour before they would start running.

The main approach to fix this situation within our old solution was to bump up the number and size of the build agents substantially to reduce the total build time. However, this solution was not scalable as it did not address the core issue of build speed. We decided to look for a new solution that could do it better and faster.

As a replacement for our CI solution, we looked for a system that would support our whole tech stack and help us standardize the way we build and test our applications. We use Java and Python for backend applications development, GitHub as our VCS, as well as Kotlin, Swift, and React Native for mobile development.

What alternative solutions did you consider? What were your key criteria for a CI/CD tool?

We started with a long-list of 10 solution providers and reduced it to a shortlist of 3: CircleCI, TeamCity, and GitHub Actions. Our main goal was to minimize the queue times and improve the CI performance by reducing build time.

Another requirement was to have self-hosted agents. During the analysis, we realized that our old solution didn’t upgrade the tooling on agents very quickly. We were spending 2–5 minutes every single build just installing tools. Considering an average build takes around 9 minutes, it doesn’t make sense to spend another 3 minutes installing the tooling beforehand. That’s how we arrived at the idea of having self-hosted agents where we’d have the latest tooling, appropriate performance, and pretty much everything we needed to just start our builds.

The CI solution being in the cloud was also a prerequisite for us. At Picnic, we’re trying to manage as few systems as possible. With CI, we wanted to have a managed solution that we wouldn’t have to run ourselves. At the same time, self-hosted agents in the cloud would allow us to have control over which software we’re running and the exact tooling we’re using. It gives us the flexibility to control the whole build environment – the hardware that’s used and the exact tooling that’s pre-installed – while not having to manage the CI server.


Quantitative and CI/CD metrics

We have around 300 CI users, most of whom are active and commit quite intensively on almost 200 projects. These projects follow a fairly hierarchical structure in TeamCity, with over 120 VCS roots.

Next to that we have 40 build agents for a maximum of 40 concurrent builds. We run almost 1,100 builds a day, with the total build time per day getting close to 10,000 minutes. Agents need around 2 minutes to spin up, and builds are rarely queued longer than that.

Over a month, this all adds up to almost 300,000 minutes of build time.


Which part of your development process is supported by TeamCity?

We use TeamCity for all internal development. Once you open a PR in GitHub, the CI builds it, runs unit, integration, and component tests, and gives you feedback. That’s roughly where it stops. TeamCity is used up to the point where it produces a build artifact, which is then pushed to a repository for later deployment.


What, how, and where do you deploy to?

All the Linux and regular build agents are running in our cloud provider – AWS. For iOS applications, we have a few Mac minis that are hooked to the network in our office and connected to TeamCity. In fact, the iOS team already used TeamCity on-site before we migrated to TeamCity Cloud.

We run agents with on-demand instances. Spot instances are still a very attractive option on the horizon for future improvement.

We prepare our own images and use TeamCity Cloud Profiles to run self-hosted agents. These are, in fact, EC2 machines in AWS. We use TeamCity and Packer to build AWS images of the agents.

Then, we use those agents together with launch templates and cloud profiles in TeamCity to start build agents on demand. At the moment, they’re short-lived and only used for a single build.

We noticed that the queue time was taking 2 minutes, 1.5 minutes of which was waiting for the build agent to be provisioned in AWS. Having long-lived reusable agents would help us eliminate this problem. With that, we would start looking at spot instances.


Do you use configuration as code or the Kotlin DSL? If so, what’s your approach to it?

Our policy is to disable editing in TeamCity’s UI. One of our goals is to standardize pipelines. We achieve that by using configuration as code.

All build configurations are stored in their respective repositories as Kotlin code. We’ve built our own DSL on top of the TeamCity Kotlin DSL, which allows us to define pipelines using 20 lines of code or fewer.

Picnic code snippet
This is the simplest configuration for a basic pipeline that includes a regular build of the project, a separate build for additional PR verifications, and a build for scheduled static analysis.

For example, you’d get the main build that builds your Java application. It will run all the tests and component tests. If everything is successful, it will upload the build artifacts to the appropriate repository.

Besides that, we have a separate job for static analysis of the code and one more for the nightly static analysis of the code to be used in SonarCloud. These are all part of a standard pipeline. For Java applications, that’s what you get from the get-go if you just define the basic things for your pipelines.

Some of our teams have a lot of component tests (BDD). We write those in Python, whereas the main app is written in Java. The app is packaged into a Docker image. We then spin up a Docker Compose environment on the build agent with real infrastructure (databases, etc.) while third-party services are mocked. Behave tests are run against the app running in Docker Compose.

In the Kotlin DSL provided by TeamCity, you can perform whatever tweaks you want to the build pipeline. You can change the steps, the triggers, and many other things.

By default, we give teams a golden standard of the pipelines the way the Platform team sees it. For example, we run the component tests as part of the build pipeline on the same agent, as part of the same job.

There are teams that have hundreds of component tests, which unfortunately are not yet parallelized. In those cases, we allow splitting the Behave component tests into test suites and configuring them as a build chain.

Picnic code snippet
This is a more advanced configuration with extensive component tests split across three parallel builds to speed up the pipeline.

So, the main build produces the artifact and executes unit tests and simple fast tests. If that succeeds, you can run as many parallel test suites as you need that would test with multiple instances on different agents.

A visual representation of the build chain generated by the configuration above.
This is a visual representation of the build chain generated by the configuration above.

What are your top gains and pains of using TeamCity?

Here are the things we love the most about using TeamCity:

Setup of default build agents. When we started looking at TeamCity, we noticed that the system uses compute-optimized agents with an SSD attached to it. This provides top performance compared to other CIs. Even though we don’t use the provided agents, this prompted us to use compute-optimized instances for our own agents as well, providing better performance.

We also love the performance monitor feature. One of the big struggles for actual teams who use those pipelines was: How do we decide whether we use a small, medium, or large instance? Checking out the performance monitor gives nice insights into that. That makes debugging simple.

A performance chart of the build.
The performance of this build is clearly limited by the CPU. Increasing the build agent size did not give a significant enough build speed-up, so we decided to stick with the current size.

As for the pains, having metrics across the projects that we’re building in TeamCity would be great, because at the moment TeamCity provides this information per project only. There’s no way for us to see the metrics across all Java projects, for example.

Also, the Kotlin DSL involves quite a steep learning curve, especially for engineers who aren’t familiar with Java.


Future plans

Now that we have migrated all projects to TeamCity, we’d love to experiment further with build agents, spot instances, and so on.

To do that in a nice way, we love to capture build metrics in a systematic way. All these metrics are available in TeamCity. What’s blocking us from moving forward is being able to see these metrics across all projects and see how they evolve over time.

Based on this information, we would be able to make decisions for further optimization: what to improve and focus on first, and how our tweaks improve or hurt performance. For now, we’re considering building an external service that would aggregate all of this information. If it were available out of the box, that would be much better!

We’re also going to be looking at optimization and trying to get rid of those extra minutes spent on the agent start-up time, either by having long-lived agents or by introducing a pool of agents that warm up in advance.

Another thing that we’re working on at Picnic is building more end-to-end tests. That’s going to be another project that runs in TeamCity. Once we have that, we’ll need to take a fresh look at whether to move our CI/CD entirely to TeamCity.

Similar Customer Stories

Brightify

Tadeas Kriz, CTO and CoFounder, Brightify

Our Code Reviews improved significantly and we’ve been able to leverage Space’s webhooks with TeamCity to build each reviewed branch and deploy it to our QA so the branch can be tested before being merged. It’s now also easier to track who’s out of the office.

Miquido

Piotr Polus, Frontend Tech Lead, Miquido

We’ve chosen JetBrains for three reasons: convenience of use, configurability, and the easy availability of plugins.

Tangunsoft

Wooseong Kim, APAC Channel Lead and Partnership Manager, Tangunsoft

JetBrains helps you write clean, professional, maintainable code of the highest quality.

More customer stories