Automated Testing for CI/CD

Automated testing is one of the cornerstones of continuous integration and deployment. By building continuous testing into your DevOps pipelines, you can dramatically improve the quality of your software.

Continuous integration and deployment (CI/CD) is all about delivering frequent, incremental changes so you can get regular feedback on your product or service. However, delivering faster and more often shouldn’t diminish the quality of the product. After all, your users expect stable and working software, even when they’re clamoring for the next shiny thing.

That’s why a reliable and thorough automated testing process that gives you confidence in your latest build is essential to your CI/CD practices.

In this article:

Why should CI/CD testing be automated?

Testing is essential to ensuring the quality of software and has long formed part of software development practices.

In a waterfall context, the manual testing or QA stage takes place after the code is developed and integrated.

💡NB: The waterfall strategy, often referred to as the waterfall model, is a linear and sequential approach to software development and project management. It is one of the earliest models used for software development and is characterized by a structured and systematic flow through distinct phases.

One downside of this is that you can’t check that your code works as intended until long after you’ve written it. By that time, a lot more code may have been built on top of it, making it much harder to fix any problems. In turn, this slows the delivery of new features and bug fixes.

By contrast, an Agile approach favors short, iterative development cycles so you can release more frequently, get feedback from your users, and make informed decisions on what to build next. CI/CD supports this way of working by automating the steps between writing your code and releasing it to users, making the whole process more reliable and efficient.

To succeed with this DevOps methodology, you need to commit, build, and test your code changes regularly – ideally several times a day. However, even in a small team, performing a full set of manual tests once a day or more is not realistic. This is why automated testing is an essential part of any CI/CD pipeline.

Although writing automated tests requires an upfront investment of time, the work soon pays for itself, particularly as you start committing and deploying changes more frequently. Investing in automated testing provides several key benefits, including:

  • Each code change is checked to ensure it works as expected and has not introduced new bugs.
  • Feedback is delivered faster than when performing the equivalent tests manually.
  • Fixing bugs soon after they have been introduced tends to be more efficient, as the changes are fresh in your mind, and nothing else has been built on top of them.
  • Test results are more reliable than when testing manually, as asking anyone to perform the same task repeatedly inevitably results in mistakes and inconsistencies.
  • Automated tests can be run in parallel, so you can scale up your quality assurance (as far as your infrastructure allows) when time is of the essence.

While test automation cuts out a lot of boring, repetitive tasks, it doesn't make your QA engineering team redundant. In addition to defining and prioritizing related cases, the QA team should be involved in writing automated tests, often in collaboration with developers. QA engineers are also needed for the parts of testing that cannot be automated, which we’ll discuss later.

Where does testing fit into the CI/CD process?

Automated testing takes place at multiple stages throughout the pipeline.

CI/CD and QA automation is all about tight feedback loops that allow your team to discover problems as early as possible.

It’s much easier to fix an issue soon after it’s been introduced, as this helps to avoid more code being written on top of a bad foundation. It’s also more efficient for your team to make changes before they move on to the next thing and forget the context.

Many automated build testing tools support integration with CI/CD tools, so you can feed the test data into the pipeline and conduct testing in stages, with results provided after each step. Depending on your CI tool, you can also choose whether to move a build to the next stage based on the outcome of the tests in the previous step.

To get the most out of your pipeline through automated testing, it generally makes sense to order your build tests so that the fastest ones run first. This gives you feedback sooner and makes for more efficient use of test environments, as you can ensure initial tests have passed before running longer, more involved ones.

When considering how to prioritize both creation and running of automated tests, it’s helpful to think in terms of the testing pyramid.

Building a testing pyramid

The testing pyramid is a tool for prioritizing automated testing in a CI/CD pipeline, both in terms of the relative number of tests and the order in which they are performed.

Originally defined by Mike Cohn, the testing pyramid shows unit tests at the bottom, service tests in the middle, and UI tests at the top.

The testing pyramid consists of the following steps:

  • Start with a strong foundation of automated unit tests that are quick and simple to run.
  • Then, move on to tests that are more complex to write and take longer to run.
  • Finish with a small number of the most complex checks.

What types of CI/CD testing should you consider? Let’s explore the options.

Unit tests

Unit tests rightly form the basis of the testing pyramid. These tests are designed to ensure your code works as you expect by addressing the smallest possible unit of behavior. For example, if you were building a weather app, converting values from degrees Celsius to Fahrenheit might form part of a larger feature. You can use unit tests to check that the temperature conversion function returns the expected results for a range of values. Each time you change a related piece of code, you can use these unit tests to confirm this particular aspect works as intended without building and running the app each time.

For teams that have decided to invest in writing unit tests, developers typically take responsibility for adding them as they write the related code. Test-driven development (TDD) enshrines this process (as we’ll discuss below), but TDD is not a requirement for writing unit tests.

Another approach is to establish that unit tests form part of the definition of done for each development task and verify that these tests are in place when performing code reviews or by using code coverage reports.

If you’re working on an existing system and haven’t previously invested in unit testing, writing unit tests for your entire codebase from scratch can feel like an insurmountable barrier. Although wide coverage with unit tests is recommended, you can start with whatever you have and add to it over time.

If your code does not currently have good unit test coverage, consider building it up by establishing with your team to add unit tests to any piece of code you touch. This strategy ensures all new code is covered and prioritizes existing code based on what you interact with the most.

Integration tests

With integration tests, you ensure that interactions between multiple parts of your software, such as between application code and a database or calls to an API, work as expected.

It can be helpful to subdivide integration tests into broad and narrow. With the narrow approach, the interaction with another module is tested using a test double rather than the actual module. Broad integration tests use the actual component or service. To return to the example of a weather app, a broad integration test might fetch forecast data from an API, while a narrow test would use mocked data.

Depending on the complexity of your project and the number of internal and external services involved, you may want to start with a layer of narrow integration tests. These will run more quickly (and provide faster feedback) than broad integration tests, as they don’t require other parts of the system to be available.

If the narrow tests are completed successfully, you can then run a set of broad integration tests. As these tests take longer to run and involve more effort to maintain, you might want to limit them to higher-priority areas of your product or service.

End-to-end tests

Also known as full-stack tests, end-to-end tests look at the entire application. They are typically used to validate business use cases, such as whether a user can create an account or complete a transaction.

While automated end-to-end tests can be run through a GUI, they don’t have to be; an API call can also exercise multiple parts of the system (although APIs can also be checked with integration tests).

The testing pyramid recommends having a smaller number of these tests, not only because they take longer to run but also because they tend to be brittle. Any change to the user interface can break them, resulting in unhelpful noise in your build-test results and additional time required to update them. It therefore pays to design end-to-end tests carefully, with an understanding of what has already been covered by lower-level testing, so that they provide the greatest value.

Behavior-driven development

Behavior-driven development (BDD) is a collaborative approach to software development that bridges the communication gap between developers, testers, and business stakeholders. It extends TDD by emphasizing the behavior of the software rather than its implementation.

BDD can provide a useful strategy for developing both integration and end-to-end tests. Some of its key aspects include:

  • Focus on behavior: BDD focuses on specifying the behavior of a system through examples in plain language. These examples are often written in a Given-When-Then format to describe the initial state, the action, and the expected outcome.
  • Collaboration: BDD encourages collaboration among developers, testers, and business stakeholders to ensure a shared understanding of requirements. This collaborative process helps to clarify expectations and reduce misunderstandings.
  • Executable specifications: BDD scenarios are written in a way that they can be automatically executed as tests. Tools like Cucumber, SpecFlow, and JBehave are commonly used to write these scenarios in a human-readable format that can be linked to automated tests.
  • Documentation: The scenarios serve as both documentation and tests, providing a clear, living specification of the system's behavior. This makes it easier to understand the system and its requirements.
  • Iterative development: BDD supports iterative and incremental development by encouraging continuous feedback and improvement. Each iteration involves writing and implementing new scenarios, ensuring that the software evolves in alignment with business needs.

Performance tests

Although the testing pyramid makes no reference to performance tests, they are worth considering, particularly for products where stability and speed are key requirements.

Under the general heading of performance tests falls a range of testing strategies designed to check how your software will behave in a live environment:

  • Load testing checks how the system behaves when demand increases.
  • Stress testing deliberately exceeds the expected usage.
  • Soak (or endurance) testing measures performance under a continuous high load.

With these types of testing, the aim is not just to confirm that the software will cope with the defined parameters, but also to test how it behaves when those parameters are exceeded, ideally failing gracefully rather than crashing in flames.

Test environments

Both performance and end-to-end tests require environments that are very similar to production and may require build-test data. For an automated testing regime to provide confidence, it’s important for tests to be run in the same way each time. That means ensuring your test environments remain consistent between runs and updating them to match production when changes are deployed there.

Managing these environments manually can be time-consuming. Automating the process of creating and tearing down pre-production environments with each new build will both save you time and ensure a consistent and reliable automated testing regime.

How does test-driven development support automated testing?

Test-driven development (TDD) is a development approach which originated in extreme programming (XP). With TDD, the first step is to write a list of test cases for the functionality that you want to add. You then take one test case at a time, write the (failing) test for it, and then write the code to make the test pass. Finally, you refactor the code as required before moving on to the next test case. This process can be summarized as “Red, green, refactor” or “Make it work; make it right”.

One of the main advantages of TDD is that it forces you to add automated tests for any new code you write. This means your test coverage is always growing, enabling rapid and regular feedback each time you change your code. Other benefits of test-driven development include:

  • Promoting an iterative approach. Once you have identified a list of tests, you work on one test case at a time, refining the remaining list of tests as you go.
  • Encouraging you to separate interface from implementation. Following the “red, green, refactor” workflow is central to this.
  • Receiving immediate feedback as you work. Not only do you validate that your latest change passes the test, but you’ll also check that no other tests now fail.
  • Maintaining well-documented code, as the test makes the intention clear. This, in turn, makes your code easier to maintain and speeds up onboarding for new team members.

TDD is an effective way to build up your automated test coverage to support your CI/CD process. That said, TDD is not a requirement for an effective DevOps strategy, and you can maintain high levels of automated test coverage without TDD.

Working with feedback

The purpose of running automated tests as part of your CI/CD practice is to get rapid feedback on the changes that you have just made. Listening and responding to that feedback is an essential part of the process. The following best practices will help you get the most out of your automated testing regime:

  • Make your test results easy to access. CI servers typically integrate with automated testing tools so you can review all the results in a dashboard or radiator display.
  • Integrate your test reporting with your messaging platform, such as Slack, so you get automated notifications about test results.
  • When a test fails, track down the cause as quickly as possible so you can start working on a fix before more code is built on top of it. CI tools that identify the area of the codebase that the test relates to and provide access to any data produced by it – such as stack traces, output values, and screenshots – can speed up this process.
  • Design each test to check one thing, and label your tests accurately so you can easily understand what has failed.

As ever, tools and practices are only part of the equation. A really good CI/CD automation practice requires a team culture that recognizes not just the value of automated CI/CD testing, but also the importance of responding to failed tests quickly in order to keep the software in a deployable state.

Continuous testing vs. automated testing

For many teams, the starting point of automated testing is a suite of unit tests that you can trigger manually or as part of a simple continuous integration pipeline. As your DevOps culture matures, you can start working your way up the testing pyramid by adding integration tests, end-to-end tests, security tests, performance tests, and more.

Continuous testing refers to the practice of running a full range of automated tests as part of a CI/CD pipeline. With continuous testing, each set of code changes is automatically put through a series of automated tests so that any bugs are discovered as quickly as possible.

The early stages of a continuous testing process can involve tests run in the IDE before changes are even committed. For later-stage tests, continuous testing typically requires test environments that are refreshed automatically as part of the pipeline.

A fully automated continuous testing process provides maximum confidence in your code changes while speeding up releases. By putting your software through a rigorous testing regime, you significantly reduce the risk of bugs. Running that process automatically and continuously not only helps you work more efficiently but also allows you to deploy urgent fixes quickly and confidently.

Although continuous testing takes time to implement, it’s a goal you can work towards incrementally as you automate other aspects of your CI/CD pipelines and build up your test coverage.

Do CI/CD and automated testing mark the end of manual testing?

A common misconception among those new to CI/CD is that automated testing negates the need for manual testing and for professional QA engineers.

While CI/CD automation frees up some time for QA team members, it does not make them redundant. Instead of spending time on repetitive tasks, QA engineers can focus on defining test cases, writing automated tests, and applying their creativity and ingenuity to exploratory testing.

Unlike automated build tests that are carefully scripted for execution by a computer, exploratory testing requires only a loose remit. The value of exploratory testing is in finding things that a planned, structured approach to testing might miss. Essentially, you’re looking for issues that you have not already considered and written a test case for.

When deciding which areas to explore, consider both new features and the areas of your system that would result in the most harm if something were to go wrong in production. To make efficient use of testers’ time, manual testing should only take place after all automated checks have passed.

Exploratory testing should not slide into manual, repetitive testing. The intention is not to conduct the same set of checks each time. When issues are discovered during exploratory testing, you need to both fix the problem and write one or more automated tests. That way, if the problem occurs again, it will be caught much earlier in the process.

Continuous improvement for test automation

Building a test suite is not something you do once and forget. Automated tests need to be maintained to ensure they remain relevant and useful. Just as you continually improve your code, you must also continually improve your tests.

Continuing to add automated tests for new features and feeding in findings from exploratory testing will keep your test suite effective and efficient. It’s also worth taking the time to see how your tests are performing and whether you can re-order or break down parts of your test process to deliver some feedback sooner.

CI tools can provide various metrics to help you optimize your pipeline, while flaky test indicators can flag up unreliable tests that may be giving you false confidence or concern.

But while metrics can help you improve your automated testing process, you should avoid the trap of thinking test coverage is a goal in itself. The real aim is to regularly deliver working software to your users. Automation serves that goal by providing rapid, reliable feedback and giving you confidence before deploying your software to production.

Final thoughts about automated testing for CI/CD

Test automation plays a central role in any CI/CD pipeline. Running tests automatically provides rapid and reliable feedback on your code changes. In turn, this makes development more efficient, as identifying bugs earlier makes them easier to fix.

It’s a good practice to order your automated tests based on how long they take to run. Unit tests should be run first, as these will provide the quickest feedback, followed by integration tests and, finally, end-to-end tests. If you don’t have any automated tests, unit tests are the best place to start. Test-driven development (TDD) is a proven development practice that can help you improve and maintain unit test coverage.

As your DevOps culture matures, you may want to move towards continuous testing. Part of this transition will involve automating the creation and maintenance of your test environments. When writing higher-level automated tests, consider prioritizing the areas that pose the greatest risk. This might require automated performance tests, such as load, stress, or soak tests. Manual exploratory testing is a good way to identify gaps in your test coverage so you can continue to improve your CI/CD process.

How TeamCity can help

TeamCity offers extensive support for test frameworks and a range of test automation features to help you get the most out of your CI/CD process.

Speed and reliability are essential for effective test automation, and TeamCity is optimized for both. In addition to providing detailed test reports to help you get to the root of problems fast, TeamCity automatically highlights flaky tests so you can ensure only valid failures are flagged. Intelligent test reordering and parallelization deliver results even faster, while the remote run feature provides feedback before you commit.

Because TeamCity offers integrations with issue trackers, IDEs, Slack, and other platforms, you can get notifications about failed tests wherever you’re working. Finally, full support for virtual machines and Docker containers allows you to automate the management of your test environments and implement continuous testing as part of your CI/CD process.

Start a free trial Get in touch