Automation Testing in CI/CD Pipelines: Types and Stages

In a previous article, we discussed the various use cases wherein agile teams would automate their test cases. One of the cases was when teams wanted to integrate tests with every build and have a continuous integration as a part of the build process.

In this article, we will address integrating tests in a continuous integration/continuous delivery platform.

Let’s start with the basics first.

What Is Automation Testing?

Software testing refers to the execution of tests according to a set of test cases and comparing the results of actual and predicted outcomes. Several steps and testing techniques are followed in the process. Testing is essential to ensuring the quality of the product. This process is typically done manually by a team of testers. However, in automation testing, the process is automated using software tools and technology. Here, rather than resorting to a manual effort, scripts are created and test cases are run automatically. Automation testing eradicates many problems such as human error, coverage area, etc. It also saves time and enhances the ease of conducting tests with increased efficiency and effectiveness.

Automation Testing in CI/CD Pipelines

Automation testing is a core part of CI/CD pipelines because tests that run fast can provide early feedback to the developer. A problem or a bug that is identified early has the possibility to be rectified early. Therefore, the released product will be more accurate and bug-free. This enhances the overall quality of the product, which garners customer satisfaction. To put it succinctly, the advantages of automation testing in the CI/CD pipeline are as follows:

• Aligns with the CI/CD idea of "build fast, fail fast"
• Reduces manual effort, which saves time and the possibility of error.
• Test results are more accurate with the increased number of test cases and can and cover a wider area.
• Get immediate feedback on any problem
• Multiple test results can be generated and compared to ensure quality and consistency

Types of Automation Testing Used in CI/CD Pipelines

• Unit testing: It is a low-level test undertaken after a module has been coded and reviewed. Test cases are designed to test individual components. The purpose is to make sure each component works the way it is supposed to under any circumstance.
• Integration testing: Integration refers to testing the interaction of components within the application. This testing is carried out after all the modules have been unit tested. The primary objective is to test the module interfaces and check that there are no errors in parameter passing when one module invokes the functionality of another module.
• System testing: Tests are designed to validate a fully-developed system and make sure it adheres to the requirement specification document. Typically, at this stage, software is ready for use by potential users. These kinds of system tests are called alpha (carried out by a team of developers within the organization), beta (performed by a selected group of users/customers), or acceptance testing (performed by users/customers to determine acceptance of delivery of the system).

What Are Continuous Integration and Continuous Delivery?

In simple terms, continuous integration allows development teams to integrate their code into a shared repository. This helps maintain code quality and identify potential issues with the local version of the code at an early stage.

Continuous delivery is often called "Continuous Deployment" as well. Everything that is continuously merged by the development team is continuously deployed to the live environment.

Since most developers work in parallel, continuously integrating their code into one repository would mean that the master branch is continuously updated with new features. To ensure that there is no compromise in the code quality with so many changes happening rapidly, testing must happen at the same pace.

It should be no surprise that manual testing in this environment would not be the best approach to achieve this. Automated testing is the key to successful testing in a CI/CD pipeline.

9 Continuous Delivery Stages

1. Develop: The developer builds the code according to the project requirements or the feature requests.
2. Writing tests: Once the code is written, tests need to be written. At this point, these tests are usually unit tests written by the developers.
3. Local Testing: This is then locally tested to check whether all the tests pass and to ensure the code does not break. Often, a percentage is set as the pass rate that the tests running need to meet.
4. Rebase and Resolve conflict: In an actual development scenario, there will be multiple people merging their code. Developers need to make sure that their branch is updated at all times. Updating the branch with the latest merged code is called "rebasing." Once it’s rebased, there will likely be some conflicts that need to be resolved. After that, the tests are run again against the rebased code.
5. Commit: Once the tests have passed, the code is then ready to be committed with all the changes.
6. Build: The source code developed is then combined to build a deployment artifact that can be run on an instance, like the server if the environment is on-premises. This code is now ready to be deployed to different testing environments.
7. UAT: The code is then deployed to a test server where testers start to test the feature. These tests can be automated as well as manual.
8. Merge: If the commit that’s under testing is approved by the testers, this is then merged into the master branch.
9. Production deployment: Once the code is merged, it is then deployed to production.

The above process needs to be done with every build coded by the developers.

Where Does Automation Testing Fall in This CI/CD Pipeline?

Automated testing ideally happens once the build stage has been completed and the code can be deployed. Unit tests, UI tests, and integration tests can all be run at this stage. These tests help in ensuring that the code meets a standard of quality.

This phase can last from a few minutes to a couple of hours depending on how the automation is architected.

Tests can be run in parallel to execute them more quickly. If a code fails during the test phase, the build can be rejected without further investing in any manual testing time.

Tools Used for CI/CD

• Jenkins: Jenkins is an open-source tool that is used for continuous integration. It’s free to use and jobs can be configured both by the interface as well as scripts.
• Travis CI: This tool is free of cost for open-source projects, hosted by GitHub.
• Gitlab: Gitlab is a version control tool that has its own cloud-based CI methodology. It is supported on multiple platforms which have both free and paid versions.
• Bamboo: Bamboo is a CI tool by Jira. If your organization uses Jira, then it would be beneficial to check this tool out. It supports automated merging on approval of tickets as well.

Best Practices for CI/CD Pipeline to Make the Best Out of Test Automation

• Incremental changes: It is always advisable to follow a feature-by-feature approach. If the feature is really big, it is good to break it down into smaller and quicker-to-test features. This is important in terms of automation because if there is an issue, it is easier to figure out the root cause. If your commit is too big, isolating the cause of an issue would be a tough task.
• Identify what can be automated: It is very common for teams to dive fast and say, "Let’s automate everything," but this is a common mistake. We must know the purpose of automation and identify the test cases that should be automated.
• Parallel Tests: Tests should be run in parallel to make testing more efficient and timely. It can greatly reduce the time taken to run tests and thus give the results much faster. But it’s not sufficient to just execute these tests in parallel; it is also important to scale the server size where the tests are running in order to actually make them faster.

Conclusion

Automating tests is an important part of the successful deployment of projects while maintaining a standard of quality. Ensuring tests are run at every stage gives good transparency on the quality of the code. Bugs can be discovered at an early stage, and any delays that might be caused by them can be addressed promptly. Having a CI/CD pipeline with integrated tests helps in speeding up the testing and deployment process.