Running Your Load Tests With PassFail Criteria - A Guide

PassFail criteria for tests metrics and KPIs are an inherent part of performance tests, since they ensure your product is performing according to the requirements. For example, “The average response time for the users needs to be no more than 600 milliseconds,” or “I expect the average latency of my servers to be no more than 200 milliseconds.”

How can you check if a specific test breached your thresholds? In this blog post, I will elaborate about two ways: by using Taurus in PassFail mode, and by using Jenkins with Taurus or JMeter.

You can also test manually by checking if the metrics you wanted to measure reached values that are above your thresholds. But this is a thorough but very slow process, and it can be a real pain! You can also use the Apache JMeter™ AutoStop Listener (see this blog post).

Running Load Tests With PassFail Criteria in Taurus

Taurus is an open-source test automation framework. It uses simple YAML syntax to run open-source load and performance tests, for tools like JMeter, Selenium, Gatling, and Locust.

Taurus provides a module called PassFail, which helps you automate the threshold testing process. In Taurus, you can set pass/fail criteria in a YML file, and add this YML file to your Taurus execution. The Taurus pass/fail module supports many different load metrics, such as Response Time, Latency, Error Rate, and more. You can also assign the criteria to all the labels in your test, or only to specific labels. Learn about all the PassFail options in Taurus from this link.

Let's look at an example. Here is a Taurus execution configuration:

 --- execution: - concurrency: 10 hold-for: 2m ramp-up: 40s scenario: Thread Group scenarios: Thread Group: requests: - label: blazedemo method: GET url: http://blazedemo.com/ - body: fromPort: Paris toPort: Buenos Aires label: reserve method: POST url: http://blazedemo.com/reserve.php 

Save this file as execution.yml. Now, let's add pass/fail criteria.

 --- services: - module: passfail criteria: - avg-rt>10ms for 7s, stop as failed - hits of reserve >10 for 13s, continue as failed 

The above configuration sets two pass/fail criteria:

• If the average response time (for all the labels in the test) goes over 10 milliseconds for 7 seconds in a row, stop the test as failed.
• If the number of hits (number of responses) for a specific label called ‘reserve’ is more than 10 for 13 seconds in a row, continue with the test, but mark it as failed.

Save this pass/fail configuration in a different YML file and called it passfail_config.yml.

Now that we have finished with the configuration, let’s run the test.

Taurus tests are run directly from the terminal, with the ‘bzt’ command. When we have multiple YML files that we want to use in our test we simply add them one after the other:

 Bzt execution.yml passfail_config.yml 

After a few seconds, the Taurus console report will be displayed automatically.

As the average response time for the test was more than 10 milliseconds for 1 second, we get an alert. The alert appears even though we haven’t reached the threshold yet. The purpose is to let us know we might have a problem soon.

After 7 seconds from the beginning of the test, the average response time was more than 10 milliseconds for 7 seconds in a row. This means that our first criterion was met, so the test will be stopped and marked as failed as expected.

Since the first condition was met, we didn't have a chance to check out the second condition. The two are unrelated and the test checks each one separately.

By using the Taurus passfail module, you don’t have to manually examine tests results anymore. If one of your tests met the criteria you set, it will fail the test automatically and you will get a notification for this in the report and logs.

Taurus is a very good tool for test automation. Now let’s see how we can schedule Taurus tests by using open-source CI tool Jenkins.

Running Load Tests With PassFail Criteria in Jenkins

We recommend running load tests as part of your continuous integration process. This ensures you are constantly ensuring your code changes or product updates don’t crash your website or app. One of the most popular open-source tools for continuous integration is Jenkins.

Both JMeter and Taurus can be integrated into Jenkins, and Jenkins can then schedule automated tests. We will explain how to run Taurus with its PassFail configuration with Jenkins. You will see it is quite easy. To learn how to run Jenkins with JMeter, click here.

As Taurus is a command line tool, you can simply use the ‘Execute Shell’ build step in your Jenkins project, to run the same execution command we ran locally:

However, you should add  --option=modules.console.disable=true, as we don’t need the Taurus console report when running from Jenkins.

After saving our Jenkins project, we can go ahead and build it by clicking on the ‘Build Now’ button (or use a cron job for a scheduled execution).

Once the build has started, open the Console Output:

In the Console Output, you can see the results of your Taurus load test, and in this case, the failure of the test due to the pass/fail criteria being met.

As the test failed, it will fail the Jenkins build.

To schedule jobs in Jenkins, you can use the ‘Build Triggers’ section in your Jenkins project. There are a number of ways to schedule jobs: periodically by using CRON (see more details here), by clicking "Build when a change is pushed to GitHub," and more. This way, you can easily integrate your performance tests into your CI process, and see which builds failed due to a pass/fail criteria being met.

Congratulations! You now know how to run tests with thresholds by using Taurus and Jenkins.