It’s known that having automated tests as part of your build process improves the software quality and reduces the number of bugs.
Do you know if you need more unit tests? Or if your tests cover all possible branches of an if or switch statements? Or if your code coverage is decreasing over time? Especially after you join a team to work on an on-going project.
Code coverage helps to answer these questions. This post covers reporting code coverage using Maven’s jacoco-maven-plugin, a library that adds minimal overhead with a normal build.
Requirements
- Java 7+
- Maven 3.2+
- Overview of my previous post, Splitting Unit and Integration Tests using Maven and Surefire plugin because this post uses the same source code.
Sample Application
The example application has two unit test classes, DemoControllerTest
andDefaultSomeBusinessServiceTest
, and two integration tests classes, DemoControllerIT
, ApplicationTests
; this is similar to those discussed in Splitting Unit and Integration Tests Using Maven and Surefire plugin section.
Configuring jacoco-maven-plugin and Coverage Threshold
Warning: According to the JaCoCo documentation, do not set forkCount to0
or set
forkMode
to
never
as it would prevent executing the tests with the JaCoCo
javaagent
and no coverage would be recorded.
Let’s configure jacoco-maven-plugin in pom.xml
:
The prepare-agent
goal sets up the property argLine
(for most packaging types), pointing to the JaCoCo runtime agent. You can also pass argLine
as a VM argument. Maven-surefire-plugin uses argLine
to set the JVM options to run the tests.
If you are explicitly setting argLine
, make sure it allows late replacements like:
This is so that the maven-surefire-plugin picks up changes made by other Maven plugins such as jacoco-maven-plugin.
The JaCoCo Java agent will collect coverage information when maven-surefire-plugin runs the tests. It will write it to destFile
property value, if set, or target/jacoco.exec
by default. Read more at https://www.eclemma.org/jacoco/trunk/doc/prepare-agent-mojo.html.
The report
goal creates code coverage reports for tests in HTML, XML, CSV formats. Stay tuned, I’ll cover uploading code coverage reports to SonarQube in another post. This goal reads the dataFile
property value, if set, or target/jacoco.exec
. And, it writes the resulting reports to outputDirectory
property value or target/site/jacoco
. Read more at https://www.eclemma.org/jacoco/trunk/doc/report-mojo.html.
The check
goal validates that the coverage rules (discussed later) are met. In case they are not, it interrupts and fails the build unless the haltOnFailure
property is set to false
. Read more at https://www.eclemma.org/jacoco/trunk/doc/check-mojo.html.
Running the Tests and Creating the Coverage Reports
Let’s build the application and analyze the Maven command output:
Right after the clean
phase completes, jacoco-maven-plugin’s prepare-agent
goal (bound to the Maven’s Build Default Lifecycle’s initialize
phase) sets the argLine
property pointing to the JaCoCo Java agent.
Unit and Integration tests ran separately as covered in a previous post.
Next, not included in this log output, the Maven artifact is built and repackaged.
After that, jacoco-maven-plugin’s coverage-report
goal (bound to the Maven’s Build Default Lifecycle’s post-integration-test
phase) generates HTML, XML and CSV reports.
Opening the HTML report at target/site/jacoco/index.html
results in:
Code coverage report for a successful build
Lastly, jacoco-maven-plugin’s check
goal (bound to the Maven’s Build Default Lifecycle’s verify
phase) checks the code coverage metrics are met.
JaCoCo Rules
Let’s take a closer look at the jacoco-maven-plugin’s coverage-check
rules configuration in pom.xml:
Setting the rule element to CLASS
means every Java class from the application would need to meet each counter limit for the build to pass.
In this example, there is only one limit, a LINE
counter that needs coverage of at least 80 percent. I’ll cover JaCoCo Counters later.
Other JaCoCo rules you could use are:
BUNDLE | The set of counter limits would have to be met at the application as a whole |
PACKAGE | The set of counter limits would have to be met for all packages (e.g. com.asimio.demo , com.asimio.demo.rest , etc.) |
CLASS | The set of counter limits would have to be met for every Java class |
SOURCEFILE | |
METHOD | The set of counter limits would have to be met for every class method |
Notice that as you move down in the JaCoCo rules table, the check
goal becomes more constraining.
As an example, if you remove com.asimio.demo.Application
from the excludes
sections, the build fails because the LINE
counter doesn’t reach 80 percent for said class:
JaCoCo Counters
Even though I only used the LINE
counter when covering JaCoCo Rules, there are a handful of other counters you could include in the limits
set in the jacoco-maven-plugin configuration.
INSTRUCTION | The amount of code that can be executed or missed |
BRANCH | The total number of branches (if and switch statements) in a method that can be executed or missed.
|
CYCLOMATIC COMPLEXITY | https://en.wikipedia.org/wiki/Cyclomatic_complexity |
LINE | Executed when at least one instruction that is assigned to this line has been executed.
|
METHOD | Executed when at least one instruction has been executed |
CLASS | Executed when at least one of its methods has been executed |
Notice that as you move down in the JaCoCo counters table, the check
goal becomes less constraining.
Examples of associating a counter to a rule are:
A counter value
is one of:
TOTALCOUNT | COVEREDCOUNT | MISSEDCOUNT | COVEREDRATIO | MISSEDRATIO |
and a numeric minimum
or maximum
.
Conclusion
Although not a silver bullet, code coverage helps to measure what percentage of code is executed when running the test suites. And thus, it helps to reduce the number of bugs and improve the software release quality.
Keeping a certain threshold might get difficult over time as a development team adds edge cases or implement defensive programming.
JaCoCo adds minimal overhead to the build process. Jacoco-maven-plugin’s prepare-agent
goal, bound to the initialize
phase, sets the agent responsible for instrumenting the Java code before maven-surefire-plugin runs. Coverage-report
goal is bound to the post-integration-test
phase. And coverage-report
goal is bound to the verify
phase. Read more at Maven’s Build Default Lifecycle. This means, unlike other libraries, JaCoCo doesn’t need to run the tests twice.
Thanks for reading and sharing. If you found this post helpful and would like to receive updates when content like this gets published, sign up to the newsletter.
Source Code
Accompanying source code for this blog post can be found on BitBucket.
References
- https://www.eclemma.org/jacoco/trunk/doc/maven.html
- https://www.eclemma.org/jacoco/trunk/doc/prepare-agent-mojo.html
- https://www.eclemma.org/jacoco/trunk/doc/report-mojo.html
- https://www.eclemma.org/jacoco/trunk/doc/check-mojo.html
- https://www.eclemma.org/jacoco/trunk/doc/counters.html
- http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html