Extra Micrometer Practices With Quarkus

Metrics emitted from applications might contain parameters (i.e., tags or labels) for measuring a specific metric. Micrometer provides a facade over the instrumentation clients for many widespread monitoring systems like Atlas, Datadog, Graphite, Ganglia, Influx, JMX, and Prometheus. This article's scope explores some extra practices when combining Micrometer and Quarkus to tailor those metrics or give you some ideas of what you could extra measure.

For simplicity and given Prometheus popularity, the Micrometer Prometheus registry will be used as an example to demonstrate most of the use cases.

Prerequisites

• You can apply the concepts from this article by generating your project via https://code.quarkus.io/.
• JDK 11 installed with JAVA_HOME configured appropriately 
• If interested in concepts about working with Quarkus and Micrometer, please take a look at https://quarkus.io/guides/micrometer.
• To further observe the metrics in Prometheus and create Grafana panels, please consider provisioning those tools. However, you can run the code locally as well.

Global Tags Setup

By simply adding micrometer-registry-prometheus extension as a dependency to your Quarkus code, your application code will benefit from some Out Of The Box metrics, accessible on localhost via localhost:8080/q/metrics.

Suppose your application is deployed across multiple regions and the business logic inside it needs to consider specific details (like connecting to third-party services that offer data in different formats). In that case, you can add your global tags to help further inspect performance statistics per region.  In the example below, the global metrics should be all properties prefixed with  global and defined separately in a class.

@StaticInitSafe
@ConfigMapping(prefix = "global")
interface GlobalTagsConfig {

   String PROFILE = "profile";
   String REGION = "region";

   String region();
   String customMeter();
}

Furthermore, you can define those configuration properties in  application.properties file with a default value (for local development work), but the expectation is for those to be configured via environment variables passed from Docker or Kubernetes.

xxxxxxxxxx

global.region=${REGION:CEE}

global.custom-meter=${CUSTOM_METER:''}

The last step was to instruct the Micrometer extension about the CustomConfigurationand to use its  MeterFilter CDI beans when initializing MeterRegistry instances: 

@Singleton
public class CustomConfiguration {

    @Inject
    GlobalTagsConfig tagsConfig;

    @Produces
    @Singleton
    public MeterFilter configureAllRegistries() {
        return MeterFilter.commonTags(Arrays.asList(
                Tag.of(GlobalTagsConfig.PROFILE, ProfileManager.getActiveProfile()),
                Tag.of(GlobalTagsConfig.REGION, tagsConfig.region())));
    }

    @Produces
    @Singleton
    public MeterFilter configureMeterFromEnvironment() {
        return new MeterFilter() {
            @Override
            public Meter.Id map(Meter.Id id) {
                String prefix = tagsConfig.customMeter();
                if(id.getName().startsWith(prefix)) {
                    return id.withName(prefix +"." + id.getName())
                            .withTag(new ImmutableTag(prefix+".tag", "value"));
                }
                return id;
            }
        };
    }
}

A transform filter is used to add a name prefix and an additional tag conditionally to meters, starting with the runtime environment's name.

These metrics can help generate insights about how a particular part of your application performed for a region. In Grafana, you can display a graph of database calls via a specific function (tagged by   persistency_calls_find_total ) per region by the query:

sum by(region)(rate(persistency_calls_find_total[5m]))

Measuring Database Calls 

If you are trying to measure the database calls made from within your custom service implementation, where you may have added some extra logic, you may want to do the following:

• Add  @Counted annotation on top of the methods to check the number of invocations. 
• Add  @Timed annotation on top of the expected methods to run long-running operations and deserve a closer inspection.

In the example below, @Countedwas added to trace the number of invocations done:

    @Transactional
    @Counted(value = "persistency.calls.add", extraTags = {"db", "language", "db", "content"})
    public void createMessage(MessageDTO dto) {
        Message message = new Message();
        message.setLocale(dto.getLocale());
        message.setContent(dto.getContent());
        em.persist(message);
    }

This metric can further help in observing the distribution in a Grafana panel based on the following query:

sum(increase(persistency_calls_add_total[1m]))

In the case of complex database queries or where multiple filters are applied, adding  @Timed  would give insights into those behaviors under the workload. In the example below, filtering messages by language is expected to be a long-running task:

    @Timed(value = "persistency.calls.filter", longTask = true, extraTags = {"db", "language"})
    public List<Message> filterMessages(Locale locale) {
        TypedQuery<Message> query = em.createQuery(
                "SELECT m from Message m WHERE m.locale = :locale", Message.class).
                setParameter("locale", locale);
        return query.getResultList();
    }

Measuring Performance of Requests

By default, your HTTP requests are already timed and counted. You can find those metrics having the prefix http. Depending on the complexity of the logic from methods within your API, you may want to do the following:

• Add @Countedannotation with extra tags you define on top of the methods to check the number of invocations.
• Add @Timedannotation with extra tags on top of the methods expected to have long-running operations and deserve a closer inspection.
• Measure code performance with complex business (present either inside those methods or in separate classes). You might want to consider a dynamic tagging strategy for such a case.

Implementations with more business logic would need separate metrics with more tags. The goal in such cases is to reuse some metrics and decorate them with more tags. For example, the method below has additional metrics attached to it to measure the performance of each greeting handled in a given language:

    @GET
    @Path("find/{content}")
    @Produces(MediaType.TEXT_PLAIN)
    @Timed(value = "greetings.specific", longTask = true, extraTags = {URI, API_GREET})
    @Counted(value = "get.specific.requests", extraTags = {URI, API_GREET})
    public List<Message> findGreetings(@PathParam("content") String content) {
        AtomicReference<List<Message>> messages = new AtomicReference<>();
        if (!content.isEmpty()) {
            List<Message> greetings = messageService.findMessages(content);
            messages.set(greetings);
        }
        return messages.get();
    }

For the previous API request, the average greeting retrieved over time can be measured in a Grafana panel through the query:

x

avg(increase(greetings_specific_seconds_duration_sum[1m])) without(class, endpoint, pod,instance,job,namespace,method,service,exception,uri)

The above method can also get no result for the given content, and that case should be closely observed. For this extra case, a custom Counter  was implemented were firstly the existence of a  Counter with similar features is checked and if not, increment a new instance of  Counter with additional tags attached:

xxxxxxxxxx

public class DynamicTaggedCounter extends CommonMetricDetails {

    private final String tagName;

    public DynamicTaggedCounter(String identifier, String tagName, MeterRegistry registry) {

        super(identifier, registry);

        this.tagName = tagName;

    }

    public void increment(String tagValue) {

        Counter counter = registry.counter(identifier, tagName, tagValue);

        counter.getId().withTag(new ImmutableTag(tagName, tagValue));

        counter.increment();

    }

}

Note: Similar customization was applied for DynamicTaggedTimer.

In the  ExampleEndpointthe following changes were added:

@Path("/api/v1")
public class ExampleEndpoint {

    @Inject
    protected PrometheusMeterRegistry registry;

  
    private DynamicTaggedCounter dynamicTaggedCounter;

  
    @PostConstruct
    protected void init() {
        this.dynamicTaggedCounter = new DynamicTaggedCounter("another.requests.count", CUSTOM_API_GREET, registry);
    }
 
  
    @GET
    @Path("find/{content}")
    @Produces(MediaType.TEXT_PLAIN)
    @Timed(value = "greetings.specific", longTask = true, extraTags = {URI, API_GREET})
    @Counted(value = "get.specific.requests", extraTags = {URI, API_GREET})
    public List<Message> findGreetings(@PathParam("content") String content) {
        AtomicReference<List<Message>> messages = new AtomicReference<>();
        if (!content.isEmpty()) {
            List<Message> greetings = messageService.findMessages(content);
            messages.set(greetings);
        }
        if (messages.get().size() > 0) {
            dynamicTaggedCounter.increment(content);
        } else {
            dynamicTaggedCounter.increment(EMPTY);
        }
        return messages.get();
    }
}

Based on the above implementation, the ratio of empty results requests can be outlined in Grafana via the query:

xxxxxxxxxx

sum(increase(another_requests_count_total{custom_api_greet="empty"}[5m]))/sum(increase(another_requests_count_total[5m]))

When multiple tags and values need to be added per action, the previous implementation for DynamicTaggedCounter got additional customizations: DynamicMultiTaggedCounterand DynamicMultiTaggedTimer. The DynamicMultiTaggedTimer below checks if there is an existing Timer a with the same tags and values, and if it returns a new instance of a Timer with additional tags attached:

xxxxxxxxxx

public class DynamicMultiTaggedTimer extends CommonMetricDetails {

    protected List<String> tagNames;

    public DynamicMultiTaggedTimer(String name, MeterRegistry registry, String... tags) {

        super(name, registry);

        this.tagNames = Arrays.asList(tags.clone());

    }

    public Timer decorate(String ... tagValues) {

        List<String> adaptedValues = Arrays.asList(tagValues);

        if(adaptedValues.size() != tagNames.size()) {

            throw new IllegalArgumentException("Timer tag values mismatch the tag names!"+ 

                                               "Expected args are " + tagNames.toString() +

                                               ", provided tags are " + adaptedValues);

        }

        int size = tagNames.size();

        List<Tag> tags = new ArrayList<>(size);

        for(int i = 0; i<size; i++) {

            tags.add(new ImmutableTag(tagNames.get(i), tagValues[i]));

        }

        Timer timer = registry.timer(identifier, tags);

        timer.getId().withTags(tags);

        return timer;

    }

}

The previous implementation is used below to determine the creation of exceptional content of messages in a different language: 

    @PUT
    @Path("make/{languageTag}/{content}")
    @Consumes(MediaType.TEXT_PLAIN)
    @Produces(MediaType.TEXT_PLAIN)
    @Timed(value = "greeting.generator", extraTags = {URI, API_GREET})
    @Counted(value = "put.requests", extraTags = {URI, API_GREET})
    public String generateGreeting(@PathParam("languageTag") String languageTag, @PathParam("content") String content) {
        Locale locale = Locale.forLanguageTag(languageTag);
        MessageDTO dto = new MessageDTO(locale, content);
        if (Math.random() > 0.8) {
            String exceptionalTag = "exceptional" + content;
            dynamicMultiTaggedTimer.decorate(languageTag, exceptionalTag).record(() -> {
                try {
                    Thread.sleep(1 + (long)(Math.random()*500));
                } catch (InterruptedException e) {
                    LOGGER.error("Error occured during long running operation ", e);
                }
            });
            dynamicMultiTaggedCounter.increment(languageTag, exceptionalTag);
        } else {
            dynamicMultiTaggedCounter.increment(languageTag, content);
        }
        messageService.createMessage(dto);
        return content;
    }

A Grafana panel can be created to some exceptional messages by language and content:

xxxxxxxxxx

sum by (language, custom_api_greet)(increase(other_requests_total{custom_api_greet=~"exceptional.+"}[1m]))

Final Thoughts

For sure, you can do many more tweaks with Micrometer, yet I hope this piqued your interest in Quarkus and Micrometer. The code is available here.