Metrics not appearing at prometheus endpoint when I'm using micrometer's PrometheusMeterRegistry

Issue

I'm new to micrometer and prometheus and I'm trying to build my first hello-world application to be monitored using micrometer with prometheus as monitoring backend. But I can't see the metrics by my app (Counters and Timers) appearing on the prometheus endpoint.

I'm following this tutorial for prometheus. I also followed this video for getting started with micrometer.

I downloaded prometheus from this link, extracted it and then ran prometheus to scrape using the command: ./prometheus --config.file=prometheus.yml. I'm having target set in this config file as targets: ['localhost:9090']

Then I ran my Main class which looks like this:

import cern.jet.random.Normal;
import cern.jet.random.engine.MersenneTwister64;
import cern.jet.random.engine.RandomEngine;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.Timer;
import io.micrometer.core.instrument.composite.CompositeMeterRegistry;
import io.micrometer.core.instrument.logging.LoggingMeterRegistry;
import io.micrometer.jmx.JmxMeterRegistry;
import io.micrometer.prometheus.PrometheusMeterRegistry;
import reactor.core.publisher.Flux;

import java.time.Duration;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

public class Main {

    public static void main(String[] args) throws InterruptedException {
        CompositeMeterRegistry compositeMeterRegistry = new CompositeMeterRegistry();

        LoggingMeterRegistry loggingMeterRegistry = SampleMeterRegistries.loggingMeterRegistry();
        JmxMeterRegistry jmxMeterRegistry = SampleMeterRegistries.jmxMeterRegistry();
//        AtlasMeterRegistry atlasMeterRegistry = SampleMeterRegistries.atlasMeterRegistry();
        PrometheusMeterRegistry prometheusMeterRegistry = SampleMeterRegistries.prometheus();

        compositeMeterRegistry.add(loggingMeterRegistry);
        compositeMeterRegistry.add(jmxMeterRegistry);
//        compositeMeterRegistry.add(atlasMeterRegistry);
        compositeMeterRegistry.add(prometheusMeterRegistry);


        AtomicInteger latencyForThisSecond = new AtomicInteger(0);
        Gauge gauge = Gauge.builder("my.guage", latencyForThisSecond, n -> n.get())
                .register(compositeMeterRegistry);

        Counter counter = Counter
                .builder("my.counter")
                .description("some description")
                .tags("dev", "performance")
                .register(compositeMeterRegistry);

        Timer timer = Timer.builder("timer")
                .publishPercentileHistogram()
                .sla(Duration.ofMillis(270))
                .register(compositeMeterRegistry);

        // colt/colt/1.2.0 is to be added for this.
        RandomEngine randomEngine = new MersenneTwister64(0);
        Normal incomingRequests = new Normal(0, 1, randomEngine);
        Normal duration = new Normal(250, 50, randomEngine);

        latencyForThisSecond.set(duration.nextInt());

        // For Flux you require io.projectreactor/reactor-core/3.2.3.RELEASE
        Flux.interval(Duration.ofSeconds(1))
                .doOnEach(d -> {
                    if (incomingRequests.nextDouble() + 0.4 > 0) {
                        timer.record(latencyForThisSecond.get(), TimeUnit.MILLISECONDS);
                    }
                }).blockLast();

    }
}

When I run ./prometheus --config.file=prometheus.yml, I can access the endpoint http://localhost:9090/metrics and also http://localhost:9090/graph. But when I try to execute the query on http://localhost:9090/graph sum(timer_duration_seconds_sum) / sum(timer_duration_seconds_count) it says no datapoints found.

It seems to me that I'm missing something obvious (as I'm a beginner to both of these topics).

Can someone please point out what I'm missing?

I couldn't find (where in my Main class) I have to configure the URI to publish for prometheus. Even if I'm publishing to http://localhost:9090 (which might be default hidden somewhere by micrometer) I couldn't find it.

Solution

targets: ['localhost:9090']

That's Prometheus being asked to scrape itself.

You need to add a target for the Java application's HTTP endpoint.

Answered By - brian-brazil