Write and send .Net application metrics to Elasticsearch using Prometheus

Published in

Zenika

4 min readApr 20, 2023

Good practices to properly write and send metrics to Elasticsearch, using Prometheus.

What is Prometheus?

Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Source : Prometheus

NuGet packages

The following Prometheus for .Net NuGet packages are used:

These are .NET libraries for instrumenting your applications and exporting metrics to Prometheus.

Prometheus implementation

First, you have to add the following packages in your csproj file (you can update the version to the latest available for your .Net version):

<PackageReference Include="prometheus-net" Version="8.0.0" />
<PackageReference Include="prometheus-net.AspNetCore" Version="8.0.0" />
<PackageReference Include="prometheus-net.AspNetCore.HealthChecks" Version="8.0.0" />

By default, Prometheus .Net library add some application metrics about .Net (Memory, CPU, garbaging, …). As we plan to use APM agent, we don’t want it to add this metrics, so we can suppress them. We will also add some static labels to each metrics in order to be able to add contextual information from our application, as we did it for logs:

public virtual void ConfigureServices(IServiceCollection services)
{
    // ...

    Metrics.SuppressDefaultMetrics();

    Metrics.DefaultRegistry.SetStaticLabels(new Dictionary<string, string>
    {
        { "domain", "NetClient" },
        { "domain_context", "NetClient.Elastic" }
    });

    // ...     
}

We also have to map endpoints for metrics:

public void Configure(IApplicationBuilder app)
{
    // ...

    app.UseEndpoints(endpoints =>
    {
        // ...

        endpoints.MapMetrics();
    });
}

This map exposes the /metrics endpoint with the Prometheus format.

If you need OpenMetrics format, you can easily access it with /metrics?accept=application/openmetrics-text

The result is the below:

# HELP aspnetcore_healthcheck_status ASP.NET Core health check status (0 == Unhealthy, 0.5 == Degraded, 1 == Healthy)
# TYPE aspnetcore_healthcheck_status gauge
aspnetcore_healthcheck_status{name="self",domain="NetClient",domain_context="NetClient.Elastic"} 1
# HELP myapp_gauge1 A simple gauge 1
# TYPE myapp_gauge1 gauge
myapp_gauge1{service="service1",domain="NetClient",domain_context="NetClient.Elastic"} 1028
# HELP myapp_gauge2 A simple gauge 2
# TYPE myapp_gauge2 gauge
myapp_gauge2{service="service1",domain="NetClient",domain_context="NetClient.Elastic"} 2403
# HELP myapp_gauge3 A simple gauge 3
# TYPE myapp_gauge3 gauge
myapp_gauge3{service="service1",domain="NetClient",domain_context="NetClient.Elastic"} 3872
...

Forward Health checks to Prometheus

We can easily forward our health checks (described int the previous article) to Prometheus endpoint, to avoid using http module from Metricbeat and retrieve all metrics including health checks from Metricbeat Prometheus module. By the way, we will also benefit from our static labels if defined.

This is done here in our custom extension which is used in the ConfigureServices of the Startup file:

public virtual void ConfigureServices(IServiceCollection services)
{
    // ...
    services.AddCustomHealthCheck(Configuration)
    // ...     
}

public static IServiceCollection AddCustomHealthCheck(this IServiceCollection services, IConfiguration configuration)
{
    IHealthChecksBuilder hcBuilder = services.AddHealthChecks();
    hcBuilder.AddCheck("self", () => HealthCheckResult.Healthy());

    hcBuilder.ForwardToPrometheus();

    return services;
}

Business metrics

Prometheus .Net library offers an easy way to add business metrics.

To create a new metric, you just have to instantiate an new counter, gauge, …:

private readonly Gauge Gauge1 = Metrics.CreateGauge("myapp_gauge1", "A simple gauge 1");

If you need to add attached labels, you have to add a configuration:

private static readonly GaugeConfiguration configuration = new GaugeConfiguration { LabelNames = new[] { "service" }};
private readonly Gauge Gauge2 = Metrics.CreateGauge("myapp_gauge1", "A simple gauge 1", configuration);

To apply a label and a value to a metric, use this kind of code:

Gauge1.WithLabels("service1").Set(_random.Next(1000, 2000));

Sending metrics to Elasticsearch

All the metrics are available on the /metrics endpoint.

In our example, we don’t have any Prometheus server, so metricbeat will directly access metrics from the application metrics endpoint. But if you have a Prometheus server, you can add a new target in your scrape configuration.

So, to send the metrics to Elasticseach, you will have to configure a metricbeat agent with prometheus module:

    metricbeat.modules:
    - module: prometheus
      period: 10s
      metricsets: ["collector"]
      hosts: ["host.docker.internal:8080"]
      metrics_path: /metrics

For more information about this metricbeat configuration, you can have a look to: https://github.com/ijardillier/docker-elk/blob/master/extensions/beats/metricbeat/config/metricbeat.yml