Post cover image
Observability

January 14, 2025

Mastering Advanced Logging, Tracing, and OpenTelemetry in .NET Core

When developing modern, cloud-native applications, robust observability is a must-have, not a nice-to-have. Your application’s success…

By James Joseph

7 min read

When developing modern, cloud-native applications, robust observability is a must-have, not a nice-to-have. Your application's success often hinges on how quickly you can detect, diagnose, and resolve problems in production. That's where logging, tracing, and the emerging ecosystem of OpenTelemetry come into play. In this post, we'll dive into advanced concepts of logging and distributed tracing in .NET Core, discuss the role of OpenTelemetry, and explore how these concepts tie together in a real-world application.

1. Why Advanced Logging and Tracing Matter

Logging and tracing have evolved beyond simple debugging tools. In large, distributed systems:

  • Logging provides structured context about events in your code.
  • Tracing paints a big-picture view of how requests flow across microservices.
  • OpenTelemetry standardizes instrumentation and data collection, making it easier to adopt best-of-breed observability backends.

Consider a real-world scenario: you're running a distributed e-commerce platform with multiple microservices handling orders, shipping, and payments. A user places an order and experiences a delay. Is the problem in the Order microservice, the Payment microservice, or the Shipping microservice? Advanced logging and tracing help pinpoint exactly where the bottleneck is, significantly reducing Mean Time To Recovery (MTTR).

2. High-Level Overview of .NET Core Logging

By default, .NET Core includes a powerful Logging abstraction. This abstraction supports log levels, structured logging, and logging sinks (e.g., Console, Debug, or third-party destinations). Some important concepts:

  1. ILogger — The core interface for logging in .NET.
  2. Log Levels — Trace, Debug, Information, Warning, Error, and Critical.
  3. Configuration — Usually done via appsettings.json or environment variables.
  4. Providers — Out-of-the-box providers include Console, Debug, EventSource, etc., but you can plug in libraries like Serilog, NLog, or Log4Net for more advanced capabilities.

2.1 Setting Up a Logging Provider

A common approach in .NET Core is to create a HostBuilder that configures logging providers:

public class Program
{
    public static async Task Main(string[] args)
    {
        var host = CreateHostBuilder(args).Build();
        await host.RunAsync();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureLogging((context, logging) =>
            {
                // Clear default providers
                logging.ClearProviders();

                // Add console
                logging.AddConsole();

                // Add Serilog or other providers if needed
                // logging.AddSerilog();

                // Configure from appsettings.json
                logging.AddConfiguration(context.Configuration.GetSection("Logging"));
            })
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            });
}

In your appsettings.json, you might define log levels like so:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  }
}

This configuration ensures that logs from your application are captured at Information level or higher, while logs from Microsoft framework components are only captured at Warning level or higher.

3. Structured Logging for Advanced Insights

Traditional "human-readable" logs ("User has logged in") often become a burden when you're sifting through massive logs in a production environment. Structured logging uses key-value pairs (often in JSON format) to make logs searchable and machine-readable. For example, with Serilog, you can write:

logger.LogInformation("User {UserId} has logged in from {IPAddress} at {LoginTime}",
    user.Id, user.IP, DateTime.UtcNow);

Instead of having a single text entry, each piece of data is a separate field in JSON logs. This allows you to quickly search for user sessions, login times, or other relevant details using centralized logging tools like Elastic Stack (ELK) or Azure Monitor.

3.1 Custom Log Enrichment

Serilog, for instance, supports enrichers that automatically add metadata to every log entry (e.g., machine name, thread ID, request ID). This is helpful to correlate logs across multiple microservices or pods in Kubernetes:

Log.Logger = new LoggerConfiguration()
    .Enrich.WithMachineName()
    .Enrich.WithThreadId()
    .WriteTo.Console()
    .CreateLogger();

By automatically including this data, you'll have a richer context for troubleshooting. It's an advanced yet straightforward way to level up your logging capabilities in .NET Core.

4. Distributed Tracing: Beyond Logging

Even with excellent logging, diagnosing a performance bottleneck in a microservices environment can feel like searching for a needle in a haystack. That's where distributed tracing steps in. Distributed tracing tracks the lifecycle of a request as it moves through different services. Common frameworks for distributed tracing include:

  • Zipkin
  • Jaeger
  • Azure Application Insights
  • Elastic APM

4.1 Understanding Trace Context

A trace consists of spans, which are operations within a microservice. Each span has:

  • A trace ID — a unique identifier for the entire trace.
  • A span ID — identifies the current operation.
  • A parent ID — correlates this operation with its upstream operation.

When a request enters your microservice, you'll generate a trace ID (or use an incoming one if you're continuing a distributed trace). Each downstream call propagates these IDs, ensuring all services can report spans back to the same trace.

4.2 Implementing Distributed Tracing with .NET Core

The manual approach involves injecting these IDs into HTTP headers (e.g., X-Trace-Id, X-Span-Id). However, .NET Core provides features and middlewares that automatically handle some of this for you. For instance, System.Diagnostics.Activity is commonly used for tracing within .NET.

public class TracingMiddleware
{
    private readonly RequestDelegate _next;

    public TracingMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task Invoke(HttpContext context)
    {
        using var activity = new Activity("Process Request");
        activity.Start();

        // Extract trace IDs from incoming headers if available
        if (context.Request.Headers.TryGetValue("traceparent", out var traceParent))
        {
            // parse and set parent trace details
        }

        // Perform your logic here
        await _next(context);

        activity. Stop();
    }
}

You'd also configure your HTTP clients to forward these headers downstream so that each subsequent service can correlate its spans to the parent operation.

5. Introducing OpenTelemetry

OpenTelemetry (OTel) is an open standard for collecting logs, metrics, and traces. It defines a set of APIs, libraries, agents, and collector services that help you export observability data to your choice of backend (Jaeger, Zipkin, Prometheus, etc.). OpenTelemetry aims to unify instrumentation across languages and frameworks, providing a consistent model for vendor-neutral observability.

5.1 Why OpenTelemetry?

  • Vendor Agnostic: You can switch from one observability backend to another without rewriting instrumentation.
  • Community-Driven: Backed by a large open-source community and the Cloud Native Computing Foundation (CNCF).
  • Holistic Observability: Collect traces, metrics, and logs under one specification.

5.2 Setting Up OpenTelemetry in .NET Core

To integrate OpenTelemetry in .NET Core, you'll likely reference OpenTelemetry packages such as:

  • OpenTelemetry
  • OpenTelemetry.Extensions.Hosting
  • OpenTelemetry.Trace
  • OpenTelemetry.Logs
  • OpenTelemetry.Metrics (for metrics)

Here's a sample snippet showing how to configure OpenTelemetry to collect traces and send them to Jaeger:

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureServices(services =>
        {
            services.AddOpenTelemetry()
                .WithTracing(builder =>
                {
                    builder
                        .AddSource("MyCompany.MyProduct.*")
                        .AddAspNetCoreInstrumentation()
                        .AddHttpClientInstrumentation()
                        .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MyService"))
                        .AddJaegerExporter(o =>
                        {
                            o.AgentHost = "localhost";
                            o.AgentPort = 6831;
                        });
                });
        })
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });

Breaking this down:

  • AddSource specifies which .NET Activity Sources to capture.
  • AddAspNetCoreInstrumentation automatically creates spans for incoming HTTP requests.
  • AddHttpClientInstrumentation captures spans for outgoing HTTP requests.
  • SetResourceBuilder adds metadata (like the service name) for easier identification.
  • AddJaegerExporter configures the exporter to send trace data to a Jaeger backend.

With the above configuration, your .NET Core application now sends distributed trace data to Jaeger. You can visualize request flows, measure latencies, and drill down into details for better understanding and faster debugging.

6. Advanced Scenarios

6.1 Sampling and Performance

Sampling is critical in high-throughput applications. Capturing every single request's trace can become prohibitively expensive. OpenTelemetry supports samplers like AlwaysOn, AlwaysOff, TraceIdRatioBased, and more. You can set them up like this:

services.AddOpenTelemetry().WithTracing(builder =>
{
    builder.SetSampler(new TraceIdRatioBasedSampler(0.1)) // 10% of requests
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddJaegerExporter();
});

This means only 10% of incoming requests will be traced. By tuning your sampling rate, you strike a balance between data completeness and cost.

6.2 Correlating Logs with Traces

One of the biggest wins with OpenTelemetry is the ability to correlate logs with traces. This is typically done by injecting the trace ID into log entries. Many logging frameworks and OpenTelemetry extensions can automatically enrich logs with the current Activity's ID:

logging.AddOpenTelemetry(options =>
{
    options.IncludeFormattedMessage = true;
    options.IncludeScopes = true;
    options.ParseStateValues = true;
    // Exporter configuration...
});

With correlation in place, you can filter logs by a trace ID in your observability platform and quickly jump from an error log to the corresponding trace.

6.3 Real-World Example: E-Commerce Checkout

Let's consider an E-commerce platform scenario:

  1. Checkout Service receives a request from the website to initiate an order.
  2. Checkout Service calls the Payment Service to process payment.
  3. Payment Service calls a Bank API (external) to verify funds.
  4. Payment Service returns a successful response to Checkout Service.
  5. Checkout Service logs and completes the process.

During this entire journey, an Activity is created in the Checkout Service. A trace ID is shared with the Payment Service, which in turn creates its own span. The Payment Service logs structured data ("Processing payment for {OrderId}") alongside the trace ID. Any external calls to the Bank API also carry headers with the same IDs.

If a user complains about a delay or failure, you can open your Jaeger or Zipkin dashboard, search for the relevant trace ID, and see a timeline of each span:

  • Checkout Service: 50ms
  • Payment Service: 500ms
  • External Bank API: 300ms

You notice the Payment Service spent a lot of time waiting on the Bank API. By checking logs correlated with that trace, you see an error log indicating that the external API had a high response time. Now you can respond appropriately — e.g., implement a circuit breaker or retry logic.

7. Putting It All Together

Step-by-Step Checklist

  1. Choose a Logging Framework: Decide if the built-in providers are sufficient or if you need Serilog/NLog for advanced features.
  2. Enable Structured Logging: Output logs in a format that's easy to parse, index, and query.
  3. Distribute Trace Context: Use .NET's Activity or OpenTelemetry libraries to propagate trace IDs across service boundaries.
  4. Configure Exporters: Choose your observability backend (e.g., Jaeger, Zipkin, Application Insights) and set it up.
  5. Leverage Sampling: Protect performance and costs by tracing only a subset of traffic.
  6. Correlate Logs and Traces: Inject trace IDs into logs for easier troubleshooting.
  7. Continuously Monitor and Improve: Use the insights from your logs and traces to optimize application performance and user experience.

8. Final Thoughts

Advanced logging, distributed tracing, and OpenTelemetry form the holy trinity of modern observability. By combining these techniques in your .NET Core applications, you gain a 360-degree view of your system's health, performance, and user experience. This knowledge isn't just about debugging; it's about proactively optimizing and ensuring your application meets its SLAs and user expectations.

When your microservices are properly instrumented, you can quickly detect anomalies, pinpoint latencies, and triage errors with high confidence. Logging and tracing aren't just for the ops team anymore — they're first-class citizens in your development workflow, guiding architectural and design decisions.

Implementing advanced logging, tracing, and OpenTelemetry can seem daunting, but the payoff in reduced downtime and improved user satisfaction is well worth the effort. As your services scale, so will the complexity — and these tools will be right there, helping you scale gracefully without sacrificing visibility.

In summary, remember:

  • Start small — enable structured logging, then gradually incorporate distributed tracing.
  • Adopt OpenTelemetry early — this future-proofs your instrumentation and avoids vendor lock-in.
  • Learn from your data — the insights you gather from logs and traces can guide performance tuning and architectural improvements.

Embrace these practices, and you'll be well on your way to building resilient, high-performing .NET Core applications that keep users happy and your team calm in the face of production incidents. Happy coding and tracing!