Logging & Observability

Core Principles

1. Structured logging with Serilog — Every log entry is a structured event with named properties, not a formatted string. This enables searching, filtering, and alerting. All setup (two-stage bootstrap, AddSerilog(), sinks, enrichers) lives in the serilog skill — that skill's AddSerilog()-over-UseSerilog() guidance is canonical.
2. OpenTelemetry for distributed tracing — Traces connect requests across services; metrics track system health over time. Full setup lives in the opentelemetry skill.
3. Health checks for operational readiness — Every service exposes /health endpoints for load balancers and orchestrators. Liveness and readiness are separate questions and separate endpoints.
4. Correlation IDs for request tracing — Every request gets a unique ID that flows through all log entries and downstream service calls, so one user complaint maps to one filtered log stream.

Patterns

How the Pieces Fit Together

Concern	Owner	Skill
Structured application logs	Serilog (AddSerilog())	serilog
Request summary logging	UseSerilogRequestLogging()	serilog
Traces + metrics + OTLP export	OpenTelemetry SDK	opentelemetry
Health endpoints, correlation IDs, log-level strategy	This skill	logging

Wire logging first (you need logs to debug the rest), then health checks, then tracing.

Correlation IDs

// Middleware to set correlation ID
public class CorrelationIdMiddleware(RequestDelegate next)
{
    private const string CorrelationIdHeader = "X-Correlation-Id";

    public async Task InvokeAsync(HttpContext context)
    {
        var correlationId = context.Request.Headers[CorrelationIdHeader].FirstOrDefault()
            ?? Guid.NewGuid().ToString();

        context.Items["CorrelationId"] = correlationId;
        context.Response.Headers[CorrelationIdHeader] = correlationId;

        using (LogContext.PushProperty("CorrelationId", correlationId))
        {
            await next(context);
        }
    }
}

// Program.cs — register early so every downstream log carries the ID
app.UseMiddleware<CorrelationIdMiddleware>();

Why middleware: pushing the property once at the pipeline edge attaches it to every log event in the request scope — no per-call-site plumbing. Propagate the same header on outgoing HttpClient calls via a DelegatingHandler (see the httpclient-factory skill).

Health Checks

// Program.cs
builder.Services.AddHealthChecks()
    .AddNpgSql(builder.Configuration.GetConnectionString("Default")!,
        name: "database", tags: ["ready"])
    .AddRedis(builder.Configuration.GetConnectionString("Redis")!,
        name: "redis", tags: ["ready"])
    .AddRabbitMQ(builder.Configuration.GetConnectionString("RabbitMq")!,
        name: "rabbitmq", tags: ["ready"]);

// Map endpoints
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = _ => false // No dependency checks — just "am I running?"
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready")
});

Why two endpoints: liveness failing means "restart me"; readiness failing means "stop sending traffic". Conflating them makes a slow database restart your app in a loop.

Log-Level Strategy

Level	Use for	Environment default
Debug	Diagnostic detail, payload dumps (never PII in prod)	Development only
Information	Business events: order placed, job completed	Dev + staging
Warning	Recoverable anomalies: retry fired, fallback used	Everywhere — production default
Error	Failed operations that need attention	Everywhere
Fatal/Critical	App cannot continue	Everywhere

Why Warning as the production default: Information-level request noise at scale costs real money in log storage and drowns the signals. Keep Information for genuine business events via namespace overrides (see the serilog skill's MinimumLevel.Override pattern).

Anti-patterns

Don't Log Sensitive Data

// BAD — logging credentials
logger.LogInformation("User logged in: {Email} with password {Password}", email, password);

// GOOD — log identifiers, never secrets or PII at Information level
logger.LogInformation("User {UserId} logged in", userId);

Don't Skip Health Check Tags

// BAD — all checks run for liveness AND readiness
app.MapHealthChecks("/health");

// GOOD — separate liveness (am I running?) from readiness (can I serve traffic?)
app.MapHealthChecks("/health/live", new() { Predicate = _ => false });
app.MapHealthChecks("/health/ready", new() { Predicate = c => c.Tags.Contains("ready") });

Don't Re-Implement What the Owning Skill Provides

// BAD — hand-rolling Serilog bootstrap here from memory
builder.Host.UseSerilog(...);  // legacy API — the serilog skill forbids this

// GOOD — load the serilog skill and use its two-stage AddSerilog() bootstrap
builder.Services.AddSerilog((services, lc) => lc.ReadFrom.Configuration(builder.Configuration)...);

Decision Guide

Scenario	Recommendation
Application logging setup	Load serilog — AddSerilog() two-stage bootstrap
Distributed tracing / metrics	Load opentelemetry — OTLP exporter
Custom business metrics	IMeterFactory + counters/histograms (opentelemetry skill)
Request tracing	Correlation ID middleware (this skill)
Container health	/health/live and /health/ready endpoints (this skill)
Log storage	Seq (development), Elastic/Grafana/OTLP backend (production)
Log levels	Debug in dev, Information in staging, Warning default in production