Monitoring & Observability

Health Endpoints

HyperMesh exposes standard health check endpoints for load balancers and orchestrators.

Endpoint	Auth	Purpose
`GET /health`	Required	Full health status with version, uptime, and table count
`GET /health/live`	None	Liveness probe — returns 200 if process is running
`GET /health/ready`	None	Readiness probe — returns 200 when indexes are loaded

Prometheus Metrics

The /metrics endpoint exposes Prometheus-compatible metrics:

# HELP hypermesh_records_total Total records in TPI index
# TYPE hypermesh_records_total gauge
hypermesh_records_total 1000000

# HELP hypermesh_query_duration_seconds Query execution time
# TYPE hypermesh_query_duration_seconds histogram
hypermesh_query_duration_seconds_bucket{strategy="tpi",le="0.001"} 9500
hypermesh_query_duration_seconds_bucket{strategy="tpi",le="0.01"} 9900

# HELP hypermesh_wal_size_bytes Current WAL file size
# TYPE hypermesh_wal_size_bytes gauge
hypermesh_wal_size_bytes 4194304

# HELP hypermesh_compactions_total Total compaction operations
# TYPE hypermesh_compactions_total counter
hypermesh_compactions_total 42

Key Metrics to Monitor

Metric	Alert Threshold	Action
`hypermesh_wal_size_bytes`	> 100 MB	Trigger manual compaction or reduce `compact_threshold`
`hypermesh_query_duration_seconds` (p99)	> 10 ms	Check for missing indexes or oversized time windows
`hypermesh_records_total`	Stagnant for > 1 hour	Check ingestion pipeline health
`hypermesh_compactions_total` rate	< 1/hour during ingestion	Increase autocompact frequency

Grafana Dashboard

A pre-built Grafana dashboard is available at examples/grafana/hypermesh-dashboard.json in the repository. It includes:

Query latency heatmap (by strategy: TPI, FMI, PSI)
WAL size over time with compaction markers
Records ingested per minute
Active connections and API key usage

Structured Logging

Enable JSON-formatted logs for log aggregation systems:

export HMDB_LOG_FORMAT=json
export HMDB_LOG_LEVEL=info
hmdb serve

Log output:

{"ts":"2026-06-22T10:00:01Z","level":"info","msg":"query_complete","strategy":"tpi","duration_us":42,"records":150,"table":"NetworkFlows"}

Alerting Best Practices

WAL growth — alert when WAL exceeds 50% of available disk; autocompact should prevent this
Query latency — baseline p99 during normal load, alert at 3x baseline
Ingestion lag — compare records_total growth rate against expected event rate
Disk usage — TPI files grow linearly with data; plan capacity based on retention TTL