Scaling & Performance

HyperMesh is designed for temporal hypergraph workloads: multi-party events indexed by time (TPI), node membership (FMI), and optional scalar properties (PSI). This page summarizes operational characteristics and tuning parameters. For measured benchmark tables and reproducibility steps, see Performance benchmarks.

Performance characteristics

Latencies below are from the C engine (clock_gettime), excluding Python overhead. Bulk-ingest numbers use group-commit (one fdatasync() per transaction batch), not single-record insert().

Operation	Complexity	Typical latency / throughput
Time-range query (TPI)	O(B_window + K)	25 µs – 25 ms median (1 K – 1 M records, 3 % window)
Node lookup (FMI)	O(log N + d)	1.6 ms – 1.4 s median (depends on node degree d)
Property filter (PSI)	O(log P + K)	Index-backed; scales with matching rows K
Single-record `insert()`	O(1) + fsync	~20 µs per record (crash-safe WAL)
Bulk ingest (group-commit)	O(1) per entry + batch fsync	~54k rows/s measured (3.55 M IBM AMLworld edges)
Compaction (`compact()`)	O(P + W) reconcile + O(N log N) sort	~83 ms (100 K steady-state delta); ~6.8 s (3.55 M cold build)

Symbols: B_window = time buckets overlapping the query window; K = rows returned; N = total indexed records; P = primary index size before merge; W = WAL entries; d = hyperedges touching the queried node.

Scaling write throughput

Single-record inserts pay one fdatasync() each — fine for interactive use, too slow for multi-million-row loads.

Use group-commit for bulk loads:

import hypermesh as hm

db = hm.connect("/data/aml")
db.execute("CREATE HYPEREDGE TABLE TXN () BUCKET_SECONDS 3600")

with db.transaction():
    for ts, src, dst, amount in train_edges:
        db.insert(event_ts=ts, members=[src, dst], weight=amount, table="TXN")

store = db._get_store("TXN")
store.compact()   # build TPI + FMI; truncate WAL

After a large bulk load, call compact() (or COMPACT in Cypher) before latency-sensitive reads. Compaction is synchronous — it holds the store write lock for the duration of the rebuild.

Tuning parameters

These are the knobs exposed today. There is no cache_size_mb or mmap_enabled setting in the current engine.

Table DDL (Python / Cypher / REST)

Parameter	Default	Effect
`bucket_seconds`	`10`	TPI bucket width in seconds. Smaller → more buckets, faster narrow time windows, larger index metadata. Larger → fewer buckets, better for wide scans.
`compact_threshold`	`0` (disabled)	When WAL entry count reaches this threshold, the engine triggers autocompact after the next write. Set via `CREATE HYPEREDGE TABLE …` or `POST /v1/tables`.

Python:

db.execute(
    "CREATE HYPEREDGE TABLE Events (Drone, Host) "
    "BUCKET_SECONDS 60 COMPACT_THRESHOLD 5000"
)

REST (POST /v1/tables):

{
  "name": "Events",
  "member_tables": ["Drone", "Host"],
  "bucket_seconds": 60,
  "compact_threshold": 5000
}

Runtime override (embedded engine):

store = db._get_store("Events")
store.set_autocompact(5000)   # 0 = disable autocompact

Hard limits (engine)

Limit	Value	Notes
`max_members_per_edge`	64	In-memory guard (`HM_MAX_MEMBERS`). On-disk format supports up to 255.
Max property payload	format limit	See schema guide.

Server environment

Operational server tuning (auth, TLS, rate limits) is documented in Configuration. Query timeout and backup roots are controlled via HMDB_* environment variables.

Compaction at scale

Compaction reconciles the WAL against the on-disk TPI, rebuilds TPI + FMI, and truncates the WAL. Reconciliation is O(P + W) via a hash-indexed tombstone map; the dominant term for large stores is the writer’s O(N log N) sort.

Scenario	Records	Time
Steady-state (small WAL on 100 K index)	100 K	~83 ms
Cold bulk build (IBM AMLworld train split)	3.55 M	~6.8 s

Guarded by make compact_test in hypermesh_core/ — hangs if quadratic reconciliation regresses.

Choosing `bucket_seconds`

Workload pattern	Suggested `bucket_seconds`	Rationale
Sub-second event streams (IoT, trading)	`1 – 10`	Queries often span seconds, not hours
Hourly aggregates (AML, logistics)	`3600`	Matches natural reporting windows
Batch analytics (wide time ranges)	`60 – 300`	Balance between bucket count and scan width

Rule of thumb: pick bucket width near the median query window you expect in production. The TPI seek cost is O(number of buckets overlapping the window), not O(total records).

Performance benchmarks — full TPI/FMI tables and reproducibility
Temporal range queries — how bucket pushdown works
Benchmarks vs PostgreSQL — comparative methodology
Configuration — server HMDB_* variables
REST API — tables — compact_threshold via HTTP