This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

loader — the Load stage (lab-graph RDF → QLever)

The L in the ETL. It keeps the qlever query store in sync with triplify's output: reads each RDF segment from s3://semantix/lab-graph and INSERT DATAs the ones the store doesn't have yet — so new logs become queryable automatically, no store restart.

collector → mover (s3 lab-podlogs) → triplify (RDF → s3 lab-graph) → [ loader ] → qlever (SPARQL)
  Extract                                     Transform                  Load        Store/serve

Extracting Load into its own service (rather than baking it into qlever's init) restores the clean boundary: the store is a swappable detail behind a small port (Load/Has/Drop), and the load use case is testable and observable on its own. Background: the qlever wiki Proposal — Loader service.

Where it runs

• Deployment loader in namespace pipelines (stateless — no PVC). Reads S3 with the mover-s3 creds; writes to the in-cluster store http://qlever.pipelines.svc:7001.
• Writes are token-gated — qlever-server -a (Secret qlever-access). The public /api stays read-only; only the loader (holding the token) can INSERT/DELETE.
• Repo: https://forgejo.192.168.10.46.sslip.io/forgejo_admin/loader

How it works

Level-triggered like mover/triplify — a timer + a /scan nudge only decide when a pass runs:

1. List lab-graph/**.ttl.gz (each object = a segment).
2. Read the ledger from the store: SELECT ?k ?e WHERE { ?s a col:Segment ; col:segmentKey ?k ; col:segmentEtag ?e }. The store is the single source of truth for what's loaded — reset it and the loader reloads. The ledger lives in the store, not the pod, so the loader survives restarts without re-loading.
3. For each not-yet-loaded segment: gunzip → Turtle→N-Triples (knakk/rdf), buffering the triples.
4. Coalesce & flush — once the buffer reaches BATCH_SIZE triples (spanning many segments), flush as one INSERT DATA then one marker INSERT (col:Segment, col:segmentKey/segmentEtag/ segmentTriples/loadedAt for every segment in the batch, via store.MarkSegments). Data first, markers second → a crash between them just re-loads the batch next pass (same IRIs → idempotent), and only a marked segment counts as done.

Pruned segments are skipped, not failed. triplify rotates its output (OUT_MAX_RECORDS), so an object listed at the start of a pass can be gone by the time the loader GETs it. A NoSuchKey/404 is mapped to ErrPruned and skipped (a pruned counter, not an error) — a hot signal source out-pacing the loader is a throughput/retention decision, not a load failure. See Throughput below.

Configuration (env)

Env	Default	Meaning
SRC_PREFIX	lab-graph	S3 prefix of triplify's RDF
QLEVER_ENDPOINT	http://qlever.pipelines.svc:7001	store SPARQL endpoint
QLEVER_TOKEN	—	update token (Secret qlever-access)
BATCH_SIZE	20000	triples per INSERT DATA (coalesced across many segments, not per-object)
INTERVAL_SECONDS	60	reconcile interval
S3_*	—	Hetzner S3 (Secret mover-s3)

Monitoring

GET /healthz /metrics /stats, POST /scan. Metrics: loader_segments_loaded_total, loader_triples_loaded_total, loader_bytes_read_total, loader_errors_total, loader_passes_total

• gauges loader_segments_pending (backlog), loader_store_triples, loader_segments_in_store, loader_last_pass_timestamp_seconds, loader_pass_duration_seconds. Grafana → Loader dashboard (uid loader): store growth, backlog, load rate, errors, pass duration.

Operate

• Force a pass now: curl -X POST loader:8080/scan (or wait INTERVAL_SECONDS).
• Rebuild the store from scratch: fold — kubectl -n pipelines rollout restart statefulset/qlever bulk-rebuilds the base index from all of lab-graph and re-derives the ledger (the loader then only delta-loads new segments). Full wipe (rarely needed): scale qlever to 0, delete pvc/data-qlever-0, scale back to 1; the loader repopulates.
• It's stateless — redeploy freely.

Throughput — the stall, the diagnosis, and the fix (2026-08-17)

Symptom. LoaderStalled + LoaderErrorsClimbing fired: loader_errors_total climbing, backlog growing, no pass completing for >30m.

Diagnosis (quantified). The whole lab-graph is tiny — 5,518 objects, 293k triples, 23.6 MiB; a full base-index build over it takes <1 second. The stall was never about data size or compute (CPU sat at ~1%). It was object-count × per-object round-trips, done serially:

• The loader loaded one tiny object at a time (GET → INSERT → MARK), each object ~15–90 triples, and each INSERT went into QLever's --persist-updates delta, which QLever re-processes on every write — measured at ~4 objects/min, all I/O-wait. A backlog of thousands then takes hours per pass.
• The errors were all NoSuchKey/404 on objects triplify had pruned before the loader reached them (rotation out-pacing the loader) — counted as hard errors and re-tried forever.

Layer-1 fix (done — commits 9df99fe, 0b5fb65):

1. Pruned-skip — 404 → ErrPruned → skip (see How it works). Kills the error storm.
2. Coalesced writes — buffer many segments, flush one INSERT + one marker INSERT per BATCH_SIZE triples instead of two round-trips per object. Cuts thousands of writes to a handful.
3. Bulk goes through the base rebuild, not the delta — a large backlog is folded in by restarting QLever (IndexBuilderMain bulk-builds the base + re-derives the ledger); the delta path only ever carries the small hot window. See the qlever fold cronjob.

Result (verified under a soak + chaos test): catch-up pass cleared the backlog once, then steady passes run ~20–45s at 0 pending / 0 errors; survives graceful restart, hard kill, and a 7-min store outage, recovering cleanly each time.

Still object-count-bound (→ Layer 2 / Layer 3). The remaining per-pass cost is the serial S3 GET of each new object (readSegment) — fine at the current ~250-new-per-pass delta, but it scales with object count, not data size. Two follow-ups carry the durable fix (both need design sign-off):

• qlever#1 — bounded hot-window materialized view (Layer 2): index only a time-window; keep the TB archive in S3 as sealed segments. Stops loader and fold being object-count-bound.
• triplify#2 — correlation-layer facts, not per-line triples (Layer 3): emit fewer, higher-value triples — attack the ~40× volume at source.

Runbook — LoaderStalled / LoaderErrorsClimbing

1. Is the store reachable? LoaderStalled with the pod logging waiting for the store… = the loader can't reach QLever (QLEVER_ENDPOINT). Check qlever-0 is Running/ready in pipelines.
2. Is it a big backlog (catch-up after downtime)? loader_segments_pending high but falling, 0 errors → it's recovering; let it drain. If it's not draining, fold: kubectl -n pipelines rollout restart statefulset/qlever (bulk base rebuild + ledger re-derive) — passes go fast again.
3. Errors climbing? kubectl -n pipelines logs deploy/loader | grep 'loader:'. pruned lines = rotation out-pacing the loader (benign; the Layer 2/3 tickets). insert/mark batch … : lines = a real store/write error — check QLever health + the update token (qlever-access).
4. Recovery is automatic — the ledger lives in the store, so once the store is reachable the loader reconciles the gap on its own; the incident auto-closes when a pass completes.

Out of scope (later phases)

Per-triple col:inSegment + col:epochSecond; retention (drop a segment via DELETE); a second Store adapter (Oxigraph) to prove the port.