loader — the Load stage (lab-graph RDF → QLever)

The L in the ETL. It keeps the qlever query store in sync with triplify's output: reads each RDF segment from s3://semantix/lab-graph and INSERT DATAs the ones the store doesn't have yet — so new logs become queryable automatically, no store restart.

collector → mover (s3 lab-podlogs) → triplify (RDF → s3 lab-graph) → [ loader ] → qlever (SPARQL)
  Extract                                     Transform                  Load        Store/serve

Extracting Load into its own service (rather than baking it into qlever's init) restores the clean boundary: the store is a swappable detail behind a small port (Load/Has/Drop), and the load use case is testable and observable on its own. See the qlever wiki Proposal — Loader service.

How it works

Level-triggered, like the mover/triplify — a timer + a /scan nudge only decide when a pass runs:

1. List lab-graph/**.ttl.gz (each object = a segment).
2. Read the ledger from the store: SELECT ?k WHERE { ?s a col:Segment ; col:segmentKey ?k }. The store is the single source of truth for what's loaded — reset it and the loader reloads.
3. For each segment not already loaded: gunzip → convert Turtle to N-Triples (knakk/rdf) → stream it in BATCH_SIZE-triple INSERT DATA requests → write a segment marker (col:Segment, col:segmentKey/segmentEtag/segmentTriples/loadedAt) so it isn't loaded twice.

Writes are token-gated (qlever-server -a), so the public /api stays read-only; the loader holds the token (Secret qlever-access).

Configuration (env)

Env	Default	Meaning
SRC_PREFIX	lab-graph	S3 prefix of triplify's RDF
QLEVER_ENDPOINT	http://qlever.pipelines.svc:7001	store SPARQL endpoint (in-cluster)
QLEVER_TOKEN	—	required — update access token (Secret qlever-access); startup fails loudly without it
BATCH_SIZE	20000	triples per INSERT DATA request
INTERVAL_SECONDS	60	reconcile interval
S3_ENDPOINT/S3_BUCKET/S3_ACCESS_KEY/S3_SECRET_KEY	—	Hetzner S3 (Secret mover-s3); bare aliases LOCATION/BUCKET/ACCESS_KEY/SECRET_KEY accepted; S3_REGION optional
LISTEN	:8080	HTTP: /healthz, /readyz (503 until the store answers), /metrics, /stats, POST /scan; off disables

Monitoring

HTTP API (mover-style): GET /healthz, /metrics (Prometheus), /stats (JSON), POST /scan. Metrics: loader_segments_loaded_total, loader_triples_loaded_total, loader_bytes_read_total, loader_errors_total, loader_passes_total, and gauges loader_segments_pending (backlog), loader_store_triples, loader_segments_in_store, loader_last_pass_timestamp_seconds, loader_pass_duration_seconds. Prometheus scrapes it; there's a Grafana Loader dashboard.

Build & deploy

Builds via the Temporal BuildOnly pipeline (webhook POST /build, melange/apko, signed SBOM). Pin the published loader@sha256:… digest in deploy/deployment.yaml, then kubectl apply.

Out of scope (later phases)

Delta compaction / cold-rebuild orchestration (fold the hot window into the base index); per-triple col:inSegment provenance + col:epochSecond; retention (drop a segment via DELETE); a second Store adapter (Oxigraph) to prove the port.