otel-collector — signal ingestion (Extract)

The standard Extract stage of the signal pipeline (platform wiki → Runbook — onboard a signal source). An upstream OpenTelemetry Collector (contrib) that receives signals, enriches them, and lands immutable OTLP-JSON artifacts in s3://semantix/land/<kind>/… — the seam the OTLP→RDF transform reads. It never emits RDF.

<source> → [ otel-collector: receive + enrich ] → s3://…/land/<kind>/ (OTLP-JSON) → OTLP→RDF → loader → qlever

What it does today

• Receives Kubernetes Events (k8sobjects receiver, watch). New sources are added as receiver blocks + an exporter prefix in config/config.yaml — config, not code.
• Enriches (resourcedetection + resource: simon.cluster, simon.signal). The full col:resource CMDB IRI is derived in the OTLP→RDF transform (the runbook's Anchor rule).
• Lands OTLP-JSON to land/k8s-events/year=…/… via the awss3exporter (Hetzner endpoint, path-style — verified compatible).

Monitoring (fleet standard)

• :8888/metrics — the collector's own otelcol_* self-telemetry (receiver accepted/refused, exporter sent/failed → S3, queue, uptime). Scraped by Prometheus (job otel-collector); Grafana "OTel Collector" dashboard.
• :13133/ — the health_check extension (readiness probe + Gatus, pipeline group).

Deploy / update

Config is config/config.yaml → ConfigMap otel-collector-config:

kubectl -n pipelines create configmap otel-collector-config \
  --from-file=config.yaml=config/config.yaml --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f deploy/otel-collector.yaml
kubectl -n pipelines rollout restart deploy/otel-collector   # pick up a config change

S3 creds come from the mover-s3 Secret (AWS default credential chain).

Second instance — otel-collector-syslog (network/host syslog)

A dedicated deployment of the same image accepts external syslog (a different trust boundary than the cluster-internal event watcher) and lands it in land/syslog/. See ADR-0002.

• Config: config/syslog.yaml → ConfigMap otel-collector-syslog-config; manifest deploy/otel-collector-syslog.yaml.
• Ingress: UDP+TCP 192.168.10.46:1514 (hostPort, pinned to server-46). Port 1514 (not 514) — non-root can't bind privileged ports; two receivers (syslog/udp + syslog/tcp) because one receiver only binds a single transport.
• Senders: Juniper SRX340/EX2300 + MikroTik CRS504 (native syslog), and any non-k8s host via rsyslog. Each must be in NetBox with its IP (the netbox-by-ip anchor resolves the source net.peer.ip).

kubectl -n pipelines create configmap otel-collector-syslog-config \
  --from-file=config.yaml=config/syslog.yaml --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f deploy/otel-collector-syslog.yaml

Add a source

Follow the platform wiki Runbook — onboard a signal source: add the receiver (syslog/otlp/ filelog/…), an enrichment rule for the resource IRI, and an awss3 exporter prefix land/<kind>. No new service.