project-atlas

The authoritative description of this repository. Framework docs (WORKFLOW.md, GIT.md, …) yield to this file when they disagree.

Last verified: 2026-08-31.

What this repo is

Atlas is an open semantic layer over Norwegian public data and NGO supply data, published through a public PostgREST API. It ingests ~41 public-sector sources (SSB, FHI, Bufdir, Brreg, KLASS and others), transforms them with dbt into a documented marts.* layer, and exposes a curated api_v1 view contract that anyone can query without a database role.

This repo is not the organisation and not the platform. It is the data product plus the two frontends and the documentation site that sit on top of it.

What it builds / does not build

• Builds: the ingest modules (raw.*), the dbt project (marts.*, api_v1), a customer web app, a contributor diagnostics app, and the public documentation site.
• Does not build: the platform it runs on. Postgres, the orchestrator, PostgREST deployment, networking and observability belong to the infrastructure and are operated by another agent. This repo declares what must be deployed; it never deploys.

Layout

atlas/
├── atlas-data/                  — the data product
│   ├── ingest/                  — TypeScript, one folder per source, writes raw.*
│   ├── dbt/                     — staging → marts.* → api_v1 views
│   ├── dagster/                 — assets, schedules, sensors, automation conditions
│   └── migrations/              — raw.* schema SQL
├── atlas-frontend/              — customer app; PostgREST consumer, no DB role (port 3001)
├── atlas-contributor-frontend/  — diagnostics app; direct Postgres, dev/staging only (port 4000)
└── website/                     — Docusaurus site (installed and building)
    └── docs/ai-developer/       — THIS folder. There is no second copy.

Commands

Node ≥22 is required for the ingest tests — the machine default may be older, and Vitest 4 dies at startup on Node 20 in a way that looks like a broken repo rather than an unsupported runtime.

cd atlas-data/ingest
npm install
npm run migrate               # apply ../migrations/ against $DATABASE_URL
npm run ingest:<source-id>
npm run typecheck
npm test

cd atlas-data/dbt             # dbt runs through uv, never a bare `dbt`
uv run --env-file ../ingest/.env dbt build
uv run --env-file ../ingest/.env dbt test

cd website && npm run build   # Docusaurus; onBrokenLinks is 'throw'

Git host

GitHub. origin is https://github.com/terchris/atlas. GIT.md applies; AZURE-DEVOPS.md does not.

Devcontainer

No. There is no .devcontainer in this repo and DEVCONTAINER.md does not apply. Run commands on the host. Do not invent a cage.

Contracts (non-negotiable)

• 🔴 This repository is public. Internal topology, addresses, capacity figures and runtime identifiers must not be committed here. See SECURITY.md — this is the constraint most easily broken while writing something otherwise sensible.

• api_v1 is a published contract. Adding to it is public exposure and waits for a human. private_marts.* (Red Cross personal data) stays gated.

• The frontend dogfoods the public API. atlas-frontend has no database role and reads only what an external consumer can read. Do not give it one to make something easier.

• 🔴 A new dataset is a scored decision, and the scoring already exists. Read it before proposing one. plans/backlog/INVESTIGATE-new-norwegian-public-sources.md holds the five selection criteria — report fit, geographic resolution, access mechanism, licence and attribution, ingestion complexity vs upstream cadence — and fourteen candidates already scored and tiered against them. AGENT-onboard-source.md is the how; that file is the whether, and it is where the editorial decisions live. Check whether the dataset is already a candidate before writing anything new. On 2026-09-11 this agent was asked about Brreg organisations, did not check, and wrote a fresh investigation that duplicated candidate #9 — which had already scored it Tier 2, named the correct API endpoint, and recorded a sequencing dependency. The rules were fine; they were unreachable from here.

• Every marts column is documented. The dbt-osmosis gate enforces it repo-wide; a new column without a description fails the gate.

• Prefer a build-time assertion over a rule people must remember — and make the guard fail on purpose once before trusting it. This repo has shipped a guard that protected nothing, and a green uniqueness test once masked a real fan-out bug.

• 🔴 A model's SHAPE cannot be tested here, or in CI. Send it to imac as an UPGRADE, not an install. A new column, a renamed one, a changed materialisation or incremental strategy — this agent has no database, CI never materialises over existing data, and a fresh install cannot exhibit the defect at all, because there is no old table to preserve. Only a cluster that already holds data can. On 2026-09-12 an unset on_schema_change took api_v1 to zero views on every existing install while every fresh install passed. The full account is in plans/active/PLAN-003-brreg-dim-and-frivillig.md under "The upgrade path is invisible to everyone except imac".

• 🔴 Measure in the unit the threshold is stated in, and name the unit beside the number. A question with one obvious answer often has four, and the most authoritative-looking source can be the most wrong. Asked "how big is transform_checks' plan?" on 2026-09-13, against a limit of 711: GraphQL's executionPlanOrError — the API literally named "execution plan" — returns 1, because the dbt tests all run inside one op; the repo has 813 tests; 728 of them are asset-check eligible; and the plan actually declares 675 ASSET_CHECK_EVALUATION_PLANNED events. ⚠️ Only the last is in the unit 711 is in, and the repo count overstates it by 53. 🔵 The free, correct method is to read the planned-check events from a run that has ALREADY completed — the plan declaring what it contained, rather than a construction that collapses it. Same family as the two below: a number is not an answer until it says what it counted.

• 🔴 Two objects share every Atlas tag. Label every digest to its image, always, even when only one appears. ghcr.io/terchris/atlas-data/uis:<tag> is the UIS install artifact — the thing a catalogue pin names — and ghcr.io/terchris/atlas-data:<tag> is the code-location image the artifact points at. They carry the same tag and different digests. On 2026-09-13 a pin nomination from this agent listed both images and one unlabelled digest:; it was the image's, ops-dev relayed it as the artifact's, and dev-templates refused it at the catalogue. ⚠️ The wrong value passes every fetch-side check — release asset and GHCR agree, the blob hashes to its manifest — and surfaces much later as an unexplained pull failure. The publish workflow's run summary now labels both, so a nomination copies rather than reconstructs; write them the same way:

  artifact digest       sha256:…   ghcr.io/terchris/atlas-data/uis:<tag>
  code-location digest  sha256:…   ghcr.io/terchris/atlas-data:<tag>

  🔵 The same confusion appeared three times in one night across three agents — a hash of /app/* used to check a fix living in template-info.yaml, two tree-identical commits that were identical as images and different as artifacts, and this. "Two objects share a tag and the claim does not say which" is the shape; naming the object is the whole fix.

• 🔴 Pick the tag by ANCESTRY, never by sort order. Sort order is not ancestry.

  git rev-list -n1 <tag>                       # must equal the commit you mean
  git merge-base --is-ancestor <each-sha> <tag>  # must contain every PR in the release

  ⚠️ On 2026-09-24 three PRs were squash-merged within 27 seconds, and git tag --list 'v2026*' --sort=creatordate | head -1 returned the tag for the middle one. Nominating it would have deployed a green, real, buildable artifact that was simply not the one meant — missing the third PR entirely.

  🔴 Nothing downstream catches this. The tag exists, the image builds, the digests agree, the install succeeds and every check passes; the only symptom is a change that quietly is not there. Same family as the two rules above — a value that passes every fetch-side check and is still the wrong object.

  🔵 For tags minutes apart, creation date is not even a good proxy for order. Ask the question you actually mean — does this tag contain the commits I am shipping — and ancestry answers it directly.

• 🔴 Every nomination states which Dagster job LANDS the change. Derive it, do not recall it: atlas-data/uis/lands-with.sh <range>, and paste the output beside the digests.

  LANDS WITH:
    uis dagster run transform_and_publish
               (dbt build changed marts, and the publish re-creates the
                api_v1 views over them — a publish alone would wrap the old tables)

  ⚠️ The rule this replaces failed three times in eight days, read each time by someone who had read it (urb-agents #1253, #1267, #1271): a description change is a COMMENT, COMMENTs live in the database and not in the image, and only publish_api_v1 applies them — so installing the image leaves the public API serving the previous text with every signal green. 🔵 ops-dev's diagnosis is the one to keep: a rule three readers fail to apply is not a knowledge problem, it is a missing field. A field this agent has to remember to fill in is the same rule wearing a different hat, which is why it is a script and not a checklist item.

  The mapping it encodes, and the one row that surprises:

  changed	lands with
  models/**.sql, seeds/, macros/, dbt_project.yml	transform_and_publish
  schema.yml or api_v1_generated.sql only	publish_api_v1
  dagster/, ingest/	nothing — installing the image is the deploy
  an indexes=[…] config on an incremental model	🔴 no scheduled job at all

  That last row is why the covering index on dim_brreg_enhet is a post-hook: dbt-postgres creates a model's indexes: when the table is created, so on an incremental model they arrive on the next --full-refresh and never on a scheduled run. The script flags it rather than prescribing a job that would not work.

• 🔴 An idempotence check on an already-converged system proves the weaker half. It needs a FRESH substrate. This is the mirror of the rule above and the two are easy to confuse: a shape change needs a cluster that already holds data, and a convergence check needs one that does not. On 2026-09-13 this agent asked imac to diff pg_dump --schema-only across repeated migration runs and argued that their populated cluster was the stronger test, "because it proves convergence against the state that actually exists." That was wrong. 051 exists because run 1 differed from run 2 — comments set unconditionally by 006/007, shape changed behind a guard by 008 — and a cluster already at n=k is past the point where that can appear at all; it measures the steady state n → n+1. imac ran both rather than taking the framing, and only the fresh run answered the question. If this is ever automated it wants a throwaway database: pointed at production it keeps passing right up to and including the day someone introduces the next 008. Full account in plans/completed/PLAN-001-brreg-bulk-snapshot.md.

• A test that cannot fail is not evidence. Say what made it non-vacuous. A diff of two empty schemas is also an empty diff, and a gate whose first run passes has demonstrated nothing yet. The convention here is to prove the instrument alongside the result — imac reported 134 COMMENT ON statements and 91 brreg-mentioning objects present in the database that produced the empty diff; gates in this repo are proven by being made to fail on purpose before they are trusted.

• Absence-guards are a pattern here. Look for the existing ones before inventing a third. Sometimes what makes code correct is that a statement is not there, and no ordinary test covers an absence — the code passes every test right up until someone adds the line. The answer is a test that asserts over the module's own source text. It fails on the pull request that adds the line, rather than on the cluster that loses the data.

  Two exist today, both in atlas-data/ingest/src/sources/brreg-enheter-alle/__tests__/:

  • no DELETE / TRUNCATE in the bulk loader — a re-run must upsert, and a neighbouring module (bufdir-barnefattigdom) does delete-then-insert, so copying a neighbour is a live route to introducing it.
  • no page parameter and no _links.next in the change-feed poller (PLAN-002) — Brreg's feed caps page at 20 and its HAL next link is page-based, so following the link correctly walks into an HTTP 400 after 20 hops and silently never sees a deletion.

  Both were made to fail on purpose before being trusted. If you need a third, copy the shape from these — and add it to this list, because the fourth person will not find the first three.

• A retraction must travel with the claim it retracts. On 2026-09-05 a claim was measured, found wrong and retracted in a PR thread and an investigation file. Four days later the same number was recycled — into a commit message, a PR body, and an acceptance criterion sent to another agent — because the retraction lived in one place and the number moved to another. When withdrawing something, put the withdrawal where the claim will next be read, not only where it was made.

• Never commit to main. Feature branch, then a PR, squash-merged.

• ✅ Standing authorization to merge your own PRs — BROADENED 2026-09-12 and no longer conditional. Terje, verbatim: "you have full control of your own repo and you do merge when you see fit."

  That supersedes the conditional grant below, which ended when uis template install atlas worked end to end. There is now no end condition, so the paragraphs that follow are history rather than the operative rule. They are kept because the reasoning in them still applies to how to merge, not whether.

  🔴 What this does not change, and it is the part worth re-reading when nothing is stopping you. Merging is still not self-grading: deployable work is declared with acceptance criteria a stranger can run, imac still tests it, and a green CI run is not a verification. The authorization removes the wait, not the evidence. It also does not widen the two contracts above it — api_v1 additions are still public exposure, and private_marts.* is still gated.

  ⚠️ And one that only bites once merging is unrestricted: every merge touching atlas-data/** publishes an OCI artifact. Merging freely means tagging freely, and a tag appearing while another agent is pinning or testing a nominated one is the ambiguity that cost real time on 2026-09-12. Judgement about when is now the whole of the constraint.

  Superseded: the conditional grant (urb-agents #338, #364)

• ⏳ Standing authorization to merge your own PRs (Terje, urb-agents #338, condition confirmed #364). From #364, verbatim: "keep merging your own PRs on terchris/atlas until 'uis template install atlas' works end to end. Silence from me is not a change."

  So merge your own PRs on terchris/atlas without waiting for Terje — do not stop to ask, which is the whole reason this note exists. It removes the wait for his click, not the test: deployable work is still declared with acceptance criteria and falsifications a stranger can run, imac still tests it, and this agent still does not grade its own work.

  It ends when either is true, and not before:

  1. Terje says so — anywhere; a bus task, a comment, a message. His word ends it immediately.
  2. uis template install atlas installs Atlas end to end on a clean cluster, with no manual .uis.extend edit.
     • ✅ No platform blocker remains (2026-09-09). TPL-F5 shipped in UIS 1.6.19 and was cluster-verified — a provides entry may declare a code_location and UIS writes .uis.extend/ itself (#367, #480). TPL-F7 left Atlas's path earlier: per #362 the definition uses a single-file init: and Dagster owns the migrations. The remaining work is Atlas's own, and the critical path is: publish the artifact → install it from a local registry → imac falsifies it on a cluster.
     • ⚠️ Note how this was found. Atlas held TPL-F5 as a blocker for hours after it had shipped, because the UIS CLI reference still said a code location "remains a manual edit". A stale doc kept a condition open on a standing authorization from Terje. When a blocker is load-bearing for something Terje granted, confirm it against the code or the maintainer before pacing off it.

  What does not end it: #214 shipping, uis verify postgrest going green, or the atlas instance running. The instance was already deployed before the grant was given (#263), so none of those can be what "deployed" meant. That ambiguity is settled — do not re-litigate it.

  Two things this note used to say, both now superseded — recorded so their absence looks deliberate rather than lost:

  • It warned that Terje's silence was "not agreement" and told you to ask before relying on this. #364 reverses that explicitly: silence from me is not a change. Do not go back to him on a schedule.
  • It carried a self-imposed one-month expiry (2026-10-09), added as a guard against a grant with no clock. #364 supersedes it by naming the real condition. Removed rather than kept quietly, because enforcing my own timer over his stated condition would substitute my judgement for his on something that is his.

  The one genuine gap left: if the one-command-install goal is abandoned or redefined, condition 2 becomes undefined and this grant has no end. Ask then — that is a change in the goal, not silence.

  🔵 That gap is what closed, and not the way it was expected to. The note anticipated the goal changing; what actually happened is that Terje removed the condition entirely. Recorded rather than deleted, because a grant that quietly broadens looks identical to one that was never constrained — and the difference matters to anyone auditing why this agent merges its own work.

• This agent has no cluster access. No kubectl, no docker, no cluster shell. It declares; another agent applies; a third verifies. Do not write a plan that assumes otherwise.

Always-loaded files

• Repo-root CLAUDE.md
• Repo-root AGENTS.md

URB fleet

• Agent id: atlas
• Mailbox: mailboxes/atlas/inbox/ in terchris/urb-agents
• Read protocol/communication.md remotely. Do not clone urb-agents. Do not copy protocol/ here.
• Fleet status is published to fleet/status/atlas.md with scripts/publish-status.sh.

Other documentation

• AGENT-onboard-source.md — the procedure for adding a data source.
• asgard-performance-baseline.md and baseline-2026-04-28.md — measured baselines. ⚠️ These predate the public-repo constraint above and are part of the open question named in SECURITY.md.
• plans/talk/ — legacy in-repo coordination threads. Not a fleet bus; kept because the threads record why past decisions were made.
• Product documentation for consumers lives in website/docs/ (concepts, sources, measurements).

Known-stale claims corrected on 2026-08-31

The previous version of this file said Atlas had 19 sources, that Docusaurus was not yet installed, that this folder lived at docs/ai-developer/, that orchestration was a future "v2", and that there was no repo-root CLAUDE.md. All five were out of date. If something here reads as surprising, verify it against the code before repeating it.