Backlog — priority view
Purpose: triage tool, not a roadmap. Orders what to work on next by what it unblocks. Covers both PLANs and INVESTIGATEs — an earlier version of this doc covered only INVESTIGATEs, which left the whole data-platform workstream invisible here for months.
Last updated: 2026-10-01. Re-rank whenever something moves to completed/, a new item lands,
or a blocker clears.
State: active/ is empty, and that is honest — PLAN-007 and the asgard deployment plan both
closed. Everything below is backlog or parked.
How to read the tiers: tier order is the order to start, not to finish. Tier 1 is next on deck; Tier 4 means "don't open this yet".
🔴 Waiting on Terje — not startable by atlas
These are decisions, not work. Listed first because they are the largest source of stalled items, and several have been open for days. The remainder of the UIS-installability batch does not block Tier 1.
✅ Two of these were answered on 2026-09-09, and how they got answered is the lesson. The
schemas: posture (A — the public API serves api_v1 only, Terje, urb-agents #350) and TPL-Q3
(an application ships its install definition inside its published image; the catalogue holds an
immutable pin and the pin PR is the review — #354, now PLAN-templates-002). Both had sat for days
looking owned because two agents each recorded "waiting on Terje" in a status file. A status file
is not a queue; only an auth-required hold reaches Terje's escalation. Filed as holds, both were
answered within hours.
| What | File | Since | Why it blocks |
|---|---|---|---|
| The Svalbard question — does Atlas cover Svalbard, or merely represent it? | INVESTIGATE-ssb-pseudo-regions | 2026-08-25 | Decides whether pseudo-region data flows to marts and the map. Neither suggested fix works as posed until this is answered. Source of the 17 tolerated WARNs. |
| F1 redcross data delivery — blocked on an APIM credential | PLAN-redcross-branches-private-input | 2026-08-25 | Design settled; the blocker changed shape from a static dump to an API credential, which may reopen the design. Keeps 3 of 13 views empty. |
| Phase 5 frontend | asgard deployment plan (in terchris/home) | 2026-08-25 | Held at Terje's request while he reads the code himself. |
| 🔴 Is GitOps the direction for UIS? And should that investigation be opened? | INVESTIGATE-atlas-as-a-uis-application | 2026-09-07 | The discussion Terje asked for has happened and produced an answer for everything except this. UIS has two deployment paths that do not meet and Atlas straddles them. If GitOps is the direction, per-workload secrets become prerequisites, not by-products. The UIS maintainer declined to open a Tier 1 platform-direction investigation on one message without Terje in it. Blocks nothing — A proceeds regardless. |
| Should imac's cluster start with its host? | INVESTIGATE-atlas-as-a-uis-application | 2026-09-07 | The API disappears silently on every reboot and a frontend is about to be built against it. All cluster state survives; only the process does not. One host setting, ~5 minutes — Rancher Desktop autostart or k3s under systemd. Not platform work, queues behind nothing. |
| NLOD attribution across everything Atlas publishes | INVESTIGATE-nlod-attribution | 2026-09-11 | Split out of the Brreg work at Terje's direction so it is not treated as Brreg-specific. Repo-wide: every upstream is NLOD and requires attribution. Machinery exists (meta_sources publishes licence and attribution); seed-sources/brreg-enheter/ has no manifest at all. Deliberately parked — recorded so it is not lost, not being worked. |
| Public-docs topology — internal detail in a public repo | raised in terchris/home talk | 2026-08-26 | asgard-performance-baseline.md is world-readable and names infrastructure. Proposed a platform-facts-to-home split. Still unanswered. Recurs every time a doc mentions infrastructure. |
🔴 Four decisions on the analytical surface — publish Parquet at all ([Q1]), drop doc from responses ([Q4]), host on R2 ([Q5]), accept a prerelease duckdb-wasm tag ([Q8]) | INVESTIGATE-parquet-duckdb-wasm-surface | 2026-09-23 | Decides whether Atlas gets a second, file-based query surface or stays single-surface. Does not block the hardening half — Option A (db-max-rows, statement_timeout, indexes) is correct regardless and needs no decision. [Q5] touches Terje's Cloudflare account, so it is his either way. |
| 🔴 Does the public API stay pointed at imac? | INVESTIGATE-parquet-duckdb-wasm-surface [Q15] | 2026-09-23 | api-atlas.urbalurba.com is served from imac (test) today, while PLAN-atlas-asgard-001 puts production on asgard against Odin pg and imac's charter is explicitly a place to try things without touching production. Inherited rather than chosen. Pairs with the reboot question two rows up — a public endpoint that vanishes on every reboot is the same decision seen from the other end. |
| NGO activity taxonomy crosswalk — review and decide the candidate categories | INVESTIGATE-ngo-activity-taxonomy [Q5], [Q6] | 2026-10-04 | 500 activities, all 11 NGOs, already mapped and structurally verified (atlas, urb-agents #1844) — deliberately not adopted. [Q5]: free equipment lending (BUA etc.), public-service help, practical help, hospital volunteers, elder residential care, alternative education. [Q6]: load policy while review is in progress (0 of 589 rows reviewed today). Blocks the taxonomy's own first child PLAN, not the handover's Phase 1 (PLAN-001, below). |
🔴 Fleet-wide constraint, recorded 2026-09-05
No one can observe asgard. kubectl is absent on tecMacDev and on the ops host; the ops host
has the kubeconfig but no client; huginn runs inside the cluster but is excluded pending login.
The tester's green Dagster verification of 2026-09-04 describes its own cluster, not ours.
asgard's Dagster is unverified since 2026-08-30. The public API hostnames also do not resolve
from this machine, so the data cannot be probed from outside either.
✅ The struck sentence is no longer true, corrected 2026-09-23. api-atlas.urbalurba.com
resolves and answers from tecMacDev today, and was measured end to end — latency by query shape,
row counts, payload sizes, allowed methods and the Cloudflare edge configuration. See
INVESTIGATE-parquet-duckdb-wasm-surface. ⚠️ The
rest of this constraint stands: that hostname is served from imac, not asgard, so probing it
says nothing about asgard. The observability gap is unchanged.
This is not an Atlas defect and not something Atlas can fix. It is recorded here because it bounds what any item below can claim: no plan may treat "verified" as meaning verified on the instance that serves data, unless it names who ran the check and where.
Tier 1 — do next
Data platform (from the 2026-08-30 Sunday tick)
| # | Item | Effort | Why this tier |
|---|---|---|---|
| — | 🔴 Brreg: all 1.17M organisations + automated updates — INVESTIGATE-all-brreg-organisations | L | Decided by Terje 2026-09-11: full Enhetsregisteret, no new public endpoint. Needs a PLAN. Architecture is settled — bulk snapshot + /oppdateringer change feed polled by ?oppdateringsid=, append-only into raw.*, reconciled by a dbt incremental model. Two Brreg sources: Enhetsregisteret for the base, the dedicated Frivillighetsregisteret API to enrich the ~72,806 voluntary ones. ⚠️ One question surfaced after the decision and needs Terje's confirmation: Enhetsregisteret is two registers — enheter (1,174,098) and underenheter (862,903, own change feed). "Everything" was answered before that distinction was put to him. Recommendation in the file: enheter only in the first PLAN. |
| — | 🔴 lands-with.sh has been wrong in BOTH directions, so no safety margin applies to it | atlas-data/uis/lands-with.sh | 2026-09-24 |
| — | 🔴 navn is unindexed, so name search and name sort scan 1.17M rows — INVESTIGATE-parquet-duckdb-wasm-surface | S (hardening) / M (Parquet) | New 2026-09-23, measured from tecMacDev against the live host. order=navn.desc takes 5.2–7.0 s, and an ilike matching nothing takes 8.96 s; concurrent scans saturate the origin (figures on urb-agents #1438, not here). ⚠️ First diagnosis said "no indexes on marts.*" and was wrong — it came from a case-sensitive grep "CREATE INDEX" that could see neither lowercase SQL nor dbt's declarative indexes= config. dim_brreg_enhet has seven indexes; an indexed lookup matching nothing returns in 0.32 s against the same query's 8.96 s unindexed. The real gap is one column, navn — btree plus pg_trgm GIN, added the way the existing seven were. Cloudflare reports 0.01% cached (14 kB of 108 MB in 24 h); there are no Cache Rules and PostgREST sends no Cache-Control/ETag. ✅ Split it: the hardening half (db-max-rows, statement_timeout, indexes) needs no decision from Terje and removes a collapse mode anyone can trigger from a browser address bar — do that first. The Parquet half is the four decisions in the Terje table. 🔵 Directly downstream of the Brreg row above: the 1.17M decision is what makes this load-bearing. |
| 0 | INVESTIGATE-atlas-as-a-uis-application | L | 🔴 The product's target shape (Terje, 2026-09-06). Install Atlas on UIS with one command, using the Dagster/PostgreSQL/PostgREST it already ships and the config system it already has. Acceptance is stricter than anything before it: the API answers from tecMacDev over the LAN, because the frontend will be built against it from there. Two machines, one network — no public domain, no tunnel. Needs tor-agent — the platform may need an Application type it does not have. |
| 0 | INVESTIGATE-atlas-data-as-deployable-application | L | 🔴 The product's target shape (Terje, 2026-09-05). One installable application that gathers the data and makes it queryable, with the frontend as a forkable example. The container already exists and runs on UIS; what is missing is the installer, the declared query surface, and a repo split blocked by a build-time coupling from the docs site. |
| 0b | PLAN-zero-touch-ingest-automation | M | ✅ Closed, 2026-10-01 — #1793. Terje's standing requirement: "when someone installs it they should never need to manually start something." Phase 1 (code defaults to RUNNING, PR #482) AND Phase 2 (live cluster reconciled after redeploy, verified independently by ops-dev via kubectl + a fresh GraphQL query, not trusted from imac's report) both confirmed done. Phase 3 cross-references F1 (redcross-branches, unchanged). Phase 4 (bufdir masking 36 successes) traced to a transient network fetch failure and fixed with a tested retry (PR #483). Independently re-verified same day: brreg-frivillige and brreg-oppdateringer both show total_runs=1 with a fresh last_ingested_at. |
| 0c | INVESTIGATE-new-norwegian-public-sources | L | ✅ Hold lifted, 2026-10-01 — was held behind 0b, now resolved. 3 candidates shipped without a PLAN file (bufdir-barnefattigdom, ssb-10826, ssb-crime-tables); bufdir-barnevern shipped with one (see below); none of NAV (4 families), IMDi (3), Udir (5) or Husbanken exist yet. ⚠️ The "Phase 0 schema prep" this row previously pointed to was a phantom — checked the actual schema 2026-10-01: no closed provider enum exists to bump (publishers.yaml is the real, open registry) and no dim_period/dim_indicator tables exist to extend. Publisher metadata for NAV/IMDi/Udir/Husbanken landed instead (#486, real logos sourced + verified from each agency's own site), which is all the "provider" half ever needed. There is no remaining prerequisite. ✅ PLAN-003-bufdir-barnevern.md shipped end to end, 2026-10-01 — ingested (191,673 rows, zero dropped), published to api_v1, deployed to the live cluster by imac, independently re-verified against the public API. Caught two real data defects by validating against real ingested data rather than an empty schema: Barnevern's own andel/prosent vocabulary mismatch, and a Svalbard-as-kommune/fylke misclassification (the exact class of bug classify_region_code was built to fix, urb-agents #700). ✅ PLAN-004-nav-uforetrygd.md shipped end to end, 2026-10-01 — Atlas's first monthly-cadence source. Ingested (6,560 rows, zero dropped), published to api_v1, deployed to the live cluster by imac (urb-agents#1797), independently re-verified against the public API. No data defects found this round — unlike bufdir-barnevern, kommune_nr/region_kind used the established macros from the first commit. Found and corrected two more wrong investigation claims before drafting: NAV's licence is CC BY 4.0 (not NLOD), and the download is a nested fylke→kommune→bydel pivot table (not a flat sheet). ✅ PLAN-009-imdi-bosetting.md shipped end to end, 2026-10-01 — Terje authorized 2026-10-01: "IMDI is ok. we can use it." Ingested (7,848 rows, zero dropped), published to api_v1, deployed to the live cluster by imac (urb-agents#1799), independently re-verified against the public API. manifest.yml records license: NLOD as Atlas's own default for an unstated Norwegian public-sector licence, applied under that authorization — not a citation of something IMDi itself states. Two real findings during implementation: IMDi publishes kommune names only, no codes — resolved via crosswalk_kommune_name restricted to active-only rows (352/359 matched, zero ambiguity, vs. 259/359 ambiguous unrestricted); and the metric column count genuinely varies by year/kommune (Oslo's 2024 table piloted 2 extra "avtalt" metrics before they rolled out everywhere in 2026) — parser resolves by header text, not position, to represent that rather than normalize it away. ⚠️ "IMDi" is only 1 of its own 3 flagged indicators — imdi-innvandringsgrunn-kjonn/imdi-landbakgrunn ([Q42]/[Q43]) were deliberately not folded into PLAN-009, split out as INVESTIGATE-imdikator-api.md, see Tier 2. ✅ PLAN-010-udir-gsi.md shipped end to end, 2026-10-02 — Atlas's first Udir source. Ingested (18,816 rows, zero dropped), published to api_v1, deployed to the live cluster by imac (urb-agents#1806), independently re-verified against the public API, including an explicit Svalbard-split check (18,768 real kommune rows, 48 correctly-null Svalbard rows — matched exactly). Phase 1 found Udir's own public API docs name a dead hostname (api.udir-statistikkbanken.no — cert mismatch, Azure 404) and the real one is api.statistikkbanken.udir.no; from there found a single documented (if "not intended for external use") API covering GSI, Elevundersøkelsen, Nasjonale prøver, udir-fravar ([Q39]) and udir-sluttet-vgs ([Q40]) all at once — meaning the Udir follow-ups should be fast ones reusing one client, not separate research (numbered whenever drafted — PLAN-011 went to Husbanken instead, see below). One real implementation-time correction: the Phase 1 claim that kommune_nr needed no derivation macro was wrong — Svalbard sits at the same API hierarchy depth as genuine kommuner; fixed before shipping by naming the raw column region_code and resolving kommune_nr through the existing classify_region_code macro. Deliberately does not build dim_school despite the investigation's own "settles dim_school" framing for this slot — kommune-level only for v1, named as a deviation, not silently dropped. ✅ PLAN-011-husbanken-statistikkbank.md shipped end to end, 2026-10-02 — all four phases done: ingest (35,295 rows, zero dropped), dbt staging + api_v1 publication, deployed to the live cluster by imac, independently re-verified against the public API — GET /indicators__husbanken_bostotte?limit=1 returns real rows, content-range confirms 35,295. Terje confirmed the licence 2026-10-02: "Husbanken is owned by the norwegian goverment and they follow NLOD." A direct statement, not just an authorization — manifest.yml records license: NLOD as what Terje stated, still not independently verified on a Husbanken-published terms page (none was found). Phase 1 found the investigation's "Power-BI-backed" claim was wrong: statistikk.husbanken.no is Qlik Sense, backend at qlik.husbanken.no, anonymously reachable, Atlas's first WebSocket-based ingest. Real kommune_nr confirmed, no crosswalk needed — but Oslo's bydeler turned out to be present under a second, 4-digit numbering distinct from FHI's 6-digit one, caught by a real relationships test failure against loaded data; classify_region_code now handles both. The Boligsosial Monitor (homelessness/kommunal-bolig) turned out to be a separate mechanism, split off as its own future investigation, same shape as imdi-bosetting's IMDikator decision. Deploy took three attempts — a missing npm script (urb-agents#1807) then a WebSocket global absent on the deployed Node 20 (urb-agents#1808), both invisible to local testing on a newer Node, both fixed and reproduced against the deployed Node version before landing clean (urb-agents#1809). ✅ PLAN-012-nav-aap.md shipped end to end, 2026-10-02 — all four phases done: ingest (5,720 rows, zero dropped), dbt staging + api_v1 publication, deployed to the live cluster, independently re-verified against the public API — GET /indicators__nav_aap?limit=1 returns real rows, content-range confirms 5,720. Atlas's second NAV source and second monthly-cadence source, reusing the existing monthly_sources_refresh job. Phase 1 found this table's shape is simpler than nav-uforetrygd's PST302 (no fylke-sheet duplicate, no bydel nesting, no row-order inconsistency) and independently re-confirmed CC BY 4.0 for AAP specifically rather than inheriting it by assumption from the uføretrygd correction. One new finding, caught by a real test failure: the non-numeric Ukjent (unknown-region) bucket exists only in the Antall sheet, not Andel (358 regions vs 357) — falls through to classify_region_code's existing unknown branch cleanly. Deploy hit an unrelated platform incident (checksum-verified PostgreSQL corruption on dim_brreg_enhet, the second that day) — correctly escalated rather than self-repaired, verified clean, then landed (urb-agents#1810). ✅ PLAN-013-nav-sykefravaer.md shipped end to end, 2026-10-02 — all four phases done: ingest (196,770 rows, zero dropped), dbt staging + api_v1 publication, deployed to the live cluster, independently re-verified against the public API — GET /indicators__ssb_12451?limit=1 returns real rows, content-range confirms 196,770. Implements what the investigation called "nav-sykefravaer" as ssb-12451 instead: NAV's own sykefravær/sykepenger pages publish no kommune-level table at all, checked live; the real kommune-resolved data is SSB table 12451, reached via this project's existing lib/pxweb.ts — the same mechanism every ssb-* source already uses, not a NAV-family Excel parser. Licence is NLOD (SSB's own), not CC BY 4.0. One new finding, caught by querying the built table directly: 0716u ("Våle (-2001)"), SSB's own disambiguation suffix for a historical kommune whose code was later reused — falls through classify_region_code's existing branches to unknown cleanly, no macro change needed. The simplest Phase 2 of any source this session: zero relationship-test failures on the first dbt build. ✅ PLAN-014-nav-helt-ledige.md shipped end to end, 2026-10-02 — all four phases done: ingest (5,744 rows, zero dropped), dbt staging + api_v1 publication, deployed to the live cluster by imac (urb-agents#1813), independently re-verified against the public API — GET /indicators__nav_helt_ledige?limit=1 returns real rows, content-range confirms 5,744. Atlas's fourth NAV-adjacent source and third on monthly_sources_refresh. Unlike nav-sykefravaer (PLAN-013), the investigation's mechanism held exactly as described this time — NAV's own HL060 "Fylke og kommune" table is genuinely kommune-resolved. A fourth distinct NAV-Excel pivot shape, but the simplest to classify yet — neither the bare fylke header row nor the "I alt <name>" rollup carries any digit at all, so "exactly 4 leading digits" alone separates kommune rows; a fresh parser, not adapted from nav-aap's or nav-uforetrygd's. Both sentinel shapes (Svalbard's 2100, a bare Ukjent bucket) resolved exactly as predicted on the first dbt build, zero relationship-test failures. Two new findings: unlike nav-aap, Ukjent is present in BOTH the Antall and Prosent sheets here (its Prosent cells are suppressed, not omitted); and the Ukjent block's fylke-header row and kommune-leaf row share the identical bare label, with the header row correctly dropped by the ingest's existing hasData guard. CI's render-template-info.sh caught a stale raw-table count in template-info.yaml before merge — fixing it also surfaced and corrected an adjacent false claim about what generate-holdings.py actually generates. A deploy-verification mismatch (a predicted mart_indicators__nav_helt_ledige inventory entry that doesn't exist) was flagged rather than silently reconciled by imac, then confirmed to be a mistake in this plan's own request template, not a defect, by checking the same 404 exists for the long-shipped nav-aap sibling. ✅ PLAN-015-udir-elevundersokelsen-mobbing.md shipped end to end, 2026-10-03 — all four phases done: ingest (2,804 rows, zero dropped), dbt staging + api_v1 publication, deployed to the live cluster by imac (urb-agents#1822), independently re-verified against the public API — GET /indicators__udir_elevundersokelsen_mobbing?limit=1 returns real rows, content-range confirms 2,804. Atlas's second Udir source. Phase 1 found this report's shape is NOT udir-gsi's shape despite sharing one client: geography (EnhetID) is a column dimension here, not a row dimension, so udir-gsi's depth-filtered-radSti technique doesn't apply. Found and corrected an assumption carried from udir-gsi: TrinnID(-10) is NOT an "alle" sentinel for this report — it silently resolved to a single wrong grade without erroring. Resolved [Q1] (bulk-decode vs. per-kommune calls) in favour of correctness over call count — ~702 calls, one per kommune-equivalent region node per grade. The real cost of that decision turned out to be wall time, not request volume: per-call latency settles at ~3.5-6s each after an initial fast burst, not predicted in Phase 1 — 60.3 minutes in dev, 23.5 minutes in production, both well inside the "SLOW RUN, NOT STUCK" envelope flagged in the deploy request. Two genuinely new findings caught against real data: a region/grade pair can be entirely absent, not suppressed (Hægebostad has no 10th-grade cohort at all, confirmed in both dev and production); and a new sentinel, "Utlandet, uspesifisert" (2599, Norwegian schools abroad) — falls through classify_region_code's existing unspecified_within_fylke branch cleanly, alongside the already-known Svalbard (2100) sentinel. Applied PLAN-014's own closing lesson proactively — predicted only indicators__udir_elevundersokelsen_mobbing as served, with no discrepancy to flag this time. ✅ PLAN-016-udir-nasjonale-prover.md shipped end to end, 2026-10-03 — all four phases done: ingest (8,589 rows, zero dropped), dbt staging + api_v1 publication, deployed to the live cluster by imac (urb-agents#1824), independently re-verified against the public API — GET /indicators__udir_nasjonale_prover?limit=1 returns real rows, content-range confirms 8,589. Atlas's third Udir source, and its first direct learning-outcome signal. Phase 1 found this report's shape matches udir-gsi's, not udir-elevundersokelsen-mobbing's — EnhetID is the row hierarchy here (confirmed via the response's own rowHierarchy metadata, not assumed from either sibling), so the cheap radSti depth-filter technique applies — 18 calls total, 29.9s in dev, 52.72s in production, no "slow run" warning needed unlike its sibling. A real correction, caught mid-implementation not in Phase 1: the 5th-grade report's own Rapportside.gyldigeFiltre omits TrinnID, which Phase 1 read as "no TrinnID filter exists" — wrong; filterVerdier still carries one real TrinnID entry and the data endpoint accepts it explicitly with an identical result, so parse.ts discovers grade uniformly from filterVerdier for both report versions, no special-casing. Genuinely asymmetric grade×subject matrix (English only at 8th grade, not 9th — confirmed live via a real empty response) and a genuinely new finding neither sibling Udir source needed: Utlandet (schools abroad) sits under its own top-level node, a sibling of "Hele landet" rather than a descendant, so a second radSti anchor is required to reach it — and it carries real, non-suppressed data (73 pupils), so v1 includes it rather than dropping it. Both Udir sentinels (Svalbard's 2100, Utlandet, uspesifisert's 2599) resolved exactly as predicted via classify_region_code's existing branches, no macro changes needed. Licence NLOD, same portal already confirmed twice this session. A related, unrelated-source report (urb-agents#1826) surfaced during this close-out — a transient HTTP 400 on sibling udir-elevundersokelsen-mobbing — reproduced live as a one-off blip, no code change, and corrected a prior memory note's latency claim to reflect run-to-run variance even in production. ✅ PLAN-017-udir-fravar.md shipped end to end, 2026-10-03/04 — all four phases done: ingest (21,290 rows, zero dropped), dbt staging + api_v1 publication, deployed to the live cluster by imac (urb-agents#1827), independently re-verified against the public API — GET /indicators__udir_fravar?limit=1 with count=exact confirms content-range: 0-0/21290. Atlas's fourth Udir source, same EnhetID-row-hierarchy shape as udir-gsi. Two real corrections caught during Phase 2, not Phase 1: backfills all 11 available years (not latest-only — udir-gsi's own code, not its README's prose, backfills fully when the call cost is cheap, and this report's cost matches); and the Utlandet/2599 anchor's apparent "empty response" was a Phase 1 testing bug (a conflicting EnhetID(-12) filter alongside the -13.*.* radSti anchor), not a real finding — it carries real data every year, same shape udir-nasjonale-prover found. VGO_fravaer (videregående) deliberately not shipped — its EnhetID hierarchy has no kommune level at all, named as [Q4], a cross-cutting decision carried to the investigation doc rather than resolved unilaterally. Deploy took ~86 minutes (this API's own latency variance, confirmed again), correctly distinguished from a hang by polling Dagster's API directly. |
| 1 | INVESTIGATE-ingest-freshness-visibility | M | 🔴 Highest. On 2026-08-30, 15 of 41 sources silently did not refresh and every signal stayed green — the check suite returned identical numbers. We cannot currently tell "refreshed and unchanged" from "never refreshed". Monitoring that cannot distinguish those is not monitoring. The in-suite half now ships and has been seen to fail on-cluster (FAIL 28 → 24 → 2). 2026-09-08: its single 8-day threshold was wrong for the two monthly-polled KLASS sources and was measured to go falsely red from 09-13 until 10-01 — thresholds now derive from a declared meta.ingest_cadence, and brreg-enheter became a Dagster asset rather than being exempted. The remaining gap is unchanged and is what this item is still open for: if the daemon stalls, nothing runs, so nothing reports. That needs a reader outside the pipeline. |
| 1b | INVESTIGATE-failed-transform-leaves-the-api-dark | M | 🔴 Same family as #1, and filed 2026-09-21 after being promised three times. When a model fails, api_v1_surface is skipped, so the wrappers are never re-applied — and api_v1_rowcount_matches_marts is an asset check on api_v1, so the check that would notice is skipped too. The failure mode is silence, not a red check. ⚠️ The mechanism that actually removed three relations on 2026-09-21 is inferred, not measured; the file names the one measurement that settles it. Every occurrence so far was caught within the hour only because a deploy was in flight and someone was watching. |
| 1c | meta_dimensions has no catalog page — the website enumerates Atlas views from lineage.csv, which maps models to raw sources; mart_meta_dimensions reads a seed, so it has no edge and has never appeared. 17 view pages exist and it is not one. ⚠️ It is the relation whose undiscoverability produced urb-agents #1333, #1335 and #1344, and the two catalogue pages that DID appear on 2026-09-21 (meta_sources, meta_endpoints) appeared by accident, as a side effect of an unrelated lineage change. Fix: enumerate views in website/scripts/generate-sources-registry.mjs from the generated seeds/sources/api_v1_relations.csv (added 2026-09-21) rather than inferring the set from lineage; lineage stays the input for what feeds a view, not for whether it exists. Expect views with an empty lineage section — the page should say so rather than be omitted. | S | Small, well-specified, and the correct input now exists. |
| 1d | Six two-digit codes claim to be fylker and are not — INVESTIGATE-two-digit-codes-that-are-not-fylker | S-M | classify_region_code maps ^\d{2}$ to fylke; 21 Svalbard, 22 Jan Mayen, 23 Kontinentalsokkelen, 25 Utlandet, 26 Havområder, 88 Ikke bosatt i Norge are not fylker — measured against Klass 104 and SSB's own Region labels. 3,648 rows, failing relationships tests since the ssb-06913 recovery. ⚠️ The classifier already handles 21xx/22xx/23xx correctly, so 21 and 2100 disagree about the same place. Blocked only on a decision: the macro is used by 22 models and ref_region_kind is a gated closed vocabulary. Options and a recommendation are in the file. |
| 1e | ssb-12063 ingests one year and publishes as if it had all of them — its filter is Tid: "TOP(1)", so only the most recent year is ever fetched. KOSfritidredleie0000 last carried a value in 2018 and has therefore never been ingested: 1,812 non-null cells exist at SSB across 2015-2018 (408 kommuner in 2018 alone, measured by ops-dev) and none are in the fact table. ⚠️ Nothing is dropped between raw and fact — those years were never requested. It applies to the whole source, not one series. Decision: widen the filter and gain history for every series, or document the source as current-year-only. Publishing a series that looks empty is not a third option (urb-agents #1362). | S | Cause is known and one line; the choice is editorial. |
| 2 | PLAN-ingest-retry-budget | S-M | A short Retry-After overrides the backoff ladder and collapses the retry budget to ~4s; two of three HTTP clients ignore Retry-After entirely. Decides whether the next weekly tick survives a wobble. Does not help against a multi-hour outage. |
| 3 | PLAN-ingest-ci-gates phase 3 | S | Phases 1–2 shipped 2026-08-25. Only the C12 move to Node 24 remains, and it needs a real Node 24 to validate rather than an assumption. |
| 4 | INVESTIGATE-ssb-api-version-dependency | S-M | All SSB ingest depends on a beta API surface that was 503 for hours on 2026-08-30. /v2/ has now shipped and serves data identical to /v2-beta/, so moving is a durable-reliability improvement, not a fix. ⚠️ A 2026-09-05 claim that beta served stale data was wrong and has been retracted. |
Semantics and catalogue
| # | Item | Effort | Why this tier |
|---|---|---|---|
| 5 | INVESTIGATE-semantic-foundation-before-expansion | L | Highest unblocking ratio in the doc. Decides the concept-catalogue format and explicitly freezes NGO-supply expansion until resolved — blocks 4 other INVESTIGATEs from passing their own boundaries. ⚠️ Never actually decided (last updated 2026-04-25; its own [Q2], [Q9]–[Q12] are still open questions, not a standing rule) — and its [Q4] (cross-NGO activity taxonomy) is now answered by the NGO research handover's taxonomy PR (#555, 2026-10-04). Needs Terje's attention regardless of this tier's order: does the handover supersede this freeze proposal, or does this investigation still gate it? |
| 6 | INVESTIGATE-mart-meta-dimensions-cardinality | M | Catalogue UX needs cardinality + example values to render "what each column actually contains". Small investigation gap, real ship-side gap. ⚠️ Its original justification was feeding PLAN-007 phase 4 — PLAN-007 has since shipped, so re-check that the need survives before starting. |
| 4b | INVESTIGATE-agent-onboarding-friction | S | 🔴 Evidence from an outside consumer, not from us. A fresh agent given only two URLs built a dashboard in 4 minutes and filed 9 findings. Three shipped (#478 summaries, #479 map boundaries). Five remain open and need a decision, not code: the agent entry point ([Q1]), readable topics for the 40 indicators__* ([Q2] — it skipped all 40, half the catalogue), served_as semantics ([Q3] — second consumer to misread it), coverage years ([Q4]), small conveniences ([Q5]). |
| 4c | INVESTIGATE-samfunnspuls-replacement-gaps | L | 🔴 The same outside agent rebuilt samfunnspuls.rodekors.no on Atlas alone, in ~8 minutes. What stopped a full replacement is 5 decisions and a source programme: no time series in one shape ([Q1]), no official fylke/national values ([Q2] — measured: median 12.30% vs true national 11.36% for child poverty), no unit metadata ([Q3] — it guessed a unit wrong), 15 of 33 graphs with no data ([Q4]), no titles/topics ([Q5], same cause as 4b's [Q2]). ⚠️ [Q6] records a report suggestion that is WRONG and would have edited source data. |
| 4e | — | — | 🔴 2026-10-01, a THIRD independent consumer (Lovable UI) hit the same G4 gap and asked how to acquire what's missing. Verified and merged into 4c: [Q7] FHI 175/932 RESOLVED — not a data gap; both ingested fine, mart_meta_sources was stale (every Dagster schedule/sensor on the cluster stopped since a rdctl reset --vm, affecting every source's mart, not just these two — see urb-agents #1787); [Q8] the bespoke population×household extract resolved — no public SSB substitute exists, Røde Kors must supply it; [Q9] two corrections (trangbodd is FHI not Bufdir; ssb-13995's one-year reading needs re-verification once schedules resume, per [Q7]). And it independently re-hit item 16's NAV/IMDi/Udir gap, confirming that investigation's May work was never executed. |
| 4d | PLAN-010-indicator-catalogue-data-quality | S | The four items from 4c that need no decision — fylke keys on kommune_ngo_totals, publishing the ¬ classification depth, the one valueless indicator, the retired crime code system. Carved out so it is not held behind [Q1]–[Q5]. |
| 7 | INVESTIGATE-felles-datakatalog-classification | S | Already half-shipped (eu_theme: landed in PLAN-007 phase 2.10). A few hours of LOS-vocabulary mapping buys one-line interop with data.norge.no. |
Tier 2 — after Tier 1 (independent, ready, valuable)
| # | Item | Effort | Why this tier |
|---|---|---|---|
| 8 | INVESTIGATE-transform-job-decomposition | M | Re-scoped 2026-08-24. The tactical unblock shipped; layer-splitting and the CI plan-size budget (~400) are deliberately parked pending the declarative-automation direction. Reopen when that direction settles — the durable fix, not the bump. |
| 9 | INVESTIGATE-reports-and-indicators-from-catalogue | XL | Substrate for prioritising new sources; without it "what to ingest next" is gut feel. Begin once Tier 1's catalogue shape settles. |
| 10 | INVESTIGATE-multi-ngo-supply-model-extensions | M | Small schema change that unblocks two Tier-3 items together. Cheap, high downstream payoff. |
| 11 | INVESTIGATE-developer-docs-surface | M | External developers will arrive at a public API without docs. Investigate the shape before users land, not after. |
| 12 | INVESTIGATE-data-freshness-surface | M | Reader-facing freshness — deliberately distinct from #1, which is operator-facing. The two may share a max(loaded_at) per-source model; keep them separate. |
| 13 | INVESTIGATE-indicators-schema-coverage | S-M | 249 of ~566 marts columns lack descriptions (25 % coverage), mostly indicators__* pass-throughs. Settles hand-fill vs generate-from-manifest vs punt. Related: PLAN-indicators-schema-generator. |
| 14 | INVESTIGATE-sources-catalog-at-scale | M | Was missing from this doc entirely. Catalogue behaviour as source count grows. |
| 15 | PLAN-008-developer-discovery-surface | M | Ready to execute — the Atlas-native subset of data-discovery (Scalar spec viewer, lineage panel, dbt docs hosting). |
| 16 | INVESTIGATE-imdikator-api | S-M | Split out of PLAN-009-imdi-bosetting.md, 2026-10-01 — reverse-engineer IMDikator's undocumented backend to serve imdi-innvandringsgrunn-kjonn/imdi-landbakgrunn, the 2 of 3 flagged IMDi indicators imdi-bosetting doesn't cover. No prereqs, not blocking anything. Effort is genuinely unknown until someone drives a headless browser at it — could be small (an open JSON API once found) or could dead-end entirely if [Q1] (does it even carry this data) comes back no. |
Tier 3 — defer until prereqs ship
| # | Item | Waits on | Why defer |
|---|---|---|---|
| 17 | INVESTIGATE-supply-frontend-display | #10 | URL structure and viewing layers depend on schema shape. UX before schema = rework. |
| 18 | INVESTIGATE-folkehjelp-supply | #10 | Schema lands first; the second-NGO ingest then validates it. |
| 19 | INVESTIGATE-data-discovery-surface | #5 for the OpenMetadata path | The Atlas-native near-term path is split out as #15 and is ready now. Wider OpenMetadata adoption stays deferred. |
| 20 | INVESTIGATE-cloud-agent-source-onboarding | #16 | An agent that onboards sources needs a list of which sources first. |
| 21 | INVESTIGATE-private-atlas-deployments | product clarity | Worth doing only against a concrete first private tenant. Speculative architecture rots fast. |
| 22 | INVESTIGATE-deployment-pipeline | UIS/dagster direction (external) | ⚠️ Partly overtaken: the asgard deployment shipped and dagster is live with automation running. Re-read before opening — its premise may already be answered. |
Tier 4 — ideas, not investigations
| # | Item | What to do |
|---|---|---|
| 23 | INVESTIGATE-ngo-events-and-minisites | Parked in the file itself. Re-evaluate once a second NGO is in flight and the gap is visible rather than speculated. |
| 24 | INVESTIGATE-tag-indicators-sdg-icnpo | Overlaps #5's ICNPO tagging. Hold — #5's answer may absorb it entirely. |
Recently closed
- PLAN-007 (customer-frontend data display) — shipped;
INVESTIGATE-customer-frontend-data-displayneeds no further investigation work. - The asgard deployment — Phase 2 complete (2.3: 649 checks, 629 dbt PASS, 17 known WARNs, 10/13 views, 0 orphans — ⚠️ those three figures do not subtract, and ops caught it on urb-agents #632. They reconcile only if 649 is the combined pre-split suite: 629 + 17 = 646 dbt checks, plus
api_v1_checks' 3. Today the split makes it 647 intransform_checksand 3 inapi_v1_checks, measured identically on two topologies. Cite the current figures, not these) and Phase 3.2's concurrency bound verified at 4 against a real weekly fan-out on 2026-08-30. ⚠️ The observed weekly cycle failed (15 SSB sources, HTTP 429), so the bound is proven but a clean end-to-end weekly cycle has not yet been observed.
Cross-cutting notes
- ⚠️
template-info.yaml'soperational.automationtext may now be wrong about what a fresh install does. Flagged by ops-dev while landing the nav-helt-ledige pin (urb-agents#1814, 2026-10-02), not settled here. The text reads "Ships stopped. No data is fetched and no external service is contacted until an operator enables both the schedules and the sensors" — buturb-agents#1794's own redeploy (PR #482,default_status=RUNNING) measured all three schedules and sensors coming upRUNNINGon install, with the automation sensor firing an unprompted real run ~15 minutes later. ops-dev checked for a reconciling explanation (an install-time stop hook inuis/init/) and found none —001_bootstrap.sqlis one line of schema creation. So either this redeploy wasn't a realuis template install(and the text is still correct for that path), or the text is stale since PR #482 and needs to say "ships RUNNING, not stopped". dev-templates' own catalogue quotes the artifact faithfully either way and flags the question as open (uis-applications/atlas/README-atlas.mdinhelpers-no/dev-templates) — settle it here, inatlas-data/template-info.yaml, next release that touches automation. - 🔴 One check outranks everything in this doc, and it is not a decision: does the public API
accept writes? Found while measuring the Parquet item, 2026-09-23. The live API advertises
Allow: OPTIONS,GET,HEAD,POST,PATCH,DELETEon all 16 auto-updatableapi_v1views; the 4 aggregate views showGET,HEADonly — which is Postgres refusing to write through aggregates, not a privilege boundary.api_v1_generated.sqlgrantsSELECTonly, but inside a role-existence guard, so the grants can silently never apply. ⚠️ No write was attempted, and the mechanism is deliberately not spelled out in this public repo — it is on urb-agents #1438. It should be settled with\duand\dpon the host rather than by testing against a public endpoint. [Q14] in INVESTIGATE-parquet-duckdb-wasm-surface. - The data-platform cluster (#1–#4) is new and currently the most urgent, all of it from one night's evidence. #1 is the one with teeth: #2 and #4 are about preventing a missed refresh, #1 is about noticing one.
- Two long-standing workstreams still run in parallel: semantics/catalogue (#5, #9, #10) and frontend/UX (#6, #11, #12). Different files, different agents, no merge contention.
- Supply-side chain: #10 → (#17, #18). Catalogue chain: #5 → (#19, #23, #24, parts of #9).
- External blockers: only #22, and it may already be resolved by the deployment shipping.
- Terje's queue is the real bottleneck, not the backlog — four items, three of them ≥4 days old.
How to use this doc
- Take the top unstarted Tier-1 item; if Tier 1 is in flight or done, drop to Tier 2.
- INVESTIGATEs stay in
backlog/until every child PLAN ships — update theirStatus:line rather than moving the file. - When an INVESTIGATE spawns a PLAN, strike the row here and note the PLAN.
- When a Tier-3 prereq lands, promote its dependents at the next refresh.
- Re-rank after every 3 items ship, or whenever a blocker clears.