Skip to main content

End-to-end baseline — 2026-04-28

Snapshot of the full ingest → dbt → frontend pipeline taken right after PLAN-003 merged. This is the known-good starting point for further development. Re-run the same sweep before any major refactor to catch regressions.

Repo state at snapshot:

  • HEAD: c51d02c (PLAN-003 phase 1 squash merge — body covers all 6 phases)
  • Branch: main
  • Local Postgres reachable; dbt debug clean
  • All node_modules, dbt/.venv, ingest/.env present

Phase A — Ingest sweep

Result: 20 / 20 sources succeeded (after the ssb-06913 fix below).

cd atlas-data/ingest
for s in ssb-klass-kommuner ssb-klass-fylker ssb-08764 ssb-06083 ssb-06913 \
ssb-06944 ssb-06947 ssb-07459 ssb-09429 ssb-12063 ssb-12131 ssb-12132 \
ssb-12292 ssb-12944 ssb-13995 fhi-bor-alene fhi-mobbing fhi-trangbodd \
fhi-vgs-gjennomforing redcross-branches; do
npm run --silent "ingest:$s" 2>&1 | tail -1
done
SourceRowsStatus
ssb-klass-kommuner1 327
ssb-klass-fylker41
ssb-087641 790
ssb-060839 459
ssb-06913783 104✓ (after explicit-valuecodes fix; see below)
ssb-0694449 245
ssb-069475 180
ssb-07459210 728
ssb-0942941 118
ssb-1206313 365
ssb-121316 237
ssb-1213213 365
ssb-1229243 659
ssb-129444 296
ssb-1399530 294
fhi-bor-alene12 270
fhi-mobbing2 454
fhi-trangbodd36 810
fhi-vgs-gjennomforing24 540
redcross-branches391 + 2 143

ssb-06913 fix — explicit valuecodes required

During the baseline run ssb-06913 failed with PxWebAPI 400 "Parameter error: Non-existent value". Root cause: SSB's PxWebAPI v2-beta now rejects unfiltered /data calls for large tables (06913 is 1 288 × 8 × 76 ≈ 783 k cells, just under the 800 k cap). Smaller tables like 08764 (1 790 cells) still accept the no-filter default.

Fix landed (same PR as this baseline): the ingest module passes filters: { Region: "*", ContentsCode: "*", Tid: "*" } explicitly. Ingest now succeeds with 783 104 rows; downstream indicators__ssb_06913 and fact_kommune_indicators rebuild clean (51/51 tests pass).

If a similar 400 "Non-existent value" error appears on another source in the future, the same fix likely applies — add explicit * filters for every dimension.


Phase B — dbt build + test

Result: clean build, baseline match.

cd atlas-data/dbt
uv run --env-file ../ingest/.env dbt build
  • PASS: 654
  • WARN: 21 (matches the post-PLAN-002 baseline — 20 pre-existing data-quality warns + 1 expected new for Svalbard 2100 on mart_kommune_local_chapters)
  • ERROR: 0
  • TOTAL: 675

./check-osmosis.sh strict ✓ (every column in every schema.yml has a description, repo-wide).


Phase C — Frontend

Result: all 14 routes render with real data.

cd atlas-frontend
npm run typecheck # clean
npm run build # clean
npm run start # serves on :4000 (configured in package.json)

Note: Atlas's frontend serves on port 4000, not the Next.js default 3000 (port 3000 is held by the urbalurba-infrastructure Docusaurus site in this dev setup).

RoutebytesNotes
/8 648Home
/admin/supply/redcross-branches49 059Admin / supply audit
/coverage-gap/barnefattigdom60 192Map page
/data353 917Indicator index (all sources)
/data/ssb-08764/EUskala6056 342Per-indicator detail
/kommuner/0301220 145Oslo detail
/ngo54 925NGO landing
/ngo/folkehjelp12 921NGO detail (slug-based)
/ngo/redcross20 611Red Cross overview
/ngo/redcross/aktiviteter87 629Activity catalog
/ngo/redcross/chapters1 187 427All chapters (large response)
/ngo/redcross/chapters/redcross-S00817 415Chapter detail
/ngo/redcross/distrikt/redcross-D00315 967Distrikt detail
/ngo/redcross/distrikter37 831Distrikter index

Spot-checked content: /coverage-gap/barnefattigdom contains "Barnefattigdom", kommune_nr, fylke strings. /data/ssb-08764/EUskala60 contains "EUskala60", "kommune". Real renders, not error pages.


How to re-run this baseline

# Phase A — ingest (~5 minutes)
cd atlas-data/ingest
for s in ssb-klass-kommuner ssb-klass-fylker ssb-08764 ssb-06083 ssb-06913 \
ssb-06944 ssb-06947 ssb-07459 ssb-09429 ssb-12063 ssb-12131 ssb-12132 \
ssb-12292 ssb-12944 ssb-13995 fhi-bor-alene fhi-mobbing fhi-trangbodd \
fhi-vgs-gjennomforing redcross-branches; do
echo "=== $s ==="
npm run --silent "ingest:$s" 2>&1 | tail -1
done

# Phase B — dbt (~1 minute)
cd ../dbt
uv run --env-file ../ingest/.env dbt build

# Phase C — frontend (~1 minute)
cd ../../atlas-frontend
npm run typecheck && npm run build
npm run start & # serves on :4000
sleep 4
for r in "/" "/admin/supply/redcross-branches" "/coverage-gap/barnefattigdom" \
"/data" "/data/ssb-08764/EUskala60" "/kommuner/0301" \
"/ngo" "/ngo/redcross" "/ngo/redcross/aktiviteter" \
"/ngo/redcross/chapters" "/ngo/redcross/distrikter"; do
printf " %-50s %s\n" "$r" "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:4000$r)"
done
kill %1

Numbers will drift over time as upstream sources update; what matters for regression-checking is that every cell in the table above stays at "✓" / "200" / non-trivial body size.