Skip to main content

The datasets catalogue: how it's built

The public catalogue at atlas.sovereignsky.no/datasets is generated at build time from the per-source manifests. No content lives only in the website repo — every page on the catalogue is reproducible from atlas-data/ingest/src/sources/.

This page is for contributors who touch the catalogue's plumbing: adding topics, adding publishers, evolving the generator, or wiring new components into the per-dataset page.

For the "how do I add a dataset" workflow, see adding-a-source.md — Step 4b covers the manifest + regenerate dance.

A note on vocabulary: inside atlas-data/ the unit is called a source (source_id, raw.<source_id>, the sources/ folder). On the public site it's a dataset, topic, and publisher. Same things, different audience-appropriate names. See PLAN-catalog-vocabulary for the rationale.

What's generated vs hand-authored​

SurfacePathAuthored by
Per-dataset product detail pagedocs/datasets/<source-id>.mdx × 41Generated by sources:generate
Per-topic indexdocs/topics/<id>.mdx × 7Generated
Per-publisher brand pagedocs/publishers/<id>.mdx × 4Generated
Registry JSON (consumed by React components)src/data/sources-registry.jsonGenerated
Catalogue homepagesrc/pages/index.tsxHand-authored
Datasets index (browse all)docs/datasets/index.mdxHand-authored
Topics indexdocs/topics/index.mdxHand-authored
Publishers indexdocs/publishers/index.mdxHand-authored
React componentssrc/components/sources/*.tsxHand-authored
Topic metadataatlas-data/ingest/src/sources/topics.yamlHand-authored
Publisher metadataatlas-data/ingest/src/sources/publishers.yamlHand-authored

Generated files carry a {/* Generated by website/scripts/generate-sources-registry.mjs */} marker on the first MDX line. Don't hand-edit those files — your change will be overwritten next time someone touches a manifest.

The generator​

website/scripts/generate-sources-registry.mjs is the single entry point.

cd website
npm run sources:generate

What it does:

  1. Walks every atlas-data/ingest/src/sources/<id>/manifest.yml.
  2. Loads publishers.yaml and topics.yaml.
  3. Reads each source's README.md; extracts the section under ## Atlas summary (case-insensitive) if present.
  4. Resolves publisher: → publisher object and tags.topic → topic object so consumers don't need to join.
  5. Computes related_by_topic (other same-topic source_ids, alphabetical, capped at 6).
  6. Builds a default sample_query URL when the manifest doesn't supply one (first raw table + ?limit=5).
  7. Builds the citation text + BibTeX for each dataset, including the Atlas permalink /datasets/<id>.
  8. Resolves the effective feedback_url (manifest override > publisher default).
  9. Writes src/data/sources-registry.json (single file, deterministic).
  10. Scrubs stale generated MDX in each output dir so renames / deletions don't leave orphans.
  11. Writes per-dataset / per-topic / per-publisher MDX to the three output dirs.

The generator is also wired to prebuild — npm run build always regenerates before building, so local previews are never stale.

Adding a new topic​

When a new domain doesn't fit the existing 7 topics:

  1. Add an entry to atlas-data/ingest/src/sources/topics.yaml with id, name, description, emoji, order. The id MUST match what any manifest's tags.topic field uses — the manifest validator rejects mismatches.
  2. Run cd website && npm run sources:generate. The generator emits a new docs/topics/<id>.mdx.
  3. Add the new topic id to sidebars.ts under the "By topic" submenu (Atlas keeps the order explicit, not autosorted by order).
  4. Discuss in the PR — topics are editorial and we keep the list short.

Adding a new publisher​

  1. Add a <id>.svg to website/static/img/publishers/. Placeholder text-wordmarks are acceptable for v1; replace with the publisher's official asset once licensed.
  2. Add an entry to atlas-data/ingest/src/sources/publishers.yaml with id, display_name (must exactly match the publisher: string in every manifest from that publisher — validator cross-checks this), homepage, logo path, feedback_url, notes.
  3. Run cd website && npm run sources:generate. The generator emits docs/publishers/<id>.mdx.
  4. Add the new publisher id to sidebars.ts under the "By publisher" submenu.

Adding a new component to the per-dataset page​

If you want to render something new on every per-dataset page (e.g. a "Used in N Atlas models" badge):

  1. Write the component under src/components/sources/<Name>.tsx. Read from typed registry data via src/utils/sources.ts helpers — don't fetch.
  2. Update the MDX template in generate-sources-registry.mjs (the renderSourceMdx function) to import and render your component.
  3. Run npm run sources:generate — every per-dataset page picks up the new component on regeneration.
  4. Stable anchor IDs are a contract — #provenance, #in-atlas, #joined-with, #methodology, #dimensions, #sample-query, #citation, #related. Renaming any of these is a breaking change for anyone deep-linking into Atlas (sister documents, third-party tools, LLM crawlers).

The CI gate​

Two jobs in .github/workflows/check-manifests.yml:

  • check-manifests — manifest shape + cross-file resolution. See check-manifests.md.
  • check-catalog — runs the generator and git diff --exit-code against committed output. Fails if a manifest was edited without re-running the generator.

Both jobs fail the PR if anything's off. The fix message names the exact command to run locally.

What's NOT in PLAN-002 / PLAN-catalog-vocabulary​

PLAN-002 shipped the structural shell; PLAN-catalog-vocabulary fixed the public terminology. PLAN-003 adds the merchandising layer:

  • Curated collections at /datasets/collections/<slug>
  • Use-case persona pages at /datasets/for/<persona>
  • "Featured / dataset of the week" hero on the homepage
  • "New & recently refreshed" strip + per-page RSS feed
  • Social-proof badges ("Used in N Atlas models", citation counts)
  • Sample-row preview tables
  • Sparkline of time coverage; Norway thumbnail map by geo_resolution_level

When PLAN-003 lands, this doc will gain corresponding sections.

Cross-references​