Systems and tools Counts real · refreshed automatically

Data lake

Every question in this portfolio requires several datasets at once, so the lake reconciles 220 of them onto one federal crosswalk and treats standardization as a product in its own right.

Data engineering · data science. 220 datasets · dozens of providers · over a million contributors · millions of records.
01

The problem

Connecting datasets from across the labor market requires a crosswalk onto the federal taxonomy, because federal statistics, commercial postings, training records, credential rosters and employer files differ in units, level of detail, vintage and license, and each names occupations its own way. The most valuable data comes from employers, and it can only be added if the system protects their anonymity.

02

How it works

01Crosswalk map

Crosswalks through the federal taxonomy connect industry, program and skill classifications. Controlled vocabularies and the Canon, a set of naming and calculation conventions I wrote, standardize ambiguous fields such as termination reasons, employment status and facility types, so tenant data and public data match up.

Data engineering · data science

02Registry as a product

Every dataset is tagged by readiness, lifecycle stage, geographic and time resolution, and provenance. The catalog is queryable and versioned, and new work starts from it, as it is the reference for what data exists, how good it is and how it was built.

Data engineering · product

03Contribution without exposure

Employer data joins under k-anonymity floors and row-level tenant isolation, with opaque identifiers throughout. The detailed panel is the reference, and benchmarks are aggregated views of it.

Data engineering · privacy

04Constantly checked

Each change is checked against the documentation, and row counts are regenerated and committed daily. Missing or orphaned tables get flagged automatically, which keeps the registry in line with the database it describes.

Engineering
Lake one dot per dataset

The full inventory of 220 datasets, by domain and readiness.

Derived products
73
Labor market
30
Employer
30
Education & training
22
Skills
19
Demographics
18
Macroeconomic
15
Interventions
7
Sector intelligence
5
HR standardization
1
One dot per dataset. Filled is live, and hollow is gated, scaffold, or planned.
220 datasets · dozens of providers · over a million contributors · millions of records
Canon one metric, set in full

I wrote the Canon as a naming and calculation convention for the whole system. Everything ingested is standardized to it, and a metric is only shown with its definition and source attached.

vacancy rate = open ÷ (headcount + open)
measured from
Employer panel, requisition lifecycle rows, monthly
standardized by
Title resolution to the federal crosswalk, level bands, termination decomposition
suppressed when
Headcount < 10 · occupation cell < 3 · fewer than 2 employers per benchmark cell
Librarian ask in plain language

You can ask the lake a question in plain words, and it picks out the concepts and geography using a controlled vocabulary before pointing to the datasets that can answer it, with live row counts. It uses no embeddings and generates no answers, so a query can only ever land on datasets that exist. The catalog metadata is the index, and any computation is handed to a separate analyst agent.

Deterministic routing over a 50-value concept vocabulary and 220 datasets, with no model calls per query. Each routing decision is a visible concept match, so you can see why a dataset was selected.
03

Tools used

Python, pandasIngestion and the crosswalk build across 220 datasets.
Supabase, Postgres, row-level securityTenant isolation, and the detailed panel that everything else reads from.
pgvectorEmbedding search across the catalog.
k-anonymity floorsSuppression on benchmark cells below their floor.
Controlled vocabularies, the Radius CanonThe naming and calculation conventions all ingested data follows.
Automated checksDaily count regeneration, documentation drift and orphan-table audits.
04

Final state

The lake holds 220 datasets from dozens of providers, covering over a million contributing institutions and respondents and millions of records, and users can explore their own data next to everything related to it.

← Back to systems and tools What this makes possible: both digital twins →