Nigeria health facility register

Problem

Public health datasets in Nigeria often arrive in fragmented, overlapping releases across different agencies and timeframes. GRID3 published health-facility data in two major releases that cover different states and share no common identifier. Version 2.0 covers all 36 states and the Federal Capital Territory with 51,022 facilities keyed on a global identifier. Version 3.0 covers 24 states with 41,778 facilities keyed on a new unique identifier, superseding older records in those 24 states while leaving 13 states on the earlier release.

Because the two vintages share no common surrogate key, public health analysts could not track facility survival, closures, or re-surveys across versions. Furthermore, facility coverage data remained isolated from epidemiology. Planning targeted interventions requires joining facility capacity with disease prevalence, such as neglected tropical diseases and malaria, alongside population density and physical travel times across all 774 Local Government Areas.

How it works

The pipeline builds a unified authority layer in DuckDB, switching source version per state: version 3.0 where available, and version 2.0 elsewhere. Every row records its source version and flags whether the state is current or awaiting an updated survey. Staging models in dbt reconstruct lineage between releases using mutual nearest-neighbour spatial matching within a 100-metre radius on coordinates, followed by Jaro-Winkler string similarity comparison on facility names.

Downstream marts join the authoritative facility layer to disease burden across the canonical 774-LGA administrative frame. The pipeline aggregates five neglected tropical diseases from ESPEN (soil-transmitted helminths, schistosomiasis, lymphatic filariasis, onchocerciasis, and Loa loa) mapped to a 0 to 3 ordinal need rank. It enriches the frame with WorldPop population projections, Malaria Atlas Project clinical incidence and parasite rates, travel time to healthcare, and flood exposure flags.

Orchestration is handled by Dagster with an atomic promotion pattern. An automated sensor digests source files and rebuilds staging tables into run-scoped DuckDB sidecar files. Before promoting data to the live serving layer, Dagster evaluates a carry-over quality gate: if the spatial match rate drops below required thresholds, promotion is aborted and the live register remains untouched. Contracted marts are exposed read-only through Apache Superset dashboards for coverage planning.

Sources: GRID3 v2.0 & v3.0 | 774-LGA Frame | ESPEN NTD Burden | WorldPop & Malaria Atlas
   -> GeoPandas & Python: Run-scoped snapshot & geometry validation
   -> DuckDB Staging:
        - stg_facilities_authority (v3 where available, v2 elsewhere)
        - stg_version_lineage (mutual nearest-neighbour 100m + Jaro-Winkler match)
        - stg_ntd_endemicity_lga (5 NTD conditions, ordinal rank 0-3)
        - stg_access_population_lga (population, travel time, flood exposure)
   -> dbt Marts (enforced model contracts):
        - lga_coverage_burden | lga_version_drift | lga_coverage_outlook | register_summary
   -> Dagster Pipeline: Carry-over quality gate & atomic promotion (os.replace)
   -> Analytics: Apache Superset dashboards (coverage, burden, priority scores)

Hard parts

• Cross-version entity resolution without shared keys: reconstructed lineage using spatial proximity and mutual nearest-neighbour pairs within 100 metres, scoring name similarity with Jaro-Winkler into discrete confidence bands.
• Preserving missing and unsurveyed states: missing survey data, unsurveyed disease units, and unmatched administrative names are treated as unknown rather than converted to silent zeros.
• Strict administrative name reconciliation: matched heterogeneous state and LGA naming variations across datasets against a canonical 774-LGA reference frame using deterministic alias crosswalks.
• Atomic database promotion: executed transformation and validation inside isolated run-scoped DuckDB files, promoting to production via atomic filesystem renames only after all dbt model contracts and quality gates pass.
• Multi-criteria priority scoring: integrated healthcare facility density, travel times, population growth projections, flood risk, and disease endemicity into composite priority metrics for intervention planning.

Results

The repository documents verified spatial and lineage metrics evaluated against national datasets. The cross-version matcher links 89.8% of version 2.0 facilities to a version 3.0 point within 100 metres in covered states, achieving 89.9% name agreement across paired facilities.

Approximately 97% of facilities join to an administrative LGA in the canonical frame, with remaining records routed to an audit queue for manual reconciliation. All published marts conform to strict dbt model contracts with explicit type enforcement across all 774 LGAs.

• Lineage reconciliation across 51,022 version 2.0 and 41,778 version 3.0 health facilities.
• Standardized coverage, travel time, and disease burden indicators for all 774 Nigerian LGAs.
• Full test coverage with pytest unit tests, dbt schema tests, and Dagster asset checks.
• Complete containerized deployment with Docker Compose, Dagster webserver, daemon, and Apache Superset.

Artifacts

• Open-source repository on GitHub licensed under AGPL-3.0, with tagged releases through v0.7.0.
• Data transformations and marts defined in dbt with enforced model contracts and automated tests.
• Orchestration definitions, sensors, schedules, and quality gates implemented in Python with Dagster.
• Apache Superset dashboard configuration, database assets, and export bundles for geospatial analytics.
• Production deployment setup using Docker Compose, including worker queues and headless export services.