Code-Driven Manifesto: Fixing Local Gov Data with APIs, Schemas, and Pragmatic Engineering

どうも〜おかむーです！ Hi — today I'm diving into a bit of GovTech engineering with a manifesto-style take on how national and local governments should publish and operate their data and systems. エンジニア的に言うと、これはAPIとスキーマの話なんですよ〜

• Local government data often lives in PDFs and heterogeneous spreadsheets
• National guidelines (Digital Agency, MIC, Cabinet Secretariat) push standardization but execution gaps remain
• Practical fixes: API-first publishing, CSVW/JSON-LD metadata, a schema registry, and migration tooling

結論

The policy direction from Japan's Digital Agency and MIC is right: unify core business systems and make administrative data machine-readable. But in practice many datasets are still PDF-locked or inconsistent across municipalities. 要するに、方針はあるけど実装とエンジニアリングが足りないということです。エンジニアとしては、標準スキーマ＋API＋デベロッパー体験に投資すれば即効性ある改善が可能です！

Report: diagnosis, evidence, and technical prescriptions

What the gov guidance says (short)

• Digital Agency (digital.go.jp) is promoting unification/standardization of core local government systems and has liaison support for migration to standard-compliant systems.
• MIC (soumu.go.jp) has long promoted information system standardization and machine-readable statistics rules.
• Cabinet Secretariat (cas.go.jp) emphasizes making administrative data AI-ready and machine-readable.

These are great signals — but signals don't equal machine-readable endpoints.

The current technical gaps (these hit engineers every day)

• PDF-first publishing: budgets, plans, and statistics are often PDFs. This defeats automation and reproducibility.
• No stable, versioned APIs across municipalities — each city publishes differently (if at all).
• Schema inconsistency: same concept (e.g., "general account revenue") is labeled variably, with different fiscal year bases.
• Missing machine-readable metadata: no CSVW/JSON-LD, no DCAT catalog entries, no clear licensing headers.

これ見てくださいよ: MIC's guidance on machine-readable stats exists, but many local datasets still require manual scraping from PDFs — wasteful and error-prone.

Concrete, technical improvements (how to actually fix it)

API-first publishing

- Provide a REST/GraphQL endpoint for each common dataset (budget, contracts, asset registers) with OpenAPI specs and stable versioning.

- Example curl (conceptual):

curl -H "Accept: application/json" "https://data.city.example.jp/api/v1/budget?fy=2024"

Canonical schemas + schema registry

- Adopt CSVW/JSON-Schema and register schemas in a central registry (gov-schema.go.jp).

- Validate ingested files automatically; publish validation reports.

Example Python sketch (conceptual):

from jsonschema import validate

import pandas as pd

df = pd.read_csv('budget.csv')

# convert to dicts and run validate against canonical schema

Stop PDFs as primary source

- When PDFs are unavoidable, publish the underlying CSV/JSON alongside and include machine-readable tables (Tagged PDF or embedded attachments).

- Tooling: recommend tabula-py / Camelot pipelines for transitional ETL, but aim to retire them once API pipelines exist.

# tabula-py conceptual example

from tabula import read_pdf

tables = read_pdf('report.pdf', pages='1-3', multiple_tables=True)

Developer experience and incentives

- Provide SDKs, example notebooks (Colab), and a sandbox OpenData API with fake data for testing.

- Fund a "standardization liaison" role (Digital Agency already suggests this) to help small municipalities migrate.

Data quality & monitoring

- Publish machine-readable data quality metrics: completeness, schema compliance, freshness.

- Offer a central dashboard showing which municipalities meet standards (gamify adoption!).

Policy numbers vs reality

• National-level policy documents set goals for system standardization and machine-readable data (see digital.go.jp and soumu.go.jp). But on-the-ground issues — debt reporting cadence, variable fiscal definitions, and embedded PDFs — create gaps between targets and what's usable for analytics.
• Bridging this gap doesn't need new laws: it's engineering, procurement standards, and clear deliverables in contracts.

まとめ

• The direction is right: standardize core systems and make data AI-ready. But the missing pieces are practical engineering practices: API-first publication, canonical schemas (CSVW/JSON-LD), validation pipelines, and developer experience.
• Priorities: stop PDF-first, publish schemas and APIs, fund liaisons for small municipalities, and surface data quality metrics.

おかむーから一言

I'm a builder at heart — two startups, full-stack background, and GovTech is where code meets civic impact. Let's turn policy into reproducible endpoints and usable datasets, so everyone from researchers to local startups can build on top of real, reliable public data!