Code-First Manifest: Reading Japan’s Public CSVs Like an Engineer

Hey — Okamu here! Today I'm digging into a pile of public CSVs and government decisions, taking a code-first look at how Japan's public data is actually published and how it should be improved.

• This piece inspects real gov CSV links (notice.go.jp, env.go.jp, soumu.go.jp, mhlw.go.jp, inpit.go.jp) and the recent Digital Administration decisions
• Main finding: many datasets are CSV — great — but machine-readability, metadata and APIs are inconsistent, causing avoidable friction
• Concrete technical fixes: encoding standardization, CSVW/JSON-LD metadata, CI validation, lightweight API facade (FastAPI) and schema checks

結論

Public data is getting better (CSV over PDF is real progress!), but engineering hygiene is missing. If the government treats every dataset like an API-first product with schema, metadata and CI, data re-use skyrockets and policy tracking becomes measurable.

Data sources I checked

• NOTICE: https://notice.go.jp/docs/status_notice.csv (status CSV)
• Ibaraki/environment: https://www.env.go.jp/content/900398071.csv
• National list: https://www.soumu.go.jp/main_content/000323625.csv
• Rescue facilities: https://www.mhlw.go.jp/content/001682075.csv
• Domestic migration 2011: https://www.inpit.go.jp/content/100869372.csv
• Policy-level note: Digital reform meeting — machine readability rules: https://www.cas.go.jp/jp/seisaku/digital_gyozaikaikaku/kakusyoDX4/kakusyoDX4.html and summary PDF: https://www.digital.go.jp/assets/…/20260331_meeting_executive_outline_05.pdf

What I looked for (engineer's checklist)

• Format: CSV vs PDF — CSV wins, but encoding and delimiter vary
• Metadata: Is there schema/column meaning? Rarely
• API: No unified REST/GraphQL endpoints
• Versioning & provenance: Mostly absent

Check this out — quick ingestion snippet

import requests, chardet, pandas as pd
r = requests.get('https://notice.go.jp/docs/status_notice.csv')
enc = chardet.detect(r.content)['encoding']
df = pd.read_csv(pd.compat.StringIO(r.content.decode(enc)), parse_dates=True)
print(df.info())

Engineer's note: always detect encoding (Shift_JIS still common), normalize datetimes, and assert dtypes.

Validation & CI

Use Great Expectations / pandera to codify expectations: columns, types, null thresholds. Hook checks into GitHub Actions so updated CSVs fail builds if schema drifts.

API façade (pattern)

• Put CSVs in an object store (S3-like) with immutable paths
• Publish CSVW/JSON-LD describing schema + license
• Serve small FastAPI layer that returns JSON and supports filters, and exposes OpenAPI for clients

Policy gap detection (how to measure)

• Ingest time-series CSVs, compute yearly aggregates and target vs actual gaps
• Alert if deviation > X% and produce dashboards (Grafana/Metabase)

まとめ

• CSV availability is good — move from files to product: strong metadata, encoding standards, schema tests and a tiny API layer
• Adopt CSVW/JSON-LD, version assets, and automate validation
• Result: faster research, auditable policy performance, and more civic tech innovation

おかむーから一言

I'm a founder-dev who believes tech can upgrade governance. Treat data as product, not paperwork — do that and the rest follows!