Code-Telling Manifesto: Reading Japan's Public Data as Engineering

どうも〜おかむーです！ Today we're doing a little techy audit of Japan's public datasets and dashboards — "code-telling manifesto" style. エンジニア的に言うと、政策はデータとインフラで語れるんですよ〜。これ見てくださいよ、公開されてるCSVとダッシュボードを触ってみるといろいろ見えてきます！

• Japan publishes lots of CSVs (notice.go.jp, digital.go.jp, env.go.jp) but formats and encodings vary
• Machine-readability is uneven: encoding, metadata, and APIs are the bottlenecks
• Tech fixes are straightforward: standardized schemas, APIs, provenance + policy manifests

結論

Government data is becoming more open, but practical reuse is hindered by inconsistencies in format, encoding, and lack of API-first design. 要するに、データをコードで扱える形に整備すれば、政策の検証・可視化は一気に速くなるということです。

Report

What I looked at

• notice.go.jp/docs/status_notice.csv — a direct CSV endpoint is great!
• digital.go.jp resources (hospital CSV, survey answers) — files exist but show encoding/noise
• env.go.jp CSV for prefectural data — domain CSVs are present and useful
• Japan Dashboard / e-Stat dashboards — central UI for many stats, but UI != API

Practical problems (from an engineer's lens)

• Encoding hell: some CSVs contain mojibake in snippets (suggests CP932/Shift_JIS vs UTF-8 mismatches). This trips parsers.
• Missing machine metadata: many CSVs lack a machine-readable schema (column types, units, update frequency, licence).
• Fragmented access: data exists as static files rather than a consistent REST/GraphQL API. That complicates joins and time-series queries.
• Geodata: hospital CSV includes lat/long but formats vary; no GeoJSON or standardized CRS declarations.

Code patterns to consume messy CSVs

Example Python approach (pandas + encoding detection):

import chardet, requests, io
r = requests.get('https://www.digital.go.jp/.../xxxxxx_hospital.csv')
enc = chardet.detect(r.content)['encoding']
df = pd.read_csv(io.BytesIO(r.content), encoding=enc)

For large CSVs, stream with chunksize and set dtypes to reduce memory. 要するに、まずはencodingとschemaを確認するのが鉄則です。

Policy vs measurement

Many policy goals (e.g., transport targets, healthcare capacity) are not published as machine-first manifests. We have time-series CSVs (e.g., new car sales from 内閣府 CSV) but mapping them to explicit policy targets requires manual interpretation. If a ministry sets a 5-year target, that target should be a small JSON document with variable names, baseline, target value, and evaluation query.

Concrete improvement proposals

API-first publishing: provide REST endpoints returning JSON/JSON-LD and Content-Location headers; keep CSVs for bulk.

Schema registry: each dataset gets a machine-readable schema (JSON Schema), license, provenance, update cadence.

Encoding standard: mandate UTF-8 without BOM for all new datasets; provide converters for legacy CP932 files.

Policy manifest example (publish alongside law/regulation):

{
"policy_id": "ev_adoption_2030",
"indicator": "new_car_sales_ev_share",
"baseline": 0.05,
"target": 0.5,
"measurement_query": "SELECT year, ev_share FROM transport.sales WHERE prefecture='ALL'"
}

Provide tiny SDKs (python/js) that wrap APIs and handle encoding/normalization.

Reuse & ecosystem

• Japan Dashboard is a great aggregator but needs programmatic endpoints and dataset fingerprints (hash, version).
• Encourage public-private hackathons using canonical APIs and datasets like hospital lists (geo-enabled) and environmental sensors.

まとめ

Public data in Japan is getting better — CSVs and dashboards are visible — but to make policy verifiable and reusable we need standard encodings, schemas, and API-first delivery. エンジニア的に言うと、これをやれば一気に開発者と市民が恩恵を受けます！

おかむーから一言

I'm a founder and full-stack engineer — tech can make policy measurable and accountable. Let's publish policies as code and ship better civic infrastructure together!