Code-Driven Manifesto: auditing Japan’s public data with engineering eyes

How's it going — okamu here! Today I dig into some real Japanese public datasets and systems from an engineer's perspective. I look at CSVs, PDFs, APIs (or lack thereof), and propose concrete fixes so policy becomes verifiable by code.

• Quick look at actual dataset endpoints (notice.go.jp, soumu, env.go.jp) and policy docs (Digital Garden City, Cabinet Office).
• Main finding: many datasets are published (yay CSV!) but lack machine-friendly metadata, consistent encoding, and APIs — which blocks verification of KPIs.
• Actionable fixes: standardized schemas (JSON Schema/DCAT), UTF-8 publishing, simple REST APIs + OpenAPI, and time-series portals for KPIs.

結論

The data exists in many places (CSV and PDF), so this is not a "no data" problem — it's a "data usability" problem. Engineer-wise, the government often publishes raw tables (https://notice.go.jp/docs/status_notice.csv, https://www.soumu.go.jp/main_content/000420038.csv, https://www.env.go.jp/content/900398071.csv), but without consistent metadata, encoding guarantees, or APIs. That makes automated validation of policy KPIs (e.g. Digital Garden City targets) clumsy or impossible.

Report

What I inspected

• Notices CSV: https://notice.go.jp/docs/status_notice.csv — CSV available, good.
• Classification codes from Ministry of Internal Affairs: https://www.soumu.go.jp/main_content/000420038.csv — useful code lists.
• Prefectural datasets (Ibaraki): https://www.env.go.jp/content/900398071.csv — domain-specific CSVs.
• MHLW small CSV example: https://www.mhlw.go.jp/content/001429362.csv
• Policy/KPI docs: Digital Garden City strategy and evaluation PDFs (Cabinet Office pages).

These show the pattern: data is published, but often in a mix of CSV and PDF. PDFs carry the narrative and KPI summaries, CSVs carry raw rows — but they're not linked with stable metadata.

Technical issues I found

• Encoding and character set: some gov CSVs historically used Shift_JIS; modern tooling expects UTF-8. Without an explicit encoding header, automated pipelines break. In short: encoding must be declared.
• No schema or typed fields: many CSVs lack column types, units, or code dictionaries. You see headers in Japanese but no machine-readable mapping to standard identifiers (JIS codes, pref codes).
• Lack of APIs: downloading CSVs is fine for manual work, but APIs with pagination, filtering, and CORS are needed for reproducible verification and dashboards.
• Time-series fragmentation: KPI evaluation PDFs (e.g. https://www.chisou.go.jp/sousei/about/kouhukin/pdf/...) summarize outcomes, but raw time-series data aren't always exposed in the same catalog.
• Metadata and discoverability: no DCAT/JSON-LD metadata attached to many endpoints, so central portals can't automatically index datasets and relate them to KPIs.

Evidence and reproducibility

This is engineer-ish: give me an endpoint and I can script an audit. Example: quick Python snippet to load a CSV robustly and inspect encodings:

# quick check with pandas + chardet
import chardet
from urllib.request import urlopen
url = 'https://notice.go.jp/docs/status_notice.csv'
b = urlopen(url).read()
enc = chardet.detect(b)['encoding']
print('detected encoding:', enc)
import io
import pandas as pd
s = io.BytesIO(b)
df = pd.read_csv(s, encoding=enc)
print(df.head())

And if you want a tiny API wrapper to serve CSV as JSON with pagination (Node/Express):

const express = require('express');
const csv = require('csvtojson');
const app = express();
app.get('/api/notices', async (req, res) => {
const url = 'https://notice.go.jp/docs/status_notice.csv';
const json = await csv().fromStream(require('request')(url));
const page = +req.query.page || 1; const per = 100;
res.json({total: json.length, data: json.slice((page-1)per, pageper)});
});
app.listen(3000);

Concrete improvement roadmap

Publish encoding and schema: each CSV must come with a JSON Schema + sample rows and declared encoding (UTF-8 preferred).

Register datasets in a catalog with DCAT/JSON-LD metadata so portals can link datasets to policy KPIs.

Provide simple REST APIs (OpenAPI spec) with pagination, filters, and CORS enabled.

Link KPI PDFs to the raw time-series data using explicit dataset IDs — automate KPI computation and publish the calculation scripts as notebooks.

Standardize codes: use existing code lists (https://www.soumu.go.jp/main_content/000420038.csv) and reference them in schemas.

Why this matters for policy verification

Policies like the Digital Garden City strategy publish 5-year KPIs (Cabinet Office). But without machine-friendly time series, external auditors and citizens can't reproduce the numbers. That weakens accountability and slows innovation — because civic tech folks waste time cleaning and reconciling datasets instead of building insights.

まとめ

This is fixable and low-hanging. Many datasets are already published as CSVs — so the gov is halfway there. The next steps are standards, metadata, and simple APIs. Engineer move: treat each dataset like a tiny service with a schema, docs, and tests. Do that and KPIs become auditable by code.

おかむーから一言

I’ve built startups and shipped engineering teams — making data machine-friendly is boring but powerful. Give me schemas, APIs, and stable IDs and I’ll show you how quickly civic tech can turn policy into results.