Code-driven Manifest: Inspecting Japan's Public Data through an Engineer's Lens

どうも〜おかむーです！今日はちょっとエンジニアっぽい話をしますよ〜

• Government data is moving toward machine-readability, but practice lags policy.
• Several gov CSV endpoints exist (notice.go.jp, digital.go.jp, greenfinanceportal), yet format, metadata, and API gaps persist.
• Practical fixes: standardized schemas, dataset CI, OpenAPI/JSON-LD, and simple validation pipelines.

結論

Japan has clear policy momentum — see the Digital Administrative Reform Council's decision on machine-readability — but day-to-day public datasets still show variability in formats, metadata and APIs. エンジニア的に言うと、ルールはできてきたが実装と運用を自動化してこそ効果が出るんですよね。

Report

What I checked (data sources)

• CSV endpoints that are easy to fetch: https://notice.go.jp/docs/status_notice.csv (notice data), https://www5.cao.go.jp/j-j/wp/wp-je24/csv/d1-1-4.csv (Cabinet Office tables), digital.go.jp hospital CSV asset, and greenfinanceportal's issuance_list_gb.csv. These show good practice: direct CSVs with update timestamps.
• Policy docs: Cabinet Office / Digital Reform meeting materials (digital.go.jp) laying out machine-readable rules.
• Community/knowledge hubs: GovTechBridge and GovTech Tokyo notes show practitioner case studies.

これ見てくださいよ：some agencies already publish clean CSVs (green finance portal), others still publish PDFs or CSVs with messy headers, inconsistent encodings, or missing metadata.

Technical issues observed

• Format inconsistency: different column names, encodings (Shift_JIS vs UTF-8), BOMs.
• Metadata absence: no DCAT/JSON-LD descriptors, unclear update cadence and license.
• API gaps: no unified REST/GraphQL endpoints; many consumers must scrape CSV/HTML.
• Validation: no automated schema validation, so downstream pipelines break when columns shift.
• Policy vs practice: the Digital Reform decision mandates machine-readability, but rollout and enforcement details are weak.

Example: quick validation pipeline (concept)

# fetch CSV, ensure UTF-8, and validate header
curl -sS https://notice.go.jp/docs/status_notice.csv -o status.csv
iconv -f SHIFT_JIS -t UTF-8 status.csv > status.utf8.csv || cp status.csv status.utf8.csv
python3 - <<'PY'
import csv
req_cols = ['notice_id','title','date']
with open('status.utf8.csv', newline='') as f:
r = csv.reader(f)
header = next(r)
print('header:', header)
missing = [c for c in req_cols if c not in header]
print('missing:', missing)
PY

要するに：簡単なCIで多くの問題は防げます。

Concrete recommendations

Standardize on UTF-8+CSV as baseline, provide machine-readable metadata (DCAT/JSON-LD) alongside each dataset.

Publish OpenAPI/REST endpoints for high-value datasets and provide bulk-download CSVs for analytics.

Adopt schema-first approach: maintain a catalog with schema versions, and run CI to validate each publish.

Provide example client code (Python/pandas, JavaScript fetch) and canonical field mappings (e.g., ISO prefecture codes).

Encourage reusable authentication (OAuth2 for write operations) but keep read endpoints public and cacheable (CDN + S3).

Policy-number gap analysis

Policies set targets for machine-readability, but metrics matter: measure % of datasets with schema, % with update timestamps, and % with license. GovTechBridge-style scoring could be extended to an open registry (automated audit).

まとめ

• Progress exists: some portals already serve clean CSVs and policy frameworks are emerging.
• Problems remain: inconsistent formats, lack of metadata, and missing APIs make reuse costly.
• Fixes are engineering problems: schema management, automated validation, OpenAPI endpoints, and clear licensing unlock reuse quickly.

おかむーから一言

I started companies and shipped GovTech products — this is totally solvable with a few engineers and governance changes. Let's make government data not just open, but actually useful for building things!