どうも〜おかむーです！ Today I want to take a slightly nerdy, operator-first look at how Japanese national and local governments publish data — with a focus on machine-readability, APIs, and what engineers actually need to build useful services.

• Municipal open-data uptake is improving, but machine-readability remains uneven across local governments.
• Major guidelines (Digital Agency, Ministry of Internal Affairs) push for CSV/Excel and APIs, yet PDFs persist as the dominant publication format.
• Practical fixes: data catalogs, stable APIs, CSV + JSON outputs, and minimal schema standards (DCAT/CSVW) will unlock real reuse.

結論

Japan has clear policy direction — see Digital Agency draft rules on machine readability (digital.go.jp) and Ministry of Internal Affairs guidance on tabular statistics (soumu.go.jp) — but real-world adoption lags. PDFs and undocumented Excel files still dominate, which means engineers waste time parsing instead of building. 要するに、政策は正しいが実装が追いついていないということです。

Report

What the sources say (quick scan)

• Digital Agency draft: rules for machine-readability levels and recommending CSV/Excel over PDF (digital.go.jp meeting doc). This is a concrete step toward standardization.
• Ministry guidance (soumu.go.jp): unified rules for machine-readable statistical tables — useful for consistency across agencies.
• GovTech Tokyo showcases (govtechtokyo.or.jp, note.govtechtokyo.jp): practical dashboards and internal tooling that benefits from standardized datasets.
• Private-case studies page (digital.go.jp/data_case_study_private) and vendor summaries (intec.co.jp) show real use-cases but also reliance on scraping or manual cleaning.

これ見てくださいよ：official docs call for CSV/Excel and machine-readable metadata, but many municipalities still post PDFs or images embedded in PDFs. That creates enormous friction.

Technical gap analysis

• File formats: PDF > Excel/CSV. PDFs force OCR or tabular extraction (tabula/camelot) — brittle and error-prone. 要するに、PDFは人間向け、CSVは機械向けです。
• APIs: patchy. Some prefectures expose REST endpoints, but no nationwide consistent API schema. No single discovery layer (DCAT-AP-JP adoption is inconsistent).
• Metadata: inconsistent column names, missing datetimes, no schemas (JSON Schema/CSVW). That breaks automated ETL.
• Governance: policy exists (machine-readability rules draft) but lacks enforcement and developer-friendly reference implementations.

Code-first suggestions (engineer-ish)

• Always publish CSV + JSON: CSV for tabular ease, JSON for nested data. Include CSVW or JSON Schema.
• Provide a minimal OpenAPI/REST spec for any API endpoints and host a machine-readable catalog (DCAT JSON-LD). This makes discovery programmatic.
• Versioned endpoints and semantic versioning for schema changes to avoid breaking clients.

Example: quick Python pattern to load either CSV or fallback to a CSV-in-PDF via table extraction

import requests
import pandas as pd
from io import BytesIO

url_csv = 'https://example.local.gov/data.csv'
try:
df = pd.read_csv(url_csv)
except Exception:
# fallback: download PDF and try tabula (requires java + tabula-py)
pdf = requests.get('https://example.local.gov/report.pdf').content
# use tabula.read_pdf(BytesIO(pdf), pages='all') in real code
df = pd.DataFrame()  # placeholder for extracted table

print(df.head())

Policy vs reality: numbers matter

Many municipalities set targets for open-data publication and dashboard coverage (see GovTech Tokyo service pages), but the gap is visible: percentage of datasets that are machine-readable often remains below policy expectations. To evaluate properly, you need: dataset counts, formats breakdown (CSV/JSON/Excel/PDF), and API availability per municipality. Collecting that is straightforward with a crawler + schema validator.

Concrete roadmap for governments

• Mandate CSV/JSON for tabular data and publish CSVW/JSON Schema alongside files (reference: digital.go.jp rules).
• Run an annual automated audit (crawler + schema checks) and publish the audit results on a public dashboard (example: GovTech Tokyo approach).
• Provide an open-source reference implementation: a simple CRUD API and DCAT catalog that municipalities can fork.
• Offer toolchains: lightweight ETL templates (Python/pandas + CI checks) so departments can validate before publishing.

Implementation cost vs benefit

Upfront work to standardize formats and provide schemas is modest compared to downstream savings in citizen services, third-party apps, and reduced manual processing by municipal staff.

まとめ

Japan has the right policy scaffolding (digital.go.jp, soumu.go.jp, GovTech Tokyo examples), but execution needs tooling, catalogs, and enforcement. エンジニア的に言うと、API一本ときちんとしたCSVがあれば多くの課題は解決するんですよ。PDFに閉じたデータはもう終わりにしましょう！

おかむーから一言

I’ve built products that live or die on clean APIs — and trust me, governments publishing machine-readable data is the cheapest infrastructure investment you can make. Let's ship CSV + API + catalog and let builders iterate on top of that!