Code-First Manifesto: Evaluating Japan's Public Data through an Engineer's Lens

どうも〜おかむーです！ Hey, Okamu here — today we're doing a bit of GovTech autopsy with code goggles on.

• Many Japanese municipal portals publish open data (CSV/GeoJSON) but format and metadata vary widely.
• Machine-readability (PDF vs CSV), API availability, and schema consistency are the main technical bottlenecks.
• Practical fixes: standard schemas, CI for dataset quality, open APIs, and simple ETL recipes to unlock reuse.

結論

The policy direction is right: central guidance and local catalogues (Tokyo, Yokohama, Digital Agency) are pushing for "AI-ready" administrative data by 2025. But on the ground, inconsistent formats, PDF-locked tables, and spotty metadata prevent automation. エンジニア的に言うと、これはAPIとschema-driven workflowsで解決できる話なんですよ。

Report: what I looked at and what it shows

What public portals already do (good stuff)

• Tokyo's Open Data Catalog and Yokohama's portal expose many datasets in CSV and GeoJSON (source: portal.data.metro.tokyo.lg.jp, data.city.yokohama.lg.jp). That means geospatial and tabular data can be consumed directly by pandas or GIS stacks — nice!
• Digital Agency publishes case studies and standards encouraging reuse and private-sector innovation (source: digital.go.jp).

Frictions I found

• PDF-first reporting still exists in many administrative materials. Central guidance emphasizes machine-readability for an AI-ready society by 2025 (source: cas.go.jp), but practice lags.
• Metadata quality varies: missing descriptions, inconsistent field names, unclear update cadence (snippet seen in Tokyo/Yokohama listings).
• API gaps: some catalogs have CKAN-style APIs but adoption and rate-limits/SLAs are inconsistent.

要するに、データは出てるけど“使いやすさ”がバラバラということです。

Technical deep-dive: common patterns and concrete code

• Typical portal patterns: CSV/GeoJSON resources, sometimes CKAN API endpoints, sometimes just direct downloads.
• If you hit a CSV: easy. If you hit a PDF: need extraction tools.

Example: fetch a CSV and load with pandas

import requests
import pandas as pd
url = 'https://catalog.data.metro.tokyo.lg.jp/dataset/xxx/resource/yyy.csv'
resp = requests.get(url)
open('data.csv','wb').write(resp.content)
df = pd.read_csv('data.csv')
print(df.head())

Example: extract tables from a PDF (when municipal reports are PDF-only)

# tabula-py or camelot are useful
pip install tabula-py
python -c "import tabula; dfs = tabula.read_pdf('report.pdf', pages='all'); print(len(dfs))"

Example: query CKAN API (many catalogs use CKAN)

curl 'https://catalog.data.metro.tokyo.lg.jp/api/3/action/package_search?q=title:park&rows=5'

Data quality checks and pipelines

• Validate schemas with goodtables/csvlint in CI. Automate with GitHub Actions: on push, run csvlint -> report -> publish artifacts.
• Use DCAT metadata and canonical field names (e.g., datetime, lat, lon, id) so downstream code doesn't need brittle mapping.
• Version releases: attach semantic version tags (vYYYYMMDD) to datasets and keep changelogs for provenance.

Policy numbers vs reality

Central documents call for machine-readable stats and standardized table rules (soumu.go.jp guidance on machine-readable statistical tables). The target timelines (e.g., AI-ready by 2025) are credible, but local implementations need engineering support: tooling, templates, and maintenance budgets. Without those, goals stay aspirational.

Concrete improvement roadmap (tech-first)

Mandate machine-readable primary publication (CSV/JSON/GeoJSON) for all routine tables; PDFs remain secondary.

Provide open-source exporter templates per CMS used by municipalities (WordPress, Shintoshin systems, etc.).

Offer a managed CKAN/Dataverse instance for smaller municipalities with CI-enabled dataset pipelines.

Publish stable REST APIs with simple authentication and rate limits; document via OpenAPI.

Invest in data-linting pipelines (goodtables), automatic schema inference, and human review dashboards.

まとめ

This is an ops and engineering problem as much as a policy one. Japan's federal guidance and major city portals are doing the groundwork, but to get from documents to automated reuse we need: standardized schemas, API-first publishing, CI for data quality, and easy extraction tools for legacy PDFs. 技術的にはソリューションが明確なので、あとは予算と運用フローの整備がカギです！

おかむーから一言

I built products that glue data to services — public data needs the same treatment. Let's push for APIs, not PDFs; CI, not manual uploads. Tech can make policy measurable and actionable, and that's where real change starts!