Code-Savvy Audit of Japanese Municipal Open Data: From PDFs to APIs

IT Policy Proposals
Code-Savvy Audit of Japanese Municipal Open Data: From PDFs to APIs

どうも〜おかむーです!今日はちょっとエンジニアっぽい話をしますよ〜

  • Municipal sites often publish useful policy info but machine-readability is inconsistent
  • CKAN-based portals (e.g. Daisen) and e-Stat provide APIs, but many LGs still favor PDFs/HTML
  • Engineer-wise, the fix is standardized APIs, schema validation, and CI for data quality

結論

地方自治体の情報発信は“あるけど使えない”ケースがまだ多いです。これ見てくださいよ:Okazaki City(city.okazaki.lg.jp)は便利なサービス告知を出すんだけど、案内中心でデータをCSVやAPIで公開していないことが多いんです。一方で、Daisen市のCKANレジストリや東京都オープンデータカタログはAPIやCSVを提供していて、二極化が進んでます。要するに、政策は公開されているが、データ/エンジニア視点での“再利用性”が足りないということです。

Report: what I checked and how

Sources surveyed

  • National stats: e-Stat / Japan Dashboard (dashboard.e-stat.go.jp, digital.go.jp resources)
  • Prefectural catalogs: Saitama (opendata.pref.saitama.lg.jp), Tokyo (catalog.data.metro.tokyo.lg.jp)
  • Municipal: Okazaki official site (city.okazaki.lg.jp), Daisen CKAN (city.daisen.lg.jp/open-data/dataset)
  • Digital Agency case studies (digital.go.jp/resources)

これ見てくださいよ:フォーマットの分布はざっくりPDF/HTML >>> CSV/JSON/APIの順。CSVやCKAN APIを出している自治体は、データ利活用のスピードが明らかに早いです。

Technical evaluation

  • Machine-readability:
- Good: Daisen CKAN exposes datasets + CKAN API (supports search, metadata, file list).

- Mixed: Tokyo & Saitama catalogs offer CSV/XLSX but metadata quality varies.

- Poor: Many city pages (Okazaki) publish PDF notices or HTML tables without downloadable CSV/JSON.

  • API presence & format quality:
- e-Stat provides dashboards and APIs for national stats — great example to follow.

- CKAN-based registries are effective: support for dataset metadata, API access, and programmatic harvesting.

  • Versioning & provenance: rare. Few municipalities surface dataset version history or update timestamps in machine-readable fields.

Example: quick data tooling

Engineer-wise, if you have a CKAN endpoint you can fetch dataset metadata like this:

# search datasets (CKAN API)

curl -s "https://city.daisen.lg.jp/open-data/api/3/action/package_search?q=school" | jq '.result.results[] | {title, id, resources: [.resources[] | {format, url}]}'

And a simple Python snippet to load a CSV and run a basic QA:

import pandas as pd

from jsonschema import Draft7Validator

load CSV

df = pd.read_csv('https://opendata.pref.saitama.lg.jp/datasets/wako_population.csv')

basic QA

print(df.columns)

print(df.isnull().sum())

schema example (conceptual)

schema = {

'type': 'object',

'properties': {'year': {'type':'integer'}, 'age_group': {'type':'string'}}

}

validate rows as needed

Policy targets vs reality

Many policy pages announce targets (e.g., digital transformation, open-data promotion) on kantei.go.jp or digital.go.jp, but downstream implementation lacks measurable machine-readable KPIs. Without dataset-level SLAs (freshness, completeness), it's hard to audit whether targets are met. 要するに、政策目標が“人の読み物”に止まっているケースが多い。

Recommendations (practical engineering steps)

  • Mandate dataset-first publishing: every policy dataset must have CSV/JSON + metadata (DCAT) on publication.
  • Adopt CKAN or similar catalog with API, require dataset versioning and update timestamps.
  • Add schema validation CI: GitOps for data (pull-request -> tests: schema/row-count/diff checks).
  • Provide sample API endpoints and client libs (Python/JS) so developers can onboard quickly.
  • Encourage PDF->CSV tooling: automate extraction (tabula/camelot) but prefer native CSV publishing.

Roadmap (3 steps):

  • Inventory: crawl lg.jp + catalog endpoints to detect file formats and APIs.
  • Pilot: onboard 3 municipalities (one CKAN, one catalog, one PDF-heavy) and run a data pipeline.
  • Scale: publish a template repo with schema, CI, and sample clients.
  • まとめ

    • 日本のオープンデータ環境は進んでいる部分と遅れている部分が混在
    • CKAN・e-Statは模範例、でも多くの自治体はまだPDF/HTML主体
    • エンジニア的にはAPI・スキーマ・CIを導入すれば利活用が一気に進む!

    おかむーから一言

    テクノロジーで社会をアップデートするって、口で言うのは簡単だけど、まずはデータを"使える形"で出すことが一番の近道です。エンジニアなら一緒に手を動かして作りましょう!