どうも〜おかむーです！今日はちょっとエンジニアっぽい話をしますよ〜

• Government publishes many CSVs (notice.go.jp, env.go.jp, soumu.go.jp, mhlw.go.jp) but formats vary wildly
• Machine-readability rules are being formalized — good, but practical gaps remain in schema, encoding, and APIs
• エンジニア的に言うと、API + consistent CSV/JSON schemaがあればデータ活用が一気に加速するんですよ

結論

Public datasets in Japan are moving in the right direction (see recent Digital Gyozaikaikaku decisions), but in practice engineers still wrestle with encoding, inconsistent headers, missing schemas, and scarce APIs. 要するに、データはあるけど“使える”形になってない。改善案は明確で、API-first、CSVW/JSON-LDでスキーマ公開、CIでデータ品質チェックを回すことです。

Report

What I inspected

これ見てくださいよ — 公開されているCSVの例:

• NOTICE: https://notice.go.jp/docs/status_notice.csv
• Ministry of the Environment (Ibaraki excerpt): https://www.env.go.jp/content/900398071.csv
• Cabinet Secretariat / Digital reform materials: https://www.cas.go.jp/jp/seisaku/digital_gyozaikaikaku/kakusyoDX4/kakusyoDX4.html and meeting outline PDF: https://www.digital.go.jp/assets/.../20260331_meeting_executive_outline_05.pdf
• MIC nationwide CSV: https://www.soumu.go.jp/main_content/000323625.csv
• MHLW facility list: https://www.mhlw.go.jp/content/001682075.csv

These are real gov domains (go.jp) — solid provenance. But surface-level availability isn't the same as machine-readability.

Technical issues found

• Encoding and line endings: many CSVs use Shift-JIS or unknown encodings. That breaks naive UTF-8 parsers. 要するに、エンジニアの最初の時間がここで奪われる。
• Schema drift: columns change names/ordering between releases; no formal schema or versioning.
• Missing metadata: no CSVW/JSON-LD sidecar describing types, units, timezones.
• No consistent API: some datasets are only downloadable CSVs, no REST/GraphQL endpoints, no pagination, no CORS.
• PDF vs CSV: some policy documents remain PDF (see Digital Gyozaikaikaku PDFs). PDFs are unreadable for automation.

Example: practical ingestion

Code example (Python/pandas) to robustly fetch and normalize CSVs:

import requests, chardet, pandas as pd
r = requests.get('https://notice.go.jp/docs/status_notice.csv')
enc = chardet.detect(r.content)['encoding']
df = pd.read_csv(pd.io.common.BytesIO(r.content), encoding=enc)
normalize headers
df.columns = [c.strip().lower().replace(' ', '_') for c in df.columns]
basic checks
print(df.head(), df.dtypes)

Tips: always detect encoding, normalize headers, validate against a JSON Schema. Throw data CI on ingestion (unit tests for column presence, value ranges, timestamps).

Policy metrics and gaps

Many policy targets are published in PDFs or as static tables; linking the target to time series data is inconsistent. Without stable IDs and schemas you can't automatically compute achievement rates. 要するに、政策の数値目標と実績を自動で比較できないケースが多い。

Improvement roadmap (practical)

• Publish machine-readable schemas (CSVW/JSON Schema) per dataset
• Offer REST endpoints with OpenAPI specs and pagination + CORS
• Standardize encoding to UTF-8 and publish sample data
• CI pipeline: lint on schema, row-level validation, changelog/versioning
• Encourage dataset catalogs (e.g., Tokyo/Prefectural catalogs) to link to schemas and APIs

Benefits

• Faster civic tech builds and data journalism
• Automated monitoring of policy KPIs
• Lower overhead for municipalities (fewer ad-hoc data cleaning requests)

まとめ

公開データは増えてる！ただし“使える”形にするための最後の1歩が足りないんです。エンジニア的には、スキーマとAPIがあれば、データ利活用は一気に現実的になります。PDFをCSVにするだけでなく、CSVに使えるメタデータを添付して公開する、それが大事。

おかむーから一言

テクノロジーで役所をアップデートするのは可能です！小さな工程改善（CSV→UTF-8、スキーマ公開、API化）を積み重ねていきましょう〜