Stand up an Apple Music database as a service that can be stopped & started; can retrieve new Apple Music databases

[dprodger]

I loaded data from Apple Music as a one-off into the duckdb instance -- but it has been months since I refreshed it. It's time to make this a bona-fide service that we can manage and monitor etc.

In general, the iTunes API has pretty low quota limits so hitting it for a backfill is not usually going to work. And my local laptop is not the right place to have a persistent server, that holds the entirety of the apple database, and the duckdb for serving queries.

Success is:

1. a new service on Render, sized sufficiently to hold the Apple Music database (including all of the necessary supporting tables, images, artist names, etc.)
2. a job that can be invoked manually or on a schedule to refresh the Apple Music database with one-click invocation, including management, logging, and reporting of problems
3. an update to the apple match logic to query the new 'production' Apple Music database,
4. only as a last resort, relying on the iTunes API since the query limits are so low.

When we tackled this originally, we decided on duckDB -- but I am open to revisiting that idea; I'm looking for maximum ease of management; I'm willing to pay a little bit for disk or server to make this easier and more efficient,.

[dprodger]

Closing — the core "stand up the catalog as a managed service" work has landed; remaining gaps are tracked separately.

What landed against the four success criteria

1. A new service on Render — ✅ render.yaml provisions approachnote-research-worker with a 100 GB persistent disk at /data. Catalog (parquet tree + DuckDB index) lives there, surviving worker restarts.

2. One-click manual refresh — ✅ POST /admin/apple-music-catalog/refresh enqueues the chained job (apple/refresh_catalog × 3 → apple/rebuild_index). Dedup via fixed sentinel UUIDs on research_jobs so a second click during a run is a no-op. Status surfaced at /admin/apple-music-catalog.

3. Match logic queries the prod catalog — ✅ AppleMusicCatalog reads APPLE_MUSIC_CATALOG_DB (supports both local file and MotherDuck md: URIs); the worker matcher runs with local_catalog_only=True so all matching goes through the prod DuckDB.

4. iTunes API only as last resort — ✅ Same local_catalog_only=True flag. The only iTunes API calls remaining are for fetching artwork URLs after a local-catalog match (the catalog doesn't store artwork URLs); there's a separate latent bug for that documented in Apple Music album art missing when MB release title doesn't match Apple Music album #175 ("Apple Music album art missing when MB release title doesn't match Apple Music album").

Spinoffs (not done in this scope)

• Schedule periodic Apple Music catalog refresh #178 — Schedule periodic Apple Music catalog refresh. Original criterion said "manually or on a schedule." Manual is in; scheduled isn't. Apple's Feed publishes ~weekly so this matters for staleness.
• Failure alerting for research_worker jobs (esp. Apple Music catalog refresh) #179 — Failure alerting for research_worker jobs. Today's only signal for a failed refresh is admin-page log inspection. Becomes more important once Schedule periodic Apple Music catalog refresh #178 lands (silent failure during a 3 AM scheduled run).

Related

• Filter Apple Music catalog to jazz-relevant subset at index-build time #174 — Filter Apple Music catalog to jazz-relevant subset at index-build time. Would shrink the ~80 GB footprint substantially.
• Apple Music album art missing when MB release title doesn't match Apple Music album #175 — The MB-title-vs-Apple-album-title mismatch surfaced while diagnosing this work; the new admin diagnose/rematch buttons came out of it.
• Use MusicBrainz release aliases in Spotify + Apple Music matchers #176 — MB release aliases for matchers (long-tail fix for the same family of mismatches).

Worker config commits: 560670b, 36902eb, 7c38842, 7d9b715, 7ca1ed5, 283d9b2, 34c039b. Admin page: 424d32a. Sub-issue cleanup of the matcher itself: 159 (closed).