2026Solo developer — scraper and data pipeline
maps-gluten-rag — Maps list to RAG
Turns a shared Google Maps list of gluten-free places into RAG-ready files.
- Python
- Playwright
- SQLite
- GitHub Actions
- Gemini
- LLM
- RAG
- Web scraping
Problem
I keep a Google Maps list of gluten-free restaurants, but a shared list is a link, not a searchable knowledge base: I cannot ask NotebookLM what it contains, embed it, or trust it without opening Maps every time.
There is no API for someone else's shared list, and exporting by hand does not stay current. I wanted the list extracted, deduplicated and refreshed weekly, fully automatic.
Approach
Extract via the internal endpoint, with a fallback
The extractor opens the public list with Playwright and reads the internal `entitylist/getlist` endpoint, which returns every place with name, address, coordinates, `ftid` and save date. It then walks the DOM rows to add rating and category, which the endpoint does not include. If the endpoint changes, it falls back to reading the rows directly.
I rejected scraping Google's public HTML search: it is rate-limited, consent-gated and structurally unstable — the internal endpoint plus a DOM fallback is a smaller surface to maintain.
Dedup state in SQLite, kept in the workflow cache
The local source of truth is a SQLite file. In GitHub Actions it lives in the workflow cache (git-ignored), so every run upserts instead of appending: it keeps Google's `saved_on` date and only marks as new the places that did not exist before.
I rejected committing the database: a binary file in the repo defeats diff review and grows unbounded; the cache keeps history without polluting the repo.
Weekly cron, daylight-saving proof
A GitHub Actions workflow runs every Monday at 13:00 Chile time. It schedules two UTC crons (16:00 and 17:00) and a step checks the local hour, so it stays at 1 pm across daylight saving changes. It publishes the `.md`, `.jsonl` and menus file as artifacts, commits the data files if anything changed, and can copy the result into a Google Doc as a NotebookLM source.
I rejected pinning a single UTC cron: twice a year it would fire at noon or 2 pm, which breaks the 'lunch decision' use case I built this for.
Results
- Runs fully automated and free in a public repo: extraction, dedup, enrichment and publishing, Mondays at 13:00 Chile time.
- Three outputs from one list: a NotebookLM-ready Markdown file, `jsonl` for embeddings and a separate `_menus.jsonl` with dishes, prices and allergens.