Scraping Google Maps business data by category and region.
Given a business category and a city, the pipeline queries Google Maps, opens every result and returns a structured dataset: 14 fields per business, a sample of reviews, and the tracking pixels each business site runs. A derived 0–100 heuristic summarizes how much digital presence each one has. Playwright handles the extraction; the run is asynchronous, inside a 512 MB free-tier instance.
Pipeline architecture
Every run follows the same pipeline. A FastAPI endpoint validates the request, hands the job to a background worker and returns immediately; the worker executes the five stages and emails the result on completion.
The contract
One endpoint. The response is an acknowledgement rather than a result: no scraping has run when it returns.
# request { "query": "dental clinics miami", # 2–200 chars "max_businesses": 5, # 1–100 "max_reviews": 5, # 0–100 "email": "you@example.com" } # response — immediately, before any scraping happens 202 Accepted { "status": "processing", "query": "dental clinics miami", … }
Decisions & constraints
The hosting constraints drove most of the decisions below.
A run takes minutes. The endpoint validates the request, queues the job on FastAPI's BackgroundTasks and returns 202 immediately: holding an HTTP request open for five minutes would exceed standard proxy and gateway timeouts. Results are delivered by email, which avoids both a polling endpoint and any client-side session.
The free tier blocks outbound SMTP connections, so delivery goes through Brevo's HTTP API instead.
Chromium's memory footprint grows over the course of a run. Within a 512 MB limit that risks the process being OOM-killed before the job completes, so the browser is torn down and rebuilt every third business. Low-resource Chromium flags are enabled in Docker and disabled on desktop, where --single-process causes a crash.
A 90-second ceiling per business, set from a benchmark where the typical page took ~12 s and the worst case near 15 s. Past the ceiling the business is skipped rather than allowed to block the job, limiting the cost of an unresponsive profile to a single missing row.
The API's hard limits are 100 businesses and 100 reviews. The public demo below pins both to five, keeping each run within what the free tier can serve concurrently.
Output schema
Two CSVs. Every field is read from a public profile except the two computed columns, which are derived at insertion time and make the result sortable by opportunity.
businesses.csv — 14 fields
| field | type | source |
|---|---|---|
| name | str | Maps profile |
| digital_score | int | computed — 0–100 heuristic |
| digital_tier | str | computed — hot lead · established · benchmark |
| rating | str | Maps profile |
| reviews_count | str | Maps profile |
| category | str | Maps profile |
| address | str | Maps profile |
| phone | str | Maps profile |
| website | str | Maps profile |
| has_pixel | str | site probe — sim · não · sem site · erro |
| pixels | str | site probe — semicolon-separated |
| hours | str | Maps profile |
| price_range | str | Maps profile |
| google_maps_url | str | Maps profile |
reviews.csv — 7 fields
Pixel detection
Each business site is fetched and its HTML matched against nine detectors, each keyed on specific inline-install signatures rather than a loose brand string. The signatures are deliberately narrow: the trade-off accepts missing non-standard installations in order to avoid false positives.
The 0–100 heuristic
Four additive rules, evaluated as each business is inserted. The tier names are the literal values the digital_tier column takes. The weighting is transparent: every score can be reconstructed from the row that produced it.
The score is a derived column: it collapses four independent signals into one sortable measure, which is what lets a flat scrape of a category be ranked and compared.
Run the scraper
This runs the pipeline described above against live data. Enter a business category and a city, and the five stages execute and email you the two CSVs — fields, scores, pixels and reviews included.
- Any category + city (e.g. dental clinics miami)
- Takes ~5 minutes; the free-tier instance may need to cold-start
- Check spam if the email doesn't show up