Tallinja API

Malta’s public transport operator publishes live departure boards at every bus stop — the page behind the QR code on the signage — but never exposed the data as an API. I wanted structured, cacheable JSON I could build things on top of, so I reverse-engineered the board and wrapped it in a small FastAPI service.

What it does

Tallinja API mirrors service-information.publictransport.com.mt/timetable?bus_stop=<code> for a given stop code and returns clean JSON: route number, line name, minutes away, and whether the departure is live-tracked (green “active” row) or a static lower-bound estimate (+30 min). The board only shows relative times, not clock times, so the API preserves that distinction rather than inventing precision that isn’t there — a live departure gets an exact minutes_away, a static one gets null and keeps the original display text.

Since the board is server-rendered HTML, no headless browser is needed — app/parser.py targets the real markup directly (.line-item, .line-number, .line-time-container-active, …), captured in a test fixture so the parser can be verified against production without hammering the origin.

Getting past the bot protection

The interesting part of this project turned out to be getting a request through at all. The origin fingerprints the TLS handshake (JA3): it lets curl through fine but silently tarpits Python’s httpx — the handshake completes, the request goes out, and the response just never comes back. Browser-like headers don’t help, because the block happens before HTTP is even in play.

The fix was a pluggable fetch backend: shell out to curl for real traffic, keep httpx around for tests (mocked via respx, so it never touches the tarpit). It’s a small reminder that “looks like a browser” and “looks like a browser at the TLS layer” are different problems.

Staying on the right side of the rate limit

The same bot protection that tarpits httpx will also tarpit an IP that requests too aggressively — after which every request from that IP hangs, including a plain browser-UA curl. So the service is built defensively rather than just functionally:

• Response cache (30s TTL) — the board itself only refreshes ~every 60s, so this costs nothing in freshness.
• Negative cache — the origin hangs (rather than 404s) on unknown stop codes, so failures are cached too, instead of re-eating the timeout on every retry.
• Rate limiter — a minimum interval between origin fetches, enforced across all stops, not per-stop.
• Circuit breaker — opens after repeated consecutive failures and fast-fails with 503 + Retry-After instead of piling on more connections, then probes once before closing again. A 404 counts as healthy; a parse failure (markup changed) doesn’t trip it — those are different failure modes and deserve different responses.

Stack

• FastAPI + pydantic for the API surface, validation, and typed error responses
• BeautifulSoup4, with an optional lxml backend for speed
• pydantic-settings for env-driven configuration (TALLINJA_*)
• pytest + respx for testing against mocked HTML fixtures rather than the live origin

What I learned

• Bot mitigation increasingly happens below the HTTP layer — TLS fingerprinting defeats the usual “spoof the User-Agent” approach entirely
• Designing for a hostile-by-accident upstream (rate limiting, circuit breakers, negative caching) is a different discipline from designing for a well-behaved one
• Preserving a source’s own uncertainty (null vs. a lower-bound string) is more honest than normalizing it away