For a stretch of this year, a large share of our archive came in through Google News RSS. It was convenient: one feed, broad coverage, predictable format. Then it stopped HTTP-redirecting to the publisher, and everything downstream broke at once.

What actually broke

The scraper followed the redirect to the original article to extract body copy and an og:image. When the redirect stopped resolving, the extractor kept receiving a Google interstitial instead of a news page. It found no article body worth keeping and no image at all. Several hundred rows landed with a null image and a canonical URL pointing at news.google.com rather than the publisher — which is useless both to a reader and to a search engine.

Worse, the titles arrived with a " - Publisher" suffix appended, which quietly defeated our duplicate detection. The same story from the publisher's own feed and from Google's feed produced two different fingerprints, so both were stored.

The fix was subtraction

We replaced the single aggregated feed with two dozen publisher feeds read directly, plus a set of dedicated AI-industry feeds. Images now come from the feed's own media:content, media:thumbnail or enclosure elements where present, falling back to the article's og:image. Titles are normalised to strip a trailing publisher suffix before fingerprinting, so the same story from two feeds collapses to one row.

The lesson we keep relearning

An intermediary that makes ingest easy also makes it fragile in a way you do not control and cannot test against. When the aggregator changed a behaviour it never promised to keep, our pipeline had no recourse. Reading publishers directly is more configuration and more code — and it fails one feed at a time instead of all at once.

The legacy rows are still there, image-less, and they are the reason a handful of headlines on this site still carry a publisher name in the title. Cleaning them is its own project.