Google Is Crushing SERP Scrapers. AI Agents Made It Worse. Good.
Google is escalating adversarial actions against automated SERP tracking, and Search Engine Journal reports that AI agents scraping search results are giving Google even more reason to shut down automated queries. A separate but related problem is keyword inventory inflation, a long-running issue where rank trackers distort actual search volume data. This predates AI agents but is now compounding. A practitioner with 25 years of SEO experience notes the situation may be reaching a point where rank tracking becomes unsustainable.
This story demonstrates the observer effect, which I suspect most of you last encountered in a physics class you slept through. The mechanism is measurement distortion. When you scrape search results at scale to track rankings, you change the very data you are measuring. Google sees the automated queries, adjusts its defenses, and the keyword inventory data gets inflated by the tracking bots themselves. You are polluting the stream you are drinking from.
Search Engine Journal reports on Google's adversarial actions against SERP tracking, citing a practitioner with 25 years of hands-on SEO experience who observes that rank tracking may become unsustainable as AI agents compound the scraping problem.
- Open Google in an incognito window and search for a keyword you care about. Manually count how many results appear before your site. Expected outcome: a single data point of your real ranking, measured cleanly without bot interference.
- Search the same keyword in Perplexity or ChatGPT and note whether your brand appears in the AI-generated answer. Expected outcome: a measurement of AI citation visibility, which is the metric that actually matters now.
- Repeat this manually once per week for a month and record results in a simple spreadsheet. Expected outcome: a trend line based on real human observations, not distorted bot data.