Web scraping for AI agents: the practical guide

Web scraping for AI agents means giving a model a way to pull structured data out of sites that have no usable API, and in 2026 the practical answer is almost never a scraper you wrote yourself. Agents now call hosted tools that return clean JSON, which moves the proxy pool, the headless browser and the CSS selectors off your machine and out of your maintenance budget. This guide covers why the do-it-yourself route decays, what an agent can pull today, two worked flows with real per-call costs, and where the honest limits are.

Why do DIY scrapers die?

DIY scrapers die because the thing they point at changes faster than you maintain them. A scraper is a standing bet that a site's HTML, its bot defences and its rate limits will all hold still, and all three move without telling you. The worst part is that the failure is usually silent: you get an empty array back, not an exception.

• Anti-bot stopped being a user-agent check. Imperva's 2025 bad bot report put automated traffic at 51% of everything on the web, the first year bots outnumbered people, with 37% classed as bad bots. Sites reacted. Cloudflare began blocking AI crawlers by default for new domains on 1 July 2025. Detection now runs on TLS fingerprints, header order, canvas and font fingerprints and behavioural scoring, all before your code sees a single byte of HTML.
• Layout drift is structural, not occasional. Class names in a modern web app are build artifacts, regenerated on deploy. A selector is a compilation detail you copied out of devtools, not an interface anybody promised to keep. When it breaks, the field you cared about comes back null and the row still parses.
• The infrastructure bill is not only money. A headless Chromium instance wants roughly 300 MB of RAM, so twenty in parallel is a 6 GB machine before you have parsed anything. Residential proxy bandwidth is sold by the gigabyte and image-heavy pages spend it fast. Captcha solving is a fee per solve with no ceiling.
• The treadmill compounds. One scraper is a weekend. Ten scrapers across ten sites is a permanent part-time job, because the ten sites do not coordinate their redesigns with each other or with you.

Agents add a failure mode of their own. When a scraper hands back an empty list, a model does not raise an alarm. It reasons over nothing and writes a confident paragraph built on no data, and you find out three reports later.

What replaced the scraper?

Structured tool calls replaced it. Instead of your agent fetching HTML and guessing at the shape of it, the agent calls a named tool with a typed input and gets JSON back with fields it can rely on. Fetching, fingerprinting and parsing all happen on somebody else's machine, and the contract you depend on becomes a schema instead of a selector.

MCP is what made this the default shape rather than a vendor trick. A tool carries a name, a description and a JSON Schema, the model reads all three at runtime, and running one is a single call over a standard protocol. If that part is new, the explainer on MCP covers the mechanics in full.

Three things change in practice, and they are all operational:

• Discovery happens at runtime. The agent asks what tools exist when it needs one, so adding a capability does not mean redeploying the agent.
• The price is visible before the call. An agent that can read a per-call cost can budget, and you can cap it.
• Failures are typed. A blocked page comes back as an error with a reason, not as an empty list the model will happily summarise.

What can an agent pull today?

Most of the public web, through named tools rather than crawlers. The Goro catalog runs to 62 tools across 9 categories, and 46 of them read something that already exists online while the other 16 generate media. Every price below is per unit, billed on actual usage, with a $0.0020 minimum per call and a hard ceiling per call.

• Social platforms, 24 tools. instagram.profile from $0.0069 per profile, instagram.posts from $0.0045 per post, tiktok.profile from $0.0009 per result, twitter.search from $0.0012, youtube.transcript from $0.0150, reddit.posts from $0.0114. Facebook pages, groups, ads and comments are in there too.
• Ecommerce and marketplaces, 9 tools. amazon.search from $0.0003 per result, amazon.product from $0.0045, amazon.reviews from $0.0027, plus App Store and Google Play reviews at $0.0003 each and Google Shopping.
• People and companies, 6 tools. linkedin.profile from $0.03 per result, linkedin.company_employees from $0.0240 per profile, and email.finder from $0.06 per result.
• Search, maps and the open web, 7 tools. web.search from $0.0075 per results page, news.search from $0.0120, maps.places from $0.0090 per place, and web.scrape from $0.005970 per URL for any page no site-specific tool covers.

The remaining 16 are video, image and voice generation, which is a different job entirely. Full inputs, outputs and prices for all of them are on the tools pages.

Worked flow one: auditing a creator for under ten cents

A full creator audit is two calls and costs $0.0969. The first returns follower count, bio, verification and post count for the handle. The second returns the last 20 posts with their engagement counts and timestamps, so the agent can divide one by the other and produce a real engagement rate instead of a screenshot of a follower number.

run_tool({
  slug: "instagram.profile",
  input: { usernames: ["thecreator"] }
})

run_tool({
  slug: "instagram.posts",
  input: { username: ["thecreator"], resultsLimit: 20 }
})

The arithmetic: $0.0069 for the profile, then 20 posts at $0.0045 each, which is $0.0900. Two hundred creators screened this way costs $19.38 and finishes while you make coffee. The DIY version of the same job is an Instagram scraper, a proxy budget, and an ongoing subscription to whatever breaks next. This is the same shape as the lead generation playbook, which runs it end to end with enrichment attached.

Worked flow two: price monitoring for pennies a day

Price monitoring on Amazon is also two calls, at $0.0630 a sweep. One amazon.search keyword pulls a page of up to about 60 products at $0.0003 each, which caps that call at $0.0180 and hands back ASINs with prices, ratings and rank positions. Feed the ten ASINs you care about into amazon.product at $0.0045 each for the full listing, and you have a competitive set with prices.

Run it once a day for a month and the whole programme costs $1.89. That is cheap enough that the interesting question stops being budget and starts being what you do with the diff: alert on a competitor undercut, track how often a rival discounts, or watch your own listing slide down the rankings for a keyword you own.

DIY scraper or a tool gateway?

Build it yourself when you are hitting one site at volumes where a per-call margin stops making sense, or when nobody has a tool for the site you need. Use a gateway for everything else, which in practice is the long tail: fifteen sites you touch occasionally and none of which deserve their own codebase.

Is scraping public data legal?

Scraping pages that anyone can load without signing in is generally lawful in the US and the EU, but lawful is not the same as permitted. In hiQ v LinkedIn the Ninth Circuit held that scraping public profiles is not computer fraud, and the same case still ended with hiQ losing on LinkedIn's terms of use. Not a crime and not allowed are different findings. None of this is legal advice.

The working rules we hold ourselves to, and would suggest to anyone:

• Public means logged out. If a page needs an account, a paywall or a scraped session cookie to load, it is not public data and no tool should be pretending otherwise.
• Personal data stays regulated after you collect it. A name and an email being visible on a site does not make GDPR go away. You still need a lawful basis, and people can still ask you to delete it.
• Read the terms of the site, not just the robots file. They are separate documents and they disagree more often than you would expect.
• Rate discipline is the ethics part that costs you nothing. Pull what you need and stop. Nobody has ever needed a full mirror of a site to answer one question.
• Facts are fine, wholesale copies are not.Prices, counts and timestamps are data. Republishing somebody's article text is a copyright question wearing a data hat.

Agents make this sharper, not softer, because they can do it at a cadence nobody would do by hand. A monitoring loop that runs every fifteen minutes is a different footprint from a person checking a page, and the social listening playbook is worth reading with that in mind before you set an interval.