Mediapartners-Google · AdsBot Wildcard Rules · /ads.txt Reachability Audit

Robots.txt Mediapartners-Google & AdsBot Crawler Auditor

Paste your site's robots.txt to verify whether Mediapartners-Google, AdsBot-Google, and programmatic /ads.txt crawlers are permitted or accidentally blocked.

1-Click Test Cases:

Go Beyond Crawler Rules with Full 30-Section Domain Intelligence

Once Mediapartners-Google and /ads.txt are reachable, cross-check live seller authorizations in Reverse AdSense Lookup or browse Publisher Domains (/id). Need evidence-backed website teardowns? Start with the $19 Starter Pack (120 non-expiring credits · 24 reports) or upgrade to Pro ($49/mo · 400 credits).

How Mediapartners-Google and AdsBot-Google Differ from Googlebot

Google's ad-serving crawlers do not follow the exact same wildcard inheritance rules as standard web search crawlers, which causes frequent production bugs after staging migrations:

  • Mediapartners-Google (AdSense Contextual Crawler): Falls back to User-agent: * when not explicitly named. Leaving Disallow: / under * blocks AdSense contextual matching unless you add an explicit User-agent: Mediapartners-Google + Allow: / group.
  • AdsBot-Google & AdsBot-Google-Mobile: Per Google's crawler specification, AdsBot ignores wildcard User-agent: * rules and must be named explicitly if you intend to restrict or permit paths.
  • Unauthenticated /ads.txt & /app-ads.txt: Blocking root seller files in robots.txt stops Google and SSP crawlers from verifying your pub- authorization.

Diagnostic checks reflect public crawler specifications as technical publisher guidance, not a Google approval guarantee.

Frequently Asked Questions

What happens if Mediapartners-Google is blocked in robots.txt?

Blocking Mediapartners-Google prevents Google AdSense from reading page content for contextual ad targeting, often causing blank ad slots, generic low-CPM fallback ads, or site review delays.

Does AdsBot-Google obey User-agent: * wildcard rules?

No. Google documents that AdsBot-Google and AdsBot-Google-Mobile ignore wildcard (*) groups; only an explicit User-agent: AdsBot-Google block affects them.

Can robots.txt block Google from verifying /ads.txt?

Yes. If your robots.txt disallows /ads.txt or the root path for Google crawlers, AdSense cannot verify authorized seller lines and may display an Earnings at risk warning.

Does passing this robots.txt check guarantee AdSense site approval?

No. Unblocking crawlers is a technical prerequisite so reviewers and bots can fetch your pages, while site approval depends on Google's content and policy review.