Skip to content
Plumb
Light theme onCheck a URL

DocsReference

Crawlers and User Agents

Every crawler and browser the tools can send requests as: its id for the API, the token robots.txt groups match, and the header it sends.
  • 84 user agents
  • Search, AI, social, browsers

Tokens and Headers

robots.txt groups are matched by a crawler's token (User-agent: Googlebot), not by its full header. Google's specialised crawlers fall back to the Googlebot group when a file has no group of their own. The header is the user-agent string the tools send when they fetch or render a page.

The List

Search, sort and export it below. The id is what the API's agent field takes.

84 / 84
Googlebot SmartphoneGoogleGooglebotMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
GooglebotGoogleGooglebotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/131.0.0.0 Safari/537.36
Googlebot NewsGoogleGooglebot-NewsGooglebot-News
Googlebot ImagesGoogleGooglebot-ImageGooglebot-Image/1.0
Googlebot VideoGoogleGooglebot-VideoGooglebot-Video/1.0
Google StoreBot (Mobile)GoogleStorebot-GoogleMozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36
Google StoreBot (Desktop)GoogleStorebot-GoogleMozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Google-InspectionTool (Mobile)GoogleGoogle-InspectionToolMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0;)
Google-InspectionTool (Desktop)GoogleGoogle-InspectionToolMozilla/5.0 (compatible; Google-InspectionTool/1.0;)
GoogleOtherGoogleGoogleOtherGoogleOther
GoogleOther-ImageGoogleGoogleOther-ImageGoogleOther-Image/1.0
GoogleOther-VideoGoogleGoogleOther-VideoGoogleOther-Video/1.0
Google-ExtendedGoogleGoogle-ExtendedGoogle-Extended
AdsBot Mobile WebGoogleAdsBot-Google-MobileMozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
AdsBotGoogleAdsBot-GoogleAdsBot-Google (+http://www.google.com/adsbot.html)
AdSenseGoogleMediapartners-GoogleMediapartners-Google
Google-CloudVertexBotGoogleGoogle-CloudVertexBotGoogle-CloudVertexBot
BingbotBingBingbotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/131.0.0.0 Safari/537.36
Bingbot SmartphoneBingBingbotMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
MSNBotBingMSNBotmsnbot/2.0b (+http://search.msn.com/msnbot.htm)
MSNBot-MediaBingMSNBot-Mediamsnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
AdIdxBotBingAdIdxBotMozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)
BingPreviewBingBingPreviewMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 BingPreview/1.0b
Yahoo!Other search enginesSlurpMozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
DuckDuckGoOther search enginesDuckDuckBotDuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
BaiduOther search enginesBaiduspiderMozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Baidu - Render (Desktop)Other search enginesBaiduspiderMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
Baidu - Render (Mobile)Other search enginesBaiduspiderMozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
YandexOther search enginesYandexMozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
ApplebotOther search enginesApplebotMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Applebot SmartphoneOther search enginesApplebotMozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)
FacebookSocial mediafacebookexternalhitfacebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Twitter / XSocial mediaTwitterbotTwitterbot/1.0
LinkedInSocial mediaLinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)
Screaming FrogSEO toolsScreaming Frog SEO SpiderScreaming Frog SEO Spider/21.0
Screaming Frog (Mobile)SEO toolsScreaming Frog SEO SpiderMozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; Screaming Frog SEO Spider/21.0)
BotifySEO toolsBotifyMozilla/5.0 (compatible; botify; http://botify.com)
OnCrawlSEO toolsOnCrawlMozilla/5.0 (compatible; OnCrawl/1.0; +http://www.oncrawl.com/)
Moz (Campaign/Diagnostics)SEO toolsrogerbotrogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot)
Moz (Mozscape/Freshscape)SEO toolsDotBotMozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
MajesticSEO toolsMJ12botMozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
AhrefsSEO toolsAhrefsBotMozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
SemrushSEO toolsSemrushBotMozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Internet ArchiveSEO toolsia_archiveria_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
PageSpeed Insights (Mobile)SEO tools*Mozilla/5.0 (Linux; Android 11; moto g power (2022)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 Chrome-Lighthouse
PageSpeed Insights (Desktop)SEO tools*Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Chrome-Lighthouse
OpenAIAI crawlersGPTBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)
SearchGPTAI crawlersOAI-SearchBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
ChatGPT-UserAI crawlersChatGPT-UserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PerplexityAI crawlersPerplexityBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Page 1 of 2