SemrushBot
网站经常会看到：https://www.semrush.com/bot/ 来访问，它的官网是这样介绍自己的：
A bot, also known as a web robot, web spider or web crawler, is a software application designed to automatically perform simple and repetitive tasks in a more effective, structured, and concise manner than any human can ever do. The most common use of bots is in web spidering or web crawling.
SemrushBot 是一个国外的网络机器人，主要用来网页爬取或网页抓取。
如果你的站点面对的是国内客户，完全可以直接屏蔽。
下面是使用robots.txt对一些常见的网络爬虫的屏蔽:
User-agent: AhrefsBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: Uptimebot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: MegaIndex.ru
Disallow: /
User-agent: ZoominfoBot
Disallow: /
User-agent: Mail.Ru
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: ExtLinksBot
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: Researchscan
Disallow: /
User-agent: DnyzBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: YandexBot
Disallow: /
0
赞
热门推荐
-
2、 - 优质文章
-
3、 gate.io
-
8、 golang
-
9、 openharmony
-
10、 Vue中input框自动聚焦