张大佬张大佬

通过.htaccess文件屏蔽垃圾蜘蛛恶意抓取方法

下面是通过.htaccess文件屏蔽垃圾蜘蛛恶意抓取方法和代码:


RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} "^$|^-$|MSNbot|Webdup|AcoonBot|SemrushBot|CrawlDaddy|DotBot|Applebot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|DingTalkBot|DuckDuckBot|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Barkrowler|SeznamBot|Jorgee|CCBot|SWEBot|PetalBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu|EasouSpider|YYSpider|python-requests|oBot|MauiBot" [NC]
RewriteRule !(^robots\.txt$) http://en.wikipedia.org/wiki/Robots_exclusion_standard [R=403,L]


将常见的垃圾蜘蛛名称都加进去,然后把这些代码放到htaccess里,就可以屏蔽垃圾蜘蛛的爬行了。


本原创文章未经允许不得转载 | 当前页面:张大佬 » 通过.htaccess文件屏蔽垃圾蜘蛛恶意抓取方法

评论