我一直在尝试爬取一个网站,但该网站似乎已经识别并封锁了我的IP地址,并显示出429 Too many requests的响应信息。
我从这个链接(https://github.com/aivarsk/scrapy-proxies)安装了scrapy-proxies,并按照给定的说明进行操作。我从这里(http://www.gatherproxy.com/)获取了一个代理列表,现在我的settings.py和proxylist.txt看起来是这样的:
当我运行我的爬虫时,出现以下错误:
我从这个链接(https://github.com/aivarsk/scrapy-proxies)安装了scrapy-proxies,并按照给定的说明进行操作。我从这里(http://www.gatherproxy.com/)获取了一个代理列表,现在我的settings.py和proxylist.txt看起来是这样的:
Settings.py
BOT_NAME = 'project'
SPIDER_MODULES = ['project.spiders']
NEWSPIDER_MODULE = 'project.spiders'
# Retry many times since proxies often fail
RETRY_TIMES = 10
# Retry on most error codes since proxies fail for different reasons
RETRY_HTTP_CODES = [429, 500, 503, 504, 400, 403, 404, 408]
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
'scrapy_proxies.RandomProxy': 100,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}
PROXY_LIST = "filepath\proxylist.txt"
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'
CONCURRENT_REQUESTS = 1
DOWNLOAD_DELAY = 2
PROXY_MODE = 0
DOWNLOAD_HANDLERS = {'s3': None}
EXTENSIONS = {
'scrapy.telnet.TelnetConsole': None
}
proxylist.txt
http://195.208.172.20:8080
http://154.119.56.179:9999
http://124.12.50.43:8088
http://61.7.168.232:52136
http://122.193.188.236:8118
当我运行我的爬虫时,出现以下错误:
[scrapy.proxies] DEBUG: Proxy user pass not found
我试图在谷歌上搜索特定错误,但找不到任何解决方案。
非常感谢您的帮助。提前致谢。