我需要使用Scrapy爬取一些“.рф”域名下的网站。URL的结构如下:“http://сайтдляпримера.рф"(这个URL并不是真实的,只是为了举例说明)。当然,我想要爬取的网站可以通过浏览器访问。
我尝试使用start_urls
属性开始爬取,例如:
start_urls = ['http://сайтдляпримера.рф']
还有 start_requests
函数:
def start_requests(self):
return [scrapy.Request("http://сайтдляпримера.рф/", callback=self._test)]
两者都没有按照预期工作,我收到了以下控制台消息:
2016-01-01 19:02:01 [scrapy] INFO: Spider opened
2016-01-01 19:02:01 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-01-01 19:02:01 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-01 19:02:01 [scrapy] DEBUG: Retrying <GET http://%D1%81%D0%B0%D0%B9%D1%82%D0%B4%D0%BB%D1%8F%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D0%B0.%D1%80%D1%84> (failed 1 times): DNS lookup failed: address '%D1%81%D0%B0%D0%B9%D1%82%D0%B4%D0%BB%D1%8F%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D0%B0.%D1%80%D1%84' not found: [Errno -2] Name or service not known.
2016-01-01 19:02:01 [scrapy] DEBUG: Retrying <GET http://%D1%81%D0%B0%D0%B9%D1%82%D0%B4%D0%BB%D1%8F%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D0%B0.%D1%80%D1%84> (failed 2 times): DNS lookup failed: address '%D1%81%D0%B0%D0%B9%D1%82%D0%B4%D0%BB%D1%8F%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D0%B0.%D1%80%D1%84' not found: [Errno -2] Name or service not known.
2016-01-01 19:02:01 [scrapy] DEBUG: Gave up retrying <GET http://%D1%81%D0%B0%D0%B9%D1%82%D0%B4%D0%BB%D1%8F%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D0%B0.%D1%80%D1%84> (failed 3 times): DNS lookup failed: address '%D1%81%D0%B0%D0%B9%D1%82%D0%B4%D0%BB%D1%8F%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D0%B0.%D1%80%D1%84' not found: [Errno -2] Name or service not known.
2016-01-01 19:02:01 [scrapy] ERROR: Error downloading <GET http://%D1%81%D0%B0%D0%B9%D1%82%D0%B4%D0%BB%D1%8F%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D0%B0.%D1%80%D1%84>: DNS lookup failed: address '%D1%81%D0%B0%D0%B9%D1%82%D0%B4%D0%BB%D1%8F%D0%BF%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D0%B0.%D1%80%D1%84' not found: [Errno -2] Name or service not known.
2016-01-01 19:02:01 [scrapy] INFO: Closing spider (finished)
*如果有关系,我需要在基于Linux的操作系统上使用scrapy。
是否有任何解决方案?如果可能的话,由于我无法访问框架的存储库(没有处理http请求的内容被修改),是否有办法从_spider
文件中解决这个问题?
"http://сайтдляпримера.рф".decode('utf-8').encode('idna')
的操作时,出现了错误exceptions.TypeError: must be unicode, not str
。然后我得到了xn--http://-8fga3bl9al3aq0crdjw9y.xn--p1ai
,并且出现了NotSupported: Unsupported URL scheme 'xn--http': no handler available for that scheme
的错误提示。 - Helvdan'http://сайтдляпримера.рф'.encode('idna').decode('utf-8')
。这将把字节转换为Unicode字符串。 - vrsxceptions.TypeError: must be unicode, not str
,因为'somestr'.method1().method2()
是一个链式调用。所以与您原来的答案没有区别。 - Helvdan'http://' + 'сайтдляпримера.рф'.encode('idna').decode('utf-8')
。我自己用urlopen
试过了,它可以工作。希望对于scrapy
也能起作用。 - vrs'http://' + u'сайтдляпримера.рф'.encode('idna')
。我敢打赌您在Python 3.x上测试了您的示例。 - Helvdan