logo标签列表

Scrapy，Privoxy和Tor：SocketError：[Errno 61]连接被拒绝

pythonweb-scrapingscrapytor
5

我正在使用Scrapy、Privoxy和Tor。这是我的之前的问题Scrapy with Privoxy and Tor: how to renew IP，这是蜘蛛：

from scrapy.contrib.spiders import CrawlSpider
from scrapy.selector import Selector
from scrapy.http import Request

class YourCrawler(CrawlSpider):
    name = "****"
    start_urls = [
    'https://****.com/listviews/titles.php',
    ]
    allowed_domains = ["****.com"]

    def parse(self, response):
        # go to the urls in the list
        s = Selector(response)
        page_list_urls = s.xpath('///*[@id="tab7"]/article/header/h2/a/@href').extract()
        for url in page_list_urls:
            yield Request(response.urljoin(url), callback=self.parse_following_urls, dont_filter=True)

        # Return back and go to bext page in div#paginat ul li.next a::attr(href) and begin again
        next_page = response.css('ul.pagin li.presente ~ li a::attr(href)').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield Request(next_page, callback=self.parse)

    # For the urls in the list, go inside, and in div#main, take the div.ficha > div.caracteristicas > ul > li
    def parse_following_urls(self, response):
        #Parsing rules go here
        for each_book in response.css('main#main'):
            yield {
                'editor': each_book.css('header.datos1 > ul > li > h5 > a::text').extract(),
            }

在settings.py文件中，我有一个用户代理旋转和privoxy：
DOWNLOADER_MIDDLEWARES = {
        #user agent
        'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware' : None,
        '****.comm.rotate_useragent.RotateUserAgentMiddleware' :400,
        #privoxy
        'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
        '****.middlewares.ProxyMiddleware': 100
    }

在middlewares.py中，我添加了以下内容：
from stem import Signal
from stem.control import Controller

def _set_new_ip():
    with Controller.from_port(port=9051) as controller:
        controller.authenticate(password='tor_password')
        controller.signal(Signal.NEWNYM)

class ProxyMiddleware(object):
    def process_request(self, request, spider):
        _set_new_ip()
        request.meta['proxy'] = 'http://127.0.0.1:8118'
        spider.log('Proxy : %s' % request.meta['proxy'])

如果我删除middlewares.py中def _set_new_ip():方法并且在class ProxyMiddleware(object):中的调用，爬虫可以工作。但是我希望每次爬虫都调用一个新的IP地址，因此我添加了它。问题是每次尝试运行爬虫时都会返回错误SocketError: [Errno 61] Connection refused，并出现以下回溯信息：
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 37, in process_request
    response = yield method(request=request, spider=spider)
  File "/Users/nikita/scrapy/***/***/middlewares.py", line 71, in process_request
    _set_new_ip()
  File "/Users/nikita/scrapy/***/***/middlewares.py", line 65, in _set_new_ip
    with Controller.from_port(port=9051) as controller:
  File "/usr/local/lib/python2.7/site-packages/stem/control.py", line 998, in from_port
    control_port = stem.socket.ControlPort(address, port)
  File "/usr/local/lib/python2.7/site-packages/stem/socket.py", line 372, in __init__
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/stem/socket.py", line 243, in connect
    self._socket = self._make_socket()
  File "/usr/local/lib/python2.7/site-packages/stem/socket.py", line 401, in _make_socket
    raise stem.SocketError(exc)
SocketError: [Errno 61] Connection refused
2017-07-11 15:50:28 [scrapy.core.engine] INFO: Closing spider (finished)

也许问题出在使用的端口上：with Controller.from_port(port=9051) as controller:，但我不确定。如果有人有想法，那就太好了...
编辑---
好的，如果我打开浏览器并转到http://127.0.0.1:8118/，它会显示：
503 
This is Privoxy 3.0.26 on localhost (127.0.0.1), port 8118, enabled
Forwarding failure
Privoxy was unable to socks5-forward your request http://127.0.0.1:8118/ through localhost: SOCKS5 request failed

Just try again to see if this is a temporary problem, or check your forwarding settings and make sure that all forwarding servers are working correctly and listening where they are supposed to be listening.

也许与SOCKS5的配置有关… 有人知道吗？
看这里关于如何使用stem连接到Tor的信息：https://stem.torproject.org/api/connection.html - Tomáš Linhart
好的，在这个网站上他们谈论了authenticate()函数。在他们给出的例子中，首先他们创建了一个control_socket = stem.socket.ControlPort(port = 9051)，然后再执行stem.connection.authenticate(control_socket)。我应该把它们都放在ProxyMiddleware类里吗？ - user7499416
我有一些东西，更新问题。 - user7499416
你确定你已经启动了Tor，并且Privoxy与Tor的设置是正确并且正常工作的吗？ - Tomáš Linhart
我认为可以，因为当我访问 https://check.torproject.org/ 时它会显示Tor正在运行，而当我访问 http://p.p/ 时会显示Privoxy也在运行，并且在 http://www.ip2location.com/ 上我可以看到连接该网站的连接来自代理（当我抓取管理的网站时，我还可以看到来自另一个代理的IP连接）。 - user7499416
显示剩余2条评论
2个回答
2

我的猜测是：

  1. Tor没有运行。要确保Tor是否正在运行，请在终端上运行ps命令（例如：ps -ax | grep tor）和netstat命令（例如：对于Mac：netstat -an | grep 'your tor portnumber'，对于Linux，请将-an替换为-tulnp），查看Tor是否真的在运行。
  2. 你没有正确设置转发规则。根据503错误消息，看起来你没有正确设置转发规则（如果Tor正在运行）。在Privoxy的配置文件中，请确保取消注释forward-socks5t / 127.0.0.1:9050 .这一行。
0
class ProxyMiddleware(object):
    def process_request(self, request, spider):
        def _set_new_ip():
            with Controller.from_port(port=9051) as controller:
                controller.authenticate(password='PASSWORDHERE')
                controller.signal(Signal.NEWNYM)
        request.meta['proxy'] = 'http://127.0.0.1:8118'
        spider.log('Proxy : %s' % request.meta['proxy'])
你的回答目前不够清晰。请进行[编辑]以添加更多细节，帮助其他人理解这如何回答所问的问题。您可以在帮助中心找到有关撰写好答案的更多信息。 - Community
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接