我想和大家分享一下一个对我有效的解决方案(Python3,Windows10):
步骤1:开启Tor ControlPort,端口为9151
。
Tor服务默认运行在端口9150
上,而ControlPort则运行在9151
端口。当你运行netstat -an
时,你应该能够看到本地地址为127.0.0.1:9150
和127.0.0.1:9151
。
[go to windows terminal]
cd ...\Tor Browser\Browser\TorBrowser\Tor
tor --service remove
tor --service install -options ControlPort 9151
netstat -an
步骤2:Python脚本如下。
import os
import subprocess
import socket
import socks
import http.client
import time
import requests
from stem import Signal
from stem.control import Controller
import csv
import urllib
from bs4 import BeautifulSoup
import time
def launchTor():
sproc = subprocess.Popen(r'.../Tor Browser/Browser/firefox.exe')
time.sleep(30)
return sproc
def killTor(sproc):
sproc.kill()
def connectTor():
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9150, True)
socket.socket = socks.socksocket
print("Connected to Tor")
def set_new_ip():
socks.setdefaultproxy()
"""Change IP using TOR"""
with Controller.from_port(port=9151) as controller:
controller.authenticate()
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9150, True)
socket.socket = socks.socksocket
controller.signal(Signal.NEWNYM)
def checkIP():
conn = http.client.HTTPConnection("icanhazip.com")
conn.request("GET", "/")
time.sleep(3)
response = conn.getresponse()
print('current ip address :', response.read())
sproc = launchTor()
connectTor()
url_list = [list of all the urls you want to scrape]
for url in url_list:
set_new_ip()
time.sleep(5)
checkIP()
'''
[insert your scraping code here: bs4, urllib, your usual thingy]
'''
killTor(sproc)
上面的脚本将为您想要抓取的每个URL更新IP地址。只需确保睡眠时间足够长,以便IP更改。上次测试是昨天。希望这可以帮助您!