我正在使用Python脚本(Mechanize)登录代理门户。我可以成功登录,可以从
另一件让我疯狂的事情是,来自机械臂的发送连接是“: close”,即使我已经将其设置为“keep-alive”,如您在“addheaders”中所见。
read()
函数中检查到。但是,在成功登录后,我无法通过代理访问被阻止的网站。所以我从FF中检查了HTTP头,并发现Connection: Keep-alive
。但是从mechanize
中,我发现Connection: close
。我尝试使用browser.addheaders
完全模拟来自FF的HTTP头,但这也没有起作用:( 经过深入挖掘,我发现有几个建议说服务器关闭连接,因为mechanize无法完全模拟浏览器,因为网页包含JS,而这不受mechanize
支持。那么,有没有办法模拟(使服务器感觉到)mechanize是一个浏览器(支持JS),即使它不支持呢?顺便说一下,我不需要JS,我可以像上面提到的那样成功登录。请不要建议PhantomJS。我需要一个Python包来完成工作,而不是一个无头浏览器。更新:FireFox Headers:
GET xxx HTTP/1.1
Host: xxx
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: DSLastAccess=1454082611
Connection: keep-alive
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Set-Cookie: DSEPAgentInstalled=; path=/; expires=Tue, 31-Jan-2006 16:18:32 GMT; secure
Date: Fri, 29 Jan 2016 16:18:32 GMT
x-frame-options: SAMEORIGIN
Connection: Keep-Alive
Keep-Alive: timeout=15
Pragma: no-cache
Cache-Control: no-store
Expires: -1
Transfer-Encoding: chunked
Mechanize添加请求头:
browser.addheaders = [('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),\
('Accept-Language', 'en-US,en;q=0.5'),\
('Accept-Encoding', 'gzip, deflate'),\
('Host', 'xxx.net'),\
('Connection','keep-alive'),\
('Cookie', 'DSLastAccess=1454082611'),\
('User-agent', 'Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0')]
机械化头文件
send: 'CONNECT xxx.net:443 HTTP/1.0\r\n'
send: '\r\n'
send: 'GET xxx.cgi HTTP/1.1\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nHost: xxx.net\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0\r\nConnection: close\r\nCookie: DSLastAccess=1454082611\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/html; charset=utf-8
header: Set-Cookie: DSEPAgentInstalled=; path=/; expires=Tue, 31-Jan-2006 16:31:03 GMT; secure
header: Date: Fri, 29 Jan 2016 16:31:03 GMT
header: x-frame-options: SAMEORIGIN
header: Connection: close
header: Pragma: no-cache
header: Cache-Control: no-store
header: Expires: -1
另一件让我疯狂的事情是,来自机械臂的发送连接是“: close”,即使我已经将其设置为“keep-alive”,如您在“addheaders”中所见。
Keep-alive
header 设置为close
才能正常工作。 - Jared Goguen