Python的http下载页面源代码

Question

你好，我想知道是否可以连接到http主机（例如谷歌.com）并下载网页源代码？

提前感谢。

- DonJuma

Answer 1

使用urllib2下载页面。

由于谷歌会尝试阻止所有机器人，因此将阻止此请求。请在请求中添加用户代理。

import urllib2
user_agent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3'
headers = { 'User-Agent' : user_agent }
req = urllib2.Request('http://www.google.com', None, headers)
response = urllib2.urlopen(req)
page = response.read()
response.close() # its always safe to close an open connection

你也可以使用pyCurl

import sys
import pycurl

class ContentCallback:
        def __init__(self):
                self.contents = ''

        def content_callback(self, buf):
                self.contents = self.contents + buf

t = ContentCallback()
curlObj = pycurl.Curl()
curlObj.setopt(curlObj.URL, 'http://www.google.com')
curlObj.setopt(curlObj.WRITEFUNCTION, t.content_callback)
curlObj.perform()
curlObj.close()
print t.contents

- pyfunc

---

urllib2模块在Python 3中已被拆分为几个模块，命名为urllib.request和urllib.error。因此，使用上述代码会出现“没有urllib2模块”的错误。有关更新的答案，请参见https://dev59.com/5XE85IYBdhLWcg3wZyqt。 - Joris

PyCurl不需要传递用户代理吗？ - undefined

Answer 2

你可以使用urllib2模块。

import urllib2
url = "http://somewhere.com"
page = urllib2.urlopen(url)
data = page.read()
print data

查看文档以获取更多示例

- ghostdog74

Answer 3

文档中的httplib（低级）和urllib（高级）应该可以帮助你入门。选择更适合你的那一个。

- AndiDog

Answer 4

使用requests包：

使用requests包：

# Import requests
import requests

#url
url = 'https://www.google.com/'

# Create the binary string html containing the HTML source
html = requests.get(url).content

or with the urllib

from urllib.request import urlopen

#url
url = 'https://www.google.com/'

# Create the binary string html containing the HTML source
html = urlopen(url).read()

- Orkun Berk Yuzbasioglu

Answer 5

so here's another approach to this problem using mechanize. I found this to bypass a website's robot checking system. i commented out the set_all_readonly because for some reason it wasn't recognized as a module in mechanize.

import mechanize
url = 'http://www.example.com'

br = mechanize.Browser()
#br.set_all_readonly(False)    # allow everything to be written to
br.set_handle_robots(False)   # ignore robots
br.set_handle_refresh(False)  # can sometimes hang without this
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]           # [('User-agent', 'Firefox')]
response = br.open(url)
print response.read()      # the text of the page
response1 = br.response()  # get the response again
print response1.read()     # can apply lxml.html.fromstring()

- tisaconundrum