使用urllib2下载网页时出现乱码问题？（仅偶尔）

Question

为什么我访问这个网页时，只看到了HTML文本？

http://itunes.apple.com/us/app/mobile/id381057839

但是当我访问这个网页时，却看到了乱码？

http://itunes.apple.com/us/app/mobile/id375562663

我在Python中使用相同的download()函数，代码如下：

def download(source_url):
    try:
        socket.setdefaulttimeout(10)
        agent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AlexaToolbar/alxf-1.54 Firefox/3.6.10 GTB7.1"
        ree = urllib2.Request(source_url)
        ree.add_header('User-Agent',agent)
        ree.add_header("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
        ree.add_header("Accept-Language","en-us,en;q=0.5")
        ree.add_header("Accept-Charset","ISO-8859-1,utf-8;q=0.7,*;q=0.7")
        ree.add_header("Accept-Encoding","gzip,deflate")
        ree.add_header("Host","itunes.apple.com")
        resp = urllib2.urlopen(ree)
        htmlSource = resp.read()
        return htmlSource
    except Exception, e:
        print e

- TIMEX

Answer 1

解决了。这是压缩问题。

def download(source_url):
    try:
        socket.setdefaulttimeout(10)
        agents = ['Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)','Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1)','Microsoft Internet Explorer/4.0b1 (Windows 95)','Opera/8.00 (Windows NT 5.1; U; en)']
        ree = urllib2.Request(source_url)
        ree.add_header('User-Agent',random.choice(agents))
        ree.add_header('Accept-encoding', 'gzip')
        opener = urllib2.build_opener()
        h = opener.open(ree).read()
        import StringIO
        import gzip

        compressedstream = StringIO.StringIO(h)
        gzipper = gzip.GzipFile(fileobj=compressedstream)
        data = gzipper.read()
        return data

    except Exception, e:
        print e
        return ""

- TIMEX

---

嗯，也许你可以检查一下资源是否经过gzip压缩，除非你确定总是会得到gzip压缩的响应。 - Piskvor left the building