我正在编写一个脚本,用于访问链接列表并解析信息。对于大多数网站而言,它都能正常工作,但是某些网站上出现了问题,报错为“UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 13: ordinal not in range(128)”。它停在python3的urllib包中的client.py上。具体链接为:http://finance.yahoo.com/news/cafés-growing-faster-than-fast-food-peers-144512056.html。这里有很多类似的帖子,但是没有任何答案适用于我。我的代码如下:
from urllib import request
def __request(link,debug=0):
try:
html = request.urlopen(link, timeout=35).read() #made this long as I was getting lots of timeouts
unicode_html = html.decode('utf-8','ignore')
# NOTE the except HTTPError must come first, otherwise except URLError will also catch an HTTPError.
except HTTPError as e:
if debug:
print('The server couldn\'t fulfill the request for ' + link)
print('Error code: ', e.code)
return ''
except URLError as e:
if isinstance(e.reason, socket.timeout):
print('timeout')
return ''
else:
return unicode_html
这将调用请求功能
链接 = 'http://finance.yahoo.com/news/cafés-growing-faster-than-fast-food-peers-144512056.html' 页面 = __request链接
回溯是:
Traceback (most recent call last):
File "<string>", line 250, in run_nodebug
File "C:\reader\get_news.py", line 276, in <module>
main()
File "C:\reader\get_news.py", line 255, in main
body = get_article_body(item['link'],debug=0)
File "C:\reader\get_news.py", line 155, in get_article_body
page = __request('na',url)
File "C:\reader\get_news.py", line 50, in __request
html = request.urlopen(link, timeout=35).read()
File "C:\Python33\Lib\urllib\request.py", line 156, in urlopen
return opener.open(url, data, timeout)
File "C:\Python33\Lib\urllib\request.py", line 469, in open
response = self._open(req, data)
File "C:\Python33\Lib\urllib\request.py", line 487, in _open
'_open', req)
File "C:\Python33\Lib\urllib\request.py", line 447, in _call_chain
result = func(*args)
File "C:\Python33\Lib\urllib\request.py", line 1268, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python33\Lib\urllib\request.py", line 1248, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "C:\Python33\Lib\http\client.py", line 1061, in request
self._send_request(method, url, body, headers)
File "C:\Python33\Lib\http\client.py", line 1089, in _send_request
self.putrequest(method, url, **skips)
File "C:\Python33\Lib\http\client.py", line 953, in putrequest
self._output(request.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 13: ordinal not in range(128)
非常感谢您的帮助,这让我很困扰。我认为我已经尝试了所有x.decode和类似的组合。
(如果可能的话,我可以忽略有问题的字符。)
requests
确实会通过显式重新引用URL来处理其中包含非ASCII字符的URL。 - Martijn Pieters