我正在使用urllib3爬取网页。示例代码:
from urllib3 import PoolManager
pool = PoolManager()
response = pool.request("GET", url)
问题在于,我可能会偶然遇到一些下载非常大的文件的网址,而我并不想下载它。
我找到了这个问题 - 链接 - 它建议使用
urllib
和urlopen
。我不想两次联系服务器。我想限制文件大小为25MB。是否有办法使用
urllib3
实现这一点?我正在使用urllib3爬取网页。示例代码:
from urllib3 import PoolManager
pool = PoolManager()
response = pool.request("GET", url)
urllib
和urlopen
。我不想两次联系服务器。urllib3
实现这一点?Content-Length
头,那么您可以使用它来确定是否继续下载剩余的内容。如果服务器没有提供这个头,则需要流式传输响应,直到您决定不再继续为止。from urllib3 import PoolManager
pool = PoolManager()
response = pool.request("GET", url, preload_content=False)
# Maximum amount we want to read
max_bytes = 1000000
content_bytes = response.headers.get("Content-Length")
if content_bytes and int(content_bytes) < max_bytes:
# Expected body is smaller than our maximum, read the whole thing
data = response.read()
# Do something with data
...
elif content_bytes is None:
# Alternatively, stream until we hit our limit
amount_read = 0
for chunk in r.stream():
amount_read += len(chunk)
# Save chunk
...
if amount_read > max_bytes:
break
# Release the connection back into the pool
response.release_conn()