如果您已经下载了URL(使用
requests
的
GET
请求而没有使用
stream
选项),则已经拥有两个可用大小,因为整个响应已经被下载并解压缩,原始长度在标头中可用:
from __future__ import division
r = requests.get(url, headers=headers)
compressed_length = int(r.headers['content-length'])
decompressed_length = len(r.content)
ratio = compressed_length / decompressed_length
你可以将使用
Accept-Encoding: identity
的 HEAD 请求的内容长度头与使用
Accept-Encoding: gzip
的请求进行比较:
no_gzip = {'Accept-Encoding': 'identity'}
no_gzip.update(headers)
uncompressed_length = int(requests.get(url, headers=no_gzip).headers['content-length'])
force_gzip = {'Accept-Encoding': 'gzip'}
force_gzip.update(headers)
compressed_length = int(requests.get(url, headers=force_gzip).headers['content-length'])
然而,这种方法可能并不适用于所有服务器,因为动态生成内容的服务器通常会在这种情况下破坏Content-Length头部,以避免首先渲染内容。
如果您请求的是分块传输编码资源,则不会有content-length头部,在这种情况下,HEAD请求可能会或可能不会提供正确的信息。
在这种情况下,您必须流式传输整个响应,并从流的末尾提取解压后的大小(GZIP格式在末尾包括一个小端4字节无符号整数)。使用原始urllib3响应对象上的stream()
方法:
import requests
from collections import deque
if hasattr(int, 'from_bytes'):
_extract_size = lambda q: int.from_bytes(bytes(q), 'little')
else:
import struct
_le_int = struct.Struct('<I').unpack
_extract_size = lambda q: _le_int(b''.join(q))[0]
def get_content_lengths(url, headers=None, chunk_size=2048):
"""Return the compressed and uncompressed lengths for a given URL
Works for all resources accessible by GET, regardless of transfer-encoding
and discrepancies between HEAD and GET responses. This does have
to download the full request (streamed) to determine sizes.
"""
only_gzip = {'Accept-Encoding': 'gzip'}
only_gzip.update(headers or {})
r = requests.get(url, headers=only_gzip, stream=True)
r.raise_for_status()
if r.headers.get('Content-Encoding') != 'gzip':
raise ValueError('Response not gzip-compressed')
last_data = deque(maxlen=4)
compressed_length = 0
for chunk in r.raw.stream(chunk_size, decode_content=False):
compressed_length += len(chunk)
last_data.extend(chunk)
if compressed_length < 4:
raise ValueError('Not enough data loaded to determine uncompressed size')
return compressed_length, _extract_size(last_data)
演示:
>>> compressed_length, decompressed_length = get_content_lengths('http://httpbin.org/gzip')
>>> compressed_length
179
>>> decompressed_length
226
>>> compressed_length / decompressed_length
0.7920353982300885