这是个不好的主意,因为URL很长并且b64编码可能会超过文件名255字节的限制!可以压缩和b64编码,但效果有限。
from base64 import b64encode
import zlib
import bz2
from urllib.parse import quote
def url_strategies(url):
url = url.encode('utf8')
print(url.decode())
print(f'normal : {len(url)}')
print(f'quoted : {len(quote(url, ""))}')
b64url = b64encode(url)
print(f'b64 : {len(b64url)}')
url = b64encode(zlib.compress(b64url))
print(f'b64+zlib: {len(url)}')
url = b64encode(bz2.compress(b64url))
print(f'b64+bz2: {len(url)}')
这是我在angel.co上找到的一个普通链接:
URL = 'https://angel.co/job_listings/browse_startups_table?startup_ids%5B%5D=972887&startup_ids%5B%5D=365478&startup_ids%5B%5D=185570&startup_ids%5B%5D=32624&startup_ids%5B%5D=134966&startup_ids%5B%5D=722477&startup_ids%5B%5D=914250&startup_ids%5B%5D=901853&startup_ids%5B%5D=637842&startup_ids%5B%5D=305240&tab=find&page=1'
即使使用b64+zlib,它也无法符合255的限制:
normal : 316
quoted : 414
b64 : 424
b64+zlib: 304
b64+bz2 : 396
即使使用最佳的zlib压缩和b64编码策略,您仍会遇到问题。
正确的解决方案是,将url哈希化,并将其作为文件属性附加到文件中。
import os
from hashlib import sha256
def save_file(url, content, char_limit=13):
hash = sha256(url.encode()).hexdigest()[:char_limit]
filename = f'{hash}.html'
with open(filename, 'w') as f:
f.write(content)
os.setxattr(filename, 'user.url', url.encode())
然后你可以获取URL属性:
print(os.getxattr(filename, 'user.url').decode())
'https://angel.co/job_listings/browse_startups_table?startup_ids%5B%5D=972887&startup_ids%5B%5D=365478&startup_ids%5B%5D=185570&startup_ids%5B%5D=32624&startup_ids%5B%5D=134966&startup_ids%5B%5D=722477&startup_ids%5B%5D=914250&startup_ids%5B%5D=901853&startup_ids%5B%5D=637842&startup_ids%5B%5D=305240&tab=find&page=1'
注意:在Python中使用setxattr和getxattr需要添加user.
前缀
有关Python文件属性,请参见此处相关问题:https://dev59.com/4p3ha4cB1Zd3GeqPT2n2#56399698