解析原始的HTTP头部信息

51

我有一个原始的HTTP字符串,想要将其中的字段表示为一个对象。是否有办法从HTTP字符串中解析出各个头信息?

'GET /search?sourceid=chrome&ie=UTF-8&q=ergterst HTTP/1.1\r\nHost: www.google.com\r\nConnection: keep-alive\r\nAccept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\nUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.45 Safari/534.13\r\nAccept-Encoding: gzip,deflate,sdch\r\nAvail-Dictionary: GeNLY2f-\r\nAccept-Language: en-US,en;q=0.8\r\n
[...]'
9个回答

107

更新: 现在是2019年,我已经为Python 3重新编写了这个答案,因为一位程序员试图使用代码时发生了混淆的评论。原始的Python 2代码现在放在答案底部。

标准库中有优秀的工具,既能够解析RFC 821头文件,也可以解析整个HTTP请求。这里有一个示例请求字符串(请注意,Python将其视为一个大字符串,即使我们将其分成几行以提高可读性),我们可以将其提供给我的示例:

request_text = (
    b'GET /who/ken/trust.html HTTP/1.1\r\n'
    b'Host: cm.bell-labs.com\r\n'
    b'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n'
    b'Accept: text/html;q=0.9,text/plain\r\n'
    b'\r\n'
)

正如@TryPyPy所指出的,你可以使用Python的电子邮件消息库来解析标题 - 尽管我们应该补充一点,一旦你完成创建它,生成的对象就像一个头部字典:

from email.parser import BytesParser
request_line, headers_alone = request_text.split(b'\r\n', 1)
headers = BytesParser().parsebytes(headers_alone)

print(len(headers))     # -> "3"
print(headers.keys())   # -> ['Host', 'Accept-Charset', 'Accept']
print(headers['Host'])  # -> "cm.bell-labs.com"

不过,这种方法忽略了请求行,或者让您自己解析它。事实证明,有一个更好的解决方案。

如果使用标准库的BaseHTTPRequestHandler,它将为您解析HTTP。虽然其文档有点晦涩——这是标准库中整个HTTP和URL工具套件的问题——但要使其解析字符串,您只需(a)将字符串包装在BytesIO()中,(b)读取raw_requestline,以便它准备好被解析,并(c)捕获在解析期间发生的任何错误代码,而不是让它尝试将它们写回客户端(因为我们没有客户端!)。

以下是我们对标准库类的特殊化:

from http.server import BaseHTTPRequestHandler
from io import BytesIO

class HTTPRequest(BaseHTTPRequestHandler):
    def __init__(self, request_text):
        self.rfile = BytesIO(request_text)
        self.raw_requestline = self.rfile.readline()
        self.error_code = self.error_message = None
        self.parse_request()

    def send_error(self, code, message):
        self.error_code = code
        self.error_message = message

我希望标准库的开发人员能够意识到HTTP解析应该以一种不需要我们编写九行代码才能正确调用的方式分离出来,但是我们能做什么呢?以下是您如何使用这个简单类:

# Using this new class is really easy!

request = HTTPRequest(request_text)

print(request.error_code)       # None  (check this first)
print(request.command)          # "GET"
print(request.path)             # "/who/ken/trust.html"
print(request.request_version)  # "HTTP/1.1"
print(len(request.headers))     # 3
print(request.headers.keys())   # ['Host', 'Accept-Charset', 'Accept']
print(request.headers['host'])  # "cm.bell-labs.com"

如果在解析过程中出现错误,error_code 将不会是 None:

# Parsing can result in an error code and message

request = HTTPRequest(b'GET\r\nHeader: Value\r\n\r\n')

print(request.error_code)     # 400
print(request.error_message)  # "Bad request syntax ('GET')"

我更喜欢像这样使用标准库,因为我怀疑如果我试图使用正则表达式重新实现一个互联网规范,那么可能会遇到一些问题,而标准库已经遇到并解决了这些问题。

旧的Python 2代码

以下是我最初编写答案时的原始代码:

request_text = (
    'GET /who/ken/trust.html HTTP/1.1\r\n'
    'Host: cm.bell-labs.com\r\n'
    'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n'
    'Accept: text/html;q=0.9,text/plain\r\n'
    '\r\n'
    )

而且:

# Ignore the request line and parse only the headers

from mimetools import Message
from StringIO import StringIO
request_line, headers_alone = request_text.split('\r\n', 1)
headers = Message(StringIO(headers_alone))

print len(headers)     # -> "3"
print headers.keys()   # -> ['accept-charset', 'host', 'accept']
print headers['Host']  # -> "cm.bell-labs.com"

并且:

from BaseHTTPServer import BaseHTTPRequestHandler
from StringIO import StringIO

class HTTPRequest(BaseHTTPRequestHandler):
    def __init__(self, request_text):
        self.rfile = StringIO(request_text)
        self.raw_requestline = self.rfile.readline()
        self.error_code = self.error_message = None
        self.parse_request()

    def send_error(self, code, message):
        self.error_code = code
        self.error_message = message

并且:

# Using this new class is really easy!

request = HTTPRequest(request_text)

print request.error_code       # None  (check this first)
print request.command          # "GET"
print request.path             # "/who/ken/trust.html"
print request.request_version  # "HTTP/1.1"
print len(request.headers)     # 3
print request.headers.keys()   # ['accept-charset', 'host', 'accept']
print request.headers['host']  # "cm.bell-labs.com"

而且:

# Parsing can result in an error code and message

request = HTTPRequest('GET\r\nHeader: Value\r\n\r\n')

print request.error_code     # 400
print request.error_message  # "Bad request syntax ('GET')"

2
我不确定!Python用于解析的Message和请求类内部应该有一行代码来创建头字典。如果可以告诉它使用OrderedDict而不是普通的dict,那么您就会知道顺序 - 但是,我刚刚简要地浏览了代码,无法确定头字典是在哪里创建的。 - Brandon Rhodes
2
@jeffrey:从Python 3.6开始,字典的顺序是插入顺序:https://docs.python.org/3/library/stdtypes.html#typesmapping - Georg Schölly
接收方必须将HTTP消息解析为八位字节的序列,使用至少是US-ASCII字符集子集的编码。将HTTP消息解析为Unicode字符流,而不考虑特定编码,会创建安全漏洞,因为字符串处理库处理包含换行符(%x0A)的无效多字节字符序列的方式各不相同。基于字符串的解析器只能在从消息中提取协议元素后,并且在消息分析确定了各个字段之后,在协议元素内部安全地使用,例如在标头字段值中。 - Géry Ogam
2
@Maggyero,我已经更新了Python 3的代码。享受吧! - Brandon Rhodes
嗨 @Maggyero,有一个更好的选择。看看 kiss-headers 包。请访问 https://github.com/Ousret/kiss-headers。 - Ousret
显示剩余5条评论

14

mimetools自Python 2.3起已被弃用,并在Python 3中完全删除(链接)。

以下是在Python 3中的应该做法:

import email
import io
import pprint

# […]

request_line, headers_alone = request_text.split('\r\n', 1)
message = email.message_from_file(io.StringIO(headers_alone))
headers = dict(message.items())
pprint.pprint(headers, width=160)

2
email.message_from_file(io.StringIO(headers_alone)) 可以替换为 email.message_from_string(headers_alone) - eigenein

7

如果您去掉 GET 行,这似乎可以正常工作:

import mimetools
from StringIO import StringIO

he = "Host: www.google.com\r\nConnection: keep-alive\r\nAccept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\nUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.45 Safari/534.13\r\nAccept-Encoding: gzip,deflate,sdch\r\nAvail-Dictionary: GeNLY2f-\r\nAccept-Language: en-US,en;q=0.8\r\n"

m = mimetools.Message(StringIO(he))

print m.headers

一种解析示例并将第一行的信息添加到对象中的方法是:
import mimetools
from StringIO import StringIO

he = 'GET /search?sourceid=chrome&ie=UTF-8&q=ergterst HTTP/1.1\r\nHost: www.google.com\r\nConnection: keep-alive\r\n'

# Pop the first line for further processing
request, he = he.split('\r\n', 1)    

# Get the headers
m = mimetools.Message(StringIO(he))

# Add request information
m.dict['method'], m.dict['path'], m.dict['http-version'] = request.split()    

print m['method'], m['path'], m['http-version']
print m['Connection']
print m.headers
print m.dict

在Python3中有没有一种方法可以做到这一点? - Broseph
mimetools自2.3版本起已被弃用。 - demented hedgehog
@Broseph 请参考Gowtham的回答。 - jeromej

2

使用 Python3.7,urllib3.HTTPResponsehttp.client.parse_headers,并结合 curl 标志解释在这里

curl -i -L -X GET "http://httpbin.org/relative-redirect/3" |  python -c '
import sys
from io import BytesIO
from urllib3 import HTTPResponse
from http.client import parse_headers

rawresponse = sys.stdin.read().encode("utf8")
redirects = []

while True:
    header, body = rawresponse.split(b"\r\n\r\n", 1)
    if body[:4] == b"HTTP":
        redirects.append(header)
        rawresponse = body
    else:
        break

f = BytesIO(header)
# read one line for HTTP/2 STATUSCODE MESSAGE
requestline = f.readline().split(b" ")
protocol, status = requestline[:2]
headers = parse_headers(f)

resp = HTTPResponse(body, headers=headers)
resp.status = int(status)

print("headers")
print(resp.headers)

print("redirects")
print(redirects)
'

输出:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   215  100   215    0     0    435      0 --:--:-- --:--:-- --:--:--   435

headers
HTTPHeaderDict({'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Date': 'Thu, 20 Sep 2018 05:39:25 GMT', 'Content-Type': 'application/json', 'Content-Length': '215', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Via': '1.1 vegur'})
redirects
[b'HTTP/1.1 302 FOUND\r\nConnection: keep-alive\r\nServer: gunicorn/19.9.0\r\nDate: Thu, 20 Sep 2018 05:39:24 GMT\r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 0\r\nLocation: /relative-redirect/2\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Credentials: true\r\nVia: 1.1 vegur',
 b'HTTP/1.1 302 FOUND\r\nConnection: keep-alive\r\nServer: gunicorn/19.9.0\r\nDate: Thu, 20 Sep 2018 05:39:24 GMT\r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 0\r\nLocation: /relative-redirect/1\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Credentials: true\r\nVia: 1.1 vegur',
 b'HTTP/1.1 302 FOUND\r\nConnection: keep-alive\r\nServer: gunicorn/19.9.0\r\nDate: Thu, 20 Sep 2018 05:39:24 GMT\r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 0\r\nLocation: /get\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Credentials: true\r\nVia: 1.1 vegur']

notes:


在Python 3.7.7上,我得到了ValueError:not enough values to unpack(expected 2,got 1),针对header,body = rawresponse.split(b“\ r \ n \ r \ n”,1) - HappyFace

1
有没有办法从HTTP字符串中解析出单独的标头?我写了一个简单的函数,可以返回一个字典对象,希望能对你有所帮助。^_^ Python 3
def parse_request(request):
    raw_list = request.split("\r\n")
    request = {}
    for index in range(1, len(raw_list)):
        item = raw_list[index].split(":")
        if len(item) == 2:
            request.update({item[0].lstrip(' '): item[1].lstrip(' ')})
    return request


raw_request = 'GET /search?sourceid=chrome&ie=UTF-8&q=ergterst HTTP/1.1\r\nHost: www.google.com\r\nConnection: keep-alive\r\nAccept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\nUser-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.45 Safari/534.13\r\nAccept-Encoding: gzip,deflate,sdch\r\nAvail-Dictionary: GeNLY2f-\r\nAccept-Language: en-US,en;q=0.8\r\n'
request = parse_request(raw_request)
print(request)
print('\n')
print(request.keys())

输出:

{'Host': 'www.google.com', 'Connection': 'keep-alive', 'Accept': 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5', 
'User-Agent': 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.45 Safari/534.13', 'Accept-Encoding': 'gzip,deflate,sdch', 'Avail-Dictionary': 'GeNLY2f-', 'Accept-Language': 'en-US,en;q=0.8'}


dict_keys(['Host', 'Connection', 'Accept', 'User-Agent', 'Accept-Encoding', 'Avail-Dictionary', 'Accept-Language'])

1
以Pythonic的方式
request_text = (
    b'GET /who/ken/trust.html HTTP/1.1\r\n'
    b'Host: cm.bell-labs.com\r\n'
    b'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n'
    b'Accept: text/html;q=0.9,text/plain\r\n'
    b'\r\n'
)

print({ k:v.strip() for k,v in [line.split(":",1) 
        for line in request_text.decode().splitlines() if ":" in line]})

0

在Python3中

from email import message_from_string    
data = socket.recv(4096)
headers = message_from_string(str(data, 'ASCII').split('\r\n', 1)[1])
print(headers['Host'])


0
There is another way, which is simpler and safer, to handle headers that is more object-oriented and does not require manual parsing.
Short demo.
1. Parse them from "str", "bytes", "fp", "dict", "requests.Response", "email.Message", "httpx.Response", "urllib3.HTTPResponse".
from requests import get
from kiss_headers import parse_it

response = get('https://www.google.fr')
headers = parse_it(response)

headers.content_type.charset  # output: ISO-8859-1
# Its the same as
headers["content-type"]["charset"]  # output: ISO-8859-1

2. 构建它们

This

from kiss_headers import *

headers = (
    Host("developer.mozilla.org")
    + UserAgent(
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:50.0) Gecko/20100101 Firefox/50.0"
    )
    + Accept("text/html")
    + Accept("application/xhtml+xml")
    + Accept("application/xml", qualifier=0.9)
    + Accept(qualifier=0.8)
    + AcceptLanguage("en-US")
    + AcceptLanguage("en", qualifier=0.5)
    + AcceptEncoding("gzip")
    + AcceptEncoding("deflate")
    + AcceptEncoding("br")
    + Referer("https://developer.mozilla.org/testpage.html")
    + Connection(should_keep_alive=True)
    + UpgradeInsecureRequests()
    + IfModifiedSince("Mon, 18 Jul 2016 02:36:04 GMT")
    + IfNoneMatch("c561c68d0ba92bbeb8b0fff2a9199f722e3a621a")
    + CacheControl(max_age=0)
)

raw_headers = str(headers)

将会变成

Host: developer.mozilla.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:50.0) Gecko/20100101 Firefox/50.0
Accept: text/html, application/xhtml+xml, application/xml; q="0.9", */*; q="0.8"
Accept-Language: en-US, en; q="0.5"
Accept-Encoding: gzip, deflate, br
Referer: https://developer.mozilla.org/testpage.html
Connection: keep-alive
Upgrade-Insecure-Requests: 1
If-Modified-Since: Mon, 18 Jul 2016 02:36:04 GMT
If-None-Match: "c561c68d0ba92bbeb8b0fff2a9199f722e3a621a"
Cache-Control: max-age="0"

kiss-headers库的文档。


网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接