Python BaseHTTPServer，如何捕获“broken pipe”错误？

Question

我用Python构建了一个短网址转换引擎，但是我看到了很多"broken pipe"错误，并且我想知道在使用BaseHTTPServer类时如何最好地捕获它。以下代码并非全部内容，但可以让你对我目前正在做的事情有所了解：

    from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
    import memcache

    class clientThread(BaseHTTPRequestHandler):

            def do_GET(self):
                    content = None
                    http_code,response_txt,long_url = \
                            self.ag_trans_url(self.path,content,'GET')
                    self.http_output( http_code, response_txt, long_url )
                    return

            def http_output(self,http_code,response_txt,long_url):
                    self.send_response(http_code)
                    self.send_header('Content-type','text/plain')
                    if long_url:
                            self.send_header('Location', long_url)
                    self.end_headers()
                    if response_txt:
                            self.wfile.write(response_txt)
                    return

        def ag_trans_url(self, orig_short_url, post_action, getpost):
                short_url = 'http://foo.co' + orig_short_url

                # fetch it from memcache
                long_url = mc.get(short_url)

                # other magic happens to look it up from db if there was nothing
                # in memcache, etc
                return (302, None, log_url)

def populate_memcache()
        # connect to db, do lots of mc.set() calls

def main():
        populate_memcache()
        try:
                port = 8001
                if len(sys.argv) > 1:
                        port = int(sys.argv[1])
                server = HTTPServer(('',port), clientThread)
                #server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
                print '[',str(datetime.datetime.now()),'] short url processing has begun'

                server.serve_forever()
        except KeyboardInterrupt,SystemExit:
                print '^C received, shutting down server'
                server.socket.close()

这段代码本身运行良好，但在生产环境中几乎立即开始抛出错误：

Traceback (most recent call last):
  File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
    self.process_request(request, client_address)
  File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
    self.handle()
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
    self.handle_one_request()
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 310, in handle_one_request
    method()
  File "/opt/short_url_redirector/shorturl.py", line 38, in do_GET
    self.http_output( http_code, response_txt, long_url )
  File "/opt/short_url_redirector/shorturl.py", line 52, in http_output
    self.send_response(http_code)
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 370, in send_response
    self.send_header('Server', self.version_string())
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 376, in send_header
    self.wfile.write("%s: %s\r\n" % (keyword, value))
  File "/usr/lib/python2.5/socket.py", line 274, in write
    self.flush()
  File "/usr/lib/python2.5/socket.py", line 261, in flush
    self._sock.sendall(buffer)
error: (32, 'Broken pipe')

这些错误大部分似乎源于调用send_header()方法时出现问题，我只写了以下内容：

self.send_header('Location', long_url)

我很好奇在我的代码中哪里尝试捕获这个IO异常...我是在每个self.send_header/self.end_headers/self.wfile.write调用周围编写try/except调用吗？我有时看到的另一种错误是这个，但不确定要观察哪个异常才能甚至捕获它：

Traceback (most recent call last):
  File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
    self.process_request(request, client_address)
  File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
    self.handle()
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
    self.handle_one_request()
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 299, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File "/usr/lib/python2.5/socket.py", line 381, in readline
    data = self._sock.recv(self._rbufsize)
error: (104, 'Connection reset by peer')

- iandouglas

Answer 1

这似乎是SocketServer的一个错误，详见此链接 Python Bug: 14574 修复方法（适用于Python 2.7）是覆盖SocketServer.StreamRequestHandler的finish()方法，类似于以下内容：

...
def finish(self,*args,**kw):
  try:
    if not self.wfile.closed:
      self.wfile.flush()
      self.wfile.close()
  except socket.error:
    pass
  self.rfile.close()

  #Don't call the base class finish() method as it does the above
  #return SocketServer.StreamRequestHandler.finish(self)

- Jay M

---

感谢您的跟进，Jason！ - iandouglas

我的解决方案与你的类似，但在handle()方法中。 - Brent Washburne

Answer 2

“broken pipe”异常意味着你的代码试图向已关闭的套接字/管道中写入数据。如果另一端是一个Web浏览器，用户可能已经停止了请求。您可以忽略回溯;它并不表示严重问题。如果您想要抑制该消息，可以在http_output函数的所有代码周围放置try ... except块，并记录异常（如果需要）。

此外，如果您希望您的HTTP服务器同时处理多个请求，则需要您的服务器类使用SocketServer.ForkingMixIn和SocketServer.ThreadingMixIn类之一。有关详细信息，请查看SocketServer模块的文档。

添加： “connection reset by peer”异常意味着您的代码试图从死亡套接字中读取数据。如果您想要抑制回溯，则需要扩展BaseHTTPServer类并覆盖handle_one_request方法以添加try ... except块。无论如何，您都需要一个新的服务器类来实现前面关于同时处理多个请求的建议。

- Kushal Kumaran

---

谢谢！我目前不太担心多进程问题。我想知道实际上应该捕获哪个异常，是IOError异常还是其他异常？我还怀疑Broken Pipe错误是由于该系统位于HAProxy防火墙后面造成的。这是我接下来要搜索的内容。 - iandouglas

socket模块中的函数会引发socket.error异常。从Python 2.6开始，socket.error是IOError的子类（参见socket模块文档）。由于您的代码针对的是2.5版本（根据回溯中的文件名判断），因此您需要捕获socket.error异常。 - Kushal Kumaran

Answer 3

在我的应用程序中，错误发生在finish()之外，在handle()中发生。这个修复方案可以捕获broken pipe错误:

class MyHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

    ...

    def handle(self):
        try:
            BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
        except socket.error:
            pass

- Brent Washburne