uWSGI无法将Unicode数据写入从Python的stdout日志重定向的日志文件中

Question

我正在使用uWSGI（2.0.11.2）和Python（3.4.3）来在Ubuntu 14.04上提供我的Pyramid（1.5.7）应用程序。我注意到我的uWSGI日志中出现了与Unicode解码相关的错误：

#
# one of the situations when exception is raised is
# when SQLAlchemy (which has set INFO logging level)
# tries to write an SQL statement containing unicode charater
# into log file
#
2016-02-26 16:01:38,734 INFO  [sqlalchemy.engine.base.Engine][b'uWSGIWorker5Core0'] BEGIN (implicit)
2016-02-26 16:01:38,735 INFO  [sqlalchemy.engine.base.Engine][b'uWSGIWorker5Core0'] SELECT * FROM staging WHERE company_name = %(company_name_1)s AND time = %(time_1)s AND ship_name = %(ship_name_1)s
# exact place (missing line) where SQLAlchemy is trying to print out
# query parameters, which in this case include unicode character
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.4/logging/__init__.py", line 980, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode character '\xfa' in position 132: ordinal not in range(128)
Call stack:
  File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqltap/wsgi.py", line 42, in __call__
    return self.app(environ, start_response)
  File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  #
  # the stack continues...
  # full stack here -> https://bpaste.net/show/8e12af790372
  #
  File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/mk/.virtualenvs/api/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1100, in _execute_context
    sql_util._repr_params(parameters, batches=10)
Unable to print the message and arguments - possible formatting error.
Use the traceback above to help find the error.

我也注意到将相同的内容写入由Pyramid生成的日志文件（不涉及uWSGI）中完全正常，没有任何错误，并且Unicode字符被正确插入。

我正在使用以下命令运行uWSGI：

/usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals

在我的 vassals 文件夹中，我已经通过符号链接将来自我的 Pyramid 应用程序的 uWSGI 配置连接起来，它看起来像这样：

[uwsgi]

host = %h
username = mk
project_name = api
project_root = /shared/projects/python/%(project_name)

env = PYTHONIOENCODING=UTF-8

; this env var is generated based on host name
env = APP_INI_FILE=develop.ini

; folders config
home_folder = /home/%(username)
virtualenv_folder = %(home_folder)/.virtualenvs/%(project_name)
logs_folder = %(home_folder)/logs/%(project_name)
chdir = %(project_root)
socket = %(project_root)/%(project_name).sock
pidfile = %(project_root)/%(project_name).pid
virtualenv = %(virtualenv_folder)
daemonize = %(logs_folder)/uwsgi.log

; core stuff
master = true
vacuum = true
processes = 5
enable-threads = true

; socket conf
chmod-socket = 666  # invoking the One
chown-socket = %(username)
uid = %(username)
gid = %(username)

; log conf
log-reopen = true
logfile-chown = %(username)
logfile-chmod = 644

; app conf
module = wsgi:application
harakiri = 120
max-requests = 500
post-buffering = 1
paste = config:%p
paste-logger = $p

Pyramid的配置文件定义了所有日志记录，其格式如下：

###
# app configuration
# http://docs.pylonsproject.org/projects/pyramid/en/1.5-branch/narr/environment.html
###

[DEFAULT]
home_dir = /home/mk

[app:main]
use = egg:api

pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en

sqlalchemy.url = postgresql://XXX:YYY@12.13.14.15:5432/ze_database?client_encoding=utf8

[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543

###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/1.5-branch/narr/logging.html
###

[loggers]
keys = root, sqlalchemy

[handlers]
keys = console, debuglog

[formatters]
keys = generic, short

[logger_root]
level = DEBUG
handlers = console, debuglog

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither.  (Recommended for production systems.)

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = DEBUG
formatter = generic

[handler_debuglog]
class = handlers.RotatingFileHandler
args = ('%(home_dir)s/logs/api/pyramid_debug.log', 'a', 1024000000, 10)
level = DEBUG
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

[formatter_short]
format = %(asctime)s %(message)s

最后，我的Pyramid的wsgi.py文件非常简单:

import os
from pyramid.paster import get_app, setup_logging

here = os.path.dirname(os.path.abspath(__file__))
conf = os.path.join(here, os.environ.get('APP_INI_FILE'))  # APP_INI_FILE variable is set in uwsgi.ini
setup_logging(conf)

application = get_app(conf, 'main')

基本上，我正在将应用程序的日志重定向到stderr（或stdout，据我所知，两者是相同的），同时还将其写入单独的文件（pyramid_debug.log）。在我的情况下，stderr是uWSGI守护进程的日志文件，这就是错误发生的地方。

尽管系统上设置了LC_ALL和相关变量为en_EN.UTF-8，但我也尝试玩转各种与本地化相关的环境变量，并在Pyramid的wsgi应用程序中显式设置它们，但运气不太好。例如，在uWSGI配置中仅设置PYTHONIOENCODING=UTF-8变量可以解决我本地机器上的问题，但在部署后的服务器上却没有解决。

我显而易见的问题是 - 如何在此情况下正确处理uWSGI编写日志文件中的Unicode字符？

- errata

Answer 1

编辑文件 "/usr/lib/python3.4/logging/__init__.py"，第980行并更改

stream.write(msg)

到

stream.write(msg.encode('utf-8'))

最有可能的是流类型被改变了，这种方式不应该影响您的UTF-8编码能力，但由于Pythonics的原因，实际上确实会影响。（似乎无论世界如何高喊“UTF-8”，Python都会忽视这个问题。）

例如，如果您正在处理文件，则忽略所有尝试的环境变量：

# test.py
import sys
sys.stdout.write(u'\u00f6\n')

测试：

max% python test.py
ö
max% python test.py > f
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    sys.stdout.write(u'\u00f6\n')
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 0: ordinal not in range(128)

- personal_cloud

---

这在我使用wsgi + flask时发生。我不确定我是否理解了情况。每个人在使用uwsgi时都会遇到日志记录的问题吗？ - m3nda

Answer 2

设置 LANG=C.UTF-8 对我有效。

- Tsingfengli

---

你的回答可以通过提供更多支持信息来改进。请编辑以添加进一步的细节，例如引用或文档，以便他人可以确认你的答案是正确的。您可以在帮助中心中找到有关如何编写良好答案的更多信息。 - Community

Answer 3

尝试设置您的环境变量PYTHONIOENCODING为utf-8。这将使文本文件的默认编码为UTF-8而不是ASCII。

- Mark Ransom