使用可变数量的上下文管理器的contextlib.nested的替代方法

Question

我们有一些代码，根据运行时参数调用不同数量的上下文管理器：

from contextlib import nested, contextmanager

@contextmanager
def my_context(arg):
    print("entering", arg)
    try:
        yield arg
    finally:
        print("exiting", arg)

def my_fn(items): 
    with nested(*(my_context(arg) for arg in items)) as managers:
        print("processing under", managers)

my_fn(range(3))

然而，contextlib.nested自Python 2.7起已被弃用:

DeprecationWarning: With-statements now directly support multiple context managers

答案Python中的多个变量与语句表明，contextlib.nested存在一些“令人困惑且容易出错的怪癖”，但是使用多个管理器的多个with语句的建议替代方法无法适用于变量数量不定的上下文管理器(并且还破坏了向后兼容性)。

是否有任何不被弃用且(最好)没有相同缺陷的contextlib.nested替代方案？

还是应该继续使用contextlib.nested并忽略警告？如果是这样，我应该计划在将来删除contextlib.nested吗？

- ecatmur

---

你提出了一个关于管理者数量的好问题...在我看来，这个问题应该在邮件列表中被提出... - mgilson

这就是为什么Python 3引入了contextlib.ExitStack。请参见http://bugs.python.org/issue13585 - Martijn Pieters

Answer 1

新的Python 3 contextlib.ExitStack类作为contextlib.nested()的替代品被添加（请参见issue 13585）。

它的编码方式使您可以直接在Python 2中使用它：

import sys
from collections import deque


class ExitStack(object):
    """Context manager for dynamic management of a stack of exit callbacks

    For example:

        with ExitStack() as stack:
            files = [stack.enter_context(open(fname)) for fname in filenames]
            # All opened files will automatically be closed at the end of
            # the with statement, even if attempts to open files later
            # in the list raise an exception

    """
    def __init__(self):
        self._exit_callbacks = deque()

    def pop_all(self):
        """Preserve the context stack by transferring it to a new instance"""
        new_stack = type(self)()
        new_stack._exit_callbacks = self._exit_callbacks
        self._exit_callbacks = deque()
        return new_stack

    def _push_cm_exit(self, cm, cm_exit):
        """Helper to correctly register callbacks to __exit__ methods"""
        def _exit_wrapper(*exc_details):
            return cm_exit(cm, *exc_details)
        _exit_wrapper.__self__ = cm
        self.push(_exit_wrapper)

    def push(self, exit):
        """Registers a callback with the standard __exit__ method signature

        Can suppress exceptions the same way __exit__ methods can.

        Also accepts any object with an __exit__ method (registering a call
        to the method instead of the object itself)
        """
        # We use an unbound method rather than a bound method to follow
        # the standard lookup behaviour for special methods
        _cb_type = type(exit)
        try:
            exit_method = _cb_type.__exit__
        except AttributeError:
            # Not a context manager, so assume its a callable
            self._exit_callbacks.append(exit)
        else:
            self._push_cm_exit(exit, exit_method)
        return exit # Allow use as a decorator

    def callback(self, callback, *args, **kwds):
        """Registers an arbitrary callback and arguments.

        Cannot suppress exceptions.
        """
        def _exit_wrapper(exc_type, exc, tb):
            callback(*args, **kwds)
        # We changed the signature, so using @wraps is not appropriate, but
        # setting __wrapped__ may still help with introspection
        _exit_wrapper.__wrapped__ = callback
        self.push(_exit_wrapper)
        return callback # Allow use as a decorator

    def enter_context(self, cm):
        """Enters the supplied context manager

        If successful, also pushes its __exit__ method as a callback and
        returns the result of the __enter__ method.
        """
        # We look up the special methods on the type to match the with statement
        _cm_type = type(cm)
        _exit = _cm_type.__exit__
        result = _cm_type.__enter__(cm)
        self._push_cm_exit(cm, _exit)
        return result

    def close(self):
        """Immediately unwind the context stack"""
        self.__exit__(None, None, None)

    def __enter__(self):
        return self

    def __exit__(self, *exc_details):
        # We manipulate the exception state so it behaves as though
        # we were actually nesting multiple with statements
        frame_exc = sys.exc_info()[1]
        def _fix_exception_context(new_exc, old_exc):
            while 1:
                exc_context = new_exc.__context__
                if exc_context in (None, frame_exc):
                    break
                new_exc = exc_context
            new_exc.__context__ = old_exc

        # Callbacks are invoked in LIFO order to match the behaviour of
        # nested context managers
        suppressed_exc = False
        while self._exit_callbacks:
            cb = self._exit_callbacks.pop()
            try:
                if cb(*exc_details):
                    suppressed_exc = True
                    exc_details = (None, None, None)
            except:
                new_exc_details = sys.exc_info()
                # simulate the stack of exceptions by setting the context
                _fix_exception_context(new_exc_details[1], exc_details[1])
                if not self._exit_callbacks:
                    raise
                exc_details = new_exc_details
        return suppressed_exc

使用此作为上下文管理器，然后随意添加嵌套的上下文管理器：

with ExitStack() as stack:
    managers = [stack.enter_context(my_context(arg)) for arg in items]
    print("processing under", managers)

对于您的示例上下文管理器，这将打印：

>>> my_fn(range(3))
('entering', 0)
('entering', 1)
('entering', 2)
('processing under', [0, 1, 2])
('exiting', 2)
('exiting', 1)
('exiting', 0)

您也可以安装contextlib2模块；它包含了ExitStack的后端。

- Martijn Pieters

---

这个解决方案仅适用于上下文管理器在组装它们的同一上下文中“进入”的情况。例如，在此代码中，nested的结果将返回给调用者以供稍后输入。我不认为在Python 3中有实现这一点的方法。 - Jason R. Coombs

如果您想要推迟输入，那么编写一个子类，该子类接受一个队列并在 __enter__() 上调用 .enter_context()（并在发生异常时正确处理退出）并不太难。 - Martijn Pieters

你需要注意的一件事是，创建 CM 也可能会引发异常；因此，将可调用对象传递给该队列可能是最好的方法，推迟创建 CM 直到调用 __enter__。 - Martijn Pieters

好建议。我正在考虑子类化ExitStack以延迟进入上下文，但我不确定是否太过hacky。既然你推荐它，我更有信心尝试一下。谢谢！ - Jason R. Coombs

我想知道为什么他们没有在Python 2.7的最近版本中添加这个功能，因为nested已经被弃用，而ExitStack的代码可以直接在Python 2.7中使用。您认为只要我想支持Python 2.7，将其复制到我的项目中是最好的选择吗？ - Lev Levitsky

Answer 2

有一点令人烦恼的是，Python3 的维护者选择了破坏向后兼容性，因为使用 ExitStack 来实现嵌套式功能非常简单：

try:
    from contextlib import nested  # Python 2
except ImportError:
    from contextlib import ExitStack, contextmanager

    @contextmanager
    def nested(*contexts):
        """
        Reimplementation of nested in python 3.
        """
        with ExitStack() as stack:
            for ctx in contexts:
                stack.enter_context(ctx)
            yield contexts

- Lucas Wiman

---

这个实现与原始的 nested 存在相同的问题。上下文管理器已经在调用 nested 函数时创建。如果索引大于 0 的上下文管理器的构造函数抛出异常，则前面的上下文管理器将永远不会退出。 - Timothy Shields

是的，它确实存在这个“问题”。但这仅仅是对于在构造时获取资源（如open）的上下文而言是个问题。有许多上下文并不是这样运作的，而nested允许您返回一个单一的返回值，该值可用作所有子上下文的上下文。 - Lucas Wiman

Answer 3

import sys
import contextlib

class nodeA(object):

    def __init__(self):
        print( '__init__ nodeA')

    def __enter__(self):
        print( '__enter__ nodeA')

    def __exit__(self, a, b, c):
        print( '__exit__ nodeA')

class nodeB(object):

    def __init__(self):
        print( '__init__ nodeB')

    def __enter__(self):
        print( '__enter__ nodeB')

    def __exit__(self, a, b, c):
        print( '__exit__ nodeB')

class nodeC(object):

    def __init__(self):
        print( '__init__ nodeC')

    def __enter__(self):
        print( '__enter__ nodeC')

    def __exit__(self, a, b, c):
        print( '__exit__ nodeC')

print( 'Start...')

a = nodeA()
b = nodeB()
c = nodeC()

print( 'Python version: %s' % (sys.version))

if sys.version.startswith('2'):
    print('Use python 2!')
    with contextlib.nested(a, b, c):
        print('hallo?')

if sys.version.startswith('3'):
    print('Use python 3!')
    with contextlib.ExitStack() as stack:
        [stack.enter_context(arg) for arg in [a,b,c]]

print('...end!')

- Joost