我曾经遇到过同样的问题。
我们在Windows环境下经常使用 subprocess.Popen() 来调用外部工具。某个时刻，我们遇到了没有更多文件描述符可用的问题。我们深入研究了这个问题，并发现 subprocess.Popen 实例在 Windows 和 Linux 中的行为不同。
如果 Popen 实例没有被销毁（例如通过某种方式保持引用，从而不允许垃圾回收器销毁该对象），在 Windows 中创建的管道将保持打开状态，而在 Linux 中，在调用 Popen.communicate() 后，它们会自动关闭。如果继续进行进一步调用，管道中的“僵尸”文件描述符将累积，并最终导致 Python 异常 IOError: [Errno 24] Too many open files。
如何在 Python 中获取已打开的文件描述符
为了排除故障，我们需要一种获取 Python 脚本中有效文件描述符的方法。因此，我们编写了以下脚本。请注意，我们仅检查0到100之间的文件描述符，因为我们不会同时打开那么多文件。
fd_table_status.py：
import os
import stat
_fd_types = (
('REG', stat.S_ISREG),
('FIFO', stat.S_ISFIFO),
('DIR', stat.S_ISDIR),
('CHR', stat.S_ISCHR),
('BLK', stat.S_ISBLK),
('LNK', stat.S_ISLNK),
('SOCK', stat.S_ISSOCK)
)
def fd_table_status():
result = []
for fd in range(100):
try:
s = os.fstat(fd)
except:
continue
for fd_type, func in _fd_types:
if func(s.st_mode):
break
else:
fd_type = str(s.st_mode)
result.append((fd, fd_type))
return result
def fd_table_status_logify(fd_table_result):
return ('Open file handles: ' +
', '.join(['{0}: {1}'.format(*i) for i in fd_table_result]))
def fd_table_status_str():
return fd_table_status_logify(fd_table_status())
if __name__=='__main__':
print fd_table_status_str()
仅运行时，它将显示所有打开的文件描述符及其相应的类型:
$> python fd_table_status.py
Open file handles: 0: CHR, 1: CHR, 2: CHR
$>
通过Python代码调用fd_table_status_str()会得到相同的输出结果。有关“CHR”和尊重“短码”含义的详细信息，请参见《stat》的
Python文档。
测试文件描述符行为
在Linux和Windows中尝试运行以下脚本：
test_fd_handling.py :
import fd_table_status
import subprocess
import platform
fds = fd_table_status.fd_table_status_str
if platform.system()=='Windows':
python_exe = r'C:\Python27\python.exe'
else:
python_exe = 'python'
print '1) Initial file descriptors:\n' + fds()
f = open('fd_table_status.py', 'r')
print '2) After file open, before Popen:\n' + fds()
p = subprocess.Popen(['python', 'fd_table_status.py'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
print '3) After Popen, before reading piped output:\n' + fds()
result = p.communicate()
print '4) After Popen.communicate():\n' + fds()
del p
print '5) After deleting reference to Popen instance:\n' + fds()
del f
print '6) After deleting reference to file instance:\n' + fds()
print '7) child process had the following file descriptors:'
print result[0][:-1]
Linux 输出
1) Initial file descriptors:
Open file handles: 0: CHR, 1: CHR, 2: CHR
2) After file open, before Popen:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
3) After Popen, before reading piped output:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG, 5: FIFO, 6: FIFO, 8: FIFO
4) After Popen.communicate():
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
5) After deleting reference to Popen instance:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
6) After deleting reference to file instance:
Open file handles: 0: CHR, 1: CHR, 2: CHR
7) child process had the following file descriptors:
Open file handles: 0: FIFO, 1: FIFO, 2: FIFO, 3: REG
Windows输出
1) Initial file descriptors:
Open file handles: 0: CHR, 1: CHR, 2: CHR
2) After file open, before Popen:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
3) After Popen, before reading piped output:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG, 4: FIFO, 5: FIFO, 6: FIFO
4) After Popen.communicate():
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG, 5: FIFO, 6: FIFO
5) After deleting reference to Popen instance:
Open file handles: 0: CHR, 1: CHR, 2: CHR, 3: REG
6) After deleting reference to file instance:
Open file handles: 0: CHR, 1: CHR, 2: CHR
7) child process had the following file descriptors:
Open file handles: 0: FIFO, 1: FIFO, 2: FIFO
从第4步可以看出，Windows的行为与Linux不同。必须销毁Popen实例才能关闭管道。
顺便提一下，在第7步中的差异显示了有关Python解释器在Windows上的行为的不同问题，您可以在此处查看更多详细信息。
