Python在Windows中的缓冲输出和非缓冲输出的区别

Question

Python在Windows中的缓冲输出和非缓冲输出的区别

3

我使用NppExec/Notepad++运行Python脚本，在运行时连续刷新输出缓冲区，以便使用print语句更新控制台窗口（默认缓冲输出只在脚本执行完成后显示所有打印语句）。这个链接显示你只需要使用命令python -u就可以获得无缓冲输出。无论使用哪种编辑器，是否对所有我的Python脚本使用此执行模式都有不利影响？我不清楚缓冲和无缓冲输出之间的区别。编辑：我包含了这个小Python计时器脚本作为示例：

#!usr/bin/env python
import time
import threading
import sys

class Timer(threading.Thread):
    def __init__(self, seconds):
        self.runTime = seconds
        threading.Thread.__init__(self)
    def run(self):
        counter = self.runTime
        for sec in range(self.runTime):
            print counter
            time.sleep(1.0)
            counter -= 1
        print "Done."

if __name__ == '__main__':
    t = Timer(10)
    t.start()

在这种情况下，缓冲输出和非缓冲输出在效率方面会有多大差异？

- prrao

2个回答

5

我可以想到两个缺点，但它们的重要性取决于你的需求：

Unbuffered reading and writing may be significantly slower; if you're writing a text file one line at a time, your code may make hundreds more system calls to ask the OS to write the file. Depending upon the speed in which you're writing to disk, this may even mean that the last block of the file needs to be re-read from disk in order to re-save the file with a new last line. (This might be rare; but more system calls are almost always a recipe for going slower.)

Here's a simple demonstration of the number of system calls having a large influence on write speed:
```
$ cat initrd.img-2.6.38-8-generic > /dev/null
```
The first line makes sure the file is in cache, so only output speed is being measured.
```
$ dd if=initrd.img-2.6.38-8-generic of=/tmp/out bs=16 oflag=dsync
^C262766+0 records in
262766+0 records out
4204256 bytes (4.2 MB) copied, 50.7754 s, 82.8 kB/s
```
I gave up waiting for this -- it's going way too slow. This is "unbuffered" writing 16 bytes at a time to disk, and making sure each write succeeded before moving on to the next. (That's the dsync -- more on that later.)
```
$ dd if=initrd.img-2.6.38-8-generic of=/tmp/out bs=$((4096)) oflag=dsync
3218+1 records in
3218+1 records out
13181130 bytes (13 MB) copied, 3.69961 s, 3.6 MB/s
$ dd if=initrd.img-2.6.38-8-generic of=/tmp/out bs=$((4096 * 10)) oflag=dsync
321+1 records in
321+1 records out
13181130 bytes (13 MB) copied, 0.471143 s, 28.0 MB/s
```
These two commands show the effect of some buffering -- the first one writes data in 4096-byte chunks, which might be what the default buffering gives you. This is roughly fifty times faster than 16 bytes at a time. The second command is writing data in 40960 byte chunks, and it went roughly nine times faster still. (All told, it's roughly 345 times faster to write 40960 bytes at a time than 16 bytes at a time.)

If your data is small, this won't really matter. After all, it won't take much time either way. If your data is large, it might matter more, depending upon how much data you're writing at once and how often it lines up on happy "byte boundaries" for the underlying devices.
Some protocols on sockets may change their behavior based on the timing of the data you're sending. If you're building data incrementally, you may send partial data in a single packet and a packet-based receiver may not handle this gracefully. (It's harder for me to imagine a TCP-based system with this problem other than driving a game of some sort; a UDP-based system is far easier to imagine having a problem with this.)

- sarnold

你的例子有点超出了我的理解范围！能否看一下我的问题编辑，并按照那个来解释一下呢？感谢你详细的回复！ - prrao

我不是很清楚“数据是小还是大”的意思。这是否意味着我要打印出行数？我通过控制台在本地运行所有的Python文件。 - prrao

好的，我想我的问题已经得到解答了。谢谢！ - prrao

哈哈，再次证明了请求更多上下文的重要性。你的代码非常简短且功能很少，因此性能并不是真正重要的问题——但正确性却是。你可以通过在“print”语句后立即添加刷新命令来使程序正确运行，但说实话，它将完全相同地工作。只需使用“-u”选项就好了，然后开心地用吧。 :) - sarnold

我得出的结论是，如果我的代码运行时间比打印几行更新信息通知用户运行进度所需的时间要长得多，那么我可以放心地使用 python -u，不必担心任何问题。希望我理解正确。 - prrao

这更取决于你的代码要做的事情的种类；如果它正在进行大量的磁盘IO，你可能希望在用户状态消息中添加显式刷新操作。如果它完全受计算限制或者最好是空闲的，那么全不缓存就可以了。 - sarnold

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- mgilson · Accepted Answer

缓冲输出意味着计算机将输出暂存到内存中，直到累积了一定量的输出后，才一次性写入整个块。这比使用非缓冲输出更有效率，后者会在请求写入时立即写入输出。

缺点是，程序运行速度会变慢，具体取决于你要输出多少内容。如果它们是短小的程序，不会产生太多输出，你可能不会注意到任何区别...

编辑

缓冲和非缓冲输出不仅适用于Python，其他语言也适用相同的概念（和术语）。在低级语言中，这在某些方面变得更加重要 - 如果我使用缓冲输出在C程序中编写消息，然后由于编程错误导致程序崩溃，那么在错误之前暂存但未写入磁盘的任何数据都会丢失。这不是一个大问题，因为很难让Python解释器在发生错误时中止 - 即使你的代码很糟糕，解释器最终仍会清理干净...（除非你发送一个kill信号给它或类似的操作）...