logging.handlers：如何按时间或maxBytes进行翻转？

Question

logging.handlers：如何按时间或maxBytes进行翻转？

pythonlogginghandlers

23

我对日志记录有些困惑。我希望在一定时间后和达到一定大小后滚动日志。

使用TimedRotatingFileHandler可以在一定时间后滚动日志，而使用RotatingFileHandler可以在达到一定日志大小后滚动日志。

但是TimedRotatingFileHandler没有maxBytes属性，RotatingFileHandler不能在一定时间后滚动日志。我尝试将两个处理程序添加到记录器中，但结果是日志记录翻倍。

我错过了什么吗？

我还查看了logging.handlers的源代码。我尝试子类化TimedRotatingFileHandler并覆盖shouldRollover()方法，以创建具有双重功能的类：

class EnhancedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
    def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None, delay=0, utc=0, maxBytes=0):
        """ This is just a combination of TimedRotatingFileHandler and RotatingFileHandler (adds maxBytes to TimedRotatingFileHandler)  """
        # super(self). #It's old style class, so super doesn't work.
        logging.handlers.TimedRotatingFileHandler.__init__(self, filename, when='h', interval=1, backupCount=0, encoding=None, delay=0, utc=0)
        self.maxBytes=maxBytes

    def shouldRollover(self, record):
        """
        Determine if rollover should occur.

        Basically, see if the supplied record would cause the file to exceed
        the size limit we have.

        we are also comparing times        
        """
        if self.stream is None:                 # delay was set...
            self.stream = self._open()
        if self.maxBytes > 0:                   # are we rolling over?
            msg = "%s\n" % self.format(record)
            self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
            if self.stream.tell() + len(msg) >= self.maxBytes:
                return 1
        t = int(time.time())
        if t >= self.rolloverAt:
            return 1
        #print "No need to rollover: %d, %d" % (t, self.rolloverAt)
        return 0

但是这样日志会创建一个备份并被覆盖。似乎我也必须覆盖doRollover()方法，这不太容易。

还有其他的想法如何创建一个在一定时间后滚动文件并在达到一定大小后滚动文件的日志记录器吗？

- sumid

显然，除了深入研究库代码并加以解决外，没有直接的答案。幸运的是，标准库通常编写得很好且符合惯用法，因此如果必须这样做，您将会学到很多。但是，我会问任何向我提出这个问题的人：为什么您需要时间或大小的函数？您想要实现什么目标？使用情况是什么？由于“logging”非常全面，您的需求可能被认为是“非典型的”，或者该功能已经存在。 - msw

我想添加一个用例。在我的应用程序中，我希望以按天结构化的方式保留日志。我使用了Java的logback模块来保持日志的清洁。策略：基于大小的100 MB轮换和按天在单独的目录中维护。最好的部分是，它还提供了日志的归档/压缩。仍在寻找类似的Python日志记录库。 - Rohit Lal

4个回答

7

如果你真的需要这个功能，请基于TimedRotatingFileHandler编写自己的处理程序，以主要使用时间进行翻转，但将基于大小的翻转纳入现有逻辑。你已经尝试过了，但至少需要重写shouldRollover()和doRollover()方法。第一个方法确定何时翻转，第二个方法关闭当前日志文件，重命名现有文件并删除过时文件，然后打开新文件。 doRollover()逻辑可能有点棘手，但肯定可行。

- Vinay Sajip

3

以下是我使用的内容：

import logging

class  EnhancedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler, logging.handlers.RotatingFileHandler):
    '''
        cf https://dev59.com/rYnda4cB1Zd3GeqPBqCv

         Spec:
         Log files limited in size & date. I.E. when the size or date is overtaken, there is a file rollover
     '''

    ########################################


    def __init__(self, filename, mode = 'a', maxBytes = 0, backupCount = 0, encoding = None,
             delay = 0, when = 'h', interval = 1, utc = False):

        logging.handlers.TimedRotatingFileHandler.__init__(
        self, filename, when, interval, backupCount, encoding, delay, utc)

         logging.handlers.RotatingFileHandler.__init__(self, filename, mode, maxBytes, backupCount, encoding, delay)

     ########################################

     def computeRollover(self, currentTime):
         return logging.handlers.TimedRotatingFileHandler.computeRollover(self, currentTime)

    ########################################

    def getFilesToDelete(self):
        return logging.handlers.TimedRotatingFileHandler.getFilesToDelete(self)

    ########################################

    def doRollover(self):
        return logging.handlers.TimedRotatingFileHandler.doRollover(self)

    ########################################

    def shouldRollover(self, record):
         """ Determine if rollover should occur. """
         return (logging.handlers.TimedRotatingFileHandler.shouldRollover(self, record) or logging.handlers.RotatingFileHandler.shouldRollover(self, record))

- Julien Greard

这在你的生产环境中实际可行吗？在一个快速测试中，我使用range(1000)、maxBytes=1000、when='m'、interval=1记录日志，看起来它似乎缺少了几个文件/日志行块。 - Cody A. Ray

是的，我相信它可以工作，我刚刚检查过了。实际上，在我的单元测试中，我曾经遇到过与你类似的问题：这是因为日志文件的扩展名默认情况下没有毫秒数，当发生翻转时，文件模式与上一个文件相同并覆盖它。一种解决方法是更改扩展名（请参见https://dev59.com/I4Hba4cB1Zd3GeqPQFtQ）。在实践中，我将文件大小设置为约1Mo，并且我从未在1秒内拥有超过1Mo的日志。 - Julien Greard

1

啊好的，这很有道理。我刚刚编辑/更新了上面的答案，它可以与其他大小/时间间隔的组合一起使用（因为它也手动指定了基于文件大小的文件扩展名）。谢谢！继续 Stack Overflow 的协作吧！ :) - Cody A. Ray

1

我根据Julien的代码进行了修改以适应我的需求。现在它会在达到一定日志大小或一段时间后翻转。

class EnhancedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler, logging.handlers.RotatingFileHandler):

def __init__(self, filename, mode='a', maxBytes=0, backupCount=0, encoding=None,
             delay=0, when='h', interval=1, utc=False):
    logging.handlers.TimedRotatingFileHandler.__init__(
        self, filename=filename, when=when, interval=interval,
        backupCount=backupCount, encoding=encoding, delay=delay, utc=utc)

    logging.handlers.RotatingFileHandler.__init__(self, filename=filename, mode=mode, maxBytes=maxBytes,
                                                  backupCount=backupCount, encoding=encoding, delay=delay)

def computeRollover(self, current_time):
    return logging.handlers.TimedRotatingFileHandler.computeRollover(self, current_time)

def doRollover(self):
    # get from logging.handlers.TimedRotatingFileHandler.doRollover()
    current_time = int(time.time())
    dst_now = time.localtime(current_time)[-1]
    new_rollover_at = self.computeRollover(current_time)

    while new_rollover_at <= current_time:
        new_rollover_at = new_rollover_at + self.interval

    # If DST changes and midnight or weekly rollover, adjust for this.
    if (self.when == 'MIDNIGHT' or self.when.startswith('W')) and not self.utc:
        dst_at_rollover = time.localtime(new_rollover_at)[-1]
        if dst_now != dst_at_rollover:
            if not dst_now:  # DST kicks in before next rollover, so we need to deduct an hour
                addend = -3600
            else:  # DST bows out before next rollover, so we need to add an hour
                addend = 3600
            new_rollover_at += addend
    self.rolloverAt = new_rollover_at

    return logging.handlers.RotatingFileHandler.doRollover(self)

def shouldRollover(self, record):
    return logging.handlers.TimedRotatingFileHandler.shouldRollover(self, record) or logging.handlers.RotatingFileHandler.shouldRollover(self, record)

- Akinkin Denis

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- sumid · Accepted Answer

因此，我对TimedRotatingFileHandler进行了小小的修改，使其能够在时间和大小两个标准下进行滚动。我必须修改__init__、shouldRollover、doRollover和getFilesToDelete函数（见下文）。当我设置when='M'、interval=2、backupCount=20、maxBytes=1048576时，得到的结果如下：

-rw-r--r-- 1 user group  185164 Jun 10 00:54 sumid.log
-rw-r--r-- 1 user group 1048462 Jun 10 00:48 sumid.log.2011-06-10_00-48.001    
-rw-r--r-- 1 user group 1048464 Jun 10 00:48 sumid.log.2011-06-10_00-48.002    
-rw-r--r-- 1 user group 1048533 Jun 10 00:49 sumid.log.2011-06-10_00-48.003    
-rw-r--r-- 1 user group 1048544 Jun 10 00:50 sumid.log.2011-06-10_00-49.001    
-rw-r--r-- 1 user group  574362 Jun 10 00:52 sumid.log.2011-06-10_00-50.001

您可以看到前四个日志文件在达到1MB大小后被滚动，而最后一个滚动是在两分钟后发生的。到目前为止，我还没有测试过删除旧日志文件，所以它可能不会起作用。

如果backupCount>=1000，则代码肯定不会工作。我只在文件名末尾附加了三位数字。

这是修改过的代码：

class EnhancedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
    def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None, delay=0, utc=0, maxBytes=0):
        """ This is just a combination of TimedRotatingFileHandler and RotatingFileHandler (adds maxBytes to TimedRotatingFileHandler)  """
        logging.handlers.TimedRotatingFileHandler.__init__(self, filename, when, interval, backupCount, encoding, delay, utc)
        self.maxBytes=maxBytes

    def shouldRollover(self, record):
        """
        Determine if rollover should occur.

        Basically, see if the supplied record would cause the file to exceed
        the size limit we have.

        we are also comparing times        
        """
        if self.stream is None:                 # delay was set...
            self.stream = self._open()
        if self.maxBytes > 0:                   # are we rolling over?
            msg = "%s\n" % self.format(record)
            self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
            if self.stream.tell() + len(msg) >= self.maxBytes:
                return 1
        t = int(time.time())
        if t >= self.rolloverAt:
            return 1
        #print "No need to rollover: %d, %d" % (t, self.rolloverAt)
        return 0         

    def doRollover(self):
        """
        do a rollover; in this case, a date/time stamp is appended to the filename
        when the rollover happens.  However, you want the file to be named for the
        start of the interval, not the current time.  If there is a backup count,
        then we have to get a list of matching filenames, sort them and remove
        the one with the oldest suffix.
        """
        if self.stream:
            self.stream.close()
        # get the time that this sequence started at and make it a TimeTuple
        currentTime = int(time.time())
        dstNow = time.localtime(currentTime)[-1]
        t = self.rolloverAt - self.interval
        if self.utc:
            timeTuple = time.gmtime(t)
        else:
            timeTuple = time.localtime(t)
            dstThen = timeTuple[-1]
            if dstNow != dstThen:
                if dstNow:
                    addend = 3600
                else:
                    addend = -3600
                timeTuple = time.localtime(t + addend)
        dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple)
        if self.backupCount > 0:
            cnt=1
            dfn2="%s.%03d"%(dfn,cnt)
            while os.path.exists(dfn2):
                dfn2="%s.%03d"%(dfn,cnt)
                cnt+=1                
            os.rename(self.baseFilename, dfn2)
            for s in self.getFilesToDelete():
                os.remove(s)
        else:
            if os.path.exists(dfn):
                os.remove(dfn)
            os.rename(self.baseFilename, dfn)
        #print "%s -> %s" % (self.baseFilename, dfn)
        self.mode = 'w'
        self.stream = self._open()
        newRolloverAt = self.computeRollover(currentTime)
        while newRolloverAt <= currentTime:
            newRolloverAt = newRolloverAt + self.interval
        #If DST changes and midnight or weekly rollover, adjust for this.
        if (self.when == 'MIDNIGHT' or self.when.startswith('W')) and not self.utc:
            dstAtRollover = time.localtime(newRolloverAt)[-1]
            if dstNow != dstAtRollover:
                if not dstNow:  # DST kicks in before next rollover, so we need to deduct an hour
                    addend = -3600
                else:           # DST bows out before next rollover, so we need to add an hour
                    addend = 3600
                newRolloverAt += addend
        self.rolloverAt = newRolloverAt

    def getFilesToDelete(self):
        """
        Determine the files to delete when rolling over.

        More specific than the earlier method, which just used glob.glob().
        """
        dirName, baseName = os.path.split(self.baseFilename)
        fileNames = os.listdir(dirName)
        result = []
        prefix = baseName + "."
        plen = len(prefix)
        for fileName in fileNames:
            if fileName[:plen] == prefix:
                suffix = fileName[plen:-4]
                if self.extMatch.match(suffix):
                    result.append(os.path.join(dirName, fileName))
        result.sort()
        if len(result) < self.backupCount:
            result = []
        else:
            result = result[:len(result) - self.backupCount]
        return result