午夜时分,cronjob会启动logrotate去轮换4个关键的日志文件。这4个日志文件同时也被rsyslog发送到了一个日志聚合服务器。尽管在每次轮换日志后运行rsyslog logrotate命令,有时,服务器会遇到“Permission Denied”错误,从而阻止rsyslog发送属于新文件的日志。我无法弄清为什么会这样不可靠。
以下是rsyslog报告的错误。请注意,它们只会在某些日期出现:
Jul 31 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 01 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 02 00:00:04 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/nginx.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 09 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 10 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/django.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 10 00:00:01 hostname rsyslogd[26343]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]
Aug 16 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 16 00:00:02 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/django.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 19 00:00:03 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]
Aug 20 00:00:06 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/app.log': Permission denied [v8.32.0]
Aug 30 00:00:01 hostname rsyslogd[856]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Sep 03 00:00:01 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/celery.log': Permission denied [v8.32.0]
Sep 03 00:00:04 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]
Logrotate会由cron工作计划触发,使用root用户。以下是logrotate配置:
/home/ubuntu/remote_logs/*.log {
rotate 365
daily
compress
missingok
notifempty
dateext
dateformat .%Y-%m-%d
dateyesterday
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
/usr/lib/rsyslog/rsyslog-rotate
#!/bin/sh
if [ -d /run/systemd/system ]; then
systemctl kill -s HUP rsyslog.service
else
invoke-rc.d rsyslog rotate > /dev/null
fi
我知道logrotate本身是成功的,因为logrotate详细日志表明了这一点,而且我每天都能看到旋转后的日志文件。似乎rsyslog无法访问新创建的文件。
以下是logrotate的日志:
Handling 13 logs
rotating pattern: /home/ubuntu/remote_logs/*.log after 1 days (365 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 106
considering log /home/ubuntu/remote_logs/celery.log
Now: 2020-10-14 00:00
Last rotated at 2020-10-13 00:00
log needs rotating
considering log /home/ubuntu/remote_logs/django.log
Now: 2020-10-14 00:00
Last rotated at 2020-10-13 00:00
log needs rotating
considering log /home/ubuntu/remote_logs/app.log
Now: 2020-10-14 00:00
Last rotated at 2020-10-13 00:00
log needs rotating
considering log /home/ubuntu/remote_logs/nginx.log
Now: 2020-10-14 00:00
Last rotated at 2020-10-13 00:00
log needs rotating
rotating log /home/ubuntu/remote_logs/celery.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/celery.log to /home/ubuntu/remote_logs/celery.log.2020-10-13
creating new /home/ubuntu/remote_logs/celery.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/django.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/django.log to /home/ubuntu/remote_logs/django.log.2020-10-13
creating new /home/ubuntu/remote_logs/django.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/app.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/app.log to /home/ubuntu/remote_logs/app.log.2020-10-13
creating new /home/ubuntu/remote_logs/app.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/nginx.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/nginx.log to /home/ubuntu/remote_logs/nginx.log.2020-10-13
creating new /home/ubuntu/remote_logs/nginx.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
switching euid to 0 and egid to 0
rsyslog 有时无法访问日志文件的原因是什么?