Rsyslog open error permission denied

CentOS The Community ENTerprise Operating System rsyslog permission denied rsyslog permission denied Post by johnhk215 » 2020/03/23 13:28:13 syslog service can be started with error rsyslog rsyslogd[2253]: imjournal: fopen() failed for path: ‘/var/lib/rsyslog/imjournal.state.tmp’: Permission denied # # Set the default permissions for all log files. # $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 # These […]

Содержание

  1. CentOS
  2. rsyslog permission denied
  3. rsyslog permission denied
  4. Re: rsyslog permission denied
  5. Re: rsyslog permission denied
  6. Re: rsyslog permission denied
  7. Re: rsyslog permission denied
  8. rsyslog error 2433
  9. 3 thoughts on “ rsyslog error 2433 ”
  10. rsyslogd: file ‘/dev/console’: open error: Permission denied
  11. Bug Description
  12. imfile fails to send logs remoteafter logrotate #2929
  13. Comments
  14. Expected behavior
  15. Actual behavior
  16. Steps to reproduce the behavior
  17. Environment
  18. Forwarding log files with file names when we use wildcards #474
  19. Comments

CentOS

The Community ENTerprise Operating System

rsyslog permission denied

rsyslog permission denied

Post by johnhk215 » 2020/03/23 13:28:13

syslog service can be started with error
rsyslog rsyslogd[2253]: imjournal: fopen() failed for path: ‘/var/lib/rsyslog/imjournal.state.tmp’: Permission denied

#
# Set the default permissions for all log files.
#
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# These may require root
$FileOwner syslog
$FileGroup syslog
$PrivDropToUser syslog
$PrivDropToGroup syslog

if #$PrivDropToUser syslog -> no permission denied but syslog user doesn’t have /opt/rsyslog

Re: rsyslog permission denied

Post by TrevorH » 2020/03/23 14:47:44

Re: rsyslog permission denied

Post by johnhk215 » 2020/03/24 01:59:21

Yes, It is central logging server and would like to put the data in to /opt/data instead of default path /var/log.
Can I ignore the permission denied message?

Re: rsyslog permission denied

Post by pjsr2 » 2020/03/24 10:11:09

You cannot simply ignore the message.

There is a whole bunch of selinux context settings that apply to /var/log and the log files and subdirectories in it, written by various applications. If you move your log files from /var/log to /opt/log you need to get these selinux fcontext sorted out, which is a whole lot of work.

You should expect logging problems similar to the one you already observed from other applications, and they may not show up immediately as it may take time before applications feel the need to log something. Some applications don’t care when they can’t log. Other applications fail when they can’t log. If applications cannot log, that may have security implications, as you may loose important warning signals.

Re: rsyslog permission denied

Post by TrevorH » 2020/03/24 10:15:22

If you moved it to give more space then I would suggest mounting that filesystem on /var/log to begin with (though that’s more complicated than just that as you will need to copy the existing content over first and probably relabel the filesystem afterwards once it’s mounted on /var/log).

As pjsr2 says, ther are numerous selinux rules set up to allow access and you would need to duplicate or set up equivalence rules for those

Источник

rsyslog error 2433

Error other than “not found” occurred during open(): OS ERROR MESSAGE

There was a problem opening a file. The “OS ERROR MESSAGE” is taken from the OS and tells what exactly caused the issue.

A common cause is “Permission Denied” which means the rsyslog process had insufficient permissions to open the file. This can be caused by several things, with the most common being

  • access permissions to restrictive
  • SELinux policies (or similar systems, like AppArmor)

Note that in the case of SELinux and friends running as root does not solve the permission issue, because it is a design goal of these tools to be able to restrict permissions even for privileged processes.

Additional help is also available in “solving rsyslog write errors“.

3 thoughts on “ rsyslog error 2433 ”

Jan 2 12:03:26 localhost rsyslogd: file ’/mss/syslog/fgt-01.log’: open error: Permission denied [v8.24.0-34.el7 try http://www.rsyslog.com/e/2433 ]

config file:
$umask 0000
# Log remote hosts to separate log file
$fileCreateMode 0640
$fileOwner root
$fileGroup netsec
$template PerHostLog,”/mss/syslog/%HOSTNAME%.log”
$template RemoteHostFileFormat,”%PRI%1 %TIMESTAMP. date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n”
:inputname, isequal, ”imudp” ?PerHostLog;RemoteHostFileFormat
#:inputname, isequal, ”imtcp” ?PerHostLog;RemoteHostFileFormat
&

basically this error message is useless and no chown, chmod, etc has resolved this. This error message needs to be more detailed to be useful.

Well, assuming you comment on the initial sample: “Permission denied” is the actual error as it comes from the OS (this is where we obtain it). So you need to fix permissions. Note that in 2019’s Linux this may also mean things like SELinux. What exactly is the cause of the permission issue is not said by the OS, so rsyslog cannot report it.

Источник

rsyslogd: file ‘/dev/console’: open error: Permission denied

Affects Status Importance Assigned to Milestone
rsyslog (Ubuntu)

Bug Description

At the moment rsyslog cannot have access /dev/console due to a mismatch permission/ ownership between ‘/dev/console’ and the Privilege Drop User and Group ‘syslog’ in rsyslog.

* Deploy focal/20.04LTS (tested in gcloud instance)
* Install rsyslog
* systemctl restart rsyslog OR systemctl restart rsyslog
* Inspect /var/log/syslog for the following error:
syslog:Aug 4 14:37:56 rsyslogd: file ‘/dev/console’: open error: Permission denied [v8.2001.0 try https:/ /www.rsyslog. com/e/2433 ]

The Privilege Drop options ($PrivDrop*) in focal’s rsyslog both point to ‘syslog’ for the user and group, and don’t match the ownership/ permission of ‘/dev/console’ generating the following:

syslog:Aug 3 15:16:58 rsyslogd: file ‘/dev/console’: open error: Permission denied [v8.2001.0 try https:/ /www.rsyslog. com/e/2433 ]

Looking in Bionic/18.04LTS, ‘/dev/console’ used to be root:syslog[1], nowadays it’s root:tty[2]

[1] — Bionic/18.04LTS (Gcloud instance)
# ls -l /dev/console
crw—w—- 1 root syslog 5, 1 Aug 3 15:17 /dev/console

[2] — Focal/20.04LTS (Gcloud instance)
# ls -l /dev/console
crw—w—- 1 root tty 5, 1 Aug 3 17:19 /dev/console

# /etc/rsyslog.conf
$PrivDropToUser syslog
$PrivDropToGroup syslog

** As a debug exercise I did the following:
— Cannot reproduce the situation if I intentionally get rid of the PrivDrop* options.
— Cannot reproduce the situation if I intentionally add ‘syslog’ user member of ‘tty’ group.

Meaning that it’s pretty obvious with the above statement that the permission denied is caused by the permission/ ownership mismatch between ‘/dev/console’ ‘s ownership permission & syslog user (PrivDropTo[ User|Group] ).

tags: added: seg sts
description: updated

One easy fix would possibly be the following:

# debian/ rsyslog. postinst
case «$1» in
configure)
adduser —system —group —no-create-home —quiet syslog || true
adduser syslog adm || true
+adduser syslog tty || true

I have tested it in a PPA, and it works just fine:

Preparing to unpack . /rsyslog_ 8.2001. 0-1ubuntu1+ test2020307b1_ amd64.deb .
Unpacking rsyslog (8.2001. 0-1ubuntu1+ test2020307b1) over (8.2001.0-1ubuntu1) .
Setting up rsyslog (8.2001. 0-1ubuntu1+ test2020307b1) .
The user `syslog’ is already a member of `adm’.
Adding user `syslog’ to group `tty’ .
==> Adding user syslog to group tty d/disable: usr.sbin.rsyslogd
Processing triggers for man-db (2.9.1-1) .
Processing triggers for systemd (245.4-4ubuntu3.1) .

Ok, thinking about it for a moment, I can’t actually think of a way how this could have any adverse effects. rsyslog is anyway a very privileged thing (just by checking the capabilities), so adding it to tty should not really have any effect (as it already is meant to have rw access to tty’s). Maybe I’m missing something here, but so far this feels safe.

As for the SRUability of that, I think this does count as a bugfix so in theory should be SRU material.

Changed in rsyslog (Ubuntu Focal):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Eric Desrochers (slashd)
description: updated
description: updated

Thanks @sil2100 for your pre-approval comment.

I have uploaded it into focal upload queue.
It is now waiting for the official SRU team approval in order to start building in focal-proposed for the verification test phase.

Hello Eric, or anyone else affected,

Accepted rsyslog into focal-proposed. The package will build now and be available at https:/ /launchpad. net/ubuntu/ +source/ rsyslog/ 8.2001. 0-1ubuntu1. 1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https:/ /wiki.ubuntu. com/Testing/ EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification- needed- focal to verification- done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed- focal. In either case, without details of your testing we will not be able to proceed.

Источник

imfile fails to send logs remoteafter logrotate #2929

Expected behavior

Rsyslog imfine should continue to send logs to remote site after logrotate.

Actual behavior

The inode of /var/log/syslog changes during logrotate and a new state file is not created. Logs are not send to remote site.

Steps to reproduce the behavior

The new state file is not created for the new inode during logrotate. However if I issue a restart the new state file is created and logs are shipped to remote site just fine. So the difference in rsyslog getting a HUP and a restart is that the state file for the inode of the new log file only gets created
during the stop phase of the restart command.

Happens during logrotate.

Environment

  • for configuration questions/issues, include rsyslog.conf and included config files

Log file pre rotate (hostname removed):

Log file post rotate (hostname removed):

inode of logs after file rotate:

The state files in /var/spool/rsyslog

Permissions of /var/spool/rsyslog directory:

Log rotate config:

The text was updated successfully, but these errors were encountered:

I think the issues is rooted in missing permissions. From the error messages you posted:

Does logrotate create the files with the wrong ids?

Could you try to remove the privdrop statements, so that rsyslog runs as root. I would assume that it then works, what would confirm the permission issue.

Thanks for your suggestion @rgerhards
It’s the state file that doesn’t get created at all. Perhaps this could be the reason. I’ll make these changes and let you know.

The state file is only created on shutdown (except if otherwise configured). Run state is kept in memory. But if we cannot open the to-be-monitored file itself, we neither create state file nor run state simply because we cannot monitor the file as we cannot open and thus read it.

Thanks for the information on the state file. This does make sense.

I tested running rsyslog as root by commenting the Drop priv statements. The behaviour was the same. The remote logs where sent immediately after a restart however stopped sending after logrotate. The real crazy thing here is that it seems random. At first I had thought that the logs just stopped sending after logrotate, however after watching this for a while I’d also see the reverse where I would see logs in the remote location when I didn’t the day before. The syslog is rotated daily.

I’m not quite sure where to go from here. I’ve put the logs back as running under the non root user.

@ladean does the issue still exist for you? If so, can you answer the open question:

how are you rotating the file?

If I don’t hear back, I’ll close this issue.

Источник

Forwarding log files with file names when we use wildcards #474

I’m using wildcard to send logs to remote server, But not able forward log files with file names. All logs from multiple files are being dumped to single file.

Is there any way to get the original file names in Syslog ?

rsyslog client config file:

rsyslog server config file

Thanks for any help !!

The text was updated successfully, but these errors were encountered:

Let me try it relatively brief first:

You need to transmit the file name to the remote end:

  1. enable metadata inside imfile: http://www.rsyslog.com/doc/master/configuration/modules/imfile.html
  2. you now need to craft a custom template that includes the filename metadata ( $!metadata!filename )
  3. you need to apply this template to the forwarding action

On the receiver:

  1. you need to extract the metadata again (e.g. via mmjsonparse)
  2. now you can use that file name as any other variable

We need to do a writeup of a full scenario, unfortunately I do not have time at the moment. I hope this get’s you going or some others will also jump in.

I have did some changes on rsyslog-client
module(load=»imfile» mode=»inotify») #needs to be done just once
module(load=»omrelp»)

ruleset(name=»sendToLogserver») <
action(type=»omrelp» Target=»172.16.216.118″ Port=»18039″ Template=»LongTagForwardFormat»)
>

input(type=»imfile»
File=»/var/lib/docker/volumes/httpd_logs-br/data/*.log»
Tag=»tag2
»
addMetadata=»on»
Ruleset=»myapp_logs»
Severity=»notice»
Facility=»local7″)

ruleset(name=»myapp_logs») <
set $.suffix=re_extract($ !metadata!filename, «(.)/([^/])», 0, 2, «all.log»);
call sendToLogserver
>

Still it’s not working though

This is the complete file:

module(load=»imfile» mode=»inotify») #needs to be done just once

template (name=»LongTagForwardFormat» type=»string»
string=» %TIMESTAMP. date-rfc3339% %HOSTNAME% %syslogtag%%$.suffix%%msg. sp-if-no-1st-sp%%msg%»)

ruleset(name=»sendToLogserver») <
action(type=»omrelp» Target=»172.16.216.118″ Port=»18039″ Template=»LongTagForwardFormat»)
>

input(type=»imfile»
File=»/var/lib/docker/volumes/httpd_logs-br/data/*.log»
Tag=»tag2
»
addMetadata=»on»
Ruleset=»myapp_logs»
Severity=»notice»
Facility=»local7″)

ruleset(name=»myapp_logs») <
set $.suffix=re_extract($ !metadata!filename, «(.)/([^/])», 0, 2, «all.log»);
call sendToLogserver
>

And please also post the full server config. You wrote in Disqus comments, that you changed server module to «omrelp», but it should be «imrelp», if you are trying to get log, not send them.

Let’s debug this step by step:

  • try nc 18039 on clent to check, if network connectivity is OK.
  • try writing logs to local file on client and see what appears in it
  • try stopping rsyslog on server and using nc -l -p 18039 to see, what data is transferred to server

@roshu10 Are you still having this issue? Have you tried the steps that @selivan provided?

tried, But rsyslog-client is not sending the data to rsyslog-server

@roshu10 That means, it doesn’t get them locally. Do it have read permissions on /var/lib/docker/volumes/httpd_logs-br/data/*.log , and execute permissions on all folders along the way? rsyslog usually runs as syslog user, an docker often runs everything inside as root.

Источник

johnhk215

Posts: 9
Joined: 2020/02/03 06:53:09

rsyslog permission denied

syslog service can be started with error
rsyslog rsyslogd[2253]: imjournal: fopen() failed for path: ‘/var/lib/rsyslog/imjournal.state.tmp’: Permission denied

Change log path to /opt/rsyslog

—————————————————————
chcon —reference /var/log /opt/rsyslog

Rsyslog config
————————————————————
$template RemoteLogs,»/opt/rsyslog/%HOSTNAME%/%HOSTNAME%-%$YEAR%-%$MONTH%-%$DAY%.log»
*.* ?RemoteLogs
& ~

#
# Set the default permissions for all log files.
#
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# These may require root
$FileOwner syslog
$FileGroup syslog
$PrivDropToUser syslog
$PrivDropToGroup syslog

—————————————————————-

if #$PrivDropToUser syslog -> no permission denied but syslog user doesn’t have /opt/rsyslog

Any advise?

Thanks.


User avatar

TrevorH

Site Admin
Posts: 32529
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: rsyslog permission denied

Post

by TrevorH » 2020/03/23 14:47:44

Looks like your problem is caused by you trying to move it to /opt/rsyslog. Is there a reason why you’d try to do that?


johnhk215

Posts: 9
Joined: 2020/02/03 06:53:09

Re: rsyslog permission denied

Post

by johnhk215 » 2020/03/24 01:59:21

Yes, It is central logging server and would like to put the data in to /opt/data instead of default path /var/log.
Can I ignore the permission denied message?

Thanks.


pjsr2

Posts: 607
Joined: 2014/03/27 20:11:07

Re: rsyslog permission denied

Post

by pjsr2 » 2020/03/24 10:11:09

You cannot simply ignore the message.

There is a whole bunch of selinux context settings that apply to /var/log and the log files and subdirectories in it, written by various applications. If you move your log files from /var/log to /opt/log you need to get these selinux fcontext sorted out, which is a whole lot of work.

You should expect logging problems similar to the one you already observed from other applications, and they may not show up immediately as it may take time before applications feel the need to log something. Some applications don’t care when they can’t log. Other applications fail when they can’t log. If applications cannot log, that may have security implications, as you may loose important warning signals.


User avatar

TrevorH

Site Admin
Posts: 32529
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: rsyslog permission denied

Post

by TrevorH » 2020/03/24 10:15:22

If you moved it to give more space then I would suggest mounting that filesystem on /var/log to begin with (though that’s more complicated than just that as you will need to copy the existing content over first and probably relabel the filesystem afterwards once it’s mounted on /var/log).

As pjsr2 says, ther are numerous selinux rules set up to allow access and you would need to duplicate or set up equivalence rules for those

Code: Select all

# semanage fcontext -l | grep -c var_log
66


johnhk215

Posts: 9
Joined: 2020/02/03 06:53:09

Re: rsyslog permission denied

Post

by johnhk215 » 2020/03/26 15:14:50

Thank you for yours information.
Can I change Rsyslog store log path to another path without change /var/log ?


User avatar

TrevorH

Site Admin
Posts: 32529
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: rsyslog permission denied

Post

by TrevorH » 2020/03/26 15:21:05

I thought that’s what we’ve been telling you is too complicated and that you shouldn’t do? But maybe I misunderstood your question.



At midnight, a cronjob initiates logrotate to rotate 4 key log files. These 4 log files are also being sent to a log aggregation server by rsyslog. Despite running the rsyslog logrotate command after each log is rotated, sometimes, the server encounters a «Permission Denied» error that prevents rsyslog from sending the logs belonging to the new files. I cannot figure out why this is happening so unreliably.

Here are the errors reported by rsyslog. Note that they occur only on some dates:

Jul 31 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 01 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 02 00:00:04 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/nginx.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 09 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 10 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/django.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 10 00:00:01 hostname rsyslogd[26343]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]
Aug 16 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 16 00:00:02 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/django.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 19 00:00:03 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]
Aug 20 00:00:06 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/app.log': Permission denied [v8.32.0]
Aug 30 00:00:01 hostname rsyslogd[856]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Sep 03 00:00:01 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/celery.log': Permission denied [v8.32.0]
Sep 03 00:00:04 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]

Logrotate is triggered by a cron job, using the root user. Here is the logrotate config:

/home/ubuntu/remote_logs/*.log {
    rotate 365
    daily
    compress
    missingok
    notifempty
    dateext
    dateformat .%Y-%m-%d
    dateyesterday
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

/usr/lib/rsyslog/rsyslog-rotate

#!/bin/sh

if [ -d /run/systemd/system ]; then
    systemctl kill -s HUP rsyslog.service
else
    invoke-rc.d rsyslog rotate > /dev/null
fi

I know that the logrotate itself is succeeding, because the logrotate verbose logs indicate that, and I see the rotated log files every day. It seems like rsyslog cannot access the newly create files.

Here are the logrotate logs:

Handling 13 logs

rotating pattern: /home/ubuntu/remote_logs/*.log  after 1 days (365 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 106
considering log /home/ubuntu/remote_logs/celery.log
  Now: 2020-10-14 00:00
  Last rotated at 2020-10-13 00:00
  log needs rotating
considering log /home/ubuntu/remote_logs/django.log
  Now: 2020-10-14 00:00
  Last rotated at 2020-10-13 00:00
  log needs rotating
considering log /home/ubuntu/remote_logs/app.log
  Now: 2020-10-14 00:00
  Last rotated at 2020-10-13 00:00
  log needs rotating
considering log /home/ubuntu/remote_logs/nginx.log
  Now: 2020-10-14 00:00
  Last rotated at 2020-10-13 00:00
  log needs rotating
rotating log /home/ubuntu/remote_logs/celery.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/celery.log to /home/ubuntu/remote_logs/celery.log.2020-10-13
creating new /home/ubuntu/remote_logs/celery.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/django.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/django.log to /home/ubuntu/remote_logs/django.log.2020-10-13
creating new /home/ubuntu/remote_logs/django.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/app.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/app.log to /home/ubuntu/remote_logs/app.log.2020-10-13
creating new /home/ubuntu/remote_logs/app.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/nginx.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/nginx.log to /home/ubuntu/remote_logs/nginx.log.2020-10-13
creating new /home/ubuntu/remote_logs/nginx.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
switching euid to 0 and egid to 0

What is happening to rsyslog that prevents it from accessing the log files occasionally?

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account


Closed

zbjornson opened this issue

Apr 24, 2020

· 9 comments

Comments

@zbjornson

Hello! This has been going on for ages but I hadn’t taken the time to track it down until now. Our GCE VM logs (Ubuntu 19.10, nearly out-of-box) are filled with these messages:

Apr 18 00:36:45 srv-zgff systemd[1]: Started System Logging Service.
Apr 19 00:01:47 srv-zgff rsyslogd[530]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="530" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Apr 19 00:36:41 srv-zgff rsyslogd[530]: file '/dev/console': open error: Permission denied [v8.1901.0 try https://www.rsyslog.com/e/2433 ]
Apr 19 00:36:41 srv-zgff rsyslogd[530]: action 'action-7-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try h
Apr 19 00:36:41 srv-zgff rsyslogd[530]: action 'action-7-builtin:omfile' resumed (module 'builtin:omfile') [v8.1901.0 try https://www.rsyslog.com/e/2359 ]
Apr 19 00:36:41 srv-zgff rsyslogd[530]: action 'action-7-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.1901.0 try h
# these resumed/suspended lines repeat frequently

That config comes from

daemon,kern.* /dev/console

and indeed the syslog user doesn’t have access

$ ls -l /dev/console
crw--w---- 1 root tty 5, 1 Apr 19 00:01 /dev/console
$ ps aux | grep rsyslog
syslog     530  0.0  0.1 224508  5144 ?        Ssl  Apr18   0:03 /usr/sbin/rsyslogd -n -iNONE
$ groups syslog
syslog : syslog adm

Should there be a udev rule in this package to change permissions on /dev/console?

Looks like this has been happening for quite some time for at least one other user (xenial, bionic): https://irclogs.ubuntu.com/2018/11/14/%23ubuntu.html#t16:07

@kkm000

I remember there used to be an udev rule file in the old package that changed ownership of /dev/console to root:syslog. It must be somewhere down the commit history here, pre-split.

@slashdd

At first glance, it looks related to the Privilege Drop in rsyslog that both point to syslog for the user and group.
It no longer matches the ownership/permission of /dev/console.

Looking in Bionic/18.04LTS, /dev/console used to be root:syslog, nowadays it’s root:tty

# /etc/rsyslog.conf
$PrivDropToUser syslog
$PrivDropToGroup syslog

I was able to reproduce with the PrivDrop, but I can’t without it since it is obsviously always using ‘root’.

I’m still investigating, but wanted to give my 2 cents so far.

  • Eric

@slashdd

While I would not suggest getting rid of the privileges drop, unless if it’s a debug exercise in a non-production system, like I did.
As a potential workaround, one could possibly make ‘syslog’ user a member of the ‘tty’ group (by hand) and validate if it mitigates the problem while it is still under investigation.

@slashdd

@slashdd

Please see this Ubuntu bug:
https://launchpad.net/bugs/1890177

I’ll soon upload a fix in Ubuntu (Focal) that the Ubuntu SRU verification team already pre approved (pre SRU).

Note that during the verification phase of the package, I’ll need feedbacks from affected users.

Thanks in advance.

@zbjornson

Awesome, thank you.

I was mistaken then. I thought this issue was because of the rsyslog conf provided by this GCP guest OS package, but I take it that configuration should work?

@slashdd

@zbjornson

@slashdd I’ll install it on a few servers today and report there.

@bob-rove

It seems that the issue is still out there.
I’m running Ubuntu 22.04 LTS on GCE (from provided cloud images) and having the very same issue.
The only problem now is that adding syslog user to tty group doesn’t help as stated by links — it is still bleeding:

imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.2112.0]
rsyslogd's groupid changed to 111
rsyslogd's userid changed to 104
[origin software="rsyslogd" swVersion="8.2112.0" x-pid="589" x-info="https://www.rsyslog.com"] start
file '/dev/console': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]
action 'action-9-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]
action 'action-9-builtin:omfile' resumed (module 'builtin:omfile') [v8.2112.0 try https://www.rsyslog.com/e/2359 ]
action 'action-9-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]
action 'action-9-builtin:omfile' resumed (module 'builtin:omfile') [v8.2112.0 try https://www.rsyslog.com/e/2359 ]
action 'action-9-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2112.0 try https://www.rsyslog.com/e/2007 ]
...

Host details:

kkahn@host:~$ cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
kkahn@host:~$ uname -a
Linux host.example.com 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
kkahn@host:~$ ps aux | grep -i rsyslog
root       8199  0.0  0.0 224604 11456 ?        Ssl  Jun25   0:02 /usr/sbin/rsyslogd -n


kkahn@host:~$ sudo systemctl status rsyslog.service                                                                                                                                                                                                                                                                          
* rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-06-25 23:30:55 EDT; 9h ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 8199 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           `-8199 /usr/sbin/rsyslogd -n

Jun 26 08:34:40 host.example.com rsyslogd[8199]: fopen() failed: 'Permission denied', path: '/imjournal.state.tmp'  [v8.24.0 try http://www.rsyslog.com/e/2013 ]
Jun 26 08:35:01 host.example.com rsyslogd[8199]: fopen() failed: 'Permission denied', path: '/imjournal.state.tmp'  [v8.24.0 try http://www.rsyslog.com/e/2013 ]
Jun 26 08:36:01 host.example.com rsyslogd[8199]: fopen() failed: 'Permission denied', path: '/imjournal.state.tmp'  [v8.24.0 try http://www.rsyslog.com/e/2013 ]
Jun 26 08:37:01 host.example.com rsyslogd[8199]: fopen() failed: 'Permission denied', path: '/imjournal.state.tmp'  [v8.24.0 try http://www.rsyslog.com/e/2013 ]
Jun 26 08:37:40 host.example.com rsyslogd[8199]: fopen() failed: 'Permission denied', path: '/imjournal.state.tmp'  [v8.24.0 try http://www.rsyslog.com/e/2013 ]

It’s less permission denied, and more so the fact that the directory /var/lib/rsyslog (where imjournal.state.tmp would live) does not exist:

kkahn@host:~$ sudo ls -latr /var/lib/ | grep -i rsyslog
kkahn@host:~$ 

I have several other hosts of the same OS however they all have /var/lib/rsyslog/ and therefore /var/lib/rsyslog/imjournal.state.tmp. I’m not seeing anything anamolous with the config file either, here’s an m5sum comparison to a server that has /var/lib/rsyslog/:

Bad server:

kkahn@host:~$ md5sum /etc/rsyslog.conf 
0dd94a0c285fb32f41fa5b226e83c26b  /etc/rsyslog.conf

Good server:

kkahn@host:~$ md5sum /etc/rsyslog.conf 
0dd94a0c285fb32f41fa5b226e83c26b  /etc/rsyslog.conf

The directive that would be responsible for this is in /etc/rsyslog.conf even uncommented on the host:

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

I’m unable to find a way to tell if rsyslog has been properly configured/installed; I don’t know how to «true up» this installation to make sure it’s not broken in some way. I don’t think simply creating the directory is safe to do.

Rsyslog runs as root, so you should not normally get a permission denied.

Diving into /var/log/messages (or /var/log/syslog if on i.e. Ubuntu) should provide some clues.

grep 10601 /var/log/messages  # search for the port number
... setroubleshoot[29284]: SELinux is preventing /usr/sbin/rsyslogd from name_connect access on the tcp_socket port 10601. For complete SELinux messages run: sealert -l 005c986c-e0f9-481a-b3c6-0b45a9698ccd
... setroubleshoot[31103]: SELinux is preventing /usr/sbin/rsyslogd from name_connect access on the tcp_socket port 10601.#012#012*****  Plugin connect_ports (92.2 confidence) suggests   *********************#012#012If you want to allow /usr/sbin/rsyslogd to connect to network port 10601#012Then you need to modify the port type.#012Do#012# semanage port -a -t PORT_TYPE -p tcp 10601#012    where PORT_TYPE is one of the following: dns_port_t, dnssec_port_t, http_port_t, kerberos_port_t, mysqld_port_t, ocsp_port_t, postgresql_port_t, rsh_port_t, syslog_tls_port_t, syslogd_port_t, wap_wsp_port_t.#012#012*****  Plugin catchall_boolean (7.83 confidence) suggests   ******************#012#012If you want to

And there it is:

rsyslog has been configured to send to a non standard port and SELinux is denying it.

The sollution — if you keep the non-standard port — is to allow the port in SElinux.

Get details from sealert -l 005c986c-e0f9-481a-b3c6-0b45a9698ccd — the UUID is unique to you of course.

If you want to allow /usr/sbin/rsyslogd to connect to network port 10601
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 10601
    where PORT_TYPE is one of the following: dns_port_t, dnssec_port_t, http_port_t, kerberos_port_t, mysqld_port_t, ocsp_port_t, postgresql_port_t, rsh_port_t, syslog_tls_port_t, syslogd_port_t, wap_wsp_port_t.

I declare that my port is a syslogd_port_t

semanage port -a -t syslogd_port_t -p tcp 10601

# Restart rsyslog for good measure
systemctl restart rsyslog  # or "pkill -HUP rsyslog"

Logs are now sent.

Понравилась статья? Поделить с друзьями:
  • Rsyslog error 2207
  • Rsync unknown module error
  • Rsync syntax or usage error
  • Rsync sender write error broken pipe 32
  • Rsync error in rsync protocol data stream code 12 at io c