一、rsyslog 简介
1、rsyslog 在Linux上自带,兼容syslog语法,在syslog基础上增加了更多协议的支持,如数据库支持(MySQL、PostgreSQL、Oracle等)、日志内容筛选、定义日志格式模板等,配合额外module插件可以完成很多场景的使用。借用下官网的图片:
2、rsyslog提供了三种远程传输协议:
UDP 传输协议
基于传统UDP协议进行远程日志传输,也是传统syslog使用的传输协议; 可靠性比较低,但性能损耗最少, 在网络情况比较差, 或者接收服务器压力比较高情况下,
可能存在丢日志情况。 在对日志完整性要求不是很高,在可靠的局域网环境下可以使用。
TCP 传输协议
基于传统TCP协议明文传输,需要回传进行确认,可靠性比较高; 但在接收服务器宕机或者两者之间网络出问题的情况下,会出现丢日志情况。 这种协议相比于UDP在
可靠性方面已经好很多,并且rsyslog原生支持,配置简单, 同时针对可能丢日志情况,可以进行额外配置提高可靠性,因此使用比较广。
RELP 传输协议
RELP(Reliable Event Logging Protocol)是基于TCP封装的可靠日志消息传输协议; 是为了解决TCP 与 UDP 协议的缺点而在应用层实现的传输协议,也是三者
之中最可靠的。 需要多安装一个包rsyslog-relp以支持该协议。
一般使用TCP模式即可。
3、使用rsyslog的缘由:
1.防止系统崩溃无法获取系统日志分析崩溃原因,用rsyslog可以把日志传输到远程的日志服务器上
2.使用rsyslog日志可以减轻系统压力,因为使用rsyslog可以有效减轻系统的磁盘IO
3.rsyslog使用tcp传输非常可靠,可以对日志进行过滤,提取出有效的日志,rsyslog是轻量级的日志软件,在大量日志写的情况下,系统负载基本上在0.1以下。
二、rsyslog远程日志实现不同服务器记录不同目录(以Centos7系统)
1、服务器与客户端关闭防火墙和 SElinux
因为传输日志用到了 514 端口,要么端口打开,要么关闭防火墙,本测试采用关闭防火墙的方法,
查看防火墙状态
firewall-cmd –state
停止防火墙
systemctl stop firewalld.service
查看SELinux 状态
getenforce
修改
vi /etc/selinux/config
SELINUX=enforcing
改为
SELINUX=disabled
重启计算机后生效
备注:如果暂时无法重启计算机,那么可以先做如下设置,临时关闭
setenforce 0
2、服务器与客户端均确认是否已安装 rsyslogd
rpm -qa | grep rsyslog
rsyslogd -v
3、客户端设置
(1)首先在/etc/rsyslog.d/目录下新建应用配置文件classyp.conf,
内容如下:
$InputFileName /data/log/sys-info.log #要监控的日志文件
$InputFileTag sys-info 给此监控操作打一个tag(标志),用于日志接收端根据tag分别处理。
$InputFileSeverity info
$InputFileStateFile state_product_info #记录指定监控日志文件的处理状态,比如/root/.bash_history文件已经处理了多少行等状态信息,可以自定义名称。根据笔者观察,文件默认生成在/目录下。
#对指定监控文件进行轮询访问的时间间隔,这是的10是以秒为单位,每隔10秒以确认所监控的文件是否有新数据产生。
#$InputFilePollInterval 10
#表示处理多少行后更新状态文件,建议改大一点,写状态文件需要消耗系统资源。
$InputFilePersistStateInterval 250
#设置syslog消息类型,以local开始表示自定义类型,本例为local5,用于日志接收端分类处理。
$InputFileFacility local5
#设定监控操作是否生效,必选项
$InputRunFileMonitor
$InputFileName /data/log/sys-error.log
$InputFileTag sys-error
$InputFileSeverity info
$InputFileStateFile state_product_sys-error
$InputFilePersistStateInterval 250
$InputFileFacility local5
$InputRunFileMonitor
(2) 修改rsyslog配置(vi /etc/rsyslog.conf)
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
$ModLoad imfile # Load the imfile input module # 1. 加载imfile
$MaxMessageSize 256k # 设置最大消息大小,默认4k不够,会被截断
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
# 忽略掉自定义的local5,local6日志类型,避免消息跑到messages文件,导致本地磁盘爆掉
*.info;mail.none;authpriv.none;cron.none;local5.none;local6.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
//loca5是我们的自定义类型,@@表示使用tcp协议将类型为loca5的所有级别的日志消息发送到10.201.14.219的514端口上。
local5.*,local6.* @@10.201.14.219:514
$EscapeControlCharactersOnReceive off
4、服务端配置
(1)修改rsyslog配置(vi /etc/rsyslog.conf)
修改rsyslog配置(vi /etc/rsyslog.conf)
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp # 1. 加载imtcp模块,开始tcp服务
$InputTCPServerRun 514
$MaxMessageSize 256k # 设置最大消息size,默认4k不够用,会被截断
$template CleanMsgFormat,"%msg%\n" # 只保存原始消息,不需要时间戳,hostname,tag
################################################################################
# 接收info日志
$template InfoFileFormat,"/data/log/%hostname%-%syslogtag%-%$MONTH%%$DAY%.log"
if $syslogtag startswith 'sys-inf' then ?InfoFileFormat;CleanMsgFormat
& ~
# 接收error日志
$template errorFileFormat,"/data/log/%hostname%-%syslogtag%-%$MONTH%%$DAY%.log"
if $syslogtag startswith 'sys-err' then ?errorFileFormat;CleanMsgFormat
& ~
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
$EscapeControlCharactersOnReceive off # 修复收集到的日志出现“#12”错误提示