使用rsyslog收集日志(1)

作者: tcxurun 分类: 学习笔记 发布时间: 2020-01-23 11:03 ė 6 没有评论

一、rsyslog 简介

1、rsyslog 在Linux上自带,兼容syslog语法,在syslog基础上增加了更多协议的支持,如数据库支持(MySQL、PostgreSQL、Oracle等)、日志内容筛选、定义日志格式模板等,配合额外module插件可以完成很多场景的使用。借用下官网的图片:

2、rsyslog提供了三种远程传输协议:

UDP 传输协议

基于传统UDP协议进行远程日志传输,也是传统syslog使用的传输协议; 可靠性比较低,但性能损耗最少, 在网络情况比较差, 或者接收服务器压力比较高情况下,

可能存在丢日志情况。 在对日志完整性要求不是很高,在可靠的局域网环境下可以使用。

 

TCP 传输协议

基于传统TCP协议明文传输,需要回传进行确认,可靠性比较高; 但在接收服务器宕机或者两者之间网络出问题的情况下,会出现丢日志情况。 这种协议相比于UDP在

可靠性方面已经好很多,并且rsyslog原生支持,配置简单, 同时针对可能丢日志情况,可以进行额外配置提高可靠性,因此使用比较广。

 

RELP 传输协议

RELP(Reliable Event Logging Protocol)是基于TCP封装的可靠日志消息传输协议; 是为了解决TCP 与 UDP 协议的缺点而在应用层实现的传输协议,也是三者

之中最可靠的。 需要多安装一个包rsyslog-relp以支持该协议。

一般使用TCP模式即可。

3、使用rsyslog的缘由:

1.防止系统崩溃无法获取系统日志分析崩溃原因,用rsyslog可以把日志传输到远程的日志服务器上

2.使用rsyslog日志可以减轻系统压力,因为使用rsyslog可以有效减轻系统的磁盘IO

3.rsyslog使用tcp传输非常可靠,可以对日志进行过滤,提取出有效的日志,rsyslog是轻量级的日志软件,在大量日志写的情况下,系统负载基本上在0.1以下。

二、rsyslog远程日志实现不同服务器记录不同目录(以Centos7系统)

1、服务器与客户端关闭防火墙和 SElinux

因为传输日志用到了 514 端口,要么端口打开,要么关闭防火墙,本测试采用关闭防火墙的方法,

查看防火墙状态

firewall-cmd –state

停止防火墙

systemctl stop firewalld.service

查看SELinux 状态

getenforce

修改

vi /etc/selinux/config

SELINUX=enforcing

改为

SELINUX=disabled

重启计算机后生效

备注:如果暂时无法重启计算机,那么可以先做如下设置,临时关闭

setenforce 0

 

2、服务器与客户端均确认是否已安装 rsyslogd

rpm -qa | grep rsyslog

rsyslogd -v

3、客户端设置

(1)首先在/etc/rsyslog.d/目录下新建应用配置文件classyp.conf,

内容如下:

$InputFileName  /data/log/sys-info.log  #要监控的日志文件
 
$InputFileTag   sys-info 给此监控操作打一个tag(标志),用于日志接收端根据tag分别处理。
 
$InputFileSeverity info
 
$InputFileStateFile state_product_info  #记录指定监控日志文件的处理状态,比如/root/.bash_history文件已经处理了多少行等状态信息,可以自定义名称。根据笔者观察,文件默认生成在/目录下。
 
#对指定监控文件进行轮询访问的时间间隔,这是的10是以秒为单位,每隔10秒以确认所监控的文件是否有新数据产生。
 
#$InputFilePollInterval 10
 
#表示处理多少行后更新状态文件,建议改大一点,写状态文件需要消耗系统资源。
 
$InputFilePersistStateInterval 250
 
#设置syslog消息类型,以local开始表示自定义类型,本例为local5,用于日志接收端分类处理。
 
$InputFileFacility local5
 
#设定监控操作是否生效,必选项 
 
$InputRunFileMonitor
 
$InputFileName  /data/log/sys-error.log
 
$InputFileTag   sys-error
 
$InputFileSeverity info
 
$InputFileStateFile state_product_sys-error
 
$InputFilePersistStateInterval 250
 
$InputFileFacility local5
 
$InputRunFileMonitor

 

(2) 修改rsyslog配置(vi /etc/rsyslog.conf)

# rsyslog configuration file



# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html



#### MODULES ####



# The imjournal module bellow is now used as a message source instead of imuxsock.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)

$ModLoad imjournal # provides access to the systemd journal

#$ModLoad imklog # reads kernel messages (the same are read from journald)

#$ModLoad immark  # provides --MARK-- message capability



$ModLoad imfile  # Load the imfile input module                         # 1. 加载imfile



$MaxMessageSize 256k                                                    # 设置最大消息大小,默认4k不够,会被截断



# Provides UDP syslog reception

#$ModLoad imudp

#$UDPServerRun 514



# Provides TCP syslog reception

$ModLoad imtcp

$InputTCPServerRun 514



#### GLOBAL DIRECTIVES ####



# Where to place auxiliary files

$WorkDirectory /var/lib/rsyslog



# Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat



# File syncing capability is disabled by default. This feature is usually not required,

# not useful and an extreme performance hit

#$ActionFileEnableSync on



# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf



# Turn off message reception via local log socket;

# local messages are retrieved through imjournal now.

$OmitLocalLogging on



# File to store the position in the journal

$IMJournalStateFile imjournal.state





#### RULES ####



# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.*                                                 /dev/console



# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

# 忽略掉自定义的local5,local6日志类型,避免消息跑到messages文件,导致本地磁盘爆掉

*.info;mail.none;authpriv.none;cron.none;local5.none;local6.none                /var/log/messages





# The authpriv file has restricted access.

authpriv.*                                              /var/log/secure



# Log all the mail messages in one place.

mail.*                                                  -/var/log/maillog





# Log cron stuff

cron.*                                                  /var/log/cron



# Everybody gets emergency messages

*.emerg                                                 :omusrmsg:*



# Save news errors of level crit and higher in a special file.

uucp,news.crit                                          /var/log/spooler



# Save boot messages also to boot.log

local7.*                                                /var/log/boot.log





# ### begin forwarding rule ###

# The statement between the begin ... end define a SINGLE forwarding

# rule. They belong together, do NOT split them. If you create multiple

# forwarding rules, duplicate the whole block!

# Remote Logging (we use TCP for reliable delivery)

#

# An on-disk queue is created for this action. If the remote host is

# down, messages are spooled to disk and sent when it is up again.

#$ActionQueueFileName fwdRule1 # unique name prefix for spool files

#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)

#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown

#$ActionQueueType LinkedList   # run asynchronously

#$ActionResumeRetryCount -1    # infinite retries if host is down

# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional

#*.* @@remote-host:514

# ### end of the forwarding rule ###

//loca5是我们的自定义类型,@@表示使用tcp协议将类型为loca5的所有级别的日志消息发送到10.201.14.219的514端口上。

local5.*,local6.* @@10.201.14.219:514

$EscapeControlCharactersOnReceive off

 

4、服务端配置

(1)修改rsyslog配置(vi /etc/rsyslog.conf)

修改rsyslog配置(vi /etc/rsyslog.conf)

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp # 1. 加载imtcp模块,开始tcp服务
$InputTCPServerRun 514

$MaxMessageSize 256k # 设置最大消息size,默认4k不够用,会被截断

$template CleanMsgFormat,"%msg%\n" # 只保存原始消息,不需要时间戳,hostname,tag

################################################################################

# 接收info日志
$template InfoFileFormat,"/data/log/%hostname%-%syslogtag%-%$MONTH%%$DAY%.log"
if $syslogtag startswith 'sys-inf' then ?InfoFileFormat;CleanMsgFormat
& ~

# 接收error日志
$template errorFileFormat,"/data/log/%hostname%-%syslogtag%-%$MONTH%%$DAY%.log"
if $syslogtag startswith 'sys-err' then ?errorFileFormat;CleanMsgFormat
& ~

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.
authpriv.* /var/log/secure

# Log all the mail messages in one place.
mail.* -/var/log/maillog


# Log cron stuff
cron.* /var/log/cron

# Everybody gets emergency messages
*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log
local7.* /var/log/boot.log

# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
$EscapeControlCharactersOnReceive off # 修复收集到的日志出现“#12”错误提示

 

 

本文出自天一直很蓝,转载时请注明出处及相应链接。

本文永久链接: http://www.tcxurun.cn/archives/530

发表评论

电子邮件地址不会被公开。 必填项已用*标注

Ɣ回顶部