如何使用 Logstash 处理多个异构输入？

假设您有两种非常不同类型的日志，比如技术日志和业务日志，您希望:

  • 使用 gelf输出将原始技术日志路由到 graylog2服务器,
  • 使用专用的 elasticsearch_http输出将 json 业务日志存储到 elasticsearch 集群中。

我知道，以 Syslog-NG为例，配置文件允许定义几个不同的输入，然后可以在分派之前分别处理这些输入; 而 Logstash似乎无法做到这一点。即使可以使用两个特定的配置文件启动一个实例，所有日志都采用相同的通道，并且应用相同的处理..。

我是否应该运行不同类型日志的实例？

69644 次浏览
小开

Should I run as many instances as I have different types of logs?

No! You can only run one instance to handle different types of logs.

In the logstash configuration file, you can specific each input with different type. Then in the filter you can use if to distinct different processing, and also at the output you can use "if" output to different destination.

input {
file {
type => "technical"
path => "/home/technical/log"
}
file {
type => "business"
path => "/home/business/log"
}
}
filter {
if [type] == "technical" {
# processing .......
}
if [type] == "business" {
# processing .......
}
}
output {
if [type] == "technical" {
# output to gelf
}
if [type] == "business" {
# output to elasticsearch
}
}

Hope this can help you :)

小开

I think logstash can't read more than 2 files in Input section . try the below 

input {
file {
type => "technical"
path => "/home/technical/log"
}
file {
type => "business"
path => "/home/business/log"
}
file {
type => "business1"
path => "/home/business/log1"
}
}
小开

I used tags for multiple file input:

input {
file {
type => "java"
path => "/usr/aaa/logs/stdout.log"
codec => multiline {
...
},
tags => ["aaa"]
}


file {
type => "java"
path => "/usr/bbb/logs/stdout.log"
codec => multiline {
...
}
tags => ["bbb"]
}
}
output {
stdout {
codec => rubydebug
}
if "aaa" in [tags] {
elasticsearch {
hosts => ["192.168.100.211:9200"]
index => "aaa"
document_type => "aaa-%{+YYYY.MM.dd}"
}
}


if "bbb" in [tags] {
elasticsearch {
hosts => ["192.168.100.211:9200"]
index => "bbb"
document_type => "bbb-%{+YYYY.MM.dd}"
}
}
}

提交答案