Logstash mutate filter
最近在研究Logstash，透過Logstash的input, filter, output可以產生出許多不同的變化... 下面是我前陣子遇到的問題，後來找到解決方案....
例如我在抓CentO6.6的rsyslog時候，我把/var/log/message中的一些syslog重倒到logstash中
$ vi /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none @@10.240.58.95:514
另外，logstash server部分，透過input syslog module開啟514 port接聽syslog input...
input {
syslog {
port => 514
type => syslog
}
}
此時收到的syslog在rubydebug模式下，會類似：
{
"message" => "Installed: mysql-5.1.73-3.el6_5.x86_64",
"@version" => "1",
"@timestamp" => "2015-03-25T03:50:01.000Z",
"type" => "syslog",
"host" => "10.240.219.254",,
"timestamp" => "Mar 25 03:50:01"
"priority" => 14,
"logsource" => "simon-centos6",
"program" => "yum",
"pid" => "645",
"severity" => 6,
"facility" => 1,
"facility_label" => "user-level",
"severity_label" => "Informational"
}
其中，會明顯地發現有兩個timestamp的欄位... 雖然有個前面加上"@"
而這個重複的timestamp欄位會造成其他的output module錯誤，導致一些其他問題...
在找了很久，後來發現mutate這個filter module.... 透過mutate可以讓欄位改名稱、改內容、刪除欄位、增加欄位... 這樣在input與output之間的銜接，就少掉很多問題...
下面範例是透過mutate的rename function來置換我所遇到的重複key...
input {
syslog {
port => 514
type => syslog
}
}
filter {
mutate {
rename => [ "timestamp", "timestamp1" ]
}
}
output {
stdout { codec => rubydebug }
}
如果一切沒有問題，再重新執行logstash之後，可以看到新的output，其中可以看到timestamp1會被排到最後面
{
"message" => "Installed: mysql-5.1.73-3.el6_5.x86_64",
"@version" => "1",
"@timestamp" => "2015-03-25T03:50:01.000Z",
"type" => "syslog",
"host" => "10.240.219.254",
"priority" => 14,
"logsource" => "simon-centos6",
"program" => "yum",
"pid" => "645",
"severity" => 6,
"facility" => 1,
"facility_label" => "user-level",
"severity_label" => "Informational",
"timestamp1" => "Mar 25 03:50:01"
}
這樣傳給output的值就沒有問題了ㄛ :D