概要
先日Arcsight ModuleがElsaticsearch から出たということで、取り込めるかどうか試してみた。
環境
Centos7 ＠AWS環境 & proxy環境
Elasticsearch 6.0
elasticsearch のInstall
yum remove -y java-1.7.0-openjdk
yum install -y java-1.8.0-openjdk-devel
yum install -y java-1.8.0-openjdk-debuginfo --enablerepo=*debug*
# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat << EOF > /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
sudo yum -y install elasticsearch
systemctl status elasticsearch
systemctl enable elasticsearch
systemctl start elasticsearch
# curl localhost:9200
{
"name" : "QaYIMOJ",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "b2cZBZkYQ-up7dPS_LTZ1g",
"version" : {
"number" : "6.0.0",
"build_hash" : "8f0685b",
"build_date" : "2017-11-10T18:41:22.859Z",
"build_snapshot" : false,
"lucene_version" : "7.0.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
kibana のInstall
www.elastic.co
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat << EOF > /etc/yum.repos.d/kibana.repo
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
yum -y install kibana
# diff -u kibana.yml /etc/kibana/kibana.yml
--- kibana.yml 2017-11-11 03:50:42.000000000 +0900
+++ /etc/kibana/kibana.yml 2017-11-22 16:55:49.833624169 +0900
@@ -4,7 +4,7 @@
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
-#server.host: "localhost"
+server.host: "0.0.0.0"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
@@ -18,7 +18,7 @@
#server.name: "your-hostname"
# The URL of the Elasticsearch instance to use for all your queries.
-#elasticsearch.url: "http://localhost:9200"
+elasticsearch.url: "http://localhost:9200"
# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
@@ -83,7 +83,7 @@
#pid.file: /var/run/kibana.pid
# Enables you specify a file where Kibana stores log output.
-#logging.dest: stdout
+logging.dest: /var/log/kibana.log
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# systemctl restart kibana
ブラウザで起動：
http://172.20.214.47:5601
logstash のインストール
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
cat << EOF > /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
yum -y install logstash
Logstash でとりあえずログを入れてみる
/etc/logstash/conf.d/messages.conf
input {
file {
path => "/var/log/messages"
start_position => beginning
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => "localhost"
user => "elastic"
password => "changeme"
index => "logstash-2017-11-22"
}
}
- Permission がない場合は与える。問題があれば/var/log/logstash/logstash-plain.log を確認
Arcsight Module のインストール
https://www.elastic.co/guide/en/logstash/current/arcsight-module.html
ES_JAVA_OPTS="-Dhttp.proxyHost=proxyurl.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxyurl.com -Dhttps.proxyPort=3128" /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.0.0.zip
/usr/share/kibana/bin/kibana-plugin install file:x-pack-6.0.0.zip
ES_JAVA_OPTS="-Dhttp.proxyHost=proxyurl.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxyurl.com -Dhttps.proxyPort=3128" /usr/share/logstash/bin/logstash-plugin install x-pack
パスワードの設定
/usr/share/elasticsearch/bin/x-pack/setup-passwords interactive
...
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [elastic]
有効になっているか念のため確認
curl -u elastic:changeme localhost:9200
kibana.yml で下記設定をする
elasticsearch.username: "elastic"
elasticsearch.password: "changeme"
起動
modules:
- name: arcsight
var.inputs: "smartconnector"
var.input.smartconnector.port: "5000"
var.elasticsearch.hosts: "localhost:9200"
var.elasticsearch.username: "elastic"
var.elasticsearch.password: "changeme"
var.kibana.host: "localhost:5601"
var.kibana.username: "elastic"
var.kibana.password: "changeme"
Arcsight Connector 側の設定
- CEF Syslog
- Elasticsearch のIP
- Port=5000
- Protocol=Raw TCP
- Forwarder=false
- CEF Version=0.1
所感
Elasticsearch にArcsightからのログを配送できるということで試してみた。
ArcsightConnector 側で構造化されているので、これで取り込めると検索基盤として色々便利そう。
