概要
先日Arcsight ModuleがElsaticsearch から出たということで、取り込めるかどうか試してみた。
環境
Centos7 @AWS環境 & proxy環境
Elasticsearch 6.0
elasticsearch のInstall
yum remove -y java-1.7.0-openjdk yum install -y java-1.8.0-openjdk-devel yum install -y java-1.8.0-openjdk-debuginfo --enablerepo=*debug*
# java -version openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch cat << EOF > /etc/yum.repos.d/elasticsearch.repo [elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF
sudo yum -y install elasticsearch systemctl status elasticsearch systemctl enable elasticsearch systemctl start elasticsearch
# curl localhost:9200 { "name" : "QaYIMOJ", "cluster_name" : "elasticsearch", "cluster_uuid" : "b2cZBZkYQ-up7dPS_LTZ1g", "version" : { "number" : "6.0.0", "build_hash" : "8f0685b", "build_date" : "2017-11-10T18:41:22.859Z", "build_snapshot" : false, "lucene_version" : "7.0.1", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }
kibana のInstall
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch cat << EOF > /etc/yum.repos.d/kibana.repo [kibana-6.x] name=Kibana repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF
yum -y install kibana
- 設定変更(logging とアクセス制限)
# diff -u kibana.yml /etc/kibana/kibana.yml --- kibana.yml 2017-11-11 03:50:42.000000000 +0900 +++ /etc/kibana/kibana.yml 2017-11-22 16:55:49.833624169 +0900 @@ -4,7 +4,7 @@ # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. -#server.host: "localhost" +server.host: "0.0.0.0" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects # the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests @@ -18,7 +18,7 @@ #server.name: "your-hostname" # The URL of the Elasticsearch instance to use for all your queries. -#elasticsearch.url: "http://localhost:9200" +elasticsearch.url: "http://localhost:9200" # When this setting's value is true Kibana uses the hostname specified in the server.host # setting. When the value of this setting is false, Kibana uses the hostname of the host @@ -83,7 +83,7 @@ #pid.file: /var/run/kibana.pid # Enables you specify a file where Kibana stores log output. -#logging.dest: stdout +logging.dest: /var/log/kibana.log # Set the value of this setting to true to suppress all logging output. #logging.silent: false
# systemctl restart kibana
ブラウザで起動:
http://172.20.214.47:5601
logstash のインストール
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
cat << EOF > /etc/yum.repos.d/logstash.repo [logstash-6.x] name=Elastic repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF yum -y install logstash
Logstash でとりあえずログを入れてみる
/etc/logstash/conf.d/messages.conf
input { file { path => "/var/log/messages" start_position => beginning } } filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } } output { elasticsearch { hosts => "localhost" user => "elastic" password => "changeme" index => "logstash-2017-11-22" } }
- Permission がない場合は与える。問題があれば/var/log/logstash/logstash-plain.log を確認
Arcsight Module のインストール
https://www.elastic.co/guide/en/logstash/current/arcsight-module.html
ES_JAVA_OPTS="-Dhttp.proxyHost=proxyurl.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxyurl.com -Dhttps.proxyPort=3128" /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.0.0.zip /usr/share/kibana/bin/kibana-plugin install file:x-pack-6.0.0.zip ES_JAVA_OPTS="-Dhttp.proxyHost=proxyurl.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxyurl.com -Dhttps.proxyPort=3128" /usr/share/logstash/bin/logstash-plugin install x-pack
パスワードの設定
/usr/share/elasticsearch/bin/x-pack/setup-passwords interactive ... Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [elastic]
有効になっているか念のため確認
curl -u elastic:changeme localhost:9200
kibana.yml で下記設定をする
elasticsearch.username: "elastic" elasticsearch.password: "changeme"
起動
- 下記をlogstash のConfig に追加
modules: - name: arcsight var.inputs: "smartconnector" var.input.smartconnector.port: "5000" var.elasticsearch.hosts: "localhost:9200" var.elasticsearch.username: "elastic" var.elasticsearch.password: "changeme" var.kibana.host: "localhost:5601" var.kibana.username: "elastic" var.kibana.password: "changeme"
- logstash の起動
Arcsight Connector 側の設定
- CEF Syslog
- Elasticsearch のIP
- Port=5000
- Protocol=Raw TCP
- Forwarder=false
- CEF Version=0.1
所感
Elasticsearch にArcsightからのログを配送できるということで試してみた。 ArcsightConnector 側で構造化されているので、これで取り込めると検索基盤として色々便利そう。