maetesのブログ

個人用Memoです

ElasticsearchとArcsightModule を試してみた

概要

先日Arcsight ModuleがElsaticsearch から出たということで、取り込めるかどうか試してみた。

環境

Centos7 @AWS環境 & proxy環境
Elasticsearch 6.0

elasticsearch のInstall

yum remove -y java-1.7.0-openjdk
yum install -y java-1.8.0-openjdk-devel
yum install -y java-1.8.0-openjdk-debuginfo --enablerepo=*debug*
# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat << EOF >  /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
sudo yum -y install elasticsearch
systemctl status elasticsearch
systemctl enable elasticsearch
systemctl start elasticsearch
# curl localhost:9200
{
  "name" : "QaYIMOJ",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "b2cZBZkYQ-up7dPS_LTZ1g",
  "version" : {
    "number" : "6.0.0",
    "build_hash" : "8f0685b",
    "build_date" : "2017-11-10T18:41:22.859Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

kibana のInstall

www.elastic.co

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat << EOF > /etc/yum.repos.d/kibana.repo
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF
yum -y install kibana
  • 設定変更(logging とアクセス制限)
# diff -u kibana.yml /etc/kibana/kibana.yml
--- kibana.yml  2017-11-11 03:50:42.000000000 +0900
+++ /etc/kibana/kibana.yml      2017-11-22 16:55:49.833624169 +0900
@@ -4,7 +4,7 @@
 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
 # The default is 'localhost', which usually means remote machines will not be able to connect.
 # To allow connections from remote users, set this parameter to a non-loopback address.
-#server.host: "localhost"
+server.host: "0.0.0.0"

 # Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
 # the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
@@ -18,7 +18,7 @@
 #server.name: "your-hostname"

 # The URL of the Elasticsearch instance to use for all your queries.
-#elasticsearch.url: "http://localhost:9200"
+elasticsearch.url: "http://localhost:9200"

 # When this setting's value is true Kibana uses the hostname specified in the server.host
 # setting. When the value of this setting is false, Kibana uses the hostname of the host
@@ -83,7 +83,7 @@
 #pid.file: /var/run/kibana.pid

 # Enables you specify a file where Kibana stores log output.
-#logging.dest: stdout
+logging.dest: /var/log/kibana.log

 # Set the value of this setting to true to suppress all logging output.
 #logging.silent: false
# systemctl restart kibana

ブラウザで起動:
http://172.20.214.47:5601

logstash のインストール

https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

cat << EOF > /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

yum -y install logstash

Logstash でとりあえずログを入れてみる

/etc/logstash/conf.d/messages.conf

input {
  file {
    path           => "/var/log/messages"
    start_position => beginning
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}


output {
  elasticsearch {
    hosts  => "localhost"
    user   => "elastic"
    password   => "changeme"
    index => "logstash-2017-11-22"
  }
}
  • Permission がない場合は与える。問題があれば/var/log/logstash/logstash-plain.log を確認

Arcsight Module のインストール

https://www.elastic.co/guide/en/logstash/current/arcsight-module.html

ES_JAVA_OPTS="-Dhttp.proxyHost=proxyurl.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxyurl.com -Dhttps.proxyPort=3128" /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack

wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.0.0.zip
/usr/share/kibana/bin/kibana-plugin install file:x-pack-6.0.0.zip


ES_JAVA_OPTS="-Dhttp.proxyHost=proxyurl.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxyurl.com -Dhttps.proxyPort=3128" /usr/share/logstash/bin/logstash-plugin install x-pack

パスワードの設定

/usr/share/elasticsearch/bin/x-pack/setup-passwords interactive
...
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [elastic]

有効になっているか念のため確認

curl -u elastic:changeme localhost:9200

kibana.yml で下記設定をする

elasticsearch.username: "elastic"
elasticsearch.password: "changeme"

起動

  • 下記をlogstash のConfig に追加
modules:
- name: arcsight
  var.inputs: "smartconnector"
  var.input.smartconnector.port: "5000"
  var.elasticsearch.hosts: "localhost:9200"
  var.elasticsearch.username: "elastic"
  var.elasticsearch.password: "changeme"
  var.kibana.host: "localhost:5601"
  var.kibana.username: "elastic"
  var.kibana.password: "changeme"
  • logstash の起動

Arcsight Connector 側の設定

  • CEF Syslog
  • Elasticsearch のIP
  • Port=5000
  • Protocol=Raw TCP
  • Forwarder=false
  • CEF Version=0.1

所感

Elasticsearch にArcsightからのログを配送できるということで試してみた。 ArcsightConnector 側で構造化されているので、これで取り込めると検索基盤として色々便利そう。