log4j-1.2 + ELK部署

kionf

log4j-1.2 + logstash-5.0.1 + elasticsearch-5.6.0 + kibana-5.6.0

0x00 下载ELK

下载 :

1) logstash-5.0.1
2) kibana-5.6.0
3) elasticsearch-5.6.0
4) 安装jdk1.8 curl https://raw.githubusercontent.com/kalivim/Linux_shell/master/jdk-1.8.112.sh|sh

下载完成后解压相应文件,脚本安装jdk在/data/software/下

1
2
3
export JAVA_HOME=/data/jdk1.8.0_112
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

0x01 配置log4j

首先配置程序log4j配置文件log4j.properties,添加如下配置

1
2
3
4
5
6
7
8
9
10
11
#在rootLogger中添加logstash
log4j.rootLogger=info,all,logstash
#logstach
log4j.appender.logstash=org.apache.log4j.net.SocketAppender
#logstash监听端口,默认4560
log4j.appender.logstash.Port=4560
#logstash主机ip/host
log4j.appender.logstash.RemoteHost=ops
log4j.appender.logstash.ReconnectionDelay=60000
log4j.appender.logstash.LocationInfo=true

0x02 配置logstash

编辑配置文件,log4j.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
vim logstash-5.0.1/log4j.cfg
#内容
input {
log4j {
host => "172.16.0.12"
#监听地址
port => 4560
#监听端口
}
}
filter {
#过滤错误类型删除掉INFO类型错误
if [priority] == "INFO" {
drop {}
} #else if [method] == "checkSqlShardName" {
#drop {}
#}
}
output {
#输出到控制台(调试)
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["127.0.0.1:9200"]
#elasticsearch地址
index => "game-%{+YYYY.MM.dd}"
#建立索引
flush_size => 1000
}
}

注意这里有一个小bug,在input中使用log4j接收不到日志,使用tcp可以接收。这里需要改下logstash源码文件

vim logstash-core/lib/jars.rb

注释掉第二行#require_jar('org.apache.logging.log4j', 'log4j-1.2-api', '2.6.2')

注 :可以把output中的elasticsearch,注释掉,启动java程序测试下logstash控制台中是否输出日志,调试启动./bin/logstash -f log4j.cfg

0x03 配置elasticsearch

root直接启动 elasticsearch 报错如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[2017-09-18T08:51:35,959][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.0.jar:5.6.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:106) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:195) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.0.jar:5.6.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.0.jar:5.6.0]
... 6 more

elasticsearch不允许root用户运行,首先创建普通用户,设置权限

1
2
3
useradd elk
chown -R elk:elk ./elasticsearch-5.6.0

配置文件config/elasticsearch.yml第55行和59行是监听地址和端口,config/jvm.optionsjvm内存配置文件,根据需求配置

切换到普通用户elk启动elasticsearch

1
2
3
4
su - elk
cd /data/elk/elasticsearch-5.6.0
./bin/elasticsearch -d
#-d选项后台运行

0x04 配置nginx加固kibana安全

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
server {
listen 8000;
server_name localhost;
auth_basic "ELK Log Auth";
#密码文件
auth_basic_user_file /data/nginx/kibana.passwd;
#配置ssl
ssl on;
ssl_certificate /data/nginx/all.crt;
ssl_certificate_key /data/nginx/server.key;
location / {
proxy_pass http://127.0.0.1:5601;
proxy_redirect off;
}
}

1. 生成nginx认证文件

如果提示没用htpasswd就装个apache,根据提示输入两次密码就ok

1
2
3
4
htpasswd -c /data/nginx/kibana.passwd username
#> New password:
#> Re-type new password:
#> Adding password for user username

2. 生成ssl证书

1
2
3
4
5
openssl genrsa -des3 -out server.key 1024
openssl rsa -in server.key -out server1.key
openssl req -new -key server1.key -out all.csr
#会提示让输入各种信息,最后会让输入密码,如果设置启动nginx会让输入密码。
openssl x509 -req -days 365 -in all.csr -signkey server1.key -out all.crt

3. 启动nginx

1
./sbin/nginx

0x05 启动elk

使用screen 启动kibana、logstash

启动logstash:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#创建名为logstash的screen后台进程
screen -dmS logstash
#查看screen进程
screen -ls
There are screens on:
19654.logstash (Detached)
#进入会话
screen -r logstash
#启动logstash
cd /data/elk/logstash-5.0.1
./bin/logstash -f log4j.cfg
#后台运行
Ctrl + a d

启动kibana:

1
2
3
4
5
screen -dmS kibana
screen -r kibana
cd /data/elk/kibana-5.6.0-linux-x86_64
./bin/kibana
Ctrl + a + d

kibana添加索引game-(在logstash配置文件中)

kionf

自定义Visualize,显示到dashboard

kionf

0x06 这只是开始。。。。

进门
摸爬滚打