培训-ELK日志监控报警实战课件.ppt

1、ELK日志监控报警实战磁云科技张人杰2018.10.1612什么是ELKE:ElasticSearchL:LogstashK:Kibana211/28/2022运行效果当服务器宕机时,立即发送邮件通知311/28/2022环境搭建(一)ElasticSearch安装1、安装elasticsearch的yum源的密钥rpm-import https:/artifacts.elastic.co/GPG-KEY-elasticsearch2、配置elasticsearch的yum源vim/etc/yum.repos.d/elasticsearch.repoelasticsearch-6.xname=

2、Elasticsearch repository for 6.x packagesbaseurl=https:/artifacts.elastic.co/packages/6.x/yumgpgcheck=1gpgkey=https:/artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md3、安装elasticsearchyum install-y elasticsearch 411/28/2022环境搭建(一)ElasticSearch环境搭建1、需要安装jdk1.8版本以上的java-versio

3、n2、创建elasticsearch data的存放目录，并修改该目录的属主属组mkdir-p/data/es-datachown-R elasticsearch:elasticsearch/data/es-data3、修改elasticsearch的日志属主属组chown-R elasticsearch:elasticsearch/var/log/elasticsearch/4、修改elasticsearch的配置文件vim/etc/elasticsearch/elasticsearch.yml 511/28/2022/etc/elasticsearch/elasticsearch.yml编

4、辑找到配置文件中的cluster.name，打开该配置并设置集群名称cluster.name:elk-tang找到配置文件中的node.name，打开该配置并设置节点名称node.name:elk-tang-1修改data存放的路径path.data:/data/es-data修改logs日志的路径path.logs:/var/log/elasticsearch/注释配置内存使用用交换分区#bootstrap.memory_lock:true监听的网络地址network.host:0.0.0.0开启监听的端口http.port:9200增加新的参数，这样head插件可以访问es(5.x版本，如

5、果没有可以自己手动加)http.cors.enabled:truehttp.cors.allow-origin:*611/28/2022启动ElasticSearch/etc/init.d/elasticsearch start 711/28/2022创建开机自启动服务chkconfig elasticsearch on811/28/2022其他需要修改的参数vim/etc/security/limits.conf在末尾追加以下内容（elk为启动用户，当然也可以指定为*）elk soft nofile 65536elk hard nofile 65536elk soft nproc 2048e

6、lk hard nproc 2048elk soft memlock unlimitedelk hard memlock unlimitedvim/etc/security/limits.d/XXX-nproc.conf 将里面的1024改为2048（ES最少要求为2048）*soft nproc 2048vim/etc/elasticsearch/elasticsearch.yml加入以下内容bootstrap.system_call_filter:false911/28/2022再次启动/etc/init.d/elasticsearch restart1011/28/2022环境搭建（二）

7、安装elasticsearch-head插件安装node.jssudo curl-sL-o/etc/yum.repos.d/khara-nodejs.repo https:/copr.fedoraproject.org/coprs/khara/nodejs/repo/epel-7/khara-nodejs-epel-7.reposudo yum install-y nodejs nodejs-npm安装headgit clone git:/ elasticsearch-headnpm installnpm run start1111/28/2022环境搭建（三）安装Logstash环境Logs

8、tash需要安装到产生日志的服务器上rpm-import https:/artifacts.elastic.co/GPG-KEY-elasticsearchyum install-y logstashrpm-ql logstashln-s/usr/share/logstash/bin/logstash/bin/1211/28/2022Logstash配置input file path=/var/log/nginx/access.log start_position=beginning ignore_older=0 filter grok patterns_dir=/opt/logstash/p

9、atterns match=message=%NGINXACCESS add_field=“resp_code”,“%response”geoip source=http_x_forwarded_for target=geoip database=/etc/logstash/GeoLite2-City.mmdb add_field=geoipcoordinates,%geoiplongitude add_field=geoipcoordinates,%geoiplatitude mutate convert=geoipcoordinates,float convert=response,int

10、eger convert=bytes,integer replace=type=nginx_access remove_field=message date match=timestamp,dd/MMM/yyyy:HH:mm:ss Z mutate remove_field=timestamp output elasticsearch hosts=127.0.0.1:9200 index=logstash-nginx-access-%+YYYY.MM.dd stdout codec=rubydebug1311/28/2022建立grok使用的表达式mkdir-pv/opt/logstash/p

11、atternsvi/opt/logstash/patterns/nginxNGUSERNAME a-zA-Z.-+_%+NGUSER%NGUSERNAMENGINXACCESS%IPORHOST:clientip-%NOTSPACE:remote_user%HTTPDATE:timestamp(?:%WORD:verb%NOTSPACE:request HTTP/%NUMBER:httpversion|%DATA:rawrequest)%NUMBER:response(?:%NUMBER:bytes|-)%QS:referrer%QS:agent(?:%IPV4:http_x_forwarde

12、d_for|-)1411/28/2022GeoIP的数据库解析ipwget http:/ GeoLite2-City.tar.gzmv GeoLite2-City_20181030/GeoLite2-City.mmdb/etc/logstash/.1511/28/2022测试配置文件并启动Logstash服务logstash-t-f./elk.confnohup logstash f./elk.conf 2&1 /dev/null&1611/28/2022环境搭建（四）Kibanawget https:/artifacts.elastic.co/downloads/kibana/kibana-

13、6.4.2-linux-x86_64.tar.gz#注意需要与ES对应的版本tar-xzf kibana-6.4.2-linux-x86_64.tar.gzmv kibana-6.4.2-linux-x86_64/usr/localln-s/usr/local/kibana-6.4.2-linux-x86_64/usr/local/kibanavim/usr/local/kibana/config/kibana.yml1711/28/2022/usr/local/kibana/config/kibana.yml编辑server.port:5601server.host:0.0.0.0elast

14、icsearch.url:http:/localhost:9200kibana.index:.kibana 1811/28/2022安装screen,以便于kibana在后台运行yum-y install screenscreen/usr/local/kibana/bin/kibana1911/28/2022Kibana安装完成打开浏览器并设置对应的index http:/localhost:56012011/28/2022在Kibana上安装sentinl插件用于发送邮件提醒1、到https:/ install https:/ GET/HTTP/1.1 200 0-Mozilla/5.0(X11;Linux x86_64;rv:52.0)Gecko/20100101 Firefox/52.0-/var/log/nginx/access.log定期清理ES中的日志文件curl-X DELETE http:/localhost:9200/nginx-*-date+%Y-%m-%d-d-$n days2211/28/2022防火墙配置所有端口仅对内网开放2311/28/2022结束张人杰2018.11.42411/28/2022