Grok配置案例：

成都創(chuàng)新互聯(lián)公司服務(wù)項(xiàng)目包括宿豫網(wǎng)站建設(shè)、宿豫網(wǎng)站制作、宿豫網(wǎng)頁制作以及宿豫網(wǎng)絡(luò)營銷策劃等。多年來，我們專注于互聯(lián)網(wǎng)行業(yè)，利用自身積累的技術(shù)優(yōu)勢(shì)、行業(yè)經(jīng)驗(yàn)、深度合作伙伴關(guān)系等，向廣大中小型企業(yè)、政府機(jī)構(gòu)等提供互聯(lián)網(wǎng)行業(yè)的解決方案，宿豫網(wǎng)站推廣取得了明顯的社會(huì)效益與經(jīng)濟(jì)效益。目前，我們服務(wù)的客戶以成都為中心已經(jīng)輻射到宿豫省份的部分城市，未來相信會(huì)繼續(xù)擴(kuò)大服務(wù)區(qū)域并繼續(xù)獲得客戶的支持與信任！##啟動(dòng)文件配置： #?Sample?Logstash?configuration?for?creating?a?simple #?Beats?->?Logstash?->?Elasticsearch?pipeline. input?{ ??stdin{} } filter?{ grok?{ match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\ %{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"] ???} } output?{ ??stdout{ ????codec?=>?"rubydebug" ??} } ##輸出文件內(nèi)容 172.16.213.132?[07/Feb/2018:16:24:19?+0800]?"GET?/?HTTP/1.1"?403?5039 ##顯示內(nèi)容 { ??????"@version"?=>?"1", ????"@timestamp"?=>?2019-11-10T06:02:42.865Z, ??????????"host"?=>?"localhost.localdomain", ???????"message"?=>?"172.16.213.132?[07/Feb/2018:16:24:19?+0800]?\"GET?/?HTTP/1.1\"?403?5039", ?????"timestamp"?=>?"07/Feb/2018:16:24:19?+0800", ?????????"bytes"?=>?"5039", ??????"response"?=>?"403", ??????"clientip"?=>?"172.16.213.132", ??????"referrer"?=>?"\"GET?/?HTTP/1.1\"" }

Grok 過濾重復(fù)字段

##?配置文件 #?Sample?Logstash?configuration?for?creating?a?simple #?Beats?->?Logstash?->?Elasticsearch?pipeline. input?{ ??stdin{ ?} } filter?{ ??grok?{ ??match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\? ??%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"] ??remove_field?=>?["message"] ???} } output?{ ??stdout{ ??codec?=>?"rubydebug" ??} }

Grok搭配Date時(shí)間插件配置

#?Sample?Logstash?configuration?for?creating?a?simple #?Beats?->?Logstash?->?Elasticsearch?pipeline. input?{ ??stdin{ ??} } filter?{ grok?{ ?match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\? ?%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"] ?remove_field?=>?["message"] ???} date?{ ??match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"] ??} } output?{ ??stdout{ ??codec?=>?"rubydebug" ??} }

Date 過濾重復(fù)得字段配置

#?Sample?Logstash?configuration?for?creating?a?simple #?Beats?->?Logstash?->?Elasticsearch?pipeline. input?{ ??stdin{ ??} } filter?{ ?grok?{ ???match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\? ???%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"] ???remove_field?=>?["message"] ???} date?{ ??match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"] ?? ??} mutate?{ ???remove_field?=>?[?"timestamp"?]?? ??} } output?{ ?stdout{ ??codec?=>?"rubydebug" ??} }

綜合練習(xí)配置參數(shù)

#?Sample?Logstash?configuration?for?creating?a?simple #?Beats?->?Logstash?->?Elasticsearch?pipeline. input?{ ??stdin{ ??} } filter?{ ??grok?{ ???match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\? ???%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"] ???remove_field?=>?["message"] ??} ?date?{ ??match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"]? ??} ?mutate{ ????rename?=>?{"response"?=>?"response_new"} ????gsub?=>?["referrer",?"\"",?""] ????remove_field?=>?[?"timestamp"?] ????split?=>?["clientip",?"."] ??} } output?{ ?stdout{ ??codec?=>?"rubydebug" ??} }

Geoip 地理位置插件操作方式

#?Sample?Logstash?configuration?for?creating?a?simple #?Beats?->?Logstash?->?Elasticsearch?pipeline. input?{ ??stdin{ ??} } filter?{ ????grok?{ ?????match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\? ?????%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"] ?????remove_field?=>?["message"] ???} ???date?{ ????match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"]? ??} ???mutate{ ??????remove_field?=>?[?"timestamp"?] ??} ??geoip?{ ????source?=>?"clientip" ????database?=>?"/usr/local/include/GeoLite2-ASN_20191105/GeoLite2-ASN.mmdb" ???} } output?{ ??stdout{ ????codec?=>?"rubydebug" ??}? }

Geoip輸出指定屬性值

#?Sample?Logstash?configuration?for?creating?a?simple #?Beats?->?Logstash?->?Elasticsearch?pipeline. input?{ ??stdin{ ??} } filter?{ ????grok?{ ?????match?=>?["message","%{IP:clientip}\?\[%{HTTPDATE:timestamp}\]\? ?????%{QS:referrer}\?%{NUMBER:response}\?%{NUMBER:bytes}"] ?????remove_field?=>?["message"] ???} ???date?{ ????match?=>?["timestamp",?"dd/MMMM/yyyy:HH:mm:ss?Z"] ??} ???mutate{ ??????remove_field?=>?[?"timestamp"?] ??} geoip?{ source?=>?"clientip" #database?=>?"/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb" database?=>?"/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb" fields?=>?["city_name",?"region_name",?"country_name",?"ip",?"latitude",?"longitude",?"timezone"] ???} } output?{ ??stdout{ ????codec?=>?"rubydebug" ??} } 模擬數(shù)據(jù)： 36.7.152.182?[07/Feb/2018:16:24:19?+0800]?"GET?/?HTTP/1.1"?403?5039

綜合實(shí)戰(zhàn)

#?Sample?Logstash?configuration?for?creating?a?simple #?Beats?->?Logstash?->?Elasticsearch?pipeline. input?{ ??stdin{} } filter{ grok{ ??match?=>?{"message"?=>?"%{TIMESTAMP_ISO8601:localtime}\|\~\|%{IP:clientip} ??\|\~\|%{GREEDYDATA:http_user_agent}\|\~\|%{GREEDYDATA:url} ??\|\~\|%{GREEDYDATA:mediaid}\|\~\|%{GREEDYDATA:osid}"} ??remove_field?=>?[?"message"?] ???} date?{ ????match?=>?["localtime",?"yyyy-MM-dd'T'HH:mm:ssZZ"] ????target?=>?"@timestamp" ???} mutate?{ ??????remove_field?=>?["localtime"] ???} geoip?{ ?source?=>?"clientip" ?#database?=>?"/usr/local/include/GeoLite2-Country_20191015/GeoLite2-Country.mmdb" ?database?=>?"/usr/local/include/GeoLite2-City_20191105/GeoLite2-City.mmdb" ?fields?=>?["city_name",?"region_name",?"country_name",?"ip",?"latitude",?"longitude",?"timezone"] ??} } output?{ ???stdout?{ ???codec?=>?"rubydebug" ???} } 示例：2018-02-09T10:57:42+08:00|~|123.87.240.97|~|Mozilla/5.0 (iPhone;CPU?iPhone?OS?11_2_2?like?Mac?OS?X) AppleWebKit/604.4.7?Version/11.0?Mobile/15C202?Safari/604.1 |~|http://m.sina.cn/cm/ads_ck_wap.html |~|12434785489009|~|DF45566587855P

另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)cdcxhl.cn，海內(nèi)外云服務(wù)器15元起步，三天無理由+7*72小時(shí)售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢(shì)，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場(chǎng)景需求。

網(wǎng)站標(biāo)題：Logstash基礎(chǔ)操作-Filter-創(chuàng)新互聯(lián)
標(biāo)題URL：http://www.weahome.cn/article/dcepoj.html