使用Filebeat实现日志采集

发表于 分类于 阅读次数： 本文字数： 317 阅读时长 ≈ 1 分钟

使用 Filebeat 实现日志采集

logback 配置输出日志格式

1
%d{yyyy-MM-dd HH:mm:ss.SSS} ${spring:spring.application.name} ${spring:spring.profiles.active} %-5level [%t] %C#%method:%L %msg%n
1
2
3
4
5
6
7
8
9
10
11
2023-06-07 14:07:11.366 service-user dev INFO  [main] com.alibaba.cloud.nacos.refresh.NacosContextRefresher#registerNacosListener:129 [Nacos Config] Listening config: dataId=service-user.yaml, group=DEFAULT_GROUP
2023-06-07 14:07:11.367 service-user dev INFO  [main] com.alibaba.cloud.nacos.refresh.NacosContextRefresher#registerNacosListener:129 [Nacos Config] Listening config: dataId=service-user-dev.yaml, group=DEFAULT_GROUP
2023-06-07 14:09:59.182 ERROR [http-nio-9001-exec-2] org.apache.juli.logging.DirectJDKLog#log:175 Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.ArithmeticException: / by zero] with root cause
java.lang.ArithmeticException: / by zero
	at com.creek.user.api.controller.AuthController.token(AuthController.java:34) ~[classes/:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.22.jar:5.3.22]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.22.jar:5.3.22]

kibana 添加采集管道

管理名称 filebeat_pipeline
处理器 grok
字段 message

1
(?m)%{TIMESTAMP_ISO8601:createTime}%{SPACE}%{NOTSPACE:serviceName}%{SPACE}%{NOTSPACE:profile}%{SPACE}%{LOGLEVEL:level}%{SPACE}\[%{DATA:threadName}\]%{SPACE}%{JAVACLASS:javaClass}#(?<methodName>[^\s]+)%{SPACE}%{GREEDYDATA:message}

image.png

测试管道是否匹配

1
2
3
4
5
6
7
[
  {
    "_source": {
      "message": "2023-06-07 17:17:48.358 service-user dev INFO  [main] com.alibaba.cloud.nacos.registry.NacosServiceRegistry#register:75 nacos registry, DEFAULT_GROUP service-user 192.168.40.113:9001 register finished"
    }
  }
]

Feilbeat 配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
filebeat.inputs:

- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - E:\logs\*\info.log
    - E:\logs\*\*\info.log
  parsers:
    - multiline: # 多行匹配
        pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
        negate: true
        match: after

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml  
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "123456"
  pipeline: "filebeat_pipeline" # 管道名称

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

测试

生成日志文件后查询 kibana

image.png