如何通过gelf将MDC传输到graylog？

Question

我们有一个日志记录流程，在这个流程中我们的Java应用程序填充了一个MDC，通过log4j2传输到syslog并发送到一个集中式的rsyslog安装程序。在这里我们广泛使用MDC。因此，我们的设置如下：

   <Syslog name="syslog" format="RFC5424" host="localhost" port="514" protocol="UDP"
        appName="messaging_platform.${application}" mdcId="mdc" includeMDC="true" facility="LOCAL5" connectTimeoutMillis="100" ignoreExceptions="false">
    <LoggerFields>
        <KeyValuePair key="class" value="%c"/>
        <KeyValuePair key="classname" value="%c{1}"/>
        <KeyValuePair key="exception" value="%ex{full}"/>
        <KeyValuePair key="method" value="%method"/>
        <KeyValuePair key="line" value="%line"/>
        <KeyValuePair key="application_name" value="${application}"/>
        <KeyValuePair key="sequenceNumber" value="%sequenceNumber"/>
        <KeyValuePair key="application_version" value="${application.version}"/>
        <KeyValuePair key="marker" value="%marker"/>
        <KeyValuePair key="thread" value="%thread"/>
        <KeyValuePair key="system_nano_time" value="%nano"/>
        <KeyValuePair key="app_uptime" value="%relative"/>
    </LoggerFields>
</Syslog>

我正在尝试将这个转换为使用Graylog，而GELF似乎是推荐的传输协议。我已经找到多个库来完成这个任务，并从log4j2的内置开始。但是它不支持。

那么如何推荐将这些字段传输到Graylog？如果我这样做

   <Socket name="Graylog" protocol="udp" host="localhost" port="12201">
        <GelfLayout host="localhost" compressionType="GZIP" compressionThreshold="1024">
            <KeyValuePair key="class" value="%c"/>
            <KeyValuePair key="classname" value="%c{1}"/>
            <KeyValuePair key="exception" value="%ex{full}"/>
            <KeyValuePair key="method" value="%method"/>
            <KeyValuePair key="line" value="%line"/>
            <KeyValuePair key="application_name" value="${application}"/>
            <KeyValuePair key="sequenceNumber" value="%sequenceNumber"/>
            <KeyValuePair key="application_version" value="${application.version}"/>
            <KeyValuePair key="marker" value="%marker"/>
            <KeyValuePair key="thread" value="%thread"/>
            <KeyValuePair key="system_nano_time" value="%nano"/>
            <KeyValuePair key="app_uptime" value="%relative"/>
        </GelfLayout>
    </Socket>

我在Graylog中获取了一堆字段，但是值没有展开。

- Martin Schröder

Answer 1

我使用logstash-gelf找到了一个解决方案：

   <Gelf name="gelf" host="udp:localhost" port="12201" version="1.1" extractStackTrace="true"
          filterStackTrace="true" mdcProfiling="true" includeFullMdc="true" maximumMessageSize="8192"
          originHost="%host{fqdn}">
        <Field name="class" pattern="%c"/>
        <Field name="classname" pattern="%c{1}"/>
        <Field name="exception" pattern="%ex{full}"/>
        <Field name="method" pattern="%method"/>
        <Field name="line" pattern="%line"/>
        <Field name="application_name" pattern="${application}"/>
        <Field name="sequenceNumber" pattern="%sequenceNumber"/>
        <Field name="application_version" pattern="${application.version}"/>
        <Field name="marker" pattern="%marker"/>
        <Field name="thread" pattern="%thread"/>
        <Field name="system_nano_time" pattern="%nano"/>
        <Field name="app_uptime" pattern="%relative"/>
        <Field name="severity" pattern="%level{WARN=Warning, DEBUG=Debug, ERROR=Error, TRACE=Trace, INFO=Info}"/>
    </Gelf>

- Martin Schröder

---

请注意，使用includeFullMdc="true"会默认记录所有字段，包括明文密码。最好明确定义您希望记录的字段。 - Greg

这对我没有起作用 - 我复制了整个配置并运行了最简单的程序：try (CloseableThreadContext.Instance c = CloseableThreadContext.push("pushed message").put("pushed key", "pushed value")){ LogManager.getLogger().warn("warn level message seeing if it goes into graylog."); }然后当我去Graylog时，我看不到关于上下文映射或上下文堆栈的任何内容。我做错了什么吗？ - zelinka

Answer 2

我使用logstash-gelf和以下的logback配置;

logback.xml配置:

<appender name="NameOfAppender" class="biz.paluch.logging.gelf.logback.GelfLogbackAppender">
    <host>udp:10.123.33.41</host> <!-- graylog endpoint ip here -->
    <port>12201</port> <!-- graylog endpoint port here -->
    <version>1.1</version>
    <facility>java-prod</facility>
    <extractStackTrace>true</extractStackTrace>
    <filterStackTrace>true</filterStackTrace>
    <mdcProfiling>true</mdcProfiling>
    <timestampPattern>yyyy-MM-dd HH:mm:ss,SSSS</timestampPattern>
    <maximumMessageSize>8192</maximumMessageSize>

    <additionalFields>Env=prod,Operation=INTG_PERF</additionalFields>

    <!-- This are fields using MDC -->
    <mdcFields>RequestTime,ResponseTime,ElapsedTime,ServiceType,ServiceIdentifier,Endpoint,Method,Status</mdcFields>

    <dynamicMdcFields>mdc.*,(mdc|MDC)fields</dynamicMdcFields>

    <includeFullMdc>false</includeFullMdc>
    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
        <level>INFO</level>
    </filter>
</appender>

<logger name="NameOfLogger" level="INFO">
    <appender-ref ref="NameOfAppender" />
</logger> 

Java日志示例：

    org.slf4j.Logger logger = LoggerFactory.getLogger("NameOfLogger");

    MDC.put("RequestTime", new Date().toString());
    MDC.put("ResponseTime", new Date().toString());
    MDC.put("ElapsedTime", "232");
    MDC.put("ServiceType", "BANK");
    MDC.put("ServiceIdentifier", "XBANK");
    MDC.put("Endpoint", "https://www..");
    MDC.put("Method", "doSale");
    MDC.put("Status", "SUCCEED");

    logger.info("WS INTEGRATION PERF");

maven 依赖：

    <dependency>
        <groupId>biz.paluch.logging</groupId>
        <artifactId>logstash-gelf</artifactId>
        <version>1.11.1</version>
    </dependency>

- Murat