How to send logs to a particular Splunk index using Fluent Bit

Having logs for an application is good to identify errors, threats, profiling and many more critical facts. That is why we are fond of Fluent Bit. If we can analyze, query and visualize those data in a meaningful manner, then it will be more productive. That’s where you get Splunk. In this post, I am mainly concentrate on how to have these two magnificent tools link in order to get maximum outcome.

Like in all the other posts, I’ll mention the environment that I have tested this setup.

• Splunk Enterprise : Version 8.0.1
• Fluent Bit : Version 1.6.10
• Fluent Bit running OS version : Ubuntu 20.04.1 LTS

Splunk is a software for searching, monitoring, and analyzing machine-generated big data via a Web-style interface. First we will take a look at the configurations that we have to do in Splunk side. I hope the reader have a basic understanding about the Splunk related terminologies that I am using here. Let’s start with creating an Index in Splunk.

• Log in to your Splunk Enterprise application and click on the Settings menu item in the upper right hand side where you can find the Indexes option.

• Click on the New Index button at upper right hand corner, where you get a pop up window like below.

• Provide a name to your Index and keep other fields as it is if your are a naive in Splunk. Otherwise you can customize those values according to your need but that part is not relevant to the purpose that we are doing in this post.
• After saving above pop up, we need to create a HEC (HTTP Event Collector) token.
• To navigate to that window, click on the Setting like earlier and then select Data inputs.

• Click on the New Token button in upper right hand corner where you will get four steps to fill data.

first : provide a name

second : select the created index

third : review settings

fourth : complete and copy the token value

Now we are good to go with Fluent Bit related changes. It’s all about the configuration file.

[INPUT]
    Name    dummy
[FILTER]
    Name    nest
    Match   *
    Operation   nest
    Wildcard    *
    Nest_under  event
[FILTER]
    Name      modify
    Match     *
    Add index flb_index
[OUTPUT]
    Name        splunk
    Match       *
    Host        <your splunk server ip goes here>
    Port        8088
    TLS         Off
    TLS.Verify  Off
    Splunk_Token <your splunk HEC token goes here>
    Splunk_Send_Raw On

I am not going to describe the configuration parameter included here, because it has been clearly explained in the Fluent Bit documentation.

Once you run the Fluent Bit configuration file, you will be able to query the index from Splunk application.

The results will be something like below.

Yes, it is straight forward and easy as that :)