Tale of the Fluent Bit INPUT tail

This is the story about practical usage of Fluent Bit tail input plugin. I know the first question that comes to your mind “Don’t we have an excellent official documentation on this?”. Yes, you do. Here is the link. But I’ll guarantee this post will explain the content in more simpler and clear manner.

Before going in to the configuration, let me tell you about the versions and the environment that I have tried out this tests.

• Fluent Bit version : 1.6.10

fluent-bit — version

• OS version : Ubuntu 20.04.1 LTS

cat /etc/os-release

Fluent Bit tail plugin is like tail command that you encounter in Unix, Unix-like systems, FreeDOS and MSX-DOS. Therefore in order to begin, we need a file to read. We can use a default system file that get written timely. But I prefer to create our own file, even though that is out of the scope of the topic. Below shell script will read a file an write to another file in every second delay. By doing that, we have more control of what has written and what missed.

#!/bin/bash
: > OutputFile.txt
input="my_log_file.log"
while IFS= read -r line
do
  echo "$line" >> OutputFile.txt
  sleep 1
done < "$input"

Let’s begin the journey with the simplest configuration file. I have named the above script as ReadWrite.sh and content of my working directory as below.

my_log_file.log
example.conf
ReadWrite.sh

First I ran the script to write to the file.

./ReadWrite.sh

Then I executed the Fluent Bit configuration file.

fluent-bit -c example.conf

My configuration file is looks like below.

[INPUT]
    Name    tail
    Path    ./OutputFile.txt
[OUTPUT]
    Name   stdout
    Match  *

If you are coping and pasting, please make sure you have indented the code properly. Otherwise you will get “Invalid indentation level” error. Above code is the simplest configuration you can have with tail input plugin. The output will be something like below.

[3] tail.0: [1609993104.104741564, {"log"=>"some text here"}]

In more generalized form

[number] tail.0: [Unix epoch time, {"log"=>"line content"}]

Now we will try to modify the configuration file little bit.

[INPUT]
    Name    tail
    Path    ./OutputFile.txt
    Tag     mytag

Then the output will be like below

[number] mytag: [Unix epoch time, {"log"=>"line content"}]

What has happened was tail.0 was replaced by the tag we have introduced.

Before explaining a another configuration parameter, I’ll ask you to try following scenario.

• Start the script that write to a file.
• Then run the fluent-bit with above configuration file
• Stop the fluent-bit process nearly a minute and rerun the fluent-bit
• Note down the last log written in the first run of fluent-bit and first log line written in the rerun of fluent-bit session.
• Compare it with your original log file where your shell script is reading the logs. You can see, that fluent-bit has missed some log lines during the time it was not running.

What happen is, with only the above configuration parameters, fluent-bit is reading the log line that written after the fluent-bit process has started. It doesn’t keep a track of the last read line of the file. In order to address this issue, they have introduced a parameter call DB. You can update your configuration file like below.

[INPUT]
    Name    tail
    Path    ./OutputFile.txt
    Tag     mytag
    DB      ./file_status.db

You will be able to see a database file has been created in the same location where your configuration file locates. You can open that file using DB Browser for SQLite app.

The offset value is the place where fluent-bit has last read in the OutputFile.txt document. You can compare the last line written in the fluent-bit log and open the OutputFile.txt from using vim command and run below command. (Ex. :goto 1335)

:goto offset_value

If you restart the fluent-bit service, then it will read from the very next line, which will solve the issue we have described earlier.

The next scenario that I’m going to explain is little bit complex to recreate. In a real world scenario, if we try to add another log file with some content already written, whether fluent-bit able to read that file from the beginning or not. Follow below steps meaningfully,

• Edit the fluent-bit configuration file as below.

[INPUT]
    Name    tail
    Path    ./OutputFile*.txt
    Tag     mytag
    DB      ./file_status.db

What I have changed is the Path configuration parameter in order to read all the files that matches the given pattern.

• I assume the database file created above is already there, where it has the last read location of OutputFile.txt
• Start ReadWrite2.sh which writes to OutputFile2.txt. Content of ReadWrite2.sh is almost similar as the ReadWrite.sh that I have provided at the very beginning. Except I have changed the output file name and added a prefix for the log lines that writes to that file in order to separate those in fluent-bit logs. For your convenience, I have highlight the changes below.

#!/bin/bash
: > OutputFile2.txt
input="my_log_file.log"
while IFS= read -r line
do
  echo "2 $line" >> OutputFile2.txt
  sleep 1
done < "$input"

• After few seconds (nearly 10s) stop the above process.
• Next, start ReadWrite.sh which writes to OutputFile.txt
• Start fluent-bit service
• Modify the ReadWrite2.sh in order to identify new lines that writing next.

echo "2 2 $line" >> OutputFile2.txt

• Restart ReadWrite2.sh
• Check whether fluent-bit has read the very first line of the OutputFile2.txt

What you able to see is, fluent-bit has started reading from the new line written after fluent-bit process has started. To solve that issue, Fluent Bit has used the Read_from_Head configuration parameter. Let’s modify our configuration file like below.

[INPUT]
    Name    tail
    Path    ./OutputFile*.txt
    Tag     mytag
    DB      ./file_status.db
    Read_from_Head  on

Okay, now you have the solution for that issue as well. But now this post became lengthier than I expected. That is because I have shown you creating the environment that you can test by yourself. Will meet on a new post.