title

menuTitle

description

weight

killercoda

Introduction to ingesting logs into Loki using Fluentd and Fluent Bit

Configuring Fluentd and Fluent bit to send logs to Loki.

250

title

description

backend

preprocessing

Introduction to ingesting logs into Loki using Fluentd and Fluent Bit

Configuring Fluentd and Fluent bit to send logs to Loki.

imageid
ubuntu

substitutions

regexp	replacement
loki-fundamentals-([^-]+)-	loki-fundamentals_${1}_

"Introduction to ingesting logs into Loki using Fluentd and Fluent Bit" Sandbox Enviroment

Sandbox Overview

This online sandbox is part of the "Introduction to ingesting logs into Loki using Fluentd and Fluent Bit" video. In this sandbox, you will learn how to ingest logs into Loki using Fluentd and Fluent Bit. The sandbox environment consists of the following components:

Loki: Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.
Fluentd: Fluentd is an open-source data collector for unified logging layer. It allows you to unify data collection and consumption for better use and understanding of data.
Fluent Bit: Fluent Bit is a fast and lightweight log processor and forwarder for Linux, OSX, and BSD family operating systems. It has been designed to be the perfect companion for cloud-native environments.
Carnivorous Greenhouse: Carnivorous Greenhouse is a simple application that generates logs. This allows users to create an account, login and collect metrics from a series of hungry plants. All of these actions generate logs that are sent to Loki. The application can also be toggled to generate errors.
Grafana: Ofcourse the architecture would not be complete without Grafana. Grafana will be used to visualize the logs generated by the Carnivorous Greenhouse application. We will be making use of a new feature in Grafana 11 called Explore Logs.

Reporting Issues

If you encounter any issues with the environment, please report them to the GitHub repository

Step 1: Environment setup

In this step, we will set up our environment by cloning the repository that contains our demo application and spinning up our observability stack using Docker Compose.

To get started, clone the repository that contains our demo application:

git clone -b microservice-fluentd-fluentbit  https://github.com/grafana/loki-fundamentals.git

Next we will spin up our observability stack using Docker Compose:

docker-compose -f loki-fundamentals/docker-compose.yml up -d

This will spin up the following services:

✔ Container loki-fundamentals-grafana-1  Started                                                        
✔ Container loki-fundamentals-loki-1     Started                        
✔ Container loki-fundamentals-fluentd-1  Started
✔ Container loki-fundamentals-fluent-bit-1 Started

Step 2: Configure fluentd to send logs to Loki

In this step, we will configure Fluentd to send logs to Loki.

Note: Killercoda has an inbuilt Code editor which can be accessed via the Editor tab.

Source Configuration

First, we need to configure Fluentd to recive logs from the Carnivorous Greenhouse application. The carniverous greenhouse application is configured to send logs to Fluentd via the fluent python library. To recive these logs, we need to configure a source in Fluentd to listen for logs on port 24224. The type of source we will use is forward.

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

Copy and paste the above configuration into the fluentd.conf file located at loki-fundamentals/fluentd.conf.

Output Configuration

Next, we need to configure Fluentd to send logs to Loki. We will use the loki output plugin to send logs to Loki. The loki output plugin requires the following configuration:

<match service.**>
  @type loki
  url "http://loki:3100"
  extra_labels {"agent": "fluentd"}
  <label>
   service_name $.service
   instance_id $.instance_id
  </label>
  <buffer>
    flush_interval 10s
    flush_at_shutdown true
    chunk_limit_size 1m
  </buffer>
</match>

Copy and paste the above configuration into the fluentd.conf file located at loki-fundamentals/fluentd.conf. This configuration has the following properties:

@type: The type of output plugin to use. In this case, we are using the loki output plugin.
url: The URL of the Loki instance to send logs to.
extra_labels: Additional labels to add to the log stream.
label: The labels to use for the log stream.
buffer: The buffer configuration for the output plugin. Note that the service.** in the match directive is a tag that will match all logs with the tag service. We set the tag in the Carnivorous Greenhouse application to service.<service_name>.

Restart the Fluentd Container

After configuring Fluentd, we need to restart the Fluentd container to apply the changes. To restart the Fluentd container, run the following command:

docker restart loki-fundamentals-fluentd-1

Stuck? Need help?

If you get stuck or need help creating the configuration, you can copy and replace the entire fluentd.conf using the completed configuration file:

cp loki-fundamentals/completed/fluentd.conf loki-fundamentals/fluentd.conf
docker restart loki-fundamentals-fluentd-1

Step 3: Configure Fluent Bit to send logs to Loki

In this step, we will configure Fluent Bit to send logs to Loki.

Note: Killercoda has an inbuilt Code editor which can be accessed via the Editor tab.

Source Configuration

First, we need to configure Fluent Bit to recive logs from the Carnivorous Greenhouse application. Like Fluentd The carniverous greenhouse application is configured to send logs to Fluent Bit via the fluent python library. To recive these logs, we need to configure a source in Fluent bit to listen for logs on port 24224. The type of source we will use is forward. Although we will use the forward input plugin once again the config is slightly different.

[INPUT]
    Name              forward
    Listen            0.0.0.0
    Port              24224

Copy and paste the above configuration into the fluent-bit.conf file located at loki-fundamentals/fluent-bit.conf.

Output Configuration

Next, we need to configure Fluent Bit to send logs to Loki. There are two ways to send logs to Loki using Fluent Bit. The first is to use the loki output plugin and the second is to use the grafana-loki output plugin. We recommend using the grafana-loki output plugin as it is more feature-rich and actively maintained by the community. The grafana-loki output plugin requires the following configuration:

[OUTPUT]
    Name              grafana-loki
    Match             service.**
    Url               http://loki:3100/loki/api/v1/push
    Labels            {agent="fluent-bit"}
    LabelMapPath      /fluent-bit/etc/conf/logmap.json

Copy and paste the above configuration into the fluent-bit.conf file located at loki-fundamentals/fluent-bit.conf. This configuration has the following properties:

Name: The name of the output plugin to use. In this case, we are using the grafana-loki output plugin.
Match: The tag to match logs with. In this case, we are matching logs with the tag service.**.
Url: The URL of the Loki instance to send logs to.
Labels: Additional labels to add to the log stream.
LabelMapPath: The path to the label map file. The label map file is used to map log fields to labels in Loki.

Lets quickly talk about LabelMapPath and logmap.json. The LabelMapPath is a path to a file that contains a mapping of log fields to labels in Loki. The logmap.json file is used to map log fields to labels in Loki. The logmap.json file should look like this:

{
"service": "service_name",
"instance_id": "instance_id"
 }

The logmap.json file maps the service field in the log to the service_name label in Loki and the instance_id field in the log to the instance_id label in Loki.

Restart the Fluent Bit Container

After configuring Fluent Bit, we need to restart the Fluent Bit container to apply the changes. To restart the Fluent Bit container, run the following command:

docker restart loki-fundamentals-fluent-bit-1

Stuck? Need help?

If you get stuck or need help creating the configuration, you can copy and replace the entire fluent-bit.conf using the completed configuration file:

cp loki-fundamentals/completed/fluent-bit.conf loki-fundamentals/fluent-bit.conf
docker restart loki-fundamentals-fluent-bit-1

Step 4: Start the Carnivorous Greenhouse

In this step, we will start the Carnivorous Greenhouse application. To start the application, run the following command:

Note: This docker-compose file relies on the loki-fundamentals_loki docker network. If you have not started the observability stack, you will need to start it first.

docker-compose -f loki-fundamentals/greenhouse/docker-compose-micro.yml up -d --build

This will start the following services:

 ✔ Container greenhouse-db-1                 Started                                                         
 ✔ Container greenhouse-websocket_service-1  Started 
 ✔ Container greenhouse-bug_service-1        Started
 ✔ Container greenhouse-user_service-1       Started
 ✔ Container greenhouse-plant_service-1      Started
 ✔ Container greenhouse-simulation_service-1 Started
 ✔ Container greenhouse-main_app-1           Started

Once started, you can access the Carnivorous Greenhouse application at http://localhost:5005. Generate some logs by interacting with the application in the following ways:

Create a user
Log in
Create a few plants to monitor
Enable bug mode to activate the bug service. This will cause services to fail and generate additional logs.

Finally to view the logs in Loki, navigate to the Loki Logs Explore view in Grafana at http://localhost:3000/a/grafana-lokiexplore-app/explore.

Summary

Congratulations! You have completed the Introduction to Ingesting Logs with Loki (Otel) scenario.

You have successfully learned how to ingest logs into Loki using the OpenTelemetry Collector.

What's Next?

Head back to the video to continue with Introduction to Ingesting Logs using OpenTelemetry with Loki.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

intro-to-logging-fluentd-fluent-bit.md

intro-to-logging-fluentd-fluent-bit.md

"Introduction to ingesting logs into Loki using Fluentd and Fluent Bit" Sandbox Enviroment

Sandbox Overview

Reporting Issues

Step 1: Environment setup

Step 2: Configure fluentd to send logs to Loki

Source Configuration

Output Configuration

Restart the Fluentd Container

Stuck? Need help?

Step 3: Configure Fluent Bit to send logs to Loki

Source Configuration

Output Configuration

Restart the Fluent Bit Container

Stuck? Need help?

Step 4: Start the Carnivorous Greenhouse

Summary

Congratulations! You have completed the Introduction to Ingesting Logs with Loki (Otel) scenario.

What's Next?

Files

intro-to-logging-fluentd-fluent-bit.md

Latest commit

History

intro-to-logging-fluentd-fluent-bit.md

File metadata and controls

"Introduction to ingesting logs into Loki using Fluentd and Fluent Bit" Sandbox Enviroment

Sandbox Overview

Reporting Issues

Step 1: Environment setup

Step 2: Configure fluentd to send logs to Loki

Source Configuration

Output Configuration

Restart the Fluentd Container

Stuck? Need help?

Step 3: Configure Fluent Bit to send logs to Loki

Source Configuration

Output Configuration

Restart the Fluent Bit Container

Stuck? Need help?

Step 4: Start the Carnivorous Greenhouse

Summary

Congratulations! You have completed the Introduction to Ingesting Logs with Loki (Otel) scenario.

What's Next?