out_azure_kusto : fix multiple files tail issue and timeout issue

[tanmaya-panda1]

1. added kusto specific headers
2. randomized kusto ingestion resources refresh interval
3. made kusto ingestion resources refresh interval configurable
4. introduced gzip compression for payload
5. added dynamic parsing of azure kusto ingestion resources
6. fixed deadlock and added granular locks
7. added default kusto endpoints connection timeout interval configs

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[OUTPUT]
name azure_kusto
match *
tenant_id xxxxxxx
client_id xxxxxxx
client_secret xxxxxx
ingestion_endpoint https://ingest-xxxxxx.kusto.windows.net
database_name xxxxx
table_name FluentBit
ingestion_endpoint_connect_timeout 600
compression_enabled On
ingestion_resources_refresh_interval 7200

• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

https://github.com/fluent/fluent-bit-docs/pull/1405/files

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[Ripurwar80]

Hi tanmaya, facing an issue with regards to multiple tail inputs, Hope you can help me:
#8419

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[tanmaya-panda1]

Added documentation for the above changes

https://github.com/fluent/fluent-bit-docs/pull/1405/files

[kpantos]

@edsiper Given that the requested changes have been implemented, is there an ETA on when this pull request will be merged into the main branch?
Customers want to use this in their production clusters and are asking when this will be generally available.

[tanmaya-panda1]

@kpantos Since the PR is not yet merged, we have built custom Docker images from this branch that the customers can use in their environment
ARM image : [https://hub.docker.com/layers/agramachandran/fluent-bit/fix-ARM/images/sha256-f3b9575ab878d7a370364461a3c9c0b4ec250e204a16175863508ce121c9ecba?context=explore]
AMD image : [https://hub.docker.com/layers/agramachandran/fluent-bit/fix-AMD/images/sha256-3a2fe380a48d93a8d53142fd9ce3c26414f4acb48a29a435954557019d64104c?context=explore]

[kpantos]

Thank you @tanmaya-panda1 customer is already using these images.
Having said that they want to make sure this implementation is actually hitting the main branch of the official repository and that this plugin is supported as part of the official product and not a forked individual contribution.

From what I was able to see in this thread most of the reviewed items have been remediated already and there aren't any pending open issues. So, do you have any idea on when this pull request is going to be merged into the main branch?

[lecaros]

hello @tanmaya-panda1 and @kpantos,
This PR is in the queue for the next 3.2 release.

[kpantos]

Thanks @lecaros, really appreciate the response