bug: fluent-bit process once again hangs sometimes after being restarted

[jjsiv]

Describe the issue

Some time ago the fluentbit-watcher has been reworked to utilise the hot-reload feature
90d364b

This also meant removal of the SIGKILL call when the process is hanging. And so the issue that I initially reported in #510 has been reintroduced.

This is something that ideally would be fixed in fluent-bit itself (and I will report it there as well once I investigate this problem more in-depth and can reproduce it consistently...), but in the meantime I think it would be great to have handling for these situations reintroduced in fluent-operator.

To Reproduce

No clear steps to reproduce. Seems to happen when fluent-bit is restarted many times in a row, but not always

Expected behavior

Fluent-bit is restarted and works

Your Environment

- Fluent Operator version:
- Container Runtime:
- Operating system:
- Kernel version:

How did you install fluent operator?

No response

Additional context

Keeping this as somewhat of a remainder go get back to this after 18.11 or so