Scheduler Problems Due to Unknown Error in Watcher

[meneal]

Hi all, we are running Airflow 1.10.12 and the kubernetes executor, and have been seeing a number of watcher failures as in the below excerpt from the scheduler logs.

I've noticed that this behavior is quite similar to kubernetes-client/python#972 which was theoretically fixed in kubernetes-client/python-base#133 which is included in more recent versions of the python kubernetes. The current version in airflow is 11.0.0.

I'm wondering whether upgrading the python kubernetes client would be useful or whether this is expected behavior in airflow? This is coming up in the context of the scheduler container in the airflow pod crashing and not coming back up.

[2020-10-26 15:18:31,971] {kubernetes_executor.py:293} ERROR - Unknown error in KubernetesJobWatcher. Failing
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 696, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 763, in read_chunked
    self._update_chunk_length()
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 700, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 287, in run
    self.worker_uuid, self.kube_config)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 323, in _run
    for event in list_worker_pods():
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 144, in stream
    for line in iter_resp_lines(resp):
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 46, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 792, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Process KubernetesJobWatcher-3:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 696, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 763, in read_chunked
    self._update_chunk_length()
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 700, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 287, in run
    self.worker_uuid, self.kube_config)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 323, in _run
    for event in list_worker_pods():
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 144, in stream
    for line in iter_resp_lines(resp):
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 46, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 792, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
[2020-10-26 15:18:32,539] {kubernetes_executor.py:418} ERROR - Error while health checking kube watcher process. Process died for unknown reasons
[2020-10-26 15:18:32,575] {kubernetes_executor.py:302} INFO - Event: and now my watch begins starting at resource_version: 80654451
[2020-10-26 15:28:32,607] {kubernetes_executor.py:293} ERROR - Unknown error in KubernetesJobWatcher. Failing
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/response.py", line 696, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

[louison]

Hello, I have this similar error on the scheduler, I suspect a permission problem

[Gaste8]

Hi all,
we run into the same problem with kubernetes version 1.27.6 and chart version: airflow-8.9.0
anyone found the solution?
Thanks