backup fails due to not enough space for execution

[harshad16]

Describe the bug

2022-09-19 01:33:08,760 1 ERROR thoth.graph_backup_job:111: Error saving the pg dump Command exited with non-zero status code (1): pg_dump: [archiver] could not write to output file: No space left on device
.
Traceback (most recent call last):
File "app.py", line 95, in main
run_command(
File "/opt/app-root/lib64/python3.8/site-packages/thoth/analyzer/command.py", line 107, in run_command
raise CommandError(error_msg, command=command, is_json=is_json)
thoth.analyzer.command.CommandError: Command exited with non-zero status code (1): pg_dump: [archiver] could not write to output file: No space left on device

To Reproduce
Steps to reproduce the behavior:

1. Go to thoth-graph-stage
2. See error in the backup job pod

Expected behavior
Backups the database periodically.

[harshad16]

The procedure to fix this is to utilize persisted volume, instead of in memory of the pod.

• mount a volume to the graph-backup-job
• adjust the application to utilize that mounted volume for backing up the db.

/triage accepted
/sig devsecops
/priority critical-urgent

[VannTen]

Is the dump sent elsewhere after that ? If it is, maybe we can use a emptyDir instead of a persistent volume ?

I don't have good sense of what local space the openshift have and what size the dumb is.

[harshad16]

The dump is uploaded to s3 storage.
yes, we can use the emptydir, as we had pv storage so utilized that.
the dumb size is currently reached 15gb.
and node size on which emptydir depends varies from cluster to cluster.

[harshad16]

This is completed, it is synced and working.
https://console-openshift-console.apps.ocp4.prod.psi.redhat.com/k8s/ns/thoth-graph-stage/cronjobs/graph-backup/jobs
closing the issue
Thanks everyone for working on it. 💯