Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update self-hosted-with-docker docs #4518

Merged
merged 1 commit into from
Feb 12, 2025

Conversation

fvanzee
Copy link

@fvanzee fvanzee commented Feb 8, 2025

Fix scheduler arguments to prevent "no space left on device" error

Description

Adding the --etcd-data-dir=/var/lock/dapr/scheduler argument for the scheduler service in the docker-compose example.

Edit 20250211: Increase the tmpfs size from 10000 bytes to 64m. Ealier change setting --etcd-data-dir was only a workaround.

Very simple change to resolve dapr/dapr#8207

Issue reference

dapr/dapr#8207

@fvanzee fvanzee requested review from a team as code owners February 8, 2025 15:12
@fvanzee fvanzee force-pushed the self-hosted-scheduler-docs branch from dcd92b1 to f806e49 Compare February 8, 2025 15:14
msfussell
msfussell previously approved these changes Feb 10, 2025
Copy link
Member

@msfussell msfussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@msfussell msfussell dismissed their stale review February 10, 2025 17:26

Need another check

Copy link
Contributor

@cicoyle cicoyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting --etcd-data-dir helps avoid the immediate no space left on device error by redirecting writes to a different location, but it's more of a workaround. If etcd is consuming significant space, the root issue is likely insufficient storage in the container. Have you tried increasing the volume size or mounting a persistent volume with more space?

@fvanzee
Copy link
Author

fvanzee commented Feb 11, 2025

Setting --etcd-data-dir helps avoid the immediate no space left on device error by redirecting writes to a different location, but it's more of a workaround. If etcd is consuming significant space, the root issue is likely insufficient storage in the container. Have you tried increasing the volume size or mounting a persistent volume with more space?

Yes I did also try to increase the tmpfs size (if you are referring to that) to 10x the size described in the doc, which didn't resolve the issue. I will try to mount a persistent volume, and see if that does work.

@fvanzee
Copy link
Author

fvanzee commented Feb 11, 2025

Alright after some experimenting it does look like the tmpfs size was too low.

It's set to 10000 bytes which is apparently too low for etcd.
Through trial and error I noticed it should be al least set to 64m. So I wasn't even close with my 10x experiment earlier.

I'll update the documentation to set the tmpfs size to 64m to this and remove the --etcd-data-dir argument.

Increase tmpfs size to prevent "no space left on device" error

Signed-off-by: fvanzee <[email protected]>
@fvanzee fvanzee force-pushed the self-hosted-scheduler-docs branch from f806e49 to 2cfc002 Compare February 11, 2025 10:45
Copy link
Contributor

@cicoyle cicoyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the time to figure this out - Im fine increasing the size to make sure others don't run out of space as well.

Copy link
Member

@msfussell msfussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hhunter-ms hhunter-ms merged commit dea27c7 into dapr:v1.14 Feb 12, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Scheduler fails to start when started according to docker compose example in the docs
4 participants