-
Notifications
You must be signed in to change notification settings - Fork 250
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Load average values are not updated(freezes) #504
Comments
@brauner thoughts? |
This is with |
The failures to open these individual task directories is probably due to the fact that the threads exit to fast, i.e. by the time the |
Would you be ok with sharing the cpu layout, i.e. how many sockets, threads etc? |
I would think that loadavg could get problematic with increasing number of container as it uses a lot of locking. |
host loadavg:
|
Yes.
|
Hi @exsver ! Really sorry for a long delay with reply on this. Unfortunately, I can't remember any related fixes in LXCFS which can address your specific problem. I would recommend to try trace LXCFS process with |
All load average values(from /proc/loadavg) are not updated(freezes) inside containers for a long time(from a few minutes to a few hours).
LA is not working for all containers simultaneously.
one container:
another container:
grafana null options:
This issue is reproducible for a few of the same servers with 250+ containers.
Servers that run less than 250 containers work fine.
All servers have the same hardware and software.
Versions:
OS:Ubuntu 20.04.3 LTS
Kernel: 5.4.0-80-generic
LXC version "4.0.11"
LXCFS version "4.0.11"
LXD version "4.20"
Containers RUNNING: 337
Syslog
2021-12-03T03:26:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/798922/task"
2021-12-03T03:26:34+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/802521/task"
2021-12-03T03:26:34+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/802718/task"
2021-12-03T03:29:44+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/821922/task"
2021-12-03T03:30:07+02:00 lxcfs[826340]: utils.c: 290: send_creds: Invalid argument - Failed getting reply from server over socketpair: 2
2021-12-03T03:30:12+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T03:44:11+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T03:44:11+02:00 lxcfs[907178]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T03:46:08+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T03:46:08+02:00 lxcfs[918745]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T04:41:49+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1226596/task"
2021-12-03T04:42:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1227974/task"
2021-12-03T04:42:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1227977/task"
2021-12-03T04:42:59+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1231726/task"
2021-12-03T04:43:09+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1233693/task"
2021-12-03T04:43:09+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1233695/task"
2021-12-03T04:45:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1242709/task"
2021-12-03T04:45:09+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1246664/task"
2021-12-03T04:45:09+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1246665/task"
2021-12-03T04:46:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1250515/task"
2021-12-03T04:46:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1250516/task"
2021-12-03T04:46:19+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1249000/task"
2021-12-03T04:46:49+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1254582/task"
2021-12-03T04:46:54+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1255380/task"
2021-12-03T04:47:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1256138/task"
2021-12-03T04:47:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1256139/task"
2021-12-03T04:49:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1266240/task"
2021-12-03T04:50:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1271153/task"
2021-12-03T04:53:04+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T04:53:04+02:00 lxcfs[1285327]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T04:56:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1298636/task"
2021-12-03T04:56:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1298911/task"
2021-12-03T04:56:29+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1302222/task"
2021-12-03T04:57:10+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1306115/task"
2021-12-03T05:00:11+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T05:00:11+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T05:00:16+02:00 lxcfs[1323631]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T05:00:16+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1321932/task"
2021-12-03T05:00:16+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1321936/task"
2021-12-03T05:00:16+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1323888/task"
2021-12-03T05:00:16+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1323905/task"
2021-12-03T05:00:16+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1323909/task"
2021-12-03T05:05:28+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T05:05:28+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T05:05:32+02:00 lxcfs[1354204]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T05:05:32+02:00 lxcfs[1354202]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T05:08:54+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T06:14:05+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1730959/task"
2021-12-03T06:14:50+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1734712/task"
2021-12-03T06:14:50+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1734713/task"
2021-12-03T06:14:50+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1734714/task"
2021-12-03T06:17:20+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1749744/task"
2021-12-03T06:17:20+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1749745/task"
2021-12-03T06:18:17+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1753984/task"
2021-12-03T06:19:17+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1758510/task"
2021-12-03T06:20:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1760424/task"
2021-12-03T06:20:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1761060/task"
2021-12-03T06:21:14+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1768610/task"
2021-12-03T06:24:29+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/1783366/task"
2021-12-03T06:26:03+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T06:27:23+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T06:27:23+02:00 lxcfs[1796297]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T07:08:10+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:08:21+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:09:00+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:13:03+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:13:04+02:00 lxcfs[2025289]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T07:14:13+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:14:14+02:00 lxcfs[2030940]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T07:30:04+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:30:04+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:30:12+02:00 lxcfs[2108545]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T07:31:45+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:31:45+02:00 lxcfs[2116207]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T07:35:04+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:35:04+02:00 lxcfs[2130391]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T07:37:07+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T07:37:08+02:00 lxcfs[2139648]: utils.c: 315: send_creds: Connection refused - Failed at sendmsg: 2
2021-12-03T07:39:10+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2149637/task"
2021-12-03T07:39:10+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2149638/task"
2021-12-03T07:41:10+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2159346/task"
2021-12-03T07:42:09+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2163731/task"
2021-12-03T07:42:09+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2163732/task"
2021-12-03T07:45:06+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2176718/task"
2021-12-03T07:45:06+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2176727/task"
2021-12-03T07:46:25+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2186905/task"
2021-12-03T07:48:05+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2192775/task"
2021-12-03T07:48:05+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2192776/task"
2021-12-03T07:48:10+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2193908/task"
2021-12-03T07:48:10+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2193910/task"
2021-12-03T07:52:05+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2210271/task"
2021-12-03T07:52:05+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2210272/task"
2021-12-03T07:53:10+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2215296/task"
2021-12-03T07:55:15+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2224122/task"
2021-12-03T07:55:15+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2224123/task"
2021-12-03T07:58:05+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2234855/task"
2021-12-03T07:58:15+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2237000/task"
2021-12-03T07:58:15+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2237001/task"
2021-12-03T08:00:14+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2244597/task"
2021-12-03T08:00:15+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2247095/task"
2021-12-03T08:00:15+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2247096/task"
2021-12-03T08:00:19+02:00 lxcfs[2247630]: utils.c: 290: send_creds: Invalid argument - Failed getting reply from server over socketpair: 2
2021-12-03T08:00:26+02:00 lxcfs[1132]: utils.c: 254: recv_creds: Timed out waiting for scm_cred: No such process
2021-12-03T09:15:03+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2596159/task"
2021-12-03T09:15:03+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2596165/task"
2021-12-03T09:15:03+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2597481/task"
2021-12-03T09:15:03+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2597483/task"
2021-12-03T09:15:03+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2596299/task"
2021-12-03T09:15:03+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2596312/task"
2021-12-03T09:16:23+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2604626/task"
2021-12-03T09:16:23+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2604627/task"
2021-12-03T09:22:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2628332/task"
2021-12-03T09:23:24+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2635785/task"
2021-12-03T09:37:04+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2697578/task"
2021-12-03T09:39:00+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2706701/task"
2021-12-03T09:40:20+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2714267/task"
2021-12-03T09:40:40+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2715691/task"
2021-12-03T09:46:20+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2744811/task"
2021-12-03T09:54:15+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2781129/task"
2021-12-03T09:54:15+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2781130/task"
2021-12-03T10:00:13+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2807881/task"
2021-12-03T10:00:13+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2807883/task"
2021-12-03T10:00:51+02:00 lxd[715876]: 2021/12/03 10:00:51 http: superfluous response.WriteHeader call from github.com/lxc/lxd/lxd/response.(*errorResponse).Render (response.go:270)
2021-12-03T10:01:34+02:00 lxd[715876]: 2021/12/03 10:01:34 http: superfluous response.WriteHeader call from github.com/lxc/lxd/lxd/response.(*errorResponse).Render (response.go:270)
2021-12-03T10:05:03+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2834016/task"
2021-12-03T10:08:09+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2848285/task"
2021-12-03T10:14:09+02:00 lxcfs[1132]: proc_loadavg.c: 404: refresh_load: Failed to open "/proc/2874055/task"
The text was updated successfully, but these errors were encountered: