Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Experimental Local Overlay Store Issue in k8s pod #11840

Open
yaroslavkasatikov opened this issue Nov 8, 2024 · 1 comment
Open

Experimental Local Overlay Store Issue in k8s pod #11840

yaroslavkasatikov opened this issue Nov 8, 2024 · 1 comment
Assignees
@tomberek
Labels
bug

Comments

@yaroslavkasatikov
Copy link

Hello team,

I found interesting article https://blog.replit.com/super-colliding-nix-stores and read https://nix.dev/manual/nix/2.22/store/types/experimental-local-overlay-store#experimental-local-overlay-store.

Our goal is building the system based on local-overlay-store which can work in non-privileged pod (we use OKD and pod should been running under the restricted-scc).

So, I have been inspired your idea to have two stores (big with many packages and local).
My idea is having nfsd server in privileged pod, mount OverlayFS there and export nfs share for merged, upper and lower and mount them into non-privileged nix pod.

Well, but until running this idea I tried to create a privileged pod and run my first overlay-fs store test, but I was failed to start it (or maybe I understood something incorrect (I'm newbie in nix).

Here my steps for reproducing.

  1. I have created two empty pvc (for upper and lower) and mounted them into my pod. So it will be something like this:
apiVersion: v1
kind: Pod
metadata:
  name: nix-test-p
spec:
  containers:
  - name: volume-test
    env:
    - name: HOME
      value: /nix/home
    image: nixos/nix
    command:
    - tail
    - -f
    - /dev/null
    imagePullPolicy: IfNotPresent
    securityContext:
      privileged: true
    volumeMounts:
    - name: pvc-upper
      mountPath: /mnt/example/
    - name: pvc-lower
      mountPath: /mnt/example/store-a
    - name: nix
      mountPath: /etc/nix/nix.conf
      subPath: nix.conf
    ports:
    - containerPort: 80
  volumes:
  - name: pvc-lower
    persistentVolumeClaim:
      claimName: lower-p
  - name: pvc-upper
    persistentVolumeClaim:
      claimName: upper-p
  - name: nix
    configMap:
      name: nix

I have mounted nix.conf via the configmap with this content:

apiVersion: v1
data:
  nix.conf: |+
    trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
    experimental-features = nix-command flakes local-overlay-store read-only-local-store
    store = local-overlay://?root=/mnt/example/merged-store&lower-store=/mnt/example/store-a&upper-layer=/mnt/example/store-b&check-mount=false

kind: ConfigMap
metadata:
  name: nix

So I have taken fully example from the docs with the path.

First my surprise was the absence of mount command :

sh-5.2# mount
sh: mount: command not found

Okay, I have installed it with nix-env -iA nixpkgs.mount and it installed mount into /mnt/example/merged-store/nix/store/ (which is not mounted yet), but okay, I have mount now..
So the question 1 - how can I install mount in correct way if loading nix container with overlay-fs store for initial mounting?

btw, I haven't found nothing in $HOME/.nix-profiles/bin related to mount and found it with 'find' util:

sh-5.2# find / -name mount
/mnt/example/merged-store/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount
/mnt/example/merged-store/nix/store/ibgzqljiski2y3fqvg8fnj2g6bjyhqly-util-linux-2.39.4-mount/bin/mount
/mnt/example/merged-store/nix/store/2wvb4326f069mz8zan43yx6nak6lsjqk-util-linux-2.39.4-bin/share/bash-completion/completions/mount
/mnt/example/merged-store/nix/store/2wvb4326f069mz8zan43yx6nak6lsjqk-util-linux-2.39.4-bin/bin/mount
/mnt/example/merged-store/nix/store/lnk0ky82i3rqvxvi3qknky80qffldyi9-mount-util-linux-2.39.4/bin/mount

well, I have mounted overlay fs:

sh-5.2# /mnt/example/merged-store/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount -t overlay overlay   -o lowerdir=/mnt/example/store-a/nix/store   -o upperdir=/mnt/example/store-b   -o workdir=/mnt/example/workdir   /mnt/example/merged-store/nix/store
sh-5.2# ls /mnt/example/merged-store/nix/store

Ok, seems I can install packages.. btw, I mounted overlayfs according the commands from the manual, but it doesn't work without check-mount=false

So ok, I have installed packages:

sh-5.2# nix-env -iA nixpkgs.vim
installing 'vim-9.1.0377'
these 3 paths will be fetched (8.79 MiB download, 42.94 MiB unpacked):
  /nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2
  /nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377
  /nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd
copying path '/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/s83pcskzqdz78qwkpydcm33gm40ckfcx-user-environment.drv'...
sh-5.2# nix-env -iA nixpkgs.nano
installing 'nano-8.0'
these 3 paths will be fetched (0.87 MiB download, 10.89 MiB unpacked):
  /nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45
  /nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0
  /nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info
copying path '/nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/rffg19w0n2irxvfmw8paw6g8bx6hzlm0-user-environment.drv'...

all of them has been installed into correct path:

sh-5.2# find / -name vim
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/applications/editors/vim
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/test/vim
/mnt/example/store-b/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/share/vim
/mnt/example/store-b/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/bin/vim
/mnt/example/store-b/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/share/vim
/mnt/example/store-b/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/bin/vim
/mnt/example/merged-store/nix/store/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/share/vim
/mnt/example/merged-store/nix/store/0x63v0qsqws6iqxi9y151ani984j1n4d-user-environment/bin/vim
/mnt/example/merged-store/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/share/vim
/mnt/example/merged-store/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377/bin/vim
sh-5.2# find / -name nano
/nix/store/yzqnkb2mfzphmhv3248pw0pqh6f1mn6y-7sprarsdfz9qcd7859phvr9nvhi14mri-source/pkgs/applications/editors/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/doc/nano
/mnt/example/store-b/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/bin/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/share/doc/nano
/mnt/example/merged-store/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0/bin/nano

Well next I want to prepare lower storage. I plan to umount overlayfs and copy all data from upper to lower storage, cleanup database/upper and try to install data from lower-store. So I expect it won't download packages from the Network but will reuse them from lower storage.

So:

install mount again:

sh-5.2# nix-env -iA nixpkgs.mount
installing 'mount-util-linux-2.39.4'
....
sh-5.2# /mnt/example/store-b/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/umount   /mnt/example/merged-store/nix/store
sh-5.2# cp -a /mnt/example/store-b/* /mnt/example/store-a/nix/store/

found strange: database on lower layer is updated:

sh-5.2# ls -la ./mnt/example/store-a/nix/var/nix/db
total 8280
drwxr-xr-x 2 root root    4096 Nov  8 15:29 .
drwxr-xr-x 6 root root    4096 Nov  8 15:29 ..
-rw------- 1 root root       0 Nov  8 15:29 big-lock
-rw-r--r-- 1 root root   45056 Nov  8 15:29 db.sqlite
-rw-r--r-- 1 root root   32768 Nov  8 15:39 db.sqlite-shm
-rw-r--r-- 1 root root       0 Nov  8 15:29 db.sqlite-wal
-rw------- 1 root root 8388608 Nov  8 15:29 reserved
-rw-r--r-- 1 root root       2 Nov  8 15:29 schema

It is not under overlayfs - just folder on dist, but it's a strange, I thought this db should not be updated or I'm thinking incorrect?

But okay, I have removed everything from upper layer

sh-5.2# ls -la /mnt/example/store-b/
total 160
drwxrwxr-t 2 root nixbld 159744 Nov  8 15:42 .
drwxr-xr-x 7 root root     4096 Nov  8 15:20 ..
sh-5.2# ls -la /mnt/example/workdir/
total 12
drwxr-xr-x 3 root root 4096 Nov  8 15:20 .
drwxr-xr-x 7 root root 4096 Nov  8 15:20 ..
d--------- 2 root root 4096 Nov  8 15:20 work
sh-5.2# ls -la /mnt/example/workdir/work/
total 8
d--------- 2 root root 4096 Nov  8 15:20 .
drwxr-xr-x 3 root root 4096 Nov  8 15:20 ..

Remount overlayfs and trying to reinstall pacakges which stored on lower layer:

sh-5.2# /mnt/example/store-a/nix/store/5lxi14cng9r29nxlbzbszanpbjwvwbi9-util-linux-minimal-2.39.4-mount/bin/mount -t overlay overlay   -o lowerdir=/mnt/example/store-a/nix/store   -o upperdir=/mnt/example/store-b   -o workdir=/mnt/example/workdir   /mnt/example/merged-store/nix/store
sh-5.2# nix-env -iA nixpkgs.nano
installing 'nano-8.0'
these 9 paths will be fetched (8.13 MiB download, 45.52 MiB unpacked):
  /nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45
  /nix/store/87848rvrg5c7jmplpi0iapvbxyj9kfid-glibc-2.39-52
  /nix/store/ya4arqpf6vwbq4msi449314kqbhdb4l4-libidn2-2.3.7
  /nix/store/503h8rdwzj12dh0fs1b6y643qm26vqsb-libunistring-1.1
  /nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0
  /nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info
  /nix/store/qdbfy1ndpz2n868b8y1y3nlfsixgz106-ncurses-6.4
  /nix/store/19avl2s0rd4whz3b0n118qdrv6yr3b0g-xgcc-13.2.0-libgcc
  /nix/store/2k9k3q1vk8z6w7743k6nb22vnb05xv06-zlib-1.3.1
copying path '/nix/store/z8l5b4hml7fchafzj94zm2pjg7xg5jql-nano-8.0-info' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/19avl2s0rd4whz3b0n118qdrv6yr3b0g-xgcc-13.2.0-libgcc' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/503h8rdwzj12dh0fs1b6y643qm26vqsb-libunistring-1.1' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/ya4arqpf6vwbq4msi449314kqbhdb4l4-libidn2-2.3.7' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/87848rvrg5c7jmplpi0iapvbxyj9kfid-glibc-2.39-52' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/qdbfy1ndpz2n868b8y1y3nlfsixgz106-ncurses-6.4' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/2k9k3q1vk8z6w7743k6nb22vnb05xv06-zlib-1.3.1' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/kbfzchcy5x3cwa39ls1dk92hd2q6ppz2-file-5.45' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/y3wbnhsnf5lvsrlzca47l377f860q64h-nano-8.0' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/rffg19w0n2irxvfmw8paw6g8bx6hzlm0-user-environment.drv'...
sh-5.2# nix-env -iA nixpkgs.vim
installing 'vim-9.1.0377'
these 4 paths will be fetched (9.21 MiB download, 44.48 MiB unpacked):
  /nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26
  /nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2
  /nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377
  /nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd
copying path '/nix/store/wckka8fxv4h5hp74cbkhaw3fw7kbvcs1-bash-5.2p26' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/ddkcg6irdsn0w2q05gphaaw3cblkml69-gawk-5.2.2' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/xbi4fw67yyrqz4mnza64zhw16s4rgybm-vim-9.1.0377-xxd' from 'https://cache.nixos.org' to 'local-overlay://'...
copying path '/nix/store/lidpizm6ipz51494nsfss96m3jkin8k2-vim-9.1.0377' from 'https://cache.nixos.org' to 'local-overlay://'...
building '/nix/store/s83pcskzqdz78qwkpydcm33gm40ckfcx-user-environment.drv'...

And nix downloads them from Internet, but I expected them will be reused from lower-layer.

So , could you please tell me when I did a mistake? Unfortunately there are few docs about overlay-local-store, but this seems to be a killer feature.

Thank you very much.
@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/super-colliding-nix-stores/28462/30

@tomberek tomberek self-assigned this Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomberek tomberek

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tomberek @yaroslavkasatikov @nixos-discourse