minikube in gitlab ci with dind service fails

[timorickli]

What Happened?

I'm setting up minikube in a gitlab ci pipeline as described in https://minikube.sigs.k8s.io/docs/tutorials/continuous_integration/.

.gitlab-ci.yml

.default-minikube:  
  image: docker:cli
  variables:
    MINIKUBE_VERSION: v1.33.1
  before_script:
    - curl -Lo /usr/local/bin/minikube https://storage.googleapis.com/minikube/releases/${MINIKUBE_VERSION}/minikube-linux-amd64
    - chmod +x /usr/local/bin/minikube
    - minikube start --force -alsologtostderr -v=2
  services:
    # Use docker in docker as a service
    - name: docker:dind
      alias: docker

output

W0826 13:33:07.969299      82 out.go:239] ! Listening to 0.0.0.0 on external docker host docker. Please be advised
! Listening to 0.0.0.0 on external docker host docker. Please be advised
I0826 13:33:07.969349      82 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0826 13:33:07.979377      82 cli_runner.go:164] Run: docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0826 13:33:07.990011      82 oci.go:103] Successfully created a docker volume minikube
I0826 13:33:07.990068      82 cli_runner.go:164] Run: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -d /var/lib
I0826 13:33:08.276702      82 oci.go:107] Successfully prepared a docker volume minikube
I0826 13:33:08.276736      82 preload.go:132] Checking if preload exists for k8s version v1.30.0 and runtime docker
I0826 13:33:08.276765      82 kic.go:194] Starting extracting preloaded images to volume ...
I0826 13:33:08.276829      82 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /root/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -I lz4 -xf /preloaded.tar -C /extractDir
W0826 13:33:08.556452      82 cli_runner.go:211] docker run --rm --entrypoint /usr/bin/tar -v /root/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -I lz4 -xf /preloaded.tar -C /extractDir returned with exit code 2
I0826 13:33:08.556476      82 kic.go:201] Unable to extract preloaded tarball to volume: docker run --rm --entrypoint /usr/bin/tar -v /root/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -I lz4 -xf /preloaded.tar -C /extractDir: exit status 2
stdout:
stderr:
tar (child): /preloaded.tar: Cannot read: Is a directory
tar (child): At beginning of tape, quitting now
tar (child): Error is not recoverable: exiting now
/usr/bin/tar: Child returned status 2
/usr/bin/tar: Error is not recoverable: exiting now
W0826 13:33:08.556534      82 cgroups_linux.go:77] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
W0826 13:33:08.556563      82 oci.go:243] Your kernel does not support CPU cfs period/quota or the cgroup is not mounted.
I0826 13:33:08.556591      82 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0826 13:33:08.589044      82 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.58.2 --volume minikube:/var --security-opt apparmor=unconfined --memory=2200mb -e container=docker --expose 8443 --publish=0.0.0.0::8443 --publish=0.0.0.0::22 --publish=0.0.0.0::2376 --publish=0.0.0.0::5000 --publish=0.0.0.0::32443 gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e
I0826 13:33:08.799275      82 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Running}}
I0826 13:33:08.809134      82 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Status}}
I0826 13:33:08.819587      82 cli_runner.go:164] Run: docker exec minikube stat /var/lib/dpkg/alternatives/iptables
I0826 13:33:08.856210      82 oci.go:144] the created container "minikube" has a running status.
I0826 13:33:08.856233      82 kic.go:225] Creating ssh key for kic: /root/.minikube/machines/minikube/id_rsa...
I0826 13:33:09.023284      82 kic_runner.go:191] docker (temp): /root/.minikube/machines/minikube/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0826 13:33:09.037434      82 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Status}}
I0826 13:33:09.049711      82 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0826 13:33:09.049734      82 kic_runner.go:114] Args: [docker exec --privileged minikube chown docker:docker /home/docker/.ssh/authorized_keys]
I0826 13:33:09.087966      82 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Status}}
I0826 13:33:09.099477      82 machine.go:94] provisionDockerMachine start ...
I0826 13:33:09.099541      82 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0826 13:33:09.111690      82 main.go:141] libmachine: Using SSH client type: native
I0826 13:33:09.111848      82 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d6e0] 0x830440 <nil>  [] 0s} docker 32773 <nil> <nil>}
I0826 13:33:09.111855      82 main.go:141] libmachine: About to run SSH command:
hostname
I0826 13:34:01.809269      82 main.go:141] libmachine: Error dialing TCP: dial tcp 172.17.0.2:32768: connect: connection timed out
I0826 13:34:04.809595      82 main.go:141] libmachine: Error dialing TCP: dial tcp 172.17.0.2:32768: connect: connection refused

I see that minikube wants to start a container and mount the preloaded.tar into the container -> "W0826 13:33:08.556452 82 cli_runner.go:211] docker run --rm --entrypoint /usr/bin/tar -v /root/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/".
But this is not possible do to dind is spawned as a service which runs outside the job in the gitlab runner and the job only attaches to the docker daemon. The root/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-amd64.tar.lz4 isn't available there so a new dir gets created.

I suppose to make this run we would need to create a volume and cp the preloaded-images-k8s-v18-v1.30.0-docker-overlay2-amd64.tar.lz4 into it and then we could use the volume to make the tar available. (https://github.com/kubernetes/minikube/blob/master/pkg/drivers/kic/oci/volumes.go#L133)

Attach the log file

log.txt

Operating System

None

Driver

None

[afbjorklund]

You can disable the preload, with --preload=false. Then it will use the regular cache instead.

Though you might have the same issues with that, if you don't have a home directory (or cache)

[timorickli]

this fixed the issue with the preload. But i still can't startup minikube. I assume it's due to the fact that all certs get generated in the root dir of the runner. So the dind service which runs as a seperate container can't access certs. I guess its the same issue why the preload is not working.

I0827 09:34:09.886814     125 ubuntu.go:169] provisioning hostname "minikube"
I0827 09:34:09.886866     125 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0827 09:34:09.897268     125 main.go:141] libmachine: Using SSH client type: native
I0827 09:34:09.897393     125 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d6e0] 0x830440 <nil>  [] 0s} docker 32768 <nil> <nil>}
I0827 09:34:09.897409     125 main.go:141] libmachine: About to run SSH command:
sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname
I0827 09:34:10.039306     125 main.go:141] libmachine: SSH cmd err, output: <nil>: minikube

I0827 09:34:10.039408     125 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0827 09:34:10.057589     125 main.go:141] libmachine: Using SSH client type: native
I0827 09:34:10.057805     125 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d6e0] 0x830440 <nil>  [] 0s} docker 32768 <nil> <nil>}
I0827 09:34:10.057843     125 main.go:141] libmachine: About to run SSH command:

		if ! grep -xq '.*\sminikube' /etc/hosts; then
			if grep -xq '127.0.1.1\s.*' /etc/hosts; then
				sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 minikube/g' /etc/hosts;
			else 
				echo '127.0.1.1 minikube' | sudo tee -a /etc/hosts; 
			fi
		fi

I0827 09:36:43.846074     125 main.go:141] libmachine: SSH cmd err, output: <nil>: 
I0827 09:36:43.846101     125 ubuntu.go:175] set auth options {CertDir:/root/.minikube CaCertPath:/root/.minikube/certs/ca.pem CaPrivateKeyPath:/root/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/root/.minikube/machines/server.pem ServerKeyPath:/root/.minikube/machines/server-key.pem ClientKeyPath:/root/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/root/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/root/.minikube}
I0827 09:36:43.846130     125 ubuntu.go:177] setting up certificates
I0827 09:36:43.846147     125 provision.go:84] configureAuth start
I0827 09:36:43.846195     125 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube
I0827 09:36:43.857182     125 provision.go:143] copyHostCerts
I0827 09:36:43.858850     125 exec_runner.go:151] cp: /root/.minikube/certs/ca.pem --> /root/.minikube/ca.pem (1082 bytes)
I0827 09:36:43.859037     125 exec_runner.go:151] cp: /root/.minikube/certs/cert.pem --> /root/.minikube/cert.pem (1123 bytes)
I0827 09:36:43.859342     125 exec_runner.go:151] cp: /root/.minikube/certs/key.pem --> /root/.minikube/key.pem (1679 bytes)
I0827 09:36:43.859881     125 provision.go:117] generating server cert: /root/.minikube/machines/server.pem ca-key=/root/.minikube/certs/ca.pem private-key=/root/.minikube/certs/ca-key.pem org=unknown.minikube san=[127.0.0.1 192.168.49.2 docker localhost minikube]
I0827 09:36:43.924217     125 provision.go:177] copyRemoteCerts
I0827 09:36:43.924369     125 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0827 09:36:43.924410     125 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0827 09:36:43.934420     125 sshutil.go:53] new ssh client: &{IP:docker Port:32768 SSHKeyPath:/root/.minikube/machines/minikube/id_rsa Username:docker}
W0827 09:38:55.429981     125 sshutil.go:64] dial failure (will retry): dial tcp 172.17.0.2:32768: connect: connection timed out
I0827 09:38:55.437076     125 provision.go:87] duration metric: took 2m11.583980121s to configureAuth
W0827 09:38:55.437091     125 ubuntu.go:180] configureAuth failed: NewSession: new client: new client: dial tcp 172.17.0.2:32768: connect: connection timed out
I0827 09:38:55.437115     125 ubuntu.go:189] Error configuring auth during provisioning Temporary Error: NewSession: new client: new client: dial tcp 172.17.0.2:32768: connect: connection timed out
I0827 09:38:55.437125     125 machine.go:97] duration metric: took 4m45.69260295s to provisionDockerMachine
I0827 09:38:55.437132     125 client.go:171] duration metric: took 4m47.412924111s to LocalClient.Create
I0827 09:38:57.438408     125 start.go:128] duration metric: took 4m49.434485436s to createHost
I0827 09:38:57.438437     125 start.go:83] releasing machines lock for "minikube", held for 4m49.438212523s
W0827 09:38:57.438919     125 start.go:713] error starting host: creating host: create: provisioning: Temporary Error: NewSession: new client: new client: dial tcp 172.17.0.2:32768: connect: connection timed out
I0827 09:38:57.439577     125 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Status}}
I0827 09:38:57.449185     125 stop.go:39] StopHost: minikube

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten