Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault on Google's Container Optimized OS (which uses musl vs glibc) #18033

Open
Nagyman opened this issue Mar 10, 2025 · 8 comments
Open

Segfault on Google's Container Optimized OS (which uses musl vs glibc) #18033

Nagyman opened this issue Mar 10, 2025 · 8 comments
Labels
atw crash An issue that could cause a crash linux An issue that occurs on Linux runtime

Comments

@Nagyman
Copy link

Nagyman commented Mar 10, 2025

How can we reproduce the crash?

Not sure if any of this helps and I can't provide access to all code.

Running environment:

  • Google Batch Job -> Google Compute Engine -> Container Optimized OS
  • Docker container oven/bun:latest (no real customizations) - full docker file pasted below
  • The crash happens somewhere during data retrieval (network requests), using Prisma as the ORM.
  • UPDATE: The crash seems to happen when using ExcelJS

Dockerfile

FROM oven/bun:latest AS runner
WORKDIR /workspace

ARG NODE_ENV="production"
ENV NODE_ENV=${NODE_ENV}

COPY package.json bun.lock ./
COPY apps/op/package.json apps/op/package.json
COPY . .

WORKDIR /workspace/apps/op

RUN bun install
RUN bunx prisma generate

CMD ["bun", "run", "jobs", "worker", "--ttl=240_000"]

Logs from the crash. There is plenty of memory on the machine, so something else is up.

Image

Relevant log output

Stack Trace (bun.report)

Bun v1.2.4 (fd9a5ea) on linux x86_64_baseline [RunCommand]

Segmentation fault at address 0x00000000

Features: tsconfig, tsconfig_paths, process_dlopen, Bun.stderr, Bun.stdin, Bun.stdout, fetch, jsc

Sentry Issue: BUN-A43

@Nagyman Nagyman added the crash An issue that could cause a crash label Mar 10, 2025
@github-actions github-actions bot added linux An issue that occurs on Linux runtime labels Mar 10, 2025
@Jarred-Sumner
Copy link
Collaborator

Jarred-Sumner commented Mar 10, 2025

Is "Container Optimized OS" a musl or glibc ? Maybe the installed native addon is musl but the bun version is glibc?

@Nagyman
Copy link
Author

Nagyman commented Mar 10, 2025

@Jarred-Sumner actually think we may be able to close this. I found another closed issue here #17897 from last week that is almost an identical situation, including running ExcelJS.

We were using oven/bun:latest so I'm re-trying with oven/bun:1.2.3

@Nagyman
Copy link
Author

Nagyman commented Mar 10, 2025

To clarify Container Optimized OS is from Google Engine Compute. https://cloud.google.com/container-optimized-os/docs

@Nagyman
Copy link
Author

Nagyman commented Mar 10, 2025

Same issue in Bun v1.2.3 (8c4d3ff) Linux x64 (baseline)

Trying 1.2.2

@Nagyman
Copy link
Author

Nagyman commented Mar 10, 2025

Is "Container Optimized OS" a musl or glibc ? Maybe the installed native addon is musl but the bun version is glibc?

Google Container Optimized OS uses musl as its standard C library, not glibc

@Nagyman
Copy link
Author

Nagyman commented Mar 10, 2025

I'm getting the same issue regardless of bun version (including canary).

I used a different boot disk instead in (allocationPolicy.instances[]) as part of the Google Batch specification e.g.:

{
  allocationPolicy: {
    instances: [{
      policy: {
        ...
        bootDisk: {
          // gcloud compute images list --project=batch-custom-image --no-standard-images
          image: 'projects/batch-custom-image/global/images/family/batch-debian-11-official',
        },
      }
    }]
    ...
  }
}

And it worked 🥳

So there's something in the default Container Optimized OS that Goole uses as the default boot image when the runnable is a docker container.

@Nagyman
Copy link
Author

Nagyman commented Mar 10, 2025

I'll close this as there's a workaround.

@Nagyman Nagyman closed this as completed Mar 10, 2025
@Electroid
Copy link
Contributor

Even if there’s a workaround, if Bun crashes, we’ll need to fix it.

@Electroid Electroid reopened this Mar 10, 2025
@Nagyman Nagyman changed the title Segfault running bun using oven/bun:latest Segfault on Google's Container Optimized OS (which uses musl vs glibc) Mar 10, 2025
@DonIsaac DonIsaac added the atw label Mar 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
atw crash An issue that could cause a crash linux An issue that occurs on Linux runtime
Projects
None yet
Development

No branches or pull requests

4 participants