Transition to ESM

[davidjgoss]

Background

The Node.js ecosystem is gradually moving from CommonJS (require, module.exports etc) to ESM (import, export etc).

CommonJS code can be consumed by ESM code, but ESM code cannot be consumed by CommonJS code (because ESM is async and CommonJS is sync). This means that once a package switches to ESM, your project must be ESM to use it. As more open source projects switch, life will get harder for the remaining projects as they'll be unable to upgrade dependencies or add new dependencies that have switched. Switching to ESM also unlocks nice things like top-level await.

As of 8.0.0, we added opt-in support for user code written in ES Modules, and made Cucumber a hybrid package - meaning it's delivered in CommonJS format but has an ESM entry point. This involved a few compromises and leaves us in a transitional state.

Plan

Here's the (draft, for now) plan of how we eventually make the full switch to ESM:

Phase 0 - where we are now

• Cucumber is a hybrid module
• The --require option loads code with require() (only works for CommonJS)
  • Transpilation is done via the --require-module option
• The --import option loads code with await import() (works for both ESM and CommonJS)
  • Transpilation is done externally via Node.js --loader option (there's no mechanism to register loaders in-process)
• The default behaviour (if neither of the above specified) uses require()
• Custom formatters/snippets and configuration files are loaded with require(), with a fallback to retry with await import()
• Deep requires (e.g. require('@cucumber/cucumber/lib/models/test_case_hook_definition')) still work in a limited way

Phase 1: establish ESM as the best and default way

(Can we done gradually - doesn't have to be in a single release.)

• Cucumber is still a hybrid module
• The --require and --import options still work as before
• The default behaviour (if neither of the above specified) uses await import()
• Custom formatters/snippets are only loaded with await import()
• Configuration files are loaded with the appropriate mechanism based on file/package type
• First-class mechanism for just-in-time transpilation (i.e. dynamic registration of loaders) First-class support for module loaders #2364
• Deep requires are gone (side note: this is good, we have control of what users are calling and can more confidently refactor internals)
  • A slight complication with some libraries relying on deep imports e.g. OpenTelemetry Cucumber Instrumentation #2174 - we may need to get plugin functionality out there to serve this need sooner than later
• Documentation and examples emphasise ESM syntax
• --require and --require-module are deprecated, users are encouraged to switch to use import() and report any difficulties

Phase 2: all-in on ESM

• Cucumber is an ESM module
• --require and --require-module are gone, code is only loaded with await import()

Stray thoughts

The time between phases 1 and 2 could be short or long; we can stay in a hybrid form for a while and take cues from the wider ecosystem. One milestone will be when ESM loaders become properly stable.

Cucumber could be quicker to do this than other test runners, because the footprint of what's affected by require vs import is relatively small, unlike some to other tools that have e.g. module-level mocking built around require.

On the other hand, we are to some extent limited by third-party frameworks that wrap Cucumber - if these move slowly towards ESM, that will give us some headaches.

[aurelien-reeves]

Looks good 👍