Consolidation of serial and parallel runtimes

[davidjgoss]

Problem

cucumber-js has two runtime implementations internally:

• Serial - used by default, everything runs in-process, scenarios are executed one-by-one as per order of discovery
• Parallel - used when the parallel option is on, tests are run on worker processes and they report their results back up to the coordinator process

This has a disadvantage in terms of complexity of the codebase, naturally - it's more code to maintain. But it's not too bad in the scheme of things.

It's a source of unexpected behaviour and bugs for users as things behave somewhat differently when using the parallel runtime e.g. global hooks.

The serial mode specifically presents problems for API use cases (with runCucumber). Because the user code is loaded in the same process and cached, subsequent runs in the same process don't load it again, resulting in . We introduced loadSupport to work around this, but it's still confusing and doesn't cover all use cases. For example, if/when we want to add support for automatically rerunning tests when the files change, we can't really do that right now.

Proposal

In cases where we currently use the serial runtime, instead just use the parallel runtime but with a single worker.

This would put us in step with other JavaScript test runners like Mocha and Jest, which also do all test execution on workers.

It would make it easier to work with ESM loaders, as we could pass a --loader option to the worker processes without affecting the main process or requiring the user to set it externally (as they have to now).

Currently the parallel runtime loads user code once on the coordinator, and then again on each worker. The coordinator ones are used to generate the canonical step definition messages and the ids are kind of "synced" down to the workers, but it would be just as easy to have one worker designated the "leader" so it can do that part. This would solve the "cached user code" issue described above.

This could be phased in by at some point making --parallel 1 the default behaviour, and allowing an override to use serial mode for a time.

Considerations

This would have an impact on tools in the near ecosystem and we'd need to ensure we have these use cases covered off before doing this change. We really don't want to break things for people.

This would be a breaking change for the API, as we'd no longer support passing in a pre-loaded support code library to runCucumber, or returning in the response. loadSupport would have no reason to exist. cc @nicojs

Tools like WDIO and Serenity are currently pretty much blocked on using the parallel runtime, see:

• Parallel Execution - SupportCodeLibraryBuilder Not Loaded with Externalised Hook Information #2327
• Parallel execution - Allow programmatic registration of per-scenario cleanup hooks #2156

I think we'll need to first come up with a solution that allows integrators to pass data between coordinator and workers to solve these issues, and unlock this change. cc @tamil777selvan @jan-molak

---

Note this is currently a proposal and not being actively worked on - I'm looking for feedback and potential gotchas before doing anything meaningful about it.

[nicojs]

Hi @davidjgoss thanks for mentioning me.

Because the user code is loaded in the same process and cached, subsequent runs in the same process don't load it again.

This is very useful for the StrykerJS use case, and I consider it a feature. It enables hot reload, making the @stryker-mutator/cucumber plugins one of the fastest for StrykerJS (much faster than jest/vitest).

In cases where we currently use the serial runtime, instead just use the parallel runtime but with a single worker.

This would put us in step with other JavaScript test runners like Mocha and Jest, which also do all test execution on workers.

This is not the case for Mocha. The CLI might do it, but using the programmatic API and disable parallel mode, it executes in-process.

I propose using the same solution as Mocha. It has an abstraction for running in-process (so, "serial" as you call it) and uses that code from the workers. If you go this route, making parallel with 1 worker the default from the CLI (when parallel is disabled / set to 1) would be fine for me. The result for the end-user would be the same.

It would make it easier to work with ESM loaders, as we could pass a --loader option to the worker processes without affecting the main process or requiring the user to set it externally (as they have to now).

This can be solved by good documentation and error messages. I'm in the camp of people who say NodeJS devs should know what a "loader" is and why it is needed/how to configure it (or choose to run cucumber on transpiled output).

[davidjgoss]

Thanks for this Nico!

I hadn't considered the fast performance angle for Stryker.

I do like the idea of having the CLI use the parallel workers but making a "virtualised" worker for in-process runs available to runCucumber. This could still reduce the complexity of the internals a fair bit, and leave the door open for running on non-Node.js environments like browsers in future. I'll do some experimenting with this.

NodeJS devs should know what a "loader" is

Yeah...I agree generally, but Cucumber users tend to skew non-technical so we try to make it as easy as possible. Between TypeScript and ESM though it's tough!

[jan-molak]

Hey @davidjgoss - this sounds like a good idea!

For context, here's how Serenity/JS integrates with Cucumber.js:

• Serenity/JS Cucumber "formatter" is selected dynamically based on the Cucumber.js major version
• For Cucumber.js 7 and above, Serenity/JS picks the Cucumber Messages listener implementation. This means that if Cucumber 11 required a different/backwards-incompatible mechanism, we could easily plug it in here too.
• Here's the tricky bit, though. Apart from translating messages from Cucumber eventBroadcaster to internal Serenity/JS domain events, the listener also registers an afterHook. This afterHook is critical as it informs Serenity/JS that the test scenario is about to finish and triggers any clean-up processes of actors' abilities.

To support parallel execution using the model you're describing, Serenity/JS would need to be able to:

1. Pass internal domain events from the workers to the reporter. However, if doing so in real time was unavailable, we could pass all the events in bulk when the scenario is finished. This is the mechanism used by Serenity/JS Playwright Test integration, where all the internal events are serialised to JSON and passed to the reporter process as a Playwright attachment.
2. Register worker-level afterHook (guaranteed to be triggered after any user-land after hooks) without the developer having to do so manually. Since Serenity would be running at the worker level, we'd also need a beforeHook (also guaranteed to be triggered before any user-land hooks) to initialise serenity instance, actors, and so on. Have a look at the serenity fixture in Serenity/JS Playwright Test integration for reference.
3. Allow the developer to register Serenity/JS reporting services, which means passing configuration to the reporter process. At the moment, we rely on the developer using a BeforeAll hook when Serenity/JS is integrated directly with Cucumber, or any existing configuration that's available. See for reference: WebdriverIO configuration mechanism, Protractor configuration mechanism, or Playwright Test configuration mechanism integration.

If we addressed the three above points, the "serial" mode could be --parallel 1 from Serenity/JS point of view. Not to mention that the integration and developer experience would be much nicer, too!

---

Thinking aloud, if Cucumber allowed for the World to have asynchronous initialise and destroy methods, Serenity/JS could provide a SerenityWorld implementation that developers could plug into their test suites, and which could address point 2. from the list above.

[jan-molak]

Also CC @christian-bromann and @nextlevelbeard as we talked about it at webdriverio/webdriverio#11101

[davidjgoss]

Thanks for all the detail Jan!

if Cucumber allowed for the World to have asynchronous initialise and destroy methods

Even aside from the specific Serenity use case, this is definitely worth revisiting. The World currently has to be defined as a class (or constructor-like function), but being able to have an async supplier with (optionally) a cleanup function would be much better for users.

I do think though, that the things Serenity needs tend towards the plugins concept I have been slowly working up - especially as plugins can be included via configuration, meaning they would be usable for people directly using the CLI and wrappers of runCucumber of which Serenity is both.

Most of the use cases I've been looking at so far have been stuff you'd do at the coordinator level, like filtering or reordering pickles. But the next step is to have a part of the plugin that runs at the worker level and can send messages up to the coordinator part of the plugin. Obviously in the serial flavour these parts would be in the same process, but the pattern should still work, and this would play nicely into @nicojs suggestion above to keep the serial flavour available but have it be a virtualised worker.

A real rough example of the kind of thing I'm thinking about:

const plugin: Plugin = {
  type: 'plugin',
  coordinator: async ({ on }) => {
    // one-time initial setup can happen here

    on('message', (envelope) => {
      // cucumber messages - map to serenity events for reporting
    })

    on('plugin:message', ({ type, message }) => {
      // handle messages from worker part
      switch (type) {
        case 'serenity.domainevent':
        // forward to reporting etc
      }
    })

    return () => {
      // one-time final teardown can happen here
    }
  },
  worker: async ({ on, emit }) => {
    on('after:usercode:register', ({ Before, After }) => {
      Before.First(async () => {
        // do setup before any user hooks
      })

      After.Last(async () => {
        // do teardown...

        emit('serenity.domainevent', {
          // emit event message to coordinator part
        })
      })
    })
  },
}

export default plugin

I think something like this would allow you to keep the Serenity-owned logic pretty much in one place?

[jan-molak]

I do think though, that the things Serenity needs tend towards the #2091 I have been slowly working up - especially as plugins can be included via configuration, meaning they would be usable for people directly using the CLI and wrappers of runCucumber of which Serenity is both.

I like the plugin idea very much, @davidjgoss! I think Serenity/JS has pushed the idea of a "formatter" much further than what it was originally intended for :-) With the plugin architecture you're describing, Serenity/JS could become a Cucumber.js plugin and have a much easier job integrating.

Even aside from the specific Serenity use case, this is definitely worth revisiting. The World currently has to be defined as a class (or constructor-like function), but being able to have an async supplier with (optionally) a cleanup function would be much better for users.

I believe so too. Also, an "async World" would make it possible for Cucumber.js to provide much better integration support for tools that require async initialisation, such as Playwright (see microsoft/playwright#11975) and WebdriverIO (running in standalone mode)

Please keep me posted on your progress in this area, and let me know if you'd like any help or feedback.