Ability to run specs in a particular order

[boris-petrov]

Is your feature request related to a problem?

I run all my other tests in randomized order - integration tests and frontend tests. I would like to run my backend-specific tests in randomized order as well. This would allow me to catch issues with dependencies between tests (which of course I don't want to be there). Just recently I spent a few hours debugging an issue in my tests that appeared only when different files were run in a specific order. I would like to protect myself from that.

Describe the solution you'd like

Something like Jupiter's junit.jupiter.testclass.order.default = org.junit.jupiter.api.ClassOrderer$Random.

Describe alternatives you've considered

There is some more context about this issue in Gitter.

Additional context

No response

[janisz]

Any updates. If I get a hint where it should be implemented I'll do it :)

[Vampire]

I guess it is relatively simple.
You add an executionOrder property to SpecInfo similar to the property in FeatureInfo.
Then in SpockEngineDiscoveryPostProcessor#postProcessEngineDescriptor you sort the stream after the processSpecNode is executed, as then the extensions were executed.

Now you should be able to use extensions to define the spec order.
I have no idea whether it works that way, but I think it should.

[janisz]

Thanks. That helps a lot. At least I know how to create proof of concept. After adding new field to

spock/spock-core/src/main/java/org/spockframework/runtime/model/SpecInfo.java

Line 35 in e97e2ed

public class SpecInfo extends SpecElementInfo<NodeInfo, Class<?>> implements IMethodNameMapper {

I should sort it by this new field:

spock/spock-core/src/main/java/org/spockframework/runtime/SpockEngineDiscoveryPostProcessor.java

Lines 12 to 18 in e97e2ed

	SpockEngineDescriptor postProcessEngineDescriptor(UniqueId uniqueId, RunContext runContext,
	SpockEngineDescriptor engineDescriptor) {
	SpockEngineDescriptor processedEngineDescriptor = new SpockEngineDescriptor(uniqueId, runContext);
	engineDescriptor.getChildren().stream()
	.map(child -> processSpecNode(child, runContext))
	.forEach(processedEngineDescriptor::addChild);
	return processedEngineDescriptor;

The field could be set by extension in visitSpec(SpecInfo spec) (for example user can define "rank", or seed as annotation parameter).

[kriegaex]

@Vampire, it works nicely with these changes in Spock:

--- a/spock-core/src/main/java/org/spockframework/runtime/model/SpecInfo.java	(revision Staged)
+++ b/spock-core/src/main/java/org/spockframework/runtime/model/SpecInfo.java	(date 1681293989411)
@@ -48,6 +48,8 @@
   private ExecutionMode executionMode = null;
   private ExecutionMode childExecutionMode = null;
 
+  private int executionOrder;
+
   private String pkg;
   private String filename;
   private String narrative;
@@ -66,6 +68,14 @@
 
   private final List<FeatureInfo> features = new ArrayList<>();
 
+  public int getExecutionOrder() {
+    return executionOrder;
+  }
+
+  public void setExecutionOrder(int executionOrder) {
+    this.executionOrder = executionOrder;
+  }
+
   public String getPackage() {
     return pkg;
   }
--- a/spock-core/src/main/java/org/spockframework/runtime/SpockEngineDiscoveryPostProcessor.java	(revision Staged)
+++ b/spock-core/src/main/java/org/spockframework/runtime/SpockEngineDiscoveryPostProcessor.java	(date 1681299546435)
@@ -5,6 +5,8 @@
 
 import org.junit.platform.engine.*;
 
+import static java.util.Comparator.comparingInt;
+
 class SpockEngineDiscoveryPostProcessor {
 
   private static final Object[] EMPTY_ARGS = new Object[0];
@@ -14,6 +16,9 @@
     SpockEngineDescriptor processedEngineDescriptor = new SpockEngineDescriptor(uniqueId, runContext);
     engineDescriptor.getChildren().stream()
       .map(child -> processSpecNode(child, runContext))
+      .sorted(comparingInt((TestDescriptor child) ->
+        child instanceof SpecNode ? ((SpecNode) child).getNodeInfo().getExecutionOrder() : 0
+      ))
       .forEach(processedEngineDescriptor::addChild);
     return processedEngineDescriptor;
   }

A simple example for a global extension is:

package de.scrum_master.testing.extension

import org.spockframework.runtime.extension.IGlobalExtension
import org.spockframework.runtime.model.SpecInfo

class RandomOrderExtension implements IGlobalExtension {
  private static final Random RANDOM = new Random()

  @Override
  void visitSpec(SpecInfo spec) {
    spec.executionOrder = RANDOM.nextInt()
  }
}

And then of course META-INF/services/org.spockframework.runtime.extension.IGlobalExtension:

de.scrum_master.testing.extension.RandomOrderExtension

@leonard84, is this worth a PR in your opinion? The little extension could either be included into the Spock manual as an example or, if you think we should publish it as a standard extension, as part of Spock proper.

[Vampire]

Your last codeblock is wrong @kriegaex. :-)

If we make a built-in random order extension, maybe it should also randomize the features within a spec, or maybe have a Spock configuration file extension to configure whether to randomize only features, only specs, both, or none, probably with none as default.
Especially as it would always randomize the specs if it is a non-optional built-in extension.

Another option could be an even more generic extensions - also configurable via the config file - that has more ordering options similar to the Jupiter extension.

[kriegaex]

Your last codeblock is wrong

It is not wrong, because, like I said, the extension is a simple proof-of-concept example which only randomises the execution order of specs - no more, no less. I do agree, however, that if we think about adding this as a built-in extension and not just as an inline Spock manual snippet, we could and should make it more universal and more powerful. That would be easy enough to achieve.

[Vampire]

It is not wrong, because, like I said, the extension is a simple proof-of-concept example which only randomises the execution order in specs - no more, no less.

What I meant is

And then of course META-INF/services/org.spockframework.runtime.extension.IGlobalExtension:

org.spockframework.runtime.extension.IGlobalExtension

It should be de.scrum_master.testing.extension.RandomOrderExtension, shouldn't it?

[kriegaex]

It should be de.scrum_master.testing.extension.RandomOrderExtension, shouldn't it?

Oh, of course. Somehow, the wrong snippet must have ended up in my Windows clipboard. In my sample project, it is correct. I fixed it above.

[kriegaex]

Another quick & easy example for randomising both the order of specs and the features within them would be, adding just a single line:

package de.scrum_master.testing.extension

import org.spockframework.runtime.extension.IGlobalExtension
import org.spockframework.runtime.model.SpecInfo

class RandomOrderExtension implements IGlobalExtension {
  private static final Random RANDOM = new Random()

  @Override
  void visitSpec(SpecInfo spec) {
    spec.executionOrder = RANDOM.nextInt()
    spec.features.each { it.executionOrder = RANDOM.nextInt() }
  }
}

[janisz]

@kriegaex That's great! I think we would need 3 different options. For example let's have specs A, B, C and tests 1, 2, 3 in each:

1. Randomise specs order
   • e.g. B1 | B2 | B3 | A1 | A2 | A3 | C1 | C2 | C3
2. Randomise features in spec
   • e.g. A2 | A1 | A3 | B1 | B3 | B2 | C2 | C3 | C1
3. Randomise everything
   • e.g. C2 | A1 | B3 | A2 | A3 | C1 | B2 | C3 | B1

Plus we need to print the seed and allow configuring it in order to reproduce test.

[kriegaex]

Yes, easy enough, like I said. My latest version is basically your option 4 3. Before I further refine the example(!) into something production-ready, I would like to get feedback by @leonard84, if he would accept my PR for the patch suggested above. In the second step, I would add another PR for the randomiser extension and refine it. I have a history of PRs rejected due to code style and design disagreements, and I dislike wasting my time, preparing PRs which later someone else either completely rejects or recreates in a similar fashion.

[boris-petrov]

Thanks for taking the time to implement that feature!

I believe there is also value in a 4th way of randomizing in addition to the 3 provided by @janisz: randomize both specs and features but, unlike 3, only run a specific spec fully and only then continue with the next. For example B1 | B3 | B2 | A2 | A1 | A3 | C2 | C3 | C1. That's similar to what RSpec does when you specify random for execution order. Which makes sense I believe. How would setupSpec/setup (and their cleanup counterparts) work in option 3?

[Vampire]

I don't think option 3 is possible.
Maybe somewhat if you enable parallel execution.
But with single-threaded execution you will only be able to randomize the specs and randomize the features within, so option 4, but not mix 'n' match everything.

[kriegaex]

@boris-petrov, yes, what you are suggesting makes more sense than option 3. Actually, my example implements your option 4, not option 3. @janisz, I do not even think that option 3 makes any sense, because it would require us to run setupSpec and cleanupSpec multiple times, which could be very costly for integration tests.

Edit: @Vampire's comment was posted while I wrote mine, so our comments' contents intersect.

[kriegaex]

Note to myself: We also need to make sure that random or manual (e.g. via an annotation-driven extension) ordering vs. @Stepwise does not create any unwanted and/or hard to predict behaviour. I think that an explicit @Stepwise on spec or feature level should always trump randomisation. The devil is in the details, as usual.

[Vampire]

It will win, because global extensions are run first, annotation-driven extensions last. :-)

[kriegaex]

That might be so, but either way, there should be automated tests for that. Please also note my comment update regarding manual ordering. Then we are talking about two annotation-driven extensions.