Add BlockListener support... #1575

leonard84 · 2023-02-16T18:56:24Z

This feature allows extension authors to register a IBlockListener for
a feature to observe the execution of a feature in more detail.
This surfaces some of Spock's idiosyncrasies, for example interaction
assertions are actually setup right before entering the preceding
when-block as well as being evaluated on leaving the when-block
before actually entering the then-block.

The only valid block description is a constant String, although some
users mistakenly try to use a dynamic GString. Using anything other
than a String, will be treated as a separate statement and thus ignored.

fixes #538
fixes #111

codecov · 2023-02-16T19:03:18Z

Codecov Report

Attention: Patch coverage is 79.86111% with 29 lines in your changes missing coverage. Please review.

Project coverage is 81.86%. Comparing base (2c7db77) to head (bd9e694).
Report is 151 commits behind head on master.

Files with missing lines	Patch %	Lines
...rg/spockframework/runtime/DataIteratorFactory.java	40.00%	9 Missing ⚠️
.../java/org/spockframework/runtime/ErrorContext.java	63.15%	7 Missing ⚠️
...main/java/org/spockframework/compiler/AstUtil.java	50.00%	2 Missing ⚠️
...ava/org/spockframework/compiler/SpecAnnotator.java	66.66%	1 Missing and 1 partial ⚠️
...java/org/spockframework/compiler/SpecRewriter.java	94.59%	0 Missing and 2 partials ⚠️
...ockframework/runtime/extension/IBlockListener.java	0.00%	2 Missing ⚠️
...va/org/spockframework/runtime/model/BlockInfo.java	66.66%	2 Missing ⚠️
.../java/org/spockframework/compiler/model/Block.java	83.33%	1 Missing ⚠️
...org/spockframework/runtime/PlatformSpecRunner.java	75.00%	1 Missing ⚠️
...va/org/spockframework/runtime/model/ErrorInfo.java	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #1575      +/-   ##
============================================
+ Coverage     80.44%   81.86%   +1.41%     
- Complexity     4337     4639     +302     
============================================
  Files           441      450       +9     
  Lines         13534    14550    +1016     
  Branches       1707     1834     +127     
============================================
+ Hits          10888    11911    +1023     
+ Misses         2008     1958      -50     
- Partials        638      681      +43

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java

kriegaex · 2023-02-17T10:35:00Z

Hi @leonard84. I cannot perform a formal code review, because I am not a committer, but I hope that during the weekend I can find a small time slice to build and play around with it. I simply wanted to say thanks in advance for taking care of this feature request. It has not gone unnoticed.

Quick question: Are you planning to add more commits to this PR? Code changes? User manual? I am just asking, not demanding anything. I simply do not want to start testing too early. As for the user manual, I can of course take a look at the unit tests and take it from there. But that might not be true for all future extension developers, I am just speaking for myself.

leonard84 · 2023-03-10T10:47:32Z

@kriegaex I'll write some documentation for it, but I mainly wanted to get feedback on the feature first, like the one from @szpak concerning the usability.

leonard84 · 2023-03-10T10:50:34Z

...ecs/src/test/groovy/org/spockframework/smoke/ast/condition/ExceptionConditionsAstSpec.groovy

+        try {
+            org.spockframework.runtime.SpockRuntime.callEnterBlock(this.getSpecificationContext(), new org.spockframework.runtime.model.BlockInfo(org.spockframework.runtime.model.BlockKind.WHEN, []))
+            foobar = this.foobar()
+            org.spockframework.runtime.SpockRuntime.callExitBlock(this.getSpecificationContext(), new org.spockframework.runtime.model.BlockInfo(org.spockframework.runtime.model.BlockKind.WHEN, []))


💬 this won't be called, fixing this makes for some awkward interactions between the individual AST transformations,

I tried moving the blockListener logic where the blocks are written back in SpecRewriter#L388, however this didn't work for some cases, e.g. cleanup as it already replaced all Blocks with an anonymous block.

spock-specs/src/test/groovy/org/spockframework/smoke/ast/mock/MocksAstSpec.groovy

spock-core/src/main/java/org/spockframework/runtime/model/ErrorInfo.java

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java

szpak · 2023-10-02T15:42:21Z

As this PR seems a little bit stalled, I pushed the code I initially created ~6 months ago as a PoC for the BlockListener support. It a very raw version (with a lot of diagnostic stuff to be amended in the future), but functional in some basic scenarios. The JPA session, where requested, is flushed after the when block.

https://github.com/szpak/spock-jpa-flush-enforcer/tree/preview1

I would like to awake discussion about future shape of this PR.

@leonard84 @kriegaex WDYT?

leonard84 · 2023-10-06T14:23:29Z

It mostly hangs on the problems that adding the exit listener introduces #1575 (comment)

leonard84 · 2024-03-21T17:26:41Z

I think I've fixed the issues, but I need another fresh set of eyes to verify that everything looks good, I've stared at too many snapshots already.

spock-core/src/main/java/org/spockframework/runtime/ErrorContext.java

spock-core/src/main/java/org/spockframework/runtime/SpecificationContext.java

spock-core/src/main/java/org/spockframework/runtime/SpockRuntime.java

spock-core/src/main/java/org/spockframework/runtime/model/FeatureInfo.java

spock-core/src/main/java/org/spockframework/runtime/model/IErrorContext.java

spock-core/src/main/java/org/spockframework/runtime/ErrorContext.java

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java

leonard84 · 2024-03-21T19:09:55Z

@AndreasTu, as I mentioned earlier, I'll write some documentation and polish it later. Thanks for your comments in any case.

However, I'm looking for a review of the correctness of the implementation and generated code and general usability.

szpak

I've switched my experimental (and heavily work-in-progress) extension to the latest version and it still works as expected without any modification. Nice.

I will think about the possible corner cases, I might encounter there.

Btw, I wonder, if it is still a recommended way to decide if the block listener was intended for the current iteration?

spock-core/src/main/java/org/spockframework/runtime/SpockRuntime.java

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java

spock-specs/src/test/groovy/org/spockframework/smoke/parameterization/DataProviders.groovy

spock-core/src/main/java/org/spockframework/runtime/model/BlockInfo.java

Vampire

Some things missing in multiple places:

license headers
JavaDoc
documentation
@Beta
@since

spock-core/src/main/java/org/spockframework/compiler/AstUtil.java

spock-core/src/main/java/org/spockframework/compiler/DeepBlockRewriter.java

spock-core/src/main/java/org/spockframework/compiler/SpecRewriter.java

spock-core/src/main/java/org/spockframework/runtime/model/ErrorInfo.java

spock-core/src/main/java/org/spockframework/runtime/model/FeatureInfo.java

spock-core/src/main/java/org/spockframework/runtime/model/IErrorContext.java

Vampire · 2024-03-21T22:06:11Z

Btw. could we also support where (and upcoming filter) blocks in a sensible way?

leonard84 · 2024-03-22T00:46:42Z

Btw. could we also support where (and upcoming filter) blocks in a sensible way?

I don't think it would be intuitive or really helpful.
They run outside the normal iteration.
When would you enter or leave the block?

only once when creating the data provider
on every data iterator's iteration?

However, you are most familiar with that part of the code.

Vampire · 2024-03-22T03:57:47Z

When would you enter or leave the block?

No idea. :-D
I guess multiple times.
Maybe with an additional phase enum.
One for creating the data iterators.
One for getting the next set of data variable values.

But we can also start without and see where and how need arises.
But then it needs time to land in production version with our release cadence. :-D

leonard84 · 2024-04-19T16:13:06Z

Forgot to publish my responses 😅

spock-core/src/main/java/org/spockframework/compiler/SpecRewriter.java

spock-core/src/main/java/org/spockframework/runtime/ErrorContext.java

spock-core/src/main/java/org/spockframework/runtime/SpecificationContext.java

spock-core/src/main/java/org/spockframework/lang/ISpecificationContext.java

leonard84 · 2024-04-22T17:50:04Z

Btw, I wonder, if it is still a recommended way to decide if the block listener was intended for the current iteration?

@szpak, can you motivate this use case more? At first glance, I'd try to register a block listener that can handle all iterations instead of one per iteration.

szpak · 2024-04-25T20:00:09Z

@szpak, can you motivate this use case more? At first glance, I'd try to register a block listener that can handle all iterations instead of one per iteration.

Hmm, I think the main problem was to get the invocation instance to obtain the current field instance. As a result, I put IBlockListener inside AbstractMethodInterceptor which provides invocation. Would you propose @leonard84 to pass the invocation instance to the listener in any simpler way?

leonard84 · 2024-05-14T18:19:25Z

@szpak, can you motivate this use case more? At first glance, I'd try to register a block listener that can handle all iterations instead of one per iteration.

Hmm, I think the main problem was to get the invocation instance to obtain the current field instance. As a result, I put IBlockListener inside AbstractMethodInterceptor which provides invocation. Would you propose @leonard84 to pass the invocation instance to the listener in any simpler way?

We could easily pass in the current instance. The question is whether we should.

I've also been debating whether I should give access to the current ISpockExecution so that a listener can get an IStore.

szpak · 2024-05-18T18:19:42Z

Hmm, I think the main problem was to get the invocation instance to obtain the current field instance. As a result, I put IBlockListener inside AbstractMethodInterceptor which provides invocation. Would you propose @leonard84 to pass the invocation instance to the listener in any simpler way?

We could easily pass in the current instance. The question is whether we should.

Definitely that's the good question. What are the design difference between interceptors (which have access to invocation) and listeners (also in work there are intended for)? The first could abort the processing, while listeners should not. Listeners should only perform (external) side effects? What also is different and if invocation in the second case could "complicate" something?

I've also been debating whether I should give access to the current ISpockExecution so that a listener can get an IStore.

Do you see any good cases where it would be necessary for listeners in practice? To read the values set by the other extensions/interceptors?

szpak

Just a friendly reminder about the documentation (preferably with a small example of usage), before merging it.

szpak · 2024-10-19T21:28:12Z

spock-core/src/main/java/org/spockframework/runtime/model/BlockInfo.java

@@ -27,6 +29,15 @@ public class BlockInfo {
  private BlockKind kind;
  private List<String> texts;

+  public BlockInfo() {


Maybe we could mark it as deprecated and possibly remove it in Spock 3.0 (or so)? Spock itself no longer calls the old constructor.
The setters could be also marked as deprecated.

Vampire

Some more comments. :-)

Also, docs are of course still missing.

Vampire · 2024-11-01T01:28:19Z

spock-core/src/main/java/org/spockframework/compiler/AstNodeCache.java

@@ -107,6 +114,12 @@ public class AstNodeCache {
  public final MethodNode SpecificationContext_GetSharedInstance =
      SpecificationContext.getDeclaredMethods(org.spockframework.runtime.SpecificationContext.GET_SHARED_INSTANCE).get(0);

+  public final MethodNode SpecificationContext_GetBlockCurrentBlock =


Suggested change

public final MethodNode SpecificationContext_GetBlockCurrentBlock =

public final MethodNode SpecificationContext_GetCurrentBlock =

Vampire · 2024-11-01T01:28:43Z

spock-core/src/main/java/org/spockframework/compiler/AstNodeCache.java

+  public final MethodNode SpecificationContext_GetBlockCurrentBlock =
+      SpecificationContext.getDeclaredMethods(org.spockframework.runtime.SpecificationContext.GET_CURRENT_BLOCK).get(0);
+
+  public final MethodNode SpecificationContext_SetBlockCurrentBlock =


Suggested change

public final MethodNode SpecificationContext_SetBlockCurrentBlock =

public final MethodNode SpecificationContext_SetCurrentBlock =

Vampire · 2024-11-01T01:32:52Z

spock-core/src/main/java/org/spockframework/compiler/AstUtil.java

+
+  public static BinaryExpression createVariableIsNotNullExpression(VariableExpression var) {
+    return new BinaryExpression(
+      new VariableExpression(var),


Do we need to create a new VariableExpression from the supplied one?

Vampire · 2024-11-01T01:32:57Z

spock-core/src/main/java/org/spockframework/compiler/AstUtil.java

+
+  public static BinaryExpression createVariableIsNullExpression(VariableExpression var) {
+    return new BinaryExpression(
+      new VariableExpression(var),


Do we need to create a new VariableExpression from the supplied one?

Vampire · 2024-11-01T01:33:41Z

spock-core/src/main/java/org/spockframework/compiler/AstUtil.java

+      new ConstantExpression(null));
+  }
+
+  public static BinaryExpression createVariableIsNullExpression(VariableExpression var) {


Is it intended that this unused method is added for some future use or is it a left-over from being used at some point?

Vampire · 2024-11-01T03:14:28Z

spock-specs/src/test/groovy/org/spockframework/smoke/parameterization/DataProviders.groovy

@@ -152,7 +133,10 @@ public java.lang.Object $spock_feature_0_0proc(java.lang.Object $spock_p0) {
  }

  @Issue("https://github.com/spockframework/spock/issues/1287")
-  def "data variable with asserting closure produces error rethrower variable in data processor method"() {
+  def "data variable with asserting closure produces error rethrower variable in data processor method"(@Snapshot SpockSnapshotter snapshotter) {


Suggested change

def "data variable with asserting closure produces error rethrower variable in data processor method"(@Snapshot SpockSnapshotter snapshotter) {

def "data variable with asserting closure produces error rethrower variable in data processor method"(@Snapshot(extension = 'groovy') SpockSnapshotter snapshotter) {

Vampire · 2024-11-01T03:22:07Z

spock-core/src/main/java/org/spockframework/compiler/SpecParser.java

@@ -200,6 +200,12 @@ private void buildBlocks(Method method) throws InvalidSpecCompileException {
    checkIsValidSuccessor(method, BlockParseInfo.METHOD_END,
        method.getAst().getLastLineNumber(), method.getAst().getLastColumnNumber());

+    // set the block metaData index for each block this must be equal to the index of the block in the @BlockMetadata annotation


Suggested change

// set the block metaData index for each block this must be equal to the index of the block in the @BlockMetadata annotation

// set the block metadata index for each block this must be equal to the index of the block in the @BlockMetadata annotation

Vampire · 2024-11-01T03:24:04Z

spock-core/src/main/java/org/spockframework/compiler/model/Block.java

@@ -31,6 +31,7 @@ public abstract class Block extends Node<Method, List<Statement>> {
  private final List<String> descriptions = new ArrayList<>(3);
  private Block prev;
  private Block next;
+  private int blockMetaDataIndex = -1;


Suggested change

private int blockMetaDataIndex = -1;

private int blockMetadataIndex = -1;

Vampire · 2024-11-01T03:24:20Z

spock-core/src/main/java/org/spockframework/compiler/model/Block.java

@@ -80,5 +81,25 @@ public boolean isFirstInChain() {
    return isFirst() || getClass() != prev.getClass();
  }

+  public void setBlockMetaDataIndex(int blockMetaDataIndex) {


Suggested change

public void setBlockMetaDataIndex(int blockMetaDataIndex) {

public void setBlockMetadataIndex(int blockMetadataIndex) {

Vampire · 2024-11-01T03:24:36Z

spock-core/src/main/java/org/spockframework/compiler/model/Block.java

+    this.blockMetaDataIndex = blockMetaDataIndex;
+  }
+
+  public int getBlockMetaDataIndex() {


Suggested change

public int getBlockMetaDataIndex() {

public int getBlockMetadataIndex() {

This feature allows extension authors to register a IBlockListener for a feature to observe the execution of a feature in more detail. This surfaces some of Spock's idiosyncrasies, for example interaction assertions are actually setup right before entering the preceding `when`-block as well as being evaluated on leaving the `when`-block before actually entering the `then`-block. The only valid block description is a constant String, although some users mistakenly try to use a dynamic GString. Using anything other than a String, will be treated as a separate statement and thus ignored.

Prior to this commit, IRunListener.error(ErrorInfo) didn't give any context where the error happened.

Vampire · 2024-11-04T02:53:34Z

spock-core/src/main/java/org/spockframework/compiler/SpecRewriter.java

+
+  private void addBlockListeners(Block block) {
+    BlockParseInfo blockType = block.getParseInfo();
+    if (blockType == BlockParseInfo.WHERE


Is FILTER missing here?

leonard84 mentioned this pull request Feb 16, 2023

Adding block listener support #111

Closed

szpak reviewed Feb 16, 2023

View reviewed changes

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java Outdated Show resolved Hide resolved

leonard84 force-pushed the add-block-listener-support branch from e49b0e3 to f1d1840 Compare March 10, 2023 10:46

leonard84 commented Mar 10, 2023

View reviewed changes

szpak reviewed Mar 13, 2023

View reviewed changes

spock-core/src/main/java/org/spockframework/runtime/model/ErrorInfo.java Outdated Show resolved Hide resolved

spock-core/src/main/java/org/spockframework/runtime/extension/IBlockListener.java Outdated Show resolved Hide resolved

leonard84 force-pushed the add-block-listener-support branch from c3f30eb to 0f3197e Compare May 7, 2023 05:41

leonard84 force-pushed the add-block-listener-support branch from 0f3197e to ae38bda Compare February 25, 2024 16:53

leonard84 force-pushed the add-block-listener-support branch from ae38bda to 353cbf9 Compare March 21, 2024 17:20

leonard84 requested review from Vampire and AndreasTu March 21, 2024 17:24

AndreasTu requested changes Mar 21, 2024

View reviewed changes

szpak reviewed Mar 21, 2024

View reviewed changes

Vampire requested changes Mar 21, 2024

View reviewed changes

Vampire requested changes Apr 20, 2024

View reviewed changes

leonard84 force-pushed the add-block-listener-support branch from 5a2f3f2 to c9ee05f Compare April 28, 2024 11:35

AndreasTu previously approved these changes Oct 18, 2024

View reviewed changes

leonard84 enabled auto-merge (squash) October 19, 2024 16:54

szpak reviewed Oct 19, 2024

View reviewed changes

szpak previously approved these changes Oct 19, 2024

View reviewed changes

Vampire requested changes Nov 1, 2024

View reviewed changes

leonard84 added 21 commits November 1, 2024 11:30

Expose IErrorContext in ErrorInfo...

cc16dda

Prior to this commit, IRunListener.error(ErrorInfo) didn't give any context where the error happened.

Clear current block on exit from feature method

818f362

Update AST tests with block listeners

d3cf3da

Add blockExited to IBlockListener

68fd3bf

Add interaction example AstSpec

9a3fef4

Fix block listeners for interactions

ad3d442

Implement Review Feedback

5f321ec

Fix test in RunListenerSpec

ce7d36e

Implement Review comments

753a778

Move current block clearing to PlatformSpecRunner

1efce7c

Move DataProviders expected ast to snapshots

5b03edc

Restore failed block when cleanup runs

ccc712e

Implement Review comments re SpecificationContext

f4a3f29

Implement Review comments re ErrorContext/Info

878c8ff

Revert all changes to DeepBlockRewriter

cd30bcf

Implement Review Comments

abb8a90

Use BlockInfo from SpecInfo via an index lookup

5a78a05

Polishing

73bb879

Apply review comments

55138de

Add minimal documentation

bd9e694

leonard84 dismissed stale reviews from szpak and AndreasTu via bd9e694 November 1, 2024 14:01

leonard84 force-pushed the add-block-listener-support branch from c34f817 to bd9e694 Compare November 1, 2024 14:01

Vampire reviewed Nov 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BlockListener support... #1575

Add BlockListener support... #1575

leonard84 commented Feb 16, 2023

codecov bot commented Feb 16, 2023 •

edited

Loading

kriegaex commented Feb 17, 2023

leonard84 commented Mar 10, 2023

leonard84 Mar 10, 2023

leonard84 Mar 10, 2023

szpak commented Oct 2, 2023 •

edited

Loading

leonard84 commented Oct 6, 2023

leonard84 commented Mar 21, 2024

leonard84 commented Mar 21, 2024

szpak left a comment •

edited

Loading

Vampire left a comment

Vampire commented Mar 21, 2024

leonard84 commented Mar 22, 2024

Vampire commented Mar 22, 2024

leonard84 commented Apr 19, 2024

leonard84 commented Apr 22, 2024

szpak commented Apr 25, 2024 •

edited

Loading

leonard84 commented May 14, 2024

szpak commented May 18, 2024 •

edited

Loading

szpak left a comment •

edited

Loading

szpak Oct 19, 2024

Vampire left a comment

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 1, 2024

Vampire Nov 4, 2024

	public final MethodNode SpecificationContext_GetBlockCurrentBlock =
	public final MethodNode SpecificationContext_GetCurrentBlock =

	def "data variable with asserting closure produces error rethrower variable in data processor method"(@Snapshot SpockSnapshotter snapshotter) {
	def "data variable with asserting closure produces error rethrower variable in data processor method"(@Snapshot(extension = 'groovy') SpockSnapshotter snapshotter) {

	// set the block metaData index for each block this must be equal to the index of the block in the @BlockMetadata annotation
	// set the block metadata index for each block this must be equal to the index of the block in the @BlockMetadata annotation

	private int blockMetaDataIndex = -1;
	private int blockMetadataIndex = -1;

	public void setBlockMetaDataIndex(int blockMetaDataIndex) {
	public void setBlockMetadataIndex(int blockMetadataIndex) {

	public int getBlockMetaDataIndex() {
	public int getBlockMetadataIndex() {

Add BlockListener support... #1575

Are you sure you want to change the base?

Add BlockListener support... #1575

Conversation

leonard84 commented Feb 16, 2023

codecov bot commented Feb 16, 2023 • edited Loading

Codecov Report

kriegaex commented Feb 17, 2023

leonard84 commented Mar 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szpak commented Oct 2, 2023 • edited Loading

leonard84 commented Oct 6, 2023

leonard84 commented Mar 21, 2024

leonard84 commented Mar 21, 2024

szpak left a comment • edited Loading

Choose a reason for hiding this comment

Vampire left a comment

Choose a reason for hiding this comment

Vampire commented Mar 21, 2024

leonard84 commented Mar 22, 2024

Vampire commented Mar 22, 2024

leonard84 commented Apr 19, 2024

leonard84 commented Apr 22, 2024

szpak commented Apr 25, 2024 • edited Loading

leonard84 commented May 14, 2024

szpak commented May 18, 2024 • edited Loading

szpak left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vampire left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 16, 2023 •

edited

Loading

szpak commented Oct 2, 2023 •

edited

Loading

szpak left a comment •

edited

Loading

szpak commented Apr 25, 2024 •

edited

Loading

szpak commented May 18, 2024 •

edited

Loading

szpak left a comment •

edited

Loading