feat: add support for nested transaction rollbacks via savepoints in sql #4375

LucianBuzzo · 2023-10-17T21:59:12Z

This is my first OSS contribution for a Rust project, so I'm sure I've made some stupid mistakes, but I think it should mostly work :)

This change adds a mutable depth counter, that can track how many levels deep a transaction is, and uses savepoints to implement correct rollback behaviour. Previously, once a nested transaction was complete, it would be saved with COMMIT, meaning that even if the outer transaction was rolled back, the operations in the inner transaction would persist. With this change, if the outer transaction gets rolled back, then all inner transactions will also be rolled back.

Different flavours of SQL servers have different syntax for handling savepoints, so I've had to add new methods to the Queryable trait for getting the commit and rollback statements. These are both parameterized by the current depth.

I've additionally had to modify the begin_statement method to accept a depth parameter, as it will need to conditionally create a savepoint.

CLAassistant · 2023-10-17T22:00:40Z

All committers have signed the CLA.

LucianBuzzo · 2023-10-18T16:01:39Z

This should resolve the issue described here prisma/prisma#15212

LucianBuzzo · 2023-10-20T08:32:05Z

@miguelff @Weakky If you have a moment could you take a look a this? If it works as I think it does ( 🤞 ) it's going to fix a lot of headaches for me and my team! TIA 🙏

janpio · 2023-10-27T00:55:55Z

Would you consider this a breaking change compared to current behavior @LucianBuzzo? It smells a bit like that to me because current functionality would change. Agree?

LucianBuzzo · 2023-10-27T08:25:10Z

@janpio I think that the current behaviour is unexpected, so this would be a bug fix or new "feature". It's possible that people have made applications that rely on the current behaviour, but this is true of any bug IMO.
As an example of how I consider this a bug: the documentation for transactions shows an example where a transfer between two accounts I rolled back, but it's not explained that this will not work if the transaction is nested. If you try to make behaviour like this and couple it with something like the RLS example from the client extensions, it will fail and the user has to do a lot of debugging to find out why.

quaint/src/connector/mssql.rs

janpio · 2023-10-28T22:03:08Z

Thanks for the explanation @LucianBuzzo, I get it now!

I could also connect it to prisma/prisma#19346 then which is about the same problem.

Did you find a bug report about the incorrect behavior? Optimally this issue would close that one so we have a closed bug in our release notes when we add this, that makes it clearer that this is a "breaking change" only in the context of that it fixes a bug. (Maybe you can create the issue if non exists yet!? Thanks.)

(There are also the related prisma/prisma#9710 and prisma/prisma#12898, but I think they want to completely replace transactions with BEGIN/COMMIT with savepoints - which this PR does not do. Do you agree that there is no overlap?)

This should probably be documented in the future with a new "Nested interactive transactions" sub headline under https://www.prisma.io/docs/concepts/components/prisma-client/transactions#interactive-transactions?

What would be good test cases for prisma/prisma? (Optimally those fail right now, but will succeed when this PR here is merged to show that this properly improves things)

janpio · 2023-10-28T22:04:38Z

You can ignore all the failing "Driver Adapters" tests, and also the Vitess and MySQL on Linux ones - those are currently flaky.

But these look relevant and need to be fixed:

Lots of formatting things: https://github.com/prisma/prisma-engines/actions/runs/6553297366/job/18151456208?pr=4375
Some Rust things: https://github.com/prisma/prisma-engines/actions/runs/6553297349/job/18151455513?pr=4375

LucianBuzzo · 2023-10-29T07:40:05Z

Thanks for the review @janpio I'll get the code issues resolved.

The issue prisma/prisma#19346 describes the exact problem, I just haven't referenced it as I wanted to have this PR reviewed first. For prisma/prisma#9710 and prisma/prisma#12898 this PR will resolve those issues, as long as they are using an SQL DB.
To get integration tests in an isolated transaction, you would simply need to run your test cases inside an interactive transaction:

describe('some test', () => {
  it('should work', async () => {
    try {
    await prisma.$transaction(async (tx) => {
      //...
      expect(x).toBe(y)
      
      throw new Error('rollback')
    })
    } catch (e) {
      // ignore e
    }
  })
 })

This is a similar pattern to how I discovered this issue myself - trying to test if a user is able to perform an operation when running Yates RBAC.

I'll add some lines about nested transactions to the docs. As for a test case in https://github.com/prisma/prisma a simple way to do it would be to create a client extension that wraps all queries in a transaction and then test that the interactive transaction example in the docs (transferring money between accounts) works as expected.

janpio · 2023-10-29T10:47:00Z

Wouldn't it then also be possible to just create a standalone test case that wraps multiple transactions, without the need to change how we run tests or use an extension? (I would assume the extension just does that automatically, but it should be possible to also express that explicitly and simpler)

For prisma/prisma#9710 and prisma/prisma#12898 this PR will resolve those issues, as long as they are using an SQL DB.

Don't these suggest to use just savepoints for the actual tests, instead of normal transactions? I am still a bit unclear about the approach.

LucianBuzzo · 2023-10-29T12:01:56Z

From my reading of those issues, it seems that you could achieve the result they want using a nested transaction that utilises savepoints. I would let the original authors correct me if this is not the case though!

LucianBuzzo · 2023-10-29T12:03:16Z

And yes you could not use a client extension in the test case, I simply provided the example above as one possibility 👍

janpio · 2023-10-29T12:11:36Z

CI run is done, some more Clippy stuff to make it able to compile I guess: https://github.com/prisma/prisma-engines/actions/runs/6682077529/job/18158479734?pr=4375 (I also don't know any Rust, so unfortunately can't be of help here)

janpio · 2023-10-29T16:19:21Z

(Can you do a fake PR to README.md adding a newline or some other minimal, non-intrusive change that I can merge easily? Then your commits in this PR will automatically run tests going forward - and you don't have to wait for me to click the button😆)

LucianBuzzo · 2023-10-30T10:18:56Z

@janpio Here's a PR for the Quaint README #4399

LucianBuzzo · 2023-10-30T10:33:40Z

@janpio I've also opened a PR with a failing test case for the prisma client here prisma/prisma#21678

Jolg42 · 2023-11-01T14:19:48Z

Note: I brought the branch into the repo, it will release the engines automatically after a while
https://github.com/prisma/prisma-engines/tree/integration/sql-nested-transactions
https://buildkite.com/prisma/test-prisma-engines/builds/21364

janpio · 2023-11-01T15:54:36Z

Happened, version shared here: prisma/prisma#21678 (comment)

LucianBuzzo · 2023-11-02T08:43:50Z

@janpio @Jolg42 I think that this commit 5d2cc0a should allow us to send an existing transaction ID to the engine in the case of a nested transaction. I'm not sure how to test this behaviour in this repo - any tips on how to do this, or an existing test I could use as an example?

This change adds support for handling rollbacks in nested transactions in SQL databases. Specifically, the inner transaction should be rolled back if the outer transaction fails. To do this we keep track of the transaction ID and transaction depth so we can re-use an existing open transaction in the underlying engine. This change also allows the use of the `$transaction` method on an interactive transaction client. depends-on: prisma/prisma-engines#4375

jasonmacdonald · 2024-04-03T16:40:45Z

I anxiously await this feature. I hope this gets merged soon!

janpio · 2024-04-03T17:51:57Z

query-engine/driver-adapters/src/transaction.rs

@@ -116,6 +156,18 @@ impl Queryable for JsTransaction {

    fn requires_isolation_first(&self) -> bool {
        self.inner.requires_isolation_first()
+    } 


Suggested change

}

}

FominSergiy · 2024-05-06T14:09:12Z

hi there! faced similar issue with nested transaction rollbacks - would be great to have this feature in place :)
do you know by any chance when are you planning to add it to a future release?

luisgrases · 2024-06-06T15:07:00Z

Any updates on this feature?

FominSergiy · 2024-06-06T15:33:10Z

I wonder if it's only the comment suggestion that's missing - if that's the case, are we ok with one of us making a change? 🙂

janpio · 2024-06-06T15:44:21Z

No. Running the tests in prisma/prisma against an engine built from this branch does not succeed.
The comment is just a nitpick to get the linting to pass here probably.

LucianBuzzo · 2024-06-06T16:03:39Z

@luisgrases @FominSergiy
This PR requires additional work, as the corresponding PR to the prisma/prisma repo also needs to succeed. Unfortunately I've been flat out with my day job and haven't had much time to work on this feature as I'd like. With a little luck I should get some solid time on it early next week 👍

This change adds support for handling rollbacks in nested transactions in SQL databases. Specifically, the inner transaction should be rolled back if the outer transaction fails. To do this we keep track of the transaction ID and transaction depth so we can re-use an existing open transaction in the underlying engine. This change also allows the use of the `$transaction` method on an interactive transaction client. depends-on: prisma/prisma-engines#4375

LucianBuzzo · 2024-08-29T17:00:56Z

some solid time on it early next week

🤦
Well I finally got back to working on this - @janpio or @SevInf are you able to trigger a new build so I can work on the prisma/prisma PR? Thanks! 🙏

aqrln · 2024-08-30T16:04:28Z

@LucianBuzzo here you go: prisma/prisma#25121 (5.20.0-2.integration-nested-transactions-70e4e8eaa9b1ca0ec042fa831bc2c81f668215a7)

This change adds support for handling rollbacks in nested transactions in SQL databases. Specifically, the inner transaction should be rolled back if the outer transaction fails. To do this we keep track of the transaction ID and transaction depth so we can re-use an existing open transaction in the underlying engine. This change also allows the use of the `$transaction` method on an interactive transaction client. depends-on: prisma/prisma-engines#4375

LucianBuzzo · 2024-08-30T21:27:34Z

Makefile

@@ -407,8 +407,8 @@ ensure-prisma-present:
 		  echo "⚠️ ../prisma diverges from prisma/prisma main branch. Test results might diverge from those in CI ⚠️ "; \
 		fi \
 	else \
-		echo "git clone --depth=1 https://github.com/prisma/prisma.git --branch=$(DRIVER_ADAPTERS_BRANCH) ../prisma"; \
-		git clone --depth=1 https://github.com/prisma/prisma.git --branch=$(DRIVER_ADAPTERS_BRANCH) "../prisma" && echo "Prisma repository has been cloned to ../prisma"; \
+		echo "git clone --depth=1 https://github.com/LucianBuzzo/prisma.git --branch=lucianbuzzo/nested-rollbacks ../prisma"; \


This will need to be changed back before merge

LucianBuzzo · 2024-08-30T22:01:20Z

I've fixed the last lot of failing tests on prisma/prisma#21678 so I think this PR is finally ready 👍

This is my first OSS contribution for a Rust project, so I'm sure I've made some stupid mistakes, but I think it should mostly work :) This change adds a mutable depth counter, that can track how many levels deep a transaction is, and uses savepoints to implement correct rollback behaviour. Previously, once a nested transaction was complete, it would be saved with `COMMIT`, meaning that even if the outer transaction was rolled back, the operations in the inner transaction would persist. With this change, if the outer transaction gets rolled back, then all inner transactions will also be rolled back. Different flavours of SQL servers have different syntax for handling savepoints, so I've had to add new methods to the `Queryable` trait for getting the commit and rollback statements. These are both parameterized by the current depth. I've additionally had to modify the `begin_statement` method to accept a depth parameter, as it will need to conditionally create a savepoint. When opening a transaction via the transaction server, you can now pass the prior transaction ID to re-use the existing transaction, incrementing the depth. Signed-off-by: Lucian Buzzo <[email protected]>

This change adds support for handling rollbacks in nested transactions in SQL databases. Specifically, the inner transaction should be rolled back if the outer transaction fails. To do this we keep track of the transaction ID and transaction depth so we can re-use an existing open transaction in the underlying engine. This change also allows the use of the `$transaction` method on an interactive transaction client. depends-on: prisma/prisma-engines#4375

LucianBuzzo · 2024-09-17T21:25:10Z

@aqrln Based on your feedback, I reviewed this PR and was able to refactor it such that the transaction depth is stored entirely in the transaction rather than being borrowed from the connector, which is much cleaner 👌

The depth tracking can be encapsulated entirely inside a transaction instance, simplifying the code significantly.

LucianBuzzo requested a review from a team as a code owner October 17, 2023 21:59

LucianBuzzo requested review from miguelff and Weakky and removed request for a team October 17, 2023 21:59

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from 6f375b8 to b984ae4 Compare October 17, 2023 22:00

janpio reviewed Oct 28, 2023

View reviewed changes

quaint/src/connector/mssql.rs Outdated Show resolved Hide resolved

janpio self-assigned this Oct 28, 2023

This comment was marked as off-topic.

Sign in to view

janpio removed request for miguelff and Weakky October 29, 2023 16:19

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from da206e7 to a8af640 Compare October 30, 2023 10:16

LucianBuzzo mentioned this pull request Oct 30, 2023

feat: Add test for nested transaction rollback behaviour in SQL databases prisma/prisma#21678

Open

janpio reviewed Apr 3, 2024

View reviewed changes

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from 3a8bf35 to f46b8fb Compare August 29, 2024 13:49

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from 70e4e8e to 8f9754d Compare August 30, 2024 21:26

LucianBuzzo commented Aug 30, 2024

View reviewed changes

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from 8f9754d to 050a34b Compare August 30, 2024 21:28

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from 050a34b to f87a776 Compare September 17, 2024 10:05

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch 2 times, most recently from 18f0c3d to 6b828ff Compare September 17, 2024 21:22

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from 6b828ff to eb76cc8 Compare September 18, 2024 09:12

encapsulate transaction depth inside transaction instannce

b0bab65

The depth tracking can be encapsulated entirely inside a transaction instance, simplifying the code significantly.

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from eb76cc8 to b0bab65 Compare September 18, 2024 09:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for nested transaction rollbacks via savepoints in sql #4375

feat: add support for nested transaction rollbacks via savepoints in sql #4375

LucianBuzzo commented Oct 17, 2023

CLAassistant commented Oct 17, 2023 •

edited

Loading

LucianBuzzo commented Oct 18, 2023

LucianBuzzo commented Oct 20, 2023

janpio commented Oct 27, 2023

LucianBuzzo commented Oct 27, 2023

janpio commented Oct 28, 2023

janpio commented Oct 28, 2023

This comment was marked as off-topic.

LucianBuzzo commented Oct 29, 2023 •

edited

Loading

janpio commented Oct 29, 2023

LucianBuzzo commented Oct 29, 2023

LucianBuzzo commented Oct 29, 2023

janpio commented Oct 29, 2023

janpio commented Oct 29, 2023

LucianBuzzo commented Oct 30, 2023

LucianBuzzo commented Oct 30, 2023

Jolg42 commented Nov 1, 2023

janpio commented Nov 1, 2023

LucianBuzzo commented Nov 2, 2023

jasonmacdonald commented Apr 3, 2024

janpio Apr 3, 2024

FominSergiy commented May 6, 2024

luisgrases commented Jun 6, 2024

FominSergiy commented Jun 6, 2024

janpio commented Jun 6, 2024 •

edited

Loading

LucianBuzzo commented Jun 6, 2024

LucianBuzzo commented Aug 29, 2024 •

edited

Loading

aqrln commented Aug 30, 2024

LucianBuzzo Aug 30, 2024

LucianBuzzo commented Aug 30, 2024

LucianBuzzo commented Sep 17, 2024

feat: add support for nested transaction rollbacks via savepoints in sql #4375

Are you sure you want to change the base?

feat: add support for nested transaction rollbacks via savepoints in sql #4375

Conversation

LucianBuzzo commented Oct 17, 2023

CLAassistant commented Oct 17, 2023 • edited Loading

LucianBuzzo commented Oct 18, 2023

LucianBuzzo commented Oct 20, 2023

janpio commented Oct 27, 2023

LucianBuzzo commented Oct 27, 2023

janpio commented Oct 28, 2023

janpio commented Oct 28, 2023

This comment was marked as off-topic.

LucianBuzzo commented Oct 29, 2023 • edited Loading

janpio commented Oct 29, 2023

LucianBuzzo commented Oct 29, 2023

LucianBuzzo commented Oct 29, 2023

janpio commented Oct 29, 2023

janpio commented Oct 29, 2023

LucianBuzzo commented Oct 30, 2023

LucianBuzzo commented Oct 30, 2023

Jolg42 commented Nov 1, 2023

janpio commented Nov 1, 2023

LucianBuzzo commented Nov 2, 2023

jasonmacdonald commented Apr 3, 2024

janpio Apr 3, 2024

Choose a reason for hiding this comment

FominSergiy commented May 6, 2024

luisgrases commented Jun 6, 2024

FominSergiy commented Jun 6, 2024

janpio commented Jun 6, 2024 • edited Loading

LucianBuzzo commented Jun 6, 2024

LucianBuzzo commented Aug 29, 2024 • edited Loading

aqrln commented Aug 30, 2024

LucianBuzzo Aug 30, 2024

Choose a reason for hiding this comment

LucianBuzzo commented Aug 30, 2024

LucianBuzzo commented Sep 17, 2024

CLAassistant commented Oct 17, 2023 •

edited

Loading

LucianBuzzo commented Oct 29, 2023 •

edited

Loading

janpio commented Jun 6, 2024 •

edited

Loading

LucianBuzzo commented Aug 29, 2024 •

edited

Loading