Skip to content

Commit

Permalink
[Host.Outbox] Support for UUIDv7 values and MSSQL side ID generation,…
Browse files Browse the repository at this point in the history
… flexible PK types, refactor

Signed-off-by: Tomasz Maruszak <[email protected]>
  • Loading branch information
zarusz committed Oct 18, 2024
1 parent c2cbd1f commit 87bb9cf
Show file tree
Hide file tree
Showing 88 changed files with 784 additions and 490 deletions.
7 changes: 1 addition & 6 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,6 @@ jobs:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
steps:
# - name: Dump GitHub context
# env:
# GITHUB_CONTEXT: ${{ toJson(github) }}
# run: echo "$GITHUB_CONTEXT"

- name: PR - Checkout
if: github.event_name == 'pull_request_target'
uses: actions/checkout@v3
Expand Down Expand Up @@ -195,7 +190,7 @@ jobs:
name: .NET Tests
path: ./test-results/*.trx
reporter: dotnet-trx
fail-on-error: false
fail-on-error: true

- name: Copy NuGet packages
shell: bash
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,8 @@ SlimMessageBus is a client façade for message brokers for .NET. It comes with i
| `.Host.AspNetCore` | Integration for ASP.NET Core | [![NuGet](https://img.shields.io/nuget/v/SlimMessageBus.Host.AspNetCore.svg)](https://www.nuget.org/packages/SlimMessageBus.Host.AspNetCore) |
| `.Host.Interceptor` | Core interface for interceptors | [![NuGet](https://img.shields.io/nuget/v/SlimMessageBus.Host.Interceptor.svg)](https://www.nuget.org/packages/SlimMessageBus.Host.Interceptor) |
| `.Host.FluentValidation` | Validation for messages based on [FluentValidation](https://www.nuget.org/packages/FluentValidation) | [![NuGet](https://img.shields.io/nuget/v/SlimMessageBus.Host.FluentValidation.svg)](https://www.nuget.org/packages/SlimMessageBus.Host.FluentValidation) |
| `.Host.Outbox.Sql` | Transactional Outbox using SQL | [![NuGet](https://img.shields.io/nuget/v/SlimMessageBus.Host.Outbox.Sql.svg)](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) |
| `.Host.Outbox.DbContext` | Transactional Outbox using EF DbContext | [![NuGet](https://img.shields.io/nuget/v/SlimMessageBus.Host.Outbox.DbContext.svg)](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext) |
| `.Host.Outbox.Sql` | Transactional Outbox using MSSQL | [![NuGet](https://img.shields.io/nuget/v/SlimMessageBus.Host.Outbox.Sql.svg)](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) |
| `.Host.Outbox.Sql.DbContext` | Transactional Outbox using MSSQL with EF DataContext integration | [![NuGet](https://img.shields.io/nuget/v/SlimMessageBus.Host.Outbox.Sql.DbContext.svg)](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext) |
| `.Host.AsyncApi` | [AsyncAPI](https://www.asyncapi.com/) specification generation via [Saunter](https://github.com/tehmantra/saunter) | [![NuGet](https://img.shields.io/nuget/v/SlimMessageBus.Host.AsyncApi.svg)](https://www.nuget.org/packages/SlimMessageBus.Host.AsyncApi) |

Typically the application layers (domain model, business logic) only need to depend on `SlimMessageBus` which is the facade, and ultimately the application hosting layer (ASP.NET, Console App, Windows Service) will reference and configure the other packages (`SlimMessageBus.Host.*`) which are the messaging transport providers and additional plugins.
Expand Down
2 changes: 1 addition & 1 deletion build/tasks.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ $projects = @(

"SlimMessageBus.Host.Outbox",
"SlimMessageBus.Host.Outbox.Sql",
"SlimMessageBus.Host.Outbox.DbContext",
"SlimMessageBus.Host.Outbox.Sql.DbContext",

"SlimMessageBus.Host.AsyncApi"
)
Expand Down
19 changes: 10 additions & 9 deletions docs/plugin_outbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,30 +12,31 @@ Please read the [Introduction](intro.md) before reading this provider documentat
- [UseTransactionScope](#usetransactionscope)
- [UseSqlTransaction](#usesqltransaction)
- [How it works](#how-it-works)
- [Important note](#important-note)

## Introduction

The [`Host.Outbox`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox) introduces [Transactional Outbox](https://microservices.io/patterns/data/transactional-outbox.html) pattern to the SlimMessageBus.
It comes in two flavors:

- [`Host.Outbox.Sql`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) as integration with the System.Data.Sql client
- [`Host.Outbox.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext) as integration with Entity Framework Core
- [`Host.Outbox.Sql`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) as integration with the System.Data.Sql client (MSSQL)
- [`Host.Outbox.Sql.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext) as integration with Entity Framework Core

Outbox plugin can work in combination with any transport provider.

## Configuration

### Entity Framework

> Required: [`SlimMessageBus.Host.Outbox.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext)
> Required: [`SlimMessageBus.Host.Outbox.Sql.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext)
```cs
using SlimMessageBus.Host.Outbox.DbContext;
using SlimMessageBus.Host.Outbox.Sql.DbContext;
```

Consider the following example (from [Samples](../src/Samples/Sample.OutboxWebApi/Program.cs)):

- `services.AddOutboxUsingDbContext<CustomerContext>(...)` is used to add the [Outbox.DbContext](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext) plugin to the container.
- `services.AddOutboxUsingDbContext<CustomerContext>(...)` is used to add the [Outbox.DbContext](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext) plugin to the container.
- `CustomerContext` is the application specific Entity Framework `DbContext`.
- `CustomerCreatedEvent` is produced on the `AzureSB` child bus, the bus will deliver these events via outbox - see `.UseOutbox()`
- `CreateCustomerCommand` is consumed on the `Memory` child bus, each command is wrapped in an SQL transaction - see `UseSqlTransaction()`
Expand Down Expand Up @@ -169,10 +170,10 @@ There are two types of transaction support:

#### UseTransactionScope

> Required: [`Host.Outbox`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox)
> Required: [`SlimMessageBus.Host.Outbox`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox)
```cs
using SlimMessageBus.Host.Outbox.Sql;
using SlimMessageBus.Host.Outbox;
```

`.UseTransactionScope()` can be used on consumers (or handlers) declaration to force the consumer to start a `TransactionScope` prior the message `OnHandle` and to complete that transaction after it. Any exception raised by the consumer would cause the transaction to be rolled back.
Expand All @@ -181,7 +182,7 @@ When applied on the (child) bus level then all consumers (or handlers) will inhe

#### UseSqlTransaction

> Required: [`SlimMessageBus.Host.Outbox.Sql`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) or [`SlimMessageBus.Host.Outbox.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext)
> Required: [`SlimMessageBus.Host.Outbox.Sql`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) or [`SlimMessageBus.Host.Outbox.Sql.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext)
```cs
using SlimMessageBus.Host.Outbox.Sql;
Expand Down Expand Up @@ -218,6 +219,6 @@ When applied on the (child) bus level then all consumers (or handlers) will inhe

## Important note

As the outbox can be processed by instance of the application that did not originally process it, it is important to ensure that all active instances maintian the same message registrations (and compatible JSON schema definitions).
As the outbox can be processed by instance of the application that did not originally process it, it is important to ensure that all active instances maintain the same message registrations (and compatible JSON schema definitions).

A message that fails to deserialize will be flagged as invalid by setting the associated `DeliveryAborted` field in the `Outbox` table, to `1`. It is safe to manually reset this field value to `0` once the version incompatibility has been resolved.
21 changes: 11 additions & 10 deletions docs/plugin_outbox.t.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,30 +12,31 @@ Please read the [Introduction](intro.md) before reading this provider documentat
- [UseTransactionScope](#usetransactionscope)
- [UseSqlTransaction](#usesqltransaction)
- [How it works](#how-it-works)
- [Important note](#important-note)

## Introduction

The [`Host.Outbox`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox) introduces [Transactional Outbox](https://microservices.io/patterns/data/transactional-outbox.html) pattern to the SlimMessageBus.
It comes in two flavors:

- [`Host.Outbox.Sql`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) as integration with the System.Data.Sql client
- [`Host.Outbox.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext) as integration with Entity Framework Core
- [`Host.Outbox.Sql`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) as integration with the System.Data.Sql client (MSSQL)
- [`Host.Outbox.Sql.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext) as integration with Entity Framework Core

Outbox plugin can work in combination with any transport provider.

## Configuration

### Entity Framework

> Required: [`SlimMessageBus.Host.Outbox.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext)
> Required: [`SlimMessageBus.Host.Outbox.Sql.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext)
```cs
using SlimMessageBus.Host.Outbox.DbContext;
using SlimMessageBus.Host.Outbox.Sql.DbContext;
```

Consider the following example (from [Samples](../src/Samples/Sample.OutboxWebApi/Program.cs)):

- `services.AddOutboxUsingDbContext<CustomerContext>(...)` is used to add the [Outbox.DbContext](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext) plugin to the container.
- `services.AddOutboxUsingDbContext<CustomerContext>(...)` is used to add the [Outbox.DbContext](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext) plugin to the container.
- `CustomerContext` is the application specific Entity Framework `DbContext`.
- `CustomerCreatedEvent` is produced on the `AzureSB` child bus, the bus will deliver these events via outbox - see `.UseOutbox()`
- `CreateCustomerCommand` is consumed on the `Memory` child bus, each command is wrapped in an SQL transaction - see `UseSqlTransaction()`
Expand Down Expand Up @@ -100,10 +101,10 @@ There are two types of transaction support:

#### UseTransactionScope

> Required: [`Host.Outbox`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox)
> Required: [`SlimMessageBus.Host.Outbox`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox)
```cs
using SlimMessageBus.Host.Outbox.Sql;
using SlimMessageBus.Host.Outbox;
```

`.UseTransactionScope()` can be used on consumers (or handlers) declaration to force the consumer to start a `TransactionScope` prior the message `OnHandle` and to complete that transaction after it. Any exception raised by the consumer would cause the transaction to be rolled back.
Expand All @@ -112,7 +113,7 @@ When applied on the (child) bus level then all consumers (or handlers) will inhe

#### UseSqlTransaction

> Required: [`SlimMessageBus.Host.Outbox.Sql`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) or [`SlimMessageBus.Host.Outbox.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.DbContext)
> Required: [`SlimMessageBus.Host.Outbox.Sql`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql) or [`SlimMessageBus.Host.Outbox.Sql.DbContext`](https://www.nuget.org/packages/SlimMessageBus.Host.Outbox.Sql.DbContext)
```cs
using SlimMessageBus.Host.Outbox.Sql;
Expand Down Expand Up @@ -149,6 +150,6 @@ When applied on the (child) bus level then all consumers (or handlers) will inhe

## Important note

As the outbox can be processed by instance of the application that did not originally process it, it is important to ensure that all active instances maintian the same message registrations (and compatible JSON schema definitions).
As the outbox can be processed by instance of the application that did not originally process it, it is important to ensure that all active instances maintain the same message registrations (and compatible JSON schema definitions).

A message that fails to deserialize will be flagged as invalid by setting the associated `DeliveryAborted` field in the `Outbox` table, to `1`. It is safe to manually reset this field value to `0` once the version incompatibility has been resolved.
A message that fails to deserialize will be flagged as invalid by setting the associated `DeliveryAborted` field in the `Outbox` table, to `1`. It is safe to manually reset this field value to `0` once the version incompatibility has been resolved.
9 changes: 5 additions & 4 deletions docs/provider_kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ When troubleshooting or fine tuning it is worth reading the `librdkafka` and `co

## Configuration properties

Producer, consumer and global configuration properties are described [here](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).
Producer, consumer and global configuration properties are described [here](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md).
The configuration on the underlying Kafka client can be adjusted like so:

```cs
Expand All @@ -53,7 +53,7 @@ services.AddSlimMessageBus(mbb =>

### Minimizing message latency

There is a good description [here](https://github.com/edenhill/librdkafka/wiki/How-to-decrease-message-latency) on improving the latency by applying producer/consumer settings on librdkafka. Here is how you enter the settings using SlimMessageBus:
There is a good description [here](https://github.com/confluentinc/librdkafka/wiki/How-to-decrease-message-latency) on improving the latency by applying producer/consumer settings on librdkafka. Here is how you enter the settings using SlimMessageBus:

```cs
services.AddSlimMessageBus(mbb =>
Expand Down Expand Up @@ -176,7 +176,7 @@ public class PingConsumer : IConsumer<PingMessage>, IConsumerWithContext
{
public IConsumerContext Context { get; set; }

public Task OnHandle(PingMessage message, CancellationToken cancellationToken)
public Task OnHandle(PingMessage message)
{
// SMB Kafka transport specific extension:
var transportMessage = Context.GetTransportMessage();
Expand Down Expand Up @@ -211,7 +211,8 @@ mbb

### Offset Commit

In the current Kafka provider implementation, SMB handles the manual commit of topic-partition offsets for the consumer. This configuration is controlled through the following methods on the consumer builder:
In the current Kafka provider implementation, SMB handles the manual commit of topic-partition offsets for the consumer.Th
is configuration is controlled through the following methods on the consumer builder:

- `CheckpointEvery(int)` – Commits the offset after a specified number of processed messages.
- `CheckpointAfter(TimeSpan)` – Commits the offset after a specified time interval.
Expand Down
2 changes: 1 addition & 1 deletion src/Host.Plugin.Properties.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<Import Project="Common.NuGet.Properties.xml" />

<PropertyGroup>
<Version>3.0.0-rc11</Version>
<Version>3.0.0-rc12</Version>
</PropertyGroup>

</Project>
2 changes: 1 addition & 1 deletion src/Samples/Sample.OutboxWebApi/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@
using SlimMessageBus.Host.AzureServiceBus;
using SlimMessageBus.Host.Memory;
using SlimMessageBus.Host.Outbox;
using SlimMessageBus.Host.Outbox.DbContext;
using SlimMessageBus.Host.Outbox.Sql;
using SlimMessageBus.Host.Outbox.Sql.DbContext;
using SlimMessageBus.Host.Serialization.Json;

// Local file with secrets
Expand Down
4 changes: 1 addition & 3 deletions src/Samples/Sample.OutboxWebApi/Sample.OutboxWebApi.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@
<ProjectReference Include="..\..\SlimMessageBus.Host.AspNetCore\SlimMessageBus.Host.AspNetCore.csproj" />
<ProjectReference Include="..\..\SlimMessageBus.Host.AzureServiceBus\SlimMessageBus.Host.AzureServiceBus.csproj" />
<ProjectReference Include="..\..\SlimMessageBus.Host.Memory\SlimMessageBus.Host.Memory.csproj" />
<ProjectReference Include="..\..\SlimMessageBus.Host.Outbox.DbContext\SlimMessageBus.Host.Outbox.DbContext.csproj" />
<ProjectReference Include="..\..\SlimMessageBus.Host.Outbox.Sql\SlimMessageBus.Host.Outbox.Sql.csproj" />
<ProjectReference Include="..\..\SlimMessageBus.Host.Outbox\SlimMessageBus.Host.Outbox.csproj" />
<ProjectReference Include="..\..\SlimMessageBus.Host.Outbox.Sql.DbContext\SlimMessageBus.Host.Outbox.Sql.DbContext.csproj" />
<ProjectReference Include="..\..\SlimMessageBus.Host.Serialization.Json\SlimMessageBus.Host.Serialization.Json.csproj" />
<ProjectReference Include="..\..\Tools\SecretStore\SecretStore.csproj" />
</ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<Description>Core configuration interfaces of SlimMessageBus</Description>
<PackageTags>SlimMessageBus</PackageTags>
<RootNamespace>SlimMessageBus.Host</RootNamespace>
<Version>3.0.0-rc11</Version>
<Version>3.0.0-rc12</Version>
</PropertyGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<Import Project="../Common.NuGet.Properties.xml" />

<PropertyGroup>
<Version>3.0.0-rc11</Version>
<Version>3.0.0-rc12</Version>
<Description>Core interceptor interfaces of SlimMessageBus</Description>
<PackageTags>SlimMessageBus</PackageTags>
</PropertyGroup>
Expand Down

This file was deleted.

This file was deleted.

Loading

0 comments on commit 87bb9cf

Please sign in to comment.