Issue with AsNoTrackingWithIdentityResolution and JSON columns after upgrading to EF Core 9

[lybe86]

Bug description

After upgrade from EFCore 8 to 9 i get an error when i use AsNoTrackingWithIdentityResolution whenever i include a collection of entities that has a JSON-column. I wrote a small console application with sqlite to show the problem.

With EF Core 8 this was not an issue and I could not find anything about this in the breaking changes section. Is there anything I can do to avoid this error, other than removing the JSON column or turning off identity resolution?

Your code

using Microsoft.EntityFrameworkCore;

using var db = new MyDbContext();
db.Database.EnsureCreated();

var boom = db.Parents.Include(tl => tl.Children).AsNoTrackingWithIdentityResolution().ToList();

public class MyDbContext : DbContext
{
    public DbSet<ParentEntity> Parents { get; set; }
    public DbSet<ChildEntity> Children { get; set; }
    
    protected override void OnConfiguring(DbContextOptionsBuilder options)
        => options.UseSqlite("DataSource=:memory:");

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);
        modelBuilder.Entity<ChildEntity>().OwnsOne(e => e.Settings, buildAction =>
        {
            buildAction.ToJson();
        });
    }
}

public class ParentEntity
{
    public Guid Id { get; set; }
    public List<ChildEntity> Children { get; } = new();
}

public class ChildEntity
{
    public Guid Id { get; set; }
    public SettingsAsJson Settings { get; set; }
}

public class SettingsAsJson
{
    public string SomeSettingsHereAsJson { get; set; }
}

Stack traces

System.InvalidOperationException: Projecting queryable operations on JSON collection is not supported for 'NoTrackingWithIdentityResolution'.
    at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonCorrectOrderOfEntitiesForChangeTrackerValidator.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonCorrectOrderOfEntitiesForChangeTrackerValidator.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonCorrectOrderOfEntitiesForChangeTrackerValidator.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonCorrectOrderOfEntitiesForChangeTrackerValidator.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.JsonCorrectOrderOfEntitiesForChangeTrackerValidator.Validate(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.ProcessShaper(Expression shaperExpression, Expression& relationalCommandResolver, IReadOnlyList`1& readerColumns, LambdaExpression& relatedDataLoaders, Int32& collectionId)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.VisitShapedQuery(ShapedQueryExpression shapedQueryExpression)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.VisitExtension(Expression extensionExpression)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutorExpression[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass11_0`1.<ExecuteCore>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteCore[TResult](Expression query, Boolean async, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)

Verbose output

EF Core version

9.0.0

Database provider

sql/sqlite

Target framework

9.0

Operating system

No response

IDE

No response

[maumar]

Relevant change: 74d287f

Using query operations on JSON collections with NTWIR was disabled, due to potential data corruption (e.g. when some elements of the collection are filtered out. We test for JsonProjectionInfo inside RelationalCollectionShaperExpression. However, Include should be safe - filtered include doesn't work with owned entities, so when we are inside include will always get all the json entities in the collection (i.e. the data corruption problem doesn't manifest)

@lybe86 the only thing you can do for now is to break the query apart and stitch them on the client

            var children = await ctx.Children.Select(x => new { x, fk = EF.Property<Guid>(x, "ParentEntityId") }).AsNoTrackingWithIdentityResolution().ToListAsync();
            var parents = await ctx.Parents.AsNoTrackingWithIdentityResolution().ToListAsync();
            var stitched = parents.Select(x => new { x.Id, Children = children.Where(xx => xx.fk == x.Id).ToList() }).ToList();

but this workaround is not great.

[lybe86]

@maumar I guess my followup question is if this is something likely to change back or not, re-writing to stitch things together is ofcourse an option but also one i would like to avoid if I can simply postpone upgrading instead.

[maumar]

@lybe86 I need to do a bit more investigation if Include is indeed safe to do in all cases - if so, the fix should be easy and likely available in EF10. When it comes to patching for EF9, I will discuss with the team. It's a regression and likely an easy fix, but at the same time it's a low priority/impact scenario - those often get rejected.

Another option could be switching to regular tracking queries - what was your reason for using NTWIR instead?

[lybe86]

Well, one scenario we have is that we have some fairly dynamic rules that look through our userbase and do things to other parts of the system.

So without NoTracking we risk something changing entities that should not be changed if anything sets any properties on a user or an attribute or whatever, low risk but still. Also if the user is already in the changetracker without included things we might get wrong data.
With NoTracking we got lots of duplicates lower down the entity hierarchy that used lots of memory.
NoTrackingWithIdentityResoluton seemed to be the right fit for this as we get both the readonly properties, but also avoid the memory waste.

So for instance one such chain we had was User -> AttributeValue -> AttributeDefinition

Where the definition has some configuration about a specific attribute, and each user has lots of attributevalues but there are only a dozen or so attribute definitions.

It is very possible that the pattern we are using is inefficient and that there are better ways doing things though such as shortening the chain and not have attribute definitions below each specific attribute value but having a lookup type of thing.