Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(client): JsonSerializable and JsonDeserialized #21725

Merged
merged 107 commits into from
Mar 11, 2025

Conversation

jason-ha
Copy link
Contributor

@jason-ha jason-ha commented Jul 1, 2024

Add pair of type filters for JSON based serialization.

JsonSerializable<T> produces type representing limitations of serializing T. Incompatible elements are transformed to never or SerializationError* types that original T is not assignable to.

JsonDeserialized<T> produces type representing result of T being serialized and then deserialized.

JsonSerializable should eventually replace @fluidframework/datastore-definitions's Jsonable. That cannot done be currently as it would be a compile-time breaking change.

AB#6887

Additional changes beyond Presence implementation

  • New support added for unknown as and exactly allowed option.
    • AllowExactly option changed to tuple in support.
    • Helper added to detect template literals
    • Helper added to detect index signatures
  • Support for branded primitives (boolean, number, and string)
  • Additional comments and corrections
  • Test coverage for arrays nested in objects

Supporting Changes

Add standard test infrastructure

jason-ha added 30 commits June 20, 2024 17:22
+ test objects with certain properties

Note some new Deserialized tests are failing
+ cleanup error for required property with undefined
for deserialization tests in place of class instances
by using homomorphic mapping with `as`
and misc formatting improvements
and detach expect type from input type T to avoid influencing inference of T
reorganize for documentation and add/correct some docs
@jason-ha jason-ha requested a review from markfields February 26, 2025 19:22
@pragya91
Copy link
Contributor

pragya91 commented Mar 3, 2025

@jason-ha , for "JsonSerializable should eventually replace @fluidframework/datastore-definitions's Jsonable" , we should create a backlog item to track it, with details on which version it would be viable to do so.

@markfields
Copy link
Member

markfields commented Mar 3, 2025

It would be a cool follow-up to have JsonString type that is type branded string that implies that if you parse it you'll get JsonDeserialized or something (not sure exactly how to leverage your types here). And then we write strongly-typed wrappers for JSON.stringify and JSON.parse to yield/use JsonString so we can have strong typing even after serialization.

I've prototyped this a few times, would be very useful in ContainerRuntime layer where we pass around stringified stuff all the time.

@jason-ha
Copy link
Contributor Author

jason-ha commented Mar 4, 2025

It would be a cool follow-up to have JsonString type that is type branded string that implies that if you parse it you'll get JsonDeserialized or something (not sure exactly how to leverage your types here). And then we write strongly-typed wrappers for JSON.stringify and JSON.parse to yield/use JsonString so we can have strong typing even after serialization.

I've prototyped this a few times, would be very useful in ContainerRuntime layer where we pass around stringified stuff all the time.

Maybe I can FHL it. Something like JsonString<T> should be enough and parser/encoder can qualify what is supported for handles and such in their declarations.
function encodeWithHandles<T>(v: JsonSerializable<T, { AllowExtensionOf: IFluidHandle }>): JsonString<T>
function decodeWithHandles<T>(v: JsonString<T>): JsonDeserialized<T, { AllowExtensionOf: IFluidHandle }>
(JsonDeserialized<T, { AllowExtensionOf: [IFluidHandle] }> will just be T for perfectly round-trippable T.)

@jason-ha
Copy link
Contributor Author

jason-ha commented Mar 4, 2025

@jason-ha , for "JsonSerializable should eventually replace @fluidframework/datastore-definitions's Jsonable" , we should create a backlog item to track it, with details on which version it would be viable to do so.

Agreed. If the internal uses continue to look good, we could make a change for 3.0. (The Pages codebase may not be able to transition before - there was some cleanup needed when I made fixes to Jsonable a while back.)

jason-ha added 10 commits March 5, 2025 10:17
- remove extraneous test types
- rename test value for accuracy
- also comment corrections
Separate internal recursion detection use of AllowExactly filter control from user given so that user may specify `unknown`.

A better approach would be to turn AllowExactly into a tuple. Experimentation as needed as in the past, tuple use appeared to cause other issues.
Support primitives intersected with classes to be serialized (`JsonSerializable`). (`JsonDeserialized` already had support.)
Update `AllowExactly` option to accept tuple of types that is more flexible than union as it allows `unknown` to be mixed with other types.
Now that AllowExactly is a tuple ReplacementMaker can be added to it directly instead of requiring a separate control property even when AllowExactly may contain `unknown`.
especially for those with `unknown` value types.
… for users

Instead of some results displaying `FormDegenerate...` use direct formation to have `JsonTypeWith` show up explicitly.
@jason-ha
Copy link
Contributor Author

It would be a cool follow-up to have JsonString type that is type branded string that implies that if you parse it you'll get JsonDeserialized or something (not sure exactly how to leverage your types here). And then we write strongly-typed wrappers for JSON.stringify and JSON.parse to yield/use JsonString so we can have strong typing even after serialization.
I've prototyped this a few times, would be very useful in ContainerRuntime layer where we pass around stringified stuff all the time.

Maybe I can FHL it. Something like JsonString<T> should be enough and parser/encoder can qualify what is supported for handles and such in their declarations. function encodeWithHandles<T>(v: JsonSerializable<T, { AllowExtensionOf: IFluidHandle }>): JsonString<T> function decodeWithHandles<T>(v: JsonString<T>): JsonDeserialized<T, { AllowExtensionOf: IFluidHandle }> (JsonDeserialized<T, { AllowExtensionOf: [IFluidHandle] }> will just be T for perfectly round-trippable T.)

I worked on this during FHL which found some interesting use cases (branded strings and explicit unknown), that resulted in all of the additional changes this last week. That FHL work is not complete, but I don't think there will be needs for additional changes to implementation.
Going to strive to stop making any more changes outside of critical code review feedback.

@@ -415,13 +481,80 @@ describe("JsonDeserialized", () => {
assertIdenticalTypes(resultRead, objectWithNumberKey);
});

it("object with array of supported types (numbers) are preserved", () => {
const resultRead = passThru(objectWithArrayOfNumbers);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have generated the type name based on the variable name, but the fact that you typed out both gives me confidence for how thoroughly you've thought through these test cases!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a lot of cases where the name is exactly what is intended to be covered. Then there are some like this one that is meant to handle a set of things but only tests a specific case. We only test array of numbers, but numbers is proxy for other supported things like booleans, strings, and supported objects.

@jason-ha
Copy link
Contributor Author

All of the tests:

  • JsonDeserialized

    • positive compilation tests
      • supported primitive types are preserved
        -✔ boolean
        -✔ number
        -✔ string
        -✔ numeric enum
        -✔ string enum
        -✔ const heterogenous enum
        -✔ computed enum
        -✔ branded number
        -✔ branded string
      • unions with unsupported primitive types preserve supported types
        -✔ string | symbol
        -✔ bigint | string
        -✔ bigint | symbol
        -✔ number | bigint | symbol
      • supported literal types are preserved
        -✔ true
        -✔ false
        -✔ 0
        -✔ "string"
        -✔ null
        -✔ object with literals
        -✔ array of literals
        -✔ tuple of literals
        -✔ specific numeric enum value
        -✔ specific string enum value
        -✔ specific const heterogenous enum value
        -✔ specific computed enum value
      • arrays
        -✔ array of supported types (numbers) are preserved
        -✔ sparse array is filled in with null
        -✔ array of partially supported (numbers or undefined) is modified with null
        -✔ array of unknown becomes array of JsonTypeWith<never>
        -✔ array of partially supported (bigint or basic object) becomes basic object only
        -✔ array of partially supported (symbols or basic object) is modified with null
        -✔ array of unsupported (bigint) becomes never[]
        -✔ array of unsupported (symbols) becomes null[]
        -✔ array of unsupported (functions) becomes null[]
        -✔ array of functions with properties becomes ({...}|null)[]
        -✔ array of objects and functions becomes ({...}|null)[]
        -✔ array of bigint | symbol becomes null[]
        -✔ array of number | bigint | symbol becomes (number|null)[]
        -✔ readonly array of supported types (numbers) are preserved
      • fully supported object types are preserved
        • ✔ empty object
        • ✔ object with boolean
        • ✔ object with number
        • ✔ object with string
        • ✔ object with number key
        • ✔ object with array of supported types (numbers) are preserved
        • ✔ object with sparse array is filled in with null
        • ✔ object with branded number
        • ✔ object with branded string
        • string indexed record of numbers
        • string|number indexed record of strings
        • string indexed record of number|strings with known properties
        • string|number indexed record of strings with known number property (unassignable)
        • Partial<> string indexed record of numbers
        • ✔ templated record of numbers
        • Partial<> templated record of numbers
        • ✔ object with possible type recursion through union
        • ✔ object with optional type recursion
        • ✔ object with deep type recursion
        • ✔ object with alternating type recursion
        • ✔ simple json (JsonTypeWith<never>)
        • ✔ non-const enum
        • ✔ object with readonly
        • ✔ object with getter implemented via value
        • ✔ object with setter implemented via value
        • ✔ object with matched getter and setter implemented via value
        • ✔ object with mismatched getter and setter implemented via value
        • class instance
          -✔ with public data (propagated)
        • object with optional property (remains optional)
          -✔ without property
          -✔ with undefined value (property is removed in value)
          -✔ with defined value
      • partially supported object types are modified
        -✔ object (plain object) becomes non-null Json object
      • fully unsupported properties are removed
        -✔ object with exactly bigint
        -✔ object with exactly symbol
        -✔ object with exactly function
        -✔ object with exactly Function | symbol
        -✔ object with inherited recursion extended with unsupported properties
        -✔ object with required exact undefined
        -✔ object with optional exact undefined
        -✔ object with exactly never
        -✔ string indexed record of undefined
        -✔ string indexed record of undefined and known number property (unassignable)
      • partially unsupported properties become optional for those supported
        • ✔ object with exactly string | symbol
        • ✔ object with exactly bigint | string
        • ✔ object with exactly bigint | symbol
        • ✔ object with exactly number | bigint | symbol
        • ✔ object with symbol key
        • ✔ object with recursion and symbol unrolls 4 times and then has generic Json
        • ✔ object with exactly function with properties
        • ✔ object with exactly object and function
        • ✔ object with function object with recursion
        • ✔ object with object and function with recursion
        • ✔ object with required unknown in recursion when unknown is allowed unrolls 4 times with optional unknown
        • object with undefined
          -✔ with undefined value
          -✔ with defined value
      • partially supported array properties are modified like top-level arrays
        -✔ object with array of partially supported (numbers or undefined) is modified with null
        -✔ object with array of unknown becomes array of JsonTypeWith<never>
        -✔ object with array of partially supported (bigint or basic object) becomes basic object only
        -✔ object with array of partially supported (symbols or basic object) is modified with null
        -✔ object with array of unsupported (bigint) becomes never[]
        -✔ object with array of unsupported (symbols) becomes null[]
        -✔ object with array of unsupported (functions) becomes null[]
        -✔ object with array of functions with properties becomes ({...}|null)[]
        -✔ object with array of objects and functions becomes ({...}|null)[]
        -✔ object with array of bigint | symbol becomes null[]
        -✔ object with array of number | bigint | symbol becomes (number|null)[]
        -✔ object with readonly array of supported types (numbers) are preserved
      • function & object intersections preserve object portion
        -✔ function with properties
        -✔ object and function
        -✔ function with class instance with private data
        -✔ function with class instance with public data
        -✔ class instance with private data and is function
        -✔ class instance with public data and is function
        -✔ function object with recursion
        -✔ object and function with recursion
      • class instance methods and non-public properties are removed
        • ✔ with public method (removes method)
        • ✔ with private method (removes method)
        • ✔ with private getter (removes getter)
        • ✔ with private setter (removes setter)
        • ✔ with private data (hides private data that propagates)
        • ✔ object with recursion and handle unrolls 4 times listing public properties and then has generic Json
        • for common class instance of
          -✔ Map
          -✔ ReadonlyMap
          -✔ Set
          -✔ ReadonlySet
      • branded non-primitive types lose branding
        -✔ branded object becomes just empty
        -✔ branded object with string
      • unsupported object types
        • known defect expectations
          • ✔ array of numbers with holes
          • getters and setters preserved but do not propagate
            -✔ object with readonly implemented via getter
            -✔ object with getter
            -✔ object with setter
            -✔ object with matched getter and setter
            -✔ object with mismatched getter and setter
    • negative compilation tests
      • assumptions
        -✔ const enums are never readable
      • unsupported types
        -✔ undefined becomes never
        -✔ unknown becomes JsonTypeWith<never>
        -✔ string indexed record of unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior)
        -✔ templated record of unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior)
        -✔ string indexed record of unknown and known properties has unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior)
        -✔ string indexed record of unknown and optional known properties has unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior)
        -✔ string indexed record of unknown and required known unknown has all unknown replaced with JsonTypeWith<never> (and known becomes explicitly optional)
        -✔ string indexed record of unknown and optional known unknown has all unknown replaced with JsonTypeWith<never> (and becomes optional per current TS behavior)
        -✔ Partial<> string indexed record of unknown replaced with JsonTypeWith<never>
        -✔ Partial<> string indexed record of unknown and known properties has unknown replaced with JsonTypeWith<never>
        -✔ symbol becomes never
        -✔ unique symbol becomes never
        -✔ bigint becomes never
        -✔ function becomes never
        -✔ void becomes never
    • special cases
      • ✔ explicit any generic limits result type
      • using alternately allowed types
        • are preserved
          -✔ bigint
          -✔ object with bigint
          -✔ object with optional bigint
          -✔ array of bigints
          -✔ array of bigint or basic object
          -✔ object with specific function
          -✔ IFluidHandle
          -✔ object with IFluidHandle
          -✔ object with IFluidHandle and recursion
          -✔ unknown
          -✔ object with optional unknown
          -✔ object with optional unknown and recursion
          -✔ string indexed record of unknown
          -✔ templated record of unknown
          -✔ string indexed record of unknown and known properties
          -✔ string indexed record of unknown and optional known properties
          -✔ string indexed record of unknown and optional known unknown
          -✔ Partial<> string indexed record of unknown
          -✔ Partial<> string indexed record of unknown and known properties
          -✔ array of unknown
          -✔ object with array of unknown
        • still modifies required unknown to become optional
          -✔ object with required unknown
          -✔ object with required unknown adjacent to recursion
          -✔ mixed record of unknown
          -✔ string indexed record of unknown and required known unknown
        • continue rejecting unsupported that are not alternately allowed
          -✔ unknown (simple object) becomes JsonTypeWith<bigint>
          -✔ unknown (with bigint) becomes JsonTypeWith<bigint>
          -✔ symbol still becomes never
          -✔ object (plain object) still becomes non-null Json object
  • JsonSerializable

    • positive compilation tests
      • supported primitive types
        -✔ boolean
        -✔ number
        -✔ string
        -✔ numeric enum
        -✔ string enum
        -✔ const heterogenous enum
        -✔ computed enum
        -✔ branded number
        -✔ branded string
      • supported literal types
        -✔ true
        -✔ false
        -✔ 0
        -✔ "string"
        -✔ null
        -✔ object with literals
        -✔ array of literals
        -✔ tuple of literals
        -✔ specific numeric enum value
        -✔ specific string enum value
        -✔ specific const heterogenous enum value
        -✔ specific computed enum value
      • supported array types
        -✔ array of numbers
        -✔ readonly array of numbers
      • supported object types
        • ✔ empty object
        • ✔ object with never
        • ✔ object with boolean
        • ✔ object with number
        • ✔ object with string
        • ✔ object with number key
        • ✔ object with array of numbers
        • ✔ readonly array of numbers
        • ✔ object with branded number
        • ✔ object with branded string
        • string indexed record of numbers
        • string|number indexed record of strings
        • ✔ templated record of numbers
        • string indexed record of number|strings with known properties
        • string|number indexed record of strings with known number property (unassignable)
        • ✔ object with possible type recursion through union
        • ✔ object with optional type recursion
        • ✔ object with deep type recursion
        • ✔ object with alternating type recursion
        • ✔ simple json (JsonTypeWith)
        • ✔ non-const enums
        • ✔ object with readonly
        • ✔ object with getter implemented via value
        • ✔ object with setter implemented via value
        • ✔ object with matched getter and setter implemented via value
        • ✔ object with mismatched getter and setter implemented via value
        • class instance
          • ✔ with public data (just cares about data)
          • with ignore-inaccessible-members
            -✔ with private method ignores method
            -✔ with private getter ignores getter
            -✔ with private setter ignores setter
        • object with optional property
          -✔ without property
          -✔ with undefined value
          -✔ with defined value
      • unsupported object types
        • ✔ object with self reference throws on serialization
        • known defect expectations
          • ✔ sparse array of supported types
          • ✔ object with sparse array of supported types
          • getters and setters allowed but do not propagate
            -✔ object with readonly implemented via getter
            -✔ object with getter
            -✔ object with setter
            -✔ object with matched getter and setter
            -✔ object with mismatched getter and setter
          • class instance
            • with ignore-inaccessible-members
              -✔ with private data ignores private data (that propagates)
    • negative compilation tests
      • assumptions
        -✔ const enums are never readable
      • unsupported types cause compiler error
        • undefined
        • unknown
        • symbol
        • unique symbol
        • bigint
        • ✔ function
        • ✔ function with supported properties
        • ✔ object and function
        • ✔ object with function with supported properties
        • ✔ object with object and function
        • ✔ function with class instance with private data
        • ✔ function with class instance with public data
        • ✔ class instance with private data and is function
        • ✔ class instance with public data and is function
        • object (plain object)
        • void
        • ✔ branded object
        • ✔ branded object with string
        • unions with unsupported primitive types
          -✔ string | symbol
          -✔ bigint | string
          -✔ bigint | symbol
          -✔ number | bigint | symbol
        • array
          -✔ array of bigints
          -✔ array of symbols
          -✔ array of unknown
          -✔ array of functions
          -✔ array of functions with properties
          -✔ array of objects and functions
          -✔ array of number | undefineds
          -✔ array of bigint or basic object
          -✔ array of symbol or basic object
          -✔ array of bigint | symbols
          -✔ array of number | bigint | symbols
        • object
          • ✔ object with exactly bigint
          • ✔ object with optional bigint
          • ✔ object with exactly symbol
          • ✔ object with optional symbol
          • ✔ object with exactly function
          • ✔ object with exactly Function | symbol
          • ✔ object with exactly string | symbol
          • ✔ object with exactly bigint | string
          • ✔ object with exactly bigint | symbol
          • ✔ object with exactly number | bigint | symbol
          • ✔ object with array of bigints
          • ✔ object with array of symbols
          • ✔ object with array of unknown
          • ✔ object with array of functions
          • ✔ object with array of functions with properties
          • ✔ object with array of objects and functions
          • ✔ object with array of number | undefineds
          • ✔ object with array of bigint or basic object
          • ✔ object with array of symbol or basic object
          • ✔ object with array of bigint | symbols
          • ✔ object with symbol key
          • string indexed record of unknown
          • Partial<> string indexed record of unknown
          • Partial<> string indexed record of numbers
          • Partial<> templated record of numbers
          • ✔ object with recursion and symbol
          • ✔ function object with recursion
          • ✔ object and function with recursion
          • ✔ nested function object with recursion
          • ✔ nested object and function with recursion
          • ✔ object with inherited recursion extended with unsupported properties
          • object with undefined
            -✔ as exact property type
            -✔ in union property
            -✔ as exact property type of string indexed record
            -✔ as exact property type of string indexed record intersected with known number property (unassignable)
            -✔ as optional exact property type > varies by exactOptionalPropertyTypes setting
            -✔ under an optional property
          • object with required unknown even though exactly allowed
            -✔ as exact property type
            -✔ as exact property type adjacent to recursion
            -✔ as exact property type in recursion
          • of class instance
            -✔ with private data
            -✔ with private method
            -✔ with private getter
            -✔ with private setter
            -✔ with public method
        • common class instances
          -✔ Map
          -✔ ReadonlyMap
          -✔ Set
          -✔ ReadonlySet
    • special cases
      • ✔ explicit any generic still limits allowed types
      • number edge cases
        • supported
          -✔ MIN_SAFE_INTEGER
          -✔ MAX_SAFE_INTEGER
          -✔ MIN_VALUE
          -✔ MAX_VALUE
        • resulting in null
          -✔ NaN
          -✔ +Infinity
          -✔ -Infinity
      • using alternately allowed types
        • are supported
          -✔ bigint
          -✔ object with bigint
          -✔ object with optional bigint
          -✔ array of bigints
          -✔ array of bigint or basic object
          -✔ object with specific alternately allowed function
          -✔ IFluidHandle
          -✔ object with IFluidHandle
          -✔ object with IFluidHandle and recursion
          -✔ unknown
          -✔ array of unknown
          -✔ object with array of unknown
          -✔ object with optional unknown
          -✔ string indexed record of unknown
          -✔ templated record of unknown
          -✔ string indexed record of unknown and known properties
          -✔ string indexed record of unknown and optional known properties
          -✔ string indexed record of unknown and optional known unknown
          -✔ Partial<> string indexed record of unknown
          -✔ Partial<> string indexed record of unknown and known properties
          -✔ object with optional unknown adjacent to recursion
          -✔ object with optional unknown in recursion
        • continue rejecting unsupported that are not alternately allowed
          -✔ unknown (simple object) expects JsonTypeWith<bigint>
          -✔ unknown (with bigint) expects JsonTypeWith<bigint>
          -✔ symbol still becomes never
          -✔ object (plain object) still becomes non-null Json object
          -✔ object with non-alternately allowed too generic function
          -✔ object with non-alternately allowed too input permissive function
          -✔ object with non-alternately allowed more restrictive output function
          -✔ object with supported or non-supported function union
          -✔ string indexed record of unknown and required known unknown that must be optional
          -✔ mixed record of unknown
  • JsonSerializable under exactOptionalPropertyTypes=true

    • negative compilation tests
      • unsupported types cause compiler error
        • object
          • object with undefined
            • ✔ as optional exact property type

Copy link
Member

@markfields markfields left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disclaimer: I did not review the 1000 lines of metaprogramming, but the breadth and depth of test cases gives me a lot of confidence. And as discussed, the risks of something bad happening in production here are low - if someone thinks they're serializing something but it's not happening right and the type allows it, they should have tests to cover the runtime behavior.

@jason-ha jason-ha merged commit 944c054 into microsoft:main Mar 11, 2025
54 checks passed
chentong7 pushed a commit to chentong7/FluidFramework that referenced this pull request Mar 11, 2025
Add pair of type filters for JSON based serialization.

`JsonSerializable<T>` produces type representing limitations of
serializing `T`. Incompatible elements are transformed to `never` or
`SerializationError*` types that original `T` is not assignable to.

`JsonDeserialized<T>` produces type representing result of `T` being
serialized and then deserialized.

`JsonSerializable` should eventually replace
`@fluidframework/datastore-definitions`'s `Jsonable`. That cannot done
be currently as it would be a compile-time breaking change.


[AB#6887](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/6887)

### Additional changes beyond Presence implementation

- New support added for `unknown` as and exactly allowed option.
  - `AllowExactly` option changed to tuple in support.
  - Helper added to detect template literals
  - Helper added to detect index signatures
- Support for branded primitives (`boolean`, `number`, and `string`)
- Additional comments and corrections
- Test coverage for arrays nested in objects

### Supporting Changes

Add standard test infrastructure

---------

Co-authored-by: Daniel Lehenbauer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: framework Framework is a tag for issues involving the developer framework. Eg Aqueduct base: main PRs targeted against main branch dependencies Pull requests that update a dependency file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants