Declarative and typed specification for converters

[ghisvail]

I have got a bunch of tasks revolving around the subject of quality and integrity checking of input and output data from the converters. I have started experimenting with an encoding for some form of a typed and declarative syntax for specifying important attributes such as input / output data types, invalid values, bound checks, valid categories.

I am starting this topic for scratch notes and gathering feedback as this experiment progresses.

[ghisvail]

For instance, let's take the ADNI and AIBL datasets. As of today, we are using a bunch of plain read_csv and some resource file to reconcile columns names between the clinical and imaging dataframes. This solution is simple enough for manipulating columns regardless of other attributes than their names, but will quickly become complex as more complex criteria are introduced, such as input datatypes.

Take a typical record of the collection data for any LONI dataset, serialized as:

Image Data ID	Subject	Group	Sex	Age	Visit	Modality	Description	Type	Acq Date	Format	Downloaded
I001122	1000	Patient	F	70	1	MRI	MPRAGE	Original	1/01/2000	DCM

A lot of information is lost by manipulating data as raw strings. For instance, most ages for PET scans in AIBL are 0, so these should be converted to <NA> and be coalesced by the clinical data if available. Other attributes are clearly categorical (group, sex, modality, type, format) and should be converted as such to ensure consistency and save on space. Others are dates and need specific formatting rule, etc...

All these rules will be difficult to capture in a common tabular file.

[ghisvail]

On the topic of memory efficiency (which is an issue for large datasets like ADNI), a well typed dataframe can significantly help reduce the load. As a quick benchmark, I downloaded an AIBL collection for 4 subjects (34 records in total) and computed the usage for the untyped (4.5 KB) and typed version (3.3 KB). That's already a reduction of more than a quarter for a very limited sample size.

[ghisvail]

For reference, here is a snippet of code showing a typed read of a LONI collection:

df = pd.read_csv(
    open("collection.csv"),
    index_col="Image Data ID",
    usecols=[
        "Image Data ID",
        "Subject",
        "Group",
        "Sex",
        "Age",
        "Visit",
        "Modality",
        "Description",
        "Type",
        "Acq Date",
        "Format",
    ],
    dtype={
        "Image Data ID": pd.StringDtype(),
        "Subject": pd.StringDtype(),
        "Group": pd.CategoricalDtype(["Patient", "Control"]),
        "Sex": pd.CategoricalDtype(["F", "M"]),
        "Age": pd.UInt8Dtype(),
        "Visit": pd.UInt8Dtype(),
        "Modality": pd.CategoricalDtype(["MRI", "PET"]),
        "Description": pd.StringDtype(),
        "Type": pd.CategoricalDtype(["Original", "Processed"]),
        "Format": pd.CategoricalDtype(["DCM", "NIFTI"]),
    },
    na_values={"Age": [0]},
    parse_dates=["Acq Date"],
)

Data columns (total 10 columns):
 #   Column       Non-Null Count  Dtype
---  ------       --------------  -----
 0   Subject      34 non-null     string
 1   Group        34 non-null     string
 2   Sex          34 non-null     category
 3   Age          34 non-null     Int64
 4   Visit        34 non-null     string
 5   Modality     34 non-null     category
 6   Description  34 non-null     string
 7   Type         34 non-null     category
 8   Acq Date     34 non-null     datetime64[ns]
 9   Format       34 non-null     category
dtypes: Int64(1), category(4), datetime64[ns](1), string(4)

[ghisvail]

For instance, here is an example of a typed description (in YAML form but could be JSON or TOML) that could be parsed:

attributes:
  - name: Image Data ID
    index: true
  - name: Subject
    type: string
    pattern: '\d{3}_S_\d{4}'
  - name: Group
    type: category
    values:
      - Control
      - Patient
  - name: Sex
    type: category
    values:
       - F
       - M
  - name: Age
    type: integer
    exclude:
       - 0
  - name: Visit
    type: integer
  - name: Modality
    type: category
    values:
      - MRI
      - PET
  - name: Description
    type: string
  - name: Type
    type: category
    values:
      - Original
      - Processed
  - name: Acq Date
    type: date
  - name: Format
    type: category
    values:
      - DCM
      - NIFTI

[ghisvail]

This specification could be augmented to implement other transformation or normalization steps:

attributes:
  - name: handedness
    type: string
    replace:
      - value: Left
        by: L
      - value: Right
        by: R
     exclude:
      - Unknown

  - name: diagnosis
    type: string
    column: DX
    uppercase: true

[ghisvail]

Range of numeric values:

attributes:
  - name: mmse
    type: numeric
    minimum: 0
    maximum: 30
    increment: 1
  
  - name: cdr
    type: numeric
    minimum: 0
    maximum: 3
    increment: 0.5
 

[omar-rifai]

These typed descriptions are meant to accompany the tsv files we generate? In which case it'd could interesting to follow the BIDs recommendation for sidecar data dictionaries even if less comprehensive as a first step.

[ghisvail]

These typed descriptions are meant to accompany the tsv files we generate?

Not primarily, though they could be used to generate the sidecar data dict as you suggested indeed.

You could think of tgis declarative syntax as a way to parameterize the clinical data readers (read_csv / read_excel + some other normalization steps) which will be specific to each converter. Whereas, the sidecar data dict is generated when writing the BIDS dataset once conversion is done.

[MatthieuJoulot]

It should be noted that imported typings don't work on a function called inside a node with nipype.

[ghisvail]

Because function declarations must be standalone, and type hints in function signature may require importing the typing module. I guess that's just a limitation of nipype due to its age.