You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We need to replace the individual file readers in creator.analyses.analyzer with the ingest library's read_df method.
The ingest library's read_df is already able to read all of the file formats covered by the individual read functions in creator.analyses.analyzer so these read functions are redundant. And more importantly, the ingest library's read_df automatically detects the encoding and properly decodes the file content. Right now the CSVDictReader assumes utf-8 encoding which can be wrong, resulting in the BOM left in the file. This has resulted in silent validation failures since the BOM may be part of the first file column and then is unrecognized by the validator.
Since this change would affect critical functionality in study creator, we can simply use read_df in the validation code to start.
The text was updated successfully, but these errors were encountered:
We need to replace the individual file readers in
creator.analyses.analyzer
with the ingest library'sread_df
method.The ingest library's
read_df
is already able to read all of the file formats covered by the individual read functions increator.analyses.analyzer
so these read functions are redundant. And more importantly, the ingest library'sread_df
automatically detects the encoding and properly decodes the file content. Right now theCSVDictReader
assumes utf-8 encoding which can be wrong, resulting in the BOM left in the file. This has resulted in silent validation failures since the BOM may be part of the first file column and then is unrecognized by the validator.Since this change would affect critical functionality in study creator, we can simply use
read_df
in the validation code to start.The text was updated successfully, but these errors were encountered: