Releases: sdv-dev/SDV
v1.0.1 - 2023-04-20
This release improves the load_custom_constraint_classes
method by removing the table_name
parameter and just loading the constraint
for all tables instead. It also improves some error messages as well as removes some of the warnings that have been surfacing.
Support for sdtypes is enhanced by resolving a bug that was incorrecttly specifying Faker functions for some of them. Support for datetime formats has also been improved. Finally, the path
argument in some save
and load
methods was changed to filepath
for consistency.
New Features
- Method load_custom_constraint_classes does not need table_name parameter - Issue #1354 by @R-Palazzo
- Improve error message for invalid primary keys - Issue #1341 by @R-Palazzo
- Add functionality to find version add-on - Issue #1309 by @frances-h
Bugs Fixed
- Certain sdtypes cause Faker to raise error - Issue #1346 by @frances-h
- Change path to filepath for load and save methods - Issue #1352 by @fealho
- Some datetime formats cause InvalidDataError, even if the datetime matches the format - Issue #1136 by @amontanez24
Internal
- Inequality constraint raises RuntimeWarning (invalid value encountered in log) - Issue #1275 by @frances-h
- Pandas FutureWarning: Default dtype for Empty Series will be 'object' - Issue #1355 by @R-Palazzo
- Pandas FutureWarning: Length 1 tuple will be returned - Issue #1356 by @R-Palazzo
v1.0.0 - 2023-03-28
This is a major release that introduces a new API to the SDV
aimed at streamlining the process of synthetic data generation! To achieve this, this release includes the addition of several large features.
Metadata
Some of the most notable additions are the new SingleTableMetadata
and MultiTableMetadata
classes. These classes enable a number of features that make it easier to synthesize your data correctly such as:
- Automatic data detection - Calling
metadata.detect_from_dataframe()
ormetadata.detect_from_csv()
will populate the metadata autonomously with values it thinks represent the data. - Easy updating - Once an instance of the metadata is created, values can be easily updated using a number of methods defined in the API. For more info, view the docs.
- Metadata validation - Calling
metadata.validate()
will return a report of any invalid definitions in the metadata specification. - Upgrading - Users with the previous metadata format can easily update to the new specification using the
upgrade_metadata()
method. - Saving and loading - The metadata itself can easily be saved to a json file and loaded back up later.
Class and Module Names
Another major change is the renaming of our core modeling classes and modules. The name changes are meant to highlight the difference between the underlying machine learning models, and the objects responsible for the end-to-end workflow of generating synthetic data. The main name changes are as follows:
tabular
->single_table
relational
->multi_table
timeseries
->sequential
BaseTabularModel
->BaseSingleTableSynthesizer
GaussianCopula
->GaussianCopulaSynthesizer
CTGAN
->CTGANSynthesizer
TVAE
->TVAESynthesizer
CopulaGan
->CopulaGANSynthesizer
PAR
->PARSynthesizer
HMA1
->HMASynthesizer
In SDV
1.0, synthesizers are classes that take in metadata and handle data preprocessing, model training and model sampling. This is similar to the previous BaseTabularModel
in SDV
<1.0.
Synthetic Data Workflow
Synthesizers
in SDV
1.0 define a clear workflow for generating synthetic data.
- Synthesizers are initialized with a metadata class.
- They can then be used to transform the data and apply constraints using the
synthesizer.preprocess()
method. This step also validates that the data matches the provided metadata to avoid errors in fitting or sampling. - The processed data can then be fed into the underlying machine learning model using
synthesizer.fit_processed_data()
. (Alternatively, data can be preprocessed and fit to the model usingsynthesizer.fit()
.) - Data can then be sampled using
synthesizer.sample()
.
Each synthesizer class also provides a series of methods to help users customize the transformations their data goes through. Read more about that here.
Notice that the preprocessing and model fitting steps can now be separated. This can be helpful if preprocessing is time consuming or if the data has been processed externally.
Other Highly Requested Features
Another major addition is control over randomization. In SDV
<1.0, users could set a seed to control the randomization for only some columns. In SDV
1.0, randomization is controlled for all columns. Every new call to sample generates new data, but the synthesizer's seed can be reset to the original state using synthesizer.reset_randomization()
, enabling reproducibility.
SDV 1.0
adds accessibility and transparency into the transformers used for preprocessing and underlying machine learning models.
- Using the
synthesizer.get_transformers()
method, you can access the transformers used to preprocess each column and view their properties. This can be useful for debugging and accessing privacy information like mappings used to mask data. - Distribution parameters learned by copula models can be accessed using the
synthesizer.get_learned_distributions()
method.
PII handling is improved by the following features:
- Primary keys can be set to natural sdtypes (eg. SSN, email, name). Previously they could only be numerical or text.
- The
PseudoAnonymizedFaker
can be used to provide consistent mapping to PII columns. As mentioned before, the mapping itself can be accessed by viewing the transformers for the column usingsynthesizer.get_transformers()
. - A bug causing PII columns to slow down modeling is patched.
Finally, the synthetic data can now be easily evaluated using the evaluate_quality()
and run_diagnostic()
methods. The data can be compared visually to the actual data using the get_column_plot()
and get_column_pair_plot()
methods. For more info on how to visualize or interpret the synthetic data evaluation, read the docs here.
Issues Resolved
New Features
- Change auto_assign_transformers to handle id types - Issue #1325 by @pvk-developer
- Change 'text' sdtype to 'id' - Issue #1324 by @frances-h
- In
upgrade_metadata
, return the object instead of writing it to a JSON file - Issue #1319 by @frances-h - In
upgrade_metadata
index primary keys should be converted totext
- Issue #1318 by @amontanez24 - Add
load_from_dict
to SingleTableMetadata and MultiTableMetadata - Issue #1314 by @amontanez24 - Throw a
SynthesizerInputError
ifFixedCombinations
constraint is applied to a column that is notboolean
orcategorical
- Issue #1306 by @frances-h - Missing
save
andload
methods forHMASynthesizer
- Issue #1262 by @amontanez24 - Better input validation when creating single and multi table synthesizers - Issue #1242 by @fealho
- Better input validation on
HMASynthesizer.sample
- Issue #1241 by @R-Palazzo - Validate that relationship must be between a
primary key
andforeign key
- Issue #1236 by @fealho - Improve
update_column
validation forpii
attribute - Issue #1226 by @pvk-developer - Order the output of
get_transformers()
based on the metadata - Issue #1222 by @pvk-developer - Log if any
numerical_distributions
will not be applied - Issue #1212 by @fealho - Improve error handling for
GaussianCopulaSynthesizer
:numerical_distributions
- Issue #1211 by @fealho - Improve error handling when validating
constraints
- Issue #1210 by @fealho - Add
fake_companies
demo - Issue #1209 by @amontanez24 - Allow me to create a custom constraint class and use it in the same file - Issue #1205 by @amontanez24
- Sampling should reset after retraining the model - Issue #1201 by @pvk-developer
- Change function name
HMASynthesizer.update_table_parameters
-->set_table_parameters
- Issue #1200 by @pvk-developer - Add
get_info
method to synthesizers - Issue #1199 by @fealho - Add evaluation methods to synthesizer - Issue #1190 by @fealho
- Update
evaluate.py
to work with the newmetadata
- Issue #1186 by @fealho - Remove old code - Issue #1181 by @pvk-developer
- Drop support for python 3.6 and add support for 3.10 - Issue #1176 by @fealho
- Add constraint methods to MultiTableSynthesizers - Issue #1171 by @fealho
- Update custom constraint workflow - Issue #1169 by @pvk-developer
- Add get_constraints method to synthesizers - Issue #1168 by @pvk-developer
- Migrate adding and validating constraints to BaseSynthesizer - Issue #1163 by @pvk-developer
- Change metadata
"SCHEMA_VERSION"
-->"METADATA_SPEC_VERSION"
- Issue #1139 by @amontanez24 - Add ability to reset random sampling - Issue #1130 by @pvk-developer
- Add get_available_demos - Issue #1129 by @fealho
- Add demo loading functionality - Issue #1128 by @fealho
- Use logging instead of printing in detect methods - Issue #1107 by @fealho
- Add save and load methods to synthesizers - Issue #1106 by @pvk-developer
- Add sampling methods to PARSynthesizer - Issue #1083 by @amontanez24
- Add transformer methods to PARSynthesizer - Issue #1082 by @fealho
- Add validate to PARSynthesizer - Issue #1081 by @amontanez24
- Add preprocess and fit methods to PARSynthesizer - I...
v0.18.0 - 2023-01-24
This release adds suppport for Python 3.10 and drops support for 3.6.
Maintenance
- Drop support for python 3.6 - Issue #1177 by @amontanez24
- Support for python 3.10 - Issue #939 by @amontanez24
- Support Python >=3.10,<4 - Issue #1000 by @amontanez24
v0.17.2 - 2022-12-08
This release fixes a bug in the demo module related to loading the demo data with constraints. It also adds a name to the demo datasets. Finally, it bumps the version of SDMetrics
used.
Maintenance
New Features
- Provide a name for the default demo datasets - Issue #1124 by @amontanez24
Bugs Fixed
- Cannot load_tabular_demo with metadata - Issue #1123 by @amontanez24
v0.17.1 - 2022-09-29
v0.17.0 - 2022-09-09
This release updates the code to use RDT version 1.2.0 and greater, so that those new features are now available in SDV. This changes the transformers that are available in SDV models to be those that are in RDT version 1.2.0. As a result, some arguments for initializing models have changed.
Additionally, this release fixes bugs related to loading models with custom constraints. It also fixes a bug that added NaNs
to the index of sampled data when using sample_remaining_columns
.
Bugs Fixed
- Incorrect rounding in Custom Constraint example - Issue #941 by @amontanez24
- Can't save the model if use the custom constraint - Issue #928 by @pvk-developer
- User Guide code fixes - Issue #983 by @amontanez24
- Index contains NaNs when using sample_remaining_columns - Issue #985 by @amontanez24
- Cannot sample after loading a model with custom constraints: TypeError - Issue #984 by @pvk-developer
- Set HyperTransformer config manually, based on Metadata if given - Issue #982 by @pvk-developer
New Features
Maintenance
- Update the RDT version to 1.0 - Issue #897 by @pvk-developer
v0.16.0 - 2022-07-21
This release brings user friendly improvements and bug fixes on the SDV
constraints, to help
users generate their synthetic data easily.
Some predefined constraints have been renamed and redefined to be more user friendly & consistent.
The custom constraint API has also been updated for usability. The SDV now automatically determines
the best handling_strategy
to use for each constraint, attempting transform
by default and
falling back to reject_sampling
otherwise. The handling_strategy
parameters are no longer
included in the API.
Finally, this version of SDV
also unifies the parameters for all sampling related methods for
all models (including TabularPreset).
Changes to Constraints
-
GreatherThan
constraint is now separated in two new constraints:Inequality
, which is
intended to be used between two columns, andScalarInequality
, which is intended to be used
between a column and a scalar. -
Between
constraint is now separated in two new constraints:Range
, which is intended to
be used between three columns, andScalarRange
, which is intended to be used between a column
and low and high scalar values. -
FixedIncrements
a new constraint that makes the data increment by a certain value. -
New
create_custom_constraint
function available to create custom constraints.
Removed Constraints
Rounding
Rounding is automatically being handled by therdt.HyperTransformer
.ColumnFormula
thecreate_custom_constraint
takes place over this one and allows more
advanced usage for the end users.
New Features
- Improve error message for invalid constraints - Issue #801 by @fealho
- Numerical Instability in Constrained GaussianCopula - Issue #806 by @fealho
- Unify sampling params for reject sampling - Issue #809 by @amontanez24
- Split
GreaterThan
constraint intoInequality
andScalarInequality
- Issue #814 by @fealho - Split
Between
constraint intoRange
andScalarRange
- Issue #815 @pvk-developer - Change
columns
tocolumn_names
inOneHotEncoding
andUnique
constraints - Issue #816 by @amontanez24 - Update columns parameter in
Positive
andNegative
constraint - Issue #817 by @fealho - Create
FixedIncrements
constraint - Issue #818 by @amontanez24 - Improve datetime handling in
ScalarInequality
andScalarRange
constraints - Issue #819 by @pvk-developer - Support strict boundaries even when transform strategy is used - Issue #820 by @fealho
- Add
create_custom_constraint
factory method - Issue #836 by @fealho
Internal Improvements
- Remove
handling_strategy
parameter - Issue #833 by @amontanez24 - Remove
fit_columns_model
parameter - Issue #834 by @pvk-developer - Remove the
ColumnFormula
constraint - Issue #837 by @amontanez24 - Move
table_data.copy
to base class of constraints - Issue #845 by @fealho
Bugs Fixed
- Numerical Instability in Constrained GaussianCopula - Issue #801 by @tlranda and @fealho
- Fix error message for
FixedIncrements
- Issue #865 by @pvk-developer - Fix constraints with conditional sampling - Issue #866 by @amontanez24
- Fix error message in
ScalarInequality
- Issue #868 by @pvk-developer - Cannot use
max_tries_per_batch
on sample:TypeError: sample() got an unexpected keyword argument 'max_tries_per_batch'
- Issue #885 by @amontanez24 - Conditional sampling + batch size:
ValueError: Length of values (1) does not match length of index (5)
- Issue #886 by @amontanez24 TabularPreset
doesn't support new sampling parameters - Issue #887 by @fealho- Conditional Sampling:
batch_size
is being set toNone
by default? - Issue #889 by @amontanez24 - Conditional sampling using GaussianCopula inefficient when categories are noised - Issue #910 by @amontanez24
Documentation Changes
- Show the
API
forTabularPreset
models - Issue #854 by @katxiao - Update handling constraints doc - Pull Request #856 by @amontanez24
- Update custom costraints documentation - Pull Request #857 by @pvk-developer
v0.15.0 - 2022-05-25
This release improves the speed of the GaussianCopula
model by removing logic that previously searched for the appropriate distribution to use. It also fixes a bug that was happening when conditional sampling was used with the TabularPreset
.
The rest of the release focuses on making changes to improve constraints including changing the UniqueCombinations
constraint to FixedCombinations
, making the Unique
constraint work with missing values and erroring when null values are seen in the OneHotEncoding
constraint.
New Features
- Silence warnings coming from univariate fit in copulas - Issue #769 by @pvk-developer
- Remove parameters related to distribution search and change default - Issue #767 by @fealho
- Update the UniqueCombinations constraint - Issue #793 by @fealho
- Make Unique constraint works with nans - Issue #797 by @fealho
- Error out if nans in OneHotEncoding - Issue #800 by @amontanez24
Bugs Fixed
Documentation Changes
v0.14.1 - 2022-05-03
This release adds a TabularPreset
, available in the sdv.lite
module, which allows users to easily optimize a tabular model for speed.
In this release, we also include bug fixes for sampling with conditions, an unresolved warning, and setting field distributions. Finally,
we include documentation updates for sampling and the new TabularPreset
.
Bugs Fixed
- Sampling with conditions={column: 0.0} for float columns doesn't work - Issue #525 by @shlomihod and @tssbas
- resolved FutureWarning with Pandas replaced append by concat - Issue #759 by @Deathn0t
- Field distributions bug in CopulaGAN - Issue #747 by @katxiao
- Field distributions bug in GaussianCopula - Issue #746 by @katxiao
New Features
- Set default transformer to categorical_fuzzy - Issue #768 by @amontanez24
- Model nulls normally when tabular preset has constraints - Issue #764 by @katxiao
- Don't modify my metadata object - Issue #754 by @amontanez24
- Presets should be able to handle constraints - Issue #753 by @katxiao
- Change preset optimize_for --> name - Issue #749 by @katxiao
- Create a speed optimized Preset - Issue #716 by @katxiao
Documentation Changes
v0.14.0 - 2022-03-21
This release updates the sampling API and splits the existing functionality into three methods - sample
, sample_conditions
,
and sample_remaining_columns
. We also add support for sampling in batches, displaying a progress bar when sampling with more than one batch,
sampling deterministically, and writing the sampled results to an output file. Finally, we include fixes for sampling with conditions
and updates to the documentation.
Bugs Fixed
- Fix write to file in sampling - Issue #732 by @katxiao
- Conditional sampling doesn't work if the model has a CustomConstraint - Issue #696 by @katxiao
New Features
- Updates to GaussianCopula conditional sampling methods - Issue #729 by @katxiao
- Update conditional sampling errors - Issue #730 by @katxiao
- Enable Batch Sampling + Progress Bar - Issue #693 by @katxiao
- Create sample_remaining_columns() method - Issue #692 by @katxiao
- Create sample_conditions() method - Issue #691 by @katxiao
- Improve sample() method - Issue #690 by @katxiao
- Create Condition object - Issue #689 by @katxiao
- Is it possible to generate data with new set of primary keys? - Issue #686 by @katxiao
- No way to fix the random seed? - Issue #157 by @katxiao
- Can you set a random state for the sdv.tabular.ctgan.CTGAN.sample method? - Issue #515 by @katxiao
- generating different synthetic data while training the model multiple times. - Issue #299 by @katxiao