Merge dev into main for 2023-12-01

[zaneselvans]

• WIP first batch of changes
• WIP broken working through list
• walk back troublesome edits
• first failed attempt at adding plant_status into the util plant summary table
• nix the failed attempt
• first attempt to add in alternative plant statuses as tags
• Remove none tags
• Fix balance sheet liabilities calc
• Clean up and add a few more status tags
• Add first draft of an xbrl_factoid_plant_function_tag csv
• Add first draft of an xbrl_factoid_plant_function_tag csv
• Fix error where _ferc1 was getting propagated accross the csv as ferc2, ferc3, ferc4 etc.
• Update plant_function tags to match the values in the table_dimensions_ferc1 table
• Standardize tag compilation and column name in csv
• Fix issue with utility type of added components
• WIP
• clean up errors
• fix integer types
• Drop calculation components not observed in table_dims
• Restore assertion that was causing the actual problem
• Fix migration history
• Rename plant tag prep function
• make the tag grabbing more resilient
• Make missing calc check only flag actually missing values in data
• convert the tags resiliency to a {} and swap columns for the plant_status
• Oops clean up some cruft
• and drop parent cols var
• lol don't be a goof remove ["tags"] before .get("tags")
• Walk back new plant functions for now
• actually let's leave these unknowns as na tags for now
• add a warning for when the tags are mismatched
• Fix lefty logging message
• Undo EPDC table DBF inclusion and incorporate Christina's suggestion
• Fix index issue
• temp bandaid
• Revert other_power_supply_expense
• clean up one utils sign error
• new eia923-2022 DOI update
• Fix the override paths when running in github actions.
• Updated DOI
• Update conda-lock.yml and rendered conda environment files.
• some docs cleanup
• Use ruff instead of black for autoformatting (Use ruff instead of black for autoformatting #3060)
• [pre-commit.ci] auto fixes from pre-commit.com hooks
• Update conda-lock.yml and rendered conda environment files.
• docs fix
• Update file_map and skiprows
• Updates docs on ruff, git config, pudl-dev environment.
• Update conda-lock.yml and rendered conda environment files.
• Remove accidentally committed input-output.ts file
• Remove sandbox reference DOIs because they are now invalid since Zenodo wiped their sandbox server
• Update conda-lock.yml and rendered conda environment files.
• Simplify process of bootstrapping pudl-dev conda env. (Simplify process of bootstrapping pudl-dev conda env. #3080)
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• WIP: improve nightly build outputs.
• fix docs error
• Update EIA860M and EIA861 DOIs to non-draft versions.
• Clarify docs
• Update deployment script to watch compresson/removal. Update docs to reflect gzipped build outputs.
• Refactor PUDL to use Pydantic v2
• Pydantic m.dict() => m.model_dump(); add type hints to conftest.py
• use Pydantic strict types directly
• Remove Base class, and schema_ & fields_ aliases.
• Remove unnecessary allow_arbitrary_types configs
• Migrate from @root_validator to @model_validator in metadata classes
• Type hints, naming, and 2 non-working draft model_validators
• Migrate final @validator to @field_validator
• Suppress warning about shadowing deprecated BaseModel.schema attribute.
• Convert parent_cols into @Property; more model_validtors in XBRL Explosion.
• Address several deprecation warnings.
• Migrate PK/FK schema field validations to Pydantic v2
• Update conda-lock.yml and rendered conda environment files.
• Fix generic settings model validations
• Re-disable b0rken calc tol validator. sa.engine.Engine => sa.Engine
• Ruff formatting changes.
• Revert temporary validator method name.
• Reinstate model_config arguments across all PudlMeta classes.
• Add release notes about using v2 of Pandas, Pydantic, & SQLAlchemy.
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• Bump micromamba docker image to v1.5.3 to address curl library error.
• Update conda-lock.yml and rendered conda environment files.
• Validate default values in PudlMeta and add None to optional args.
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• Update minmax rows for new EIA923 data
• Update conda-lock.yml and rendered conda environment files.
• Remove ref: env.GITHUB_REF in zenodo-cache-sync to avoid errors related to recieving pushes from forked repos
• Add release notes note about 923 final release and quarterly update
• Fix merge failure with dev
• Update conda-lock.yml and rendered conda environment files.
• Fiddle with GHA PUDL input/output dirs.
• Update conda-lock.yml and rendered conda environment files.
• More PUDL in/out fiddling
• More PUDL in/out fiddling
• More PUDL in/out fiddling
• [pre-commit.ci] auto fixes from pre-commit.com hooks
• Dev environment cleanup and documentation tweaks
• Update conda-lock.yml and rendered conda environment files.
• Resolve core dump failure (Resolve core dump failure #3090)
• Suppress out of control warnings at module level?
• Update references to years of EIA923 data ingested by PUDL. Also update the 923 data sources page to say that we do ingest the monthly data.
• Fix the month included in the EIA923 data years in the README
• Update conda lockfiles
• [pre-commit.ci] auto fixes from pre-commit.com hooks
• Update conda-lock.yml and rendered conda environment files.
• Suppress repetitive warnings harder.
• Update conda-lock.yml and rendered conda environment files.
• Rename DBF-derived FERC SQLite DBs to ferc*_dbf.sqlite
• Update conda-lock.yml and rendered conda environment files.
• Allow .env to contain stuff that's not PudlPaths() related. (dotenv compatibility #3092)
• Clarify lockfile autorefresh process.
• make docs more concise
• Update docs/docstring refs to ferc1.sqlite to use ferc1_dbf.sqlite
• Install conda-lock before wiping old pudl-dev environment.
• Update conda-lock.yml and rendered conda environment files.
• Update conda-lock.yml and rendered conda environment files.
• Refactor datasette publish script to allow local testing.
• Update conda-lock.yml and rendered conda environment files.
• Add FERC DBF metadata for Forms 1, 2, 6, 60.
• Fix cut-and-paste error in download link titles.
• Update release notes
• Apply suggestions from code review
• Update datasette metadata integration test.
• Add ferc1_engine_xbrl fixture back into datasette metadata test.
• Update conda-lock.yml and rendered conda environment files.

[codecov]

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (c831080) 88.6% compared to head (7cc80da) 88.7%.
Report is 138 commits behind head on main.

Files	Patch %	Lines
src/pudl/transform/ferc1.py	88.4%	8 Missing ⚠️
src/pudl/cli/etl.py	33.3%	2 Missing ⚠️
src/pudl/io_managers.py	84.6%	2 Missing ⚠️
src/pudl/settings.py	98.3%	2 Missing ⚠️
src/pudl/transform/classes.py	93.9%	2 Missing ⚠️
src/pudl/metadata/classes.py	99.4%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #3104   +/-   ##
=====================================
  Coverage   88.6%   88.7%           
=====================================
  Files         91      89    -2     
  Lines      10991   11023   +32     
=====================================
+ Hits        9749    9785   +36     
+ Misses      1242    1238    -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.