Code

Transfers pipeline improvements#54

Code

Merged

Joshdpaul merged 47 commits intomainua-snap/cmip6-utils:mainfrom add_modelsua-snap/cmip6-utils:add_modelsCopy head branch name to clipboardApr 25, 2024

Commits

Commits on Mar 27, 2024

• add models

  Joshdpaul

  committed
  a3f9096

  Copy full SHA for a3f9096

Commits on Mar 28, 2024

• allow multiple grid types to be listed in esgf holdings; fix MRI-ESM2-0 institution and rm duplicate model

  Joshdpaul

  committed
  cdf0976

  Copy full SHA for cdf0976

Commits on Mar 29, 2024

• handle missing grids / versions / files in ls routines; allows for all possibilities to avoid row_di reference before assignment error

  Joshdpaul

  committed
  316d26a

  Copy full SHA for 316d26a

• draft notebook to explicitly select grids

  Joshdpaul

  committed
  329ea6d

  Copy full SHA for 329ea6d

• allow comparison of >2 grid types

  Joshdpaul

  committed
  9ea6a3f

  Copy full SHA for 9ea6a3f

• remove E3SM-2-0 from the shopping list

  Joshdpaul

  committed
  1028c28

  Copy full SHA for 1028c28

Commits on Apr 1, 2024

• allow for multiple row returns in get_filenames() and add more descriptive ls error messages; handle multiple row returns in make_holdings_table() and check for non-dict rows; incorporate retry int…

  Show description for 9c80c32
  Joshdpaul

  committed
  9c80c32

  Copy full SHA for 9c80c32

Commits on Apr 2, 2024

• add "last resort" empty row

  Joshdpaul

  committed
  4b41a80

  Copy full SHA for 4b41a80

• add in all models for full test

  Joshdpaul

  committed
  f7e8374

  Copy full SHA for f7e8374

Commits on Apr 3, 2024

• commit updated esgf holdings and script output file

  Joshdpaul

  committed
  fb7d1b7

  Copy full SHA for fb7d1b7

Commits on Apr 4, 2024

• allow for multiple institutions per model

  Joshdpaul

  committed
  8b9e8a5

  Copy full SHA for 8b9e8a5

• refactor to allow institution lists in all instances of config.model_inst_lu[model]

  Joshdpaul

  committed
  b497ab2

  Copy full SHA for b497ab2

• rm set() from row list chain

  Joshdpaul

  committed
  88b217c

  Copy full SHA for 88b217c

Commits on Apr 5, 2024

• clean holdings run with no 502s, adds MPI-ESM1-2-LR ScenarioMIP (from DKRZ inst)

  Joshdpaul

  committed
  72fa1f1

  Copy full SHA for 72fa1f1

• rerun nb with new models

  Joshdpaul

  committed
  7732f82

  Copy full SHA for 7732f82

• remove CESM2-WACCM from variant list, add doc, rerun nb and add E3SM block

  Joshdpaul

  committed
  502cf97

  Copy full SHA for 502cf97

• add "tee" instructions

  Joshdpaul

  committed
  9151bf8

  Copy full SHA for 9151bf8

• add "E3SM-1-1"

  Joshdpaul

  committed
  ade1705

  Copy full SHA for ade1705

Commits on Apr 6, 2024

• another clean run with E3SM-1-1 model added

  Joshdpaul

  committed
  e3a0c4b

  Copy full SHA for e3a0c4b

• finish select grids nb, add grid lookup to config

  Joshdpaul

  committed
  8bc0aa4

  Copy full SHA for 8bc0aa4

• update manifest; revise generate_manifest.py to allow >1 grid type per model

  Joshdpaul

  committed
  0ff5f6b

  Copy full SHA for 0ff5f6b

• update batch files with new models / grids

  Joshdpaul

  committed
  9ef1833

  Copy full SHA for 9ef1833

Commits on Apr 11, 2024

• fix typos

  Joshdpaul

  committed
  1b7f1b8

  Copy full SHA for 1b7f1b8

• rm CESM2-WACCM from lookup table

  Joshdpaul

  committed
  5a36f66

  Copy full SHA for 5a36f66

Commits on Apr 16, 2024

• Add script for E3SM holdings

  kyleredilla

  committed
  08d5451

  Copy full SHA for 08d5451

Commits on Apr 17, 2024

• format transfer utils script

  kyleredilla

  committed
  091dc30

  Copy full SHA for 091dc30

Commits on Apr 18, 2024

• Improve retry logic of holdings audit script

  kyleredilla

  committed
  63cded3

  Copy full SHA for 63cded3

• add e3sm holdings table

  kyleredilla

  committed
  3bca51d

  Copy full SHA for 3bca51d

• update main holdings table

  kyleredilla

  committed
  e6ac736

  Copy full SHA for e6ac736

• update holdings summary notebook

  kyleredilla

  committed
  619443a

  Copy full SHA for 619443a

• add E3SM models to manifest script and update manifest table

  kyleredilla

  committed
  732cb34

  Copy full SHA for 732cb34

• decouple institution and models lookup in audit scripts

  kyleredilla

  committed
  2159645

  Copy full SHA for 2159645

Commits on Apr 19, 2024

• fix e3sm model list config

  kyleredilla

  committed
  71fd94e

  Copy full SHA for 71fd94e

• update holdings tables

  kyleredilla

  committed
  3cdd638

  Copy full SHA for 3cdd638

Commits on Apr 22, 2024

• update holdings summary tables

  kyleredilla

  committed
  be294a9

  Copy full SHA for be294a9

• use e3sm prefix in batch file generation, drop node arg

  kyleredilla

  committed
  d236328

  Copy full SHA for d236328

• update manifest and batch files with e3sm

  kyleredilla

  committed
  26386e4

  Copy full SHA for 26386e4

• skip e3sm in batch file generation

  kyleredilla

  committed
  1ec0dc5

  Copy full SHA for 1ec0dc5

• update batch files

  kyleredilla

  committed
  31f3676

  Copy full SHA for 31f3676

Commits on Apr 23, 2024

• update holdings table with SNAP holdings

  kyleredilla

  committed
  b13570f

  Copy full SHA for b13570f

Commits on Apr 24, 2024

• add CESM2 back in to config

  kyleredilla

  committed
  bf44e71

  Copy full SHA for bf44e71

• update grid selection nb

  kyleredilla

  committed
  17337d6

  Copy full SHA for 17337d6

• update manifest and batch files after adding CESM2

  kyleredilla

  committed
  0ffb099

  Copy full SHA for 0ffb099

• update variatn selection notebook after improving holdings audit workflow

  kyleredilla

  committed
  a4e4c9d

  Copy full SHA for a4e4c9d

• remove unused esgf endpoint

  kyleredilla

  authored
  297906c

  Copy full SHA for 297906c

Commits on Apr 25, 2024

• remove unused code in manifest script

  kyleredilla

  authored
  3d56698

  Copy full SHA for 3d56698

• change operation_ls back to original

  Joshdpaul

  authored
  c5f3c2b

  Copy full SHA for c5f3c2b