Code

Some micro-optimizations for tf2.#94

Code

Open

Tilps wants to merge 41 commits intoLeelaChessZero:masterLeelaChessZero/lczero-training:masterfrom Tilps:migrate_2.0_part2Tilps/lczero-training:migrate_2.0_part2Copy head branch name to clipboard

+21-10

Commits

Commits on Dec 7, 2019

• Silence some warnings in 1.14 at the expense of making 1.14 the minimum version

  Tilps

  committed
  6c63629

  Copy full SHA for 6c63629

• Merge remote-tracking branch 'upstream/master' into migrate_1.14

  Tilps

  committed
  be869dd

  Copy full SHA for be869dd

• Update requirements.

  Tilps

  committed
  a9f7129

  Copy full SHA for a9f7129

• Minimal changes to get training running on 2.0

  Tilps

  committed
  7ad15ce

  Copy full SHA for 7ad15ce

• Update requirements.

  Tilps

  committed
  d170e20

  Copy full SHA for d170e20

• Remove uses of get_variable since its initializer don't seem to be called anymore...

  Tilps

  committed
  3fcbe2b

  Copy full SHA for 3fcbe2b

• Fix typo.

  Tilps

  committed
  162fa40

  Copy full SHA for 162fa40

Commits on Dec 8, 2019

• Migration in progress some more.

  Show description for ff2e256
  Tilps

  committed
  ff2e256

  Copy full SHA for ff2e256

• Add grad norm clipping back in.

  Tilps

  committed
  f202c78

  Copy full SHA for f202c78

• Add swa support and test result reporting.

  Tilps

  committed
  1398d2a

  Copy full SHA for 1398d2a

• Actually sum the gradients...

  Tilps

  committed
  2d97b80

  Copy full SHA for 2d97b80

• Fix bad bug with residual blocks, also include the stub for the incomplete save code.

  Tilps

  committed
  cd3c69d

  Copy full SHA for cd3c69d

• Constant policy map shouldn't be a variable, or it gets saved.

  Tilps

  committed
  d3f95eb

  Copy full SHA for d3f95eb

Commits on Dec 9, 2019

• Add untested network saving

  Show description for c90a9c4
  Tilps

  committed
  c90a9c4

  Copy full SHA for c90a9c4

• Fix permutation which didn't take into account the inserted gammas.

  Tilps

  committed
  61158e2

  Copy full SHA for 61158e2

• GlobalAveragePooling2D needs to know the data format.

  Tilps

  committed
  8681df6

  Copy full SHA for 8681df6

• Use tf.function on the inner loop of training for massive speed up

  Tilps

  committed
  215445d

  Copy full SHA for 215445d

• Extract test summary inner loop to tf.function for a bit more performance

  Tilps

  committed
  bdec903

  Copy full SHA for bdec903

Commits on Dec 11, 2019

• Basic tensorboard summary data writting and some other cleanup related to saving.

  Tilps

  committed
  0b17356

  Copy full SHA for 0b17356

• Remove some v1 code which I've done a second pass checking conversion.

  Tilps

  committed
  43cdaf2

  Copy full SHA for 43cdaf2

• Add a missing _v2.

  Tilps

  committed
  53c6057

  Copy full SHA for 53c6057

• Fix bug in saving swa_weights that I hadn't tested before committing...

  Tilps

  committed
  6892b1c

  Copy full SHA for 6892b1c

• Add mixed precision support to tf2 version.

  Tilps

  committed
  c610936

  Copy full SHA for c610936

• Renorm also doesn't actually support fused.

  Tilps

  committed
  5a2a95b

  Copy full SHA for 5a2a95b

• Re-add accuracy reporting.

  Tilps

  committed
  6a37e9f

  Copy full SHA for 6a37e9f

• Some more cleanup of pre-v2 code that is converted or close enough to no longer be useful, and a fix for swa summaries being written to the wrong file.

  Tilps

  committed
  f2f32a2

  Copy full SHA for f2f32a2

• Readd basic update ratios.

  Tilps

  committed
  7cbe775

  Copy full SHA for 7cbe775

• Small performance optimization to offset the cost of update ratios.

  Tilps

  committed
  47208df

  Copy full SHA for 47208df

• Optimize compute update ratio since adding update_ratio_log10 had a noticeable performance impact.

  Tilps

  committed
  d22b9b8

  Copy full SHA for d22b9b8

Commits on Dec 12, 2019

• Some more cleanup, add weight histograms back.

  Tilps

  committed
  ef06365

  Copy full SHA for ef06365

• Fix net saving for renorm mode.

  Tilps

  committed
  753b918

  Copy full SHA for 753b918

• Add net_to_model and update_steps support. Remove more unsupported code, including upgrade.py which doesn't have an obvious transition.

  Tilps

  committed
  72f7cb3

  Copy full SHA for 72f7cb3

• Some more cleanup.

  Tilps

  committed
  9f7defc

  Copy full SHA for 9f7defc

• More cleanup.

  Tilps

  committed
  7823f0d

  Copy full SHA for 7823f0d

• Small fix needed to make script work in ubuntu.

  Tilps

  committed
  5f4d8c4

  Copy full SHA for 5f4d8c4

• explicitly add the training flags to model calls - as apparently needed.

  Tilps

  committed
  da94d76

  Copy full SHA for da94d76

• More cleanup.

  Tilps

  committed
  20fe10a

  Copy full SHA for 20fe10a

Commits on Dec 13, 2019

• Pull iterator advancement out of the tf.function loops, its not supposed to work there.

  Show description for 3ac88fa
  Tilps

  committed
  3ac88fa

  Copy full SHA for 3ac88fa

• Add experimental dataset loader.

  Tilps

  committed
  681ab74

  Copy full SHA for 681ab74

• Some micro-optimizations.

  Tilps

  committed
  121545f

  Copy full SHA for 121545f

• Merge branch 'master' into migrate_2.0_part2

  Tilps

  committed
  b016a64

  Copy full SHA for b016a64