[GRPO] Log completions at every step

[lidiya-co]

What does this PR do?

Enable viewing completions at every step in wandb rather than overriding them with latest step only:

before (note my num_generations=7):

after:

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[qgallouedec]

Actually, I think you can access all the steps via a different wandb workspace config

[qgallouedec]

[lidiya-co]

@qgallouedec Oh great, is the wandb workspace config somewhere in the docs already or should I add that instead?

[qgallouedec]

TBH I really don't know how to configure the wandb workspace to get this visualization. My wandb workspace doesn't show this visualization, so I switch to one of my colleagues' workspaces. If you find the solution, please share it, I'm interested.

[semitable]

If you change runs.summary["completions"] to runs.history.concat["completions"] (directly on the wandb website shown in your screenshot that is, and click the small "run" button that appears), you can already see the past completions.

Nothing is overwritten in wandb, so there's no need to resend all the information at each timestep.