You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, energon pre-computes all shard ranges that a worker will walk through. This list can be very large and consume memory unnecessarily.
By defining a function for compute the Nth shard for a worker on-the-fly, this precomputation becomes obsolete.
When the state is saved to disk, only the current index (number of shard ranges already processed) needs to be stored, and potentially a shuffling seed.
The text was updated successfully, but these errors were encountered:
Currently, energon pre-computes all shard ranges that a worker will walk through. This list can be very large and consume memory unnecessarily.
By defining a function for compute the Nth shard for a worker on-the-fly, this precomputation becomes obsolete.
When the state is saved to disk, only the current index (number of shard ranges already processed) needs to be stored, and potentially a shuffling seed.
The text was updated successfully, but these errors were encountered: