RFC global: backup and restore features

[lnielsen]

ES 2.3+ adds a backup and restore module which e.g can snapshot an entire cluster into to a shared file systems.

Snapshot and restore a useful for:

• Disaster recovery (a restore is magnitudes faster than reindexing records from scratch).
• For development to load/dump large datasets.

Proposal:

• Programmatic API (mostly wrapper around elasticsearch_py):
  • Initialise one or more repositories which have been defined in Invenio's instance config.
  • Create a snapshot
  • Restore a snapshot
  • List snapshots and status of jobs
• Celery tasks:
  • Creating snapshots (can be installed as a Celery periodic task to have regular snapshots)
  • Expire snapshots
• CLI:

<instance> search snapshots init
<instance> search snapshots create -r <repository> snapshot_201608291200
<instance> search snapshots list -r <repository>
<instance> search snapshots status -r <repository>
<instance> search snapshots restore -r <repository> snapshot_201608291200 

Or does anyone know about already existing tools the does something similar?

More information:

• List of repository plugins