Moving Data Between S3 and EC2 Instances

Copying data from S3 to EBS

​

1. Install awscli:

    sudo apt-get install awscli

2. Obtain AWS S3 credentials for the user account. Go to Identity and Access Management at https://console.aws.amazon.com/iam/home. Select your username, go to 'User Actions', and then go to 'Manage Access Keys' and create and download new access key. It should look like:

User Name	Access Key Id	Secret Access Key
<username>	<key id>	<secret key>

3. Run aws configure and enter above credentials. For region, use 'us-east-1’.

4. Check that you have access to s3 bucket with aws s3 ls.

5. Copy entire directory. I recommend using sync rather than cp. Note this part tends to hang sometimes, so just ctrl+c and run it again. By sync, the process will resume.

     aws s3 sync s3://my-s3-bucket local_dir/

​

To copy data from EBS to S3:

​

1. Create bucket using the online console https://console.aws.amazon.com/s3/home?region=us-east-1, e.g. ‘recoli-c321-data’ (use region us-east-1 for all of our stuff) ​
2. Copy data from EBS to S3 ​

     aws s3 cp --recursive local_dir/ s3://my-s3-bucket

You can also use sync in this direction, which is useful for copying to and from S3 simultaneously (instance -> S3 -> instance).

aws s3 sync local_dir/ s3://my-s3-bucket