Using multiple existing s3 buckets as storage plugins.

[mazhigbee-pb]

Is your feature request related to a problem? Please describe.

I would like to use more than a single S3 bucket in my amplify project.

Describe the solution you'd like

Say I have two buckets, Foo and Bar.

It would be very nice if configuring another bucket was as easy as adding another AWSS3StoragePlugin. If the prefix resolver allowed me to customize the bucket information and security requirements.

I would expect to be able to add as many different buckets as I'd like.

{
  "storage": {
    "plugins": {
      "awsS3StoragePluginFoo": {
        "bucket": "FOO",
        "region": "[REGION]"
      },
      "awsS3StoragePluginBar": {
        "bucket": "BAR",
        "region": "[REGION]"
      }
    }
  }
}

let storagePluginForFooBucket = AWSS3StoragePlugin(configuration: .prefixResolver(FooPrefixResolver()))
let storagePluginForBarBucket = AWSS3StoragePlugin(configuration: .prefixResolver(BarPrefixResolver()))

try Amplify.add(plugin: storagePluginForFooBucket)
try Amplify.add(plugin: storagePluginForBarBucket)

Then when uploading to the bucket if you specify the key (or bucket-name( Amplify would look up or understand which prefix resolver to use when uploading to that bucket.

Amplify.Storage.uploadData(
        key: "Foo",
        data: data
    )

One thing of note: using Amplify.Storage.uploadData does not allow me to specify the "public-read" acl on the bucket. It would be nice if storage provided all the same options that the bucket has configured because Guest, Protected & Private doesn't match all the potentially configured security options.

Describe alternatives you've considered

As a workaround, I used the TransferManager/Utility to upload to various buckets. But this becomes hard to manage when the buckets have authentication requirements. Amplify.Storage.upload handles those requirements automatically but the TransferUtility requires me to include all that information.

AWSS3TransferUtility.default().uploadFile(fileUrl, bucket: bucketName, key: fileName, contentType: contentType...)

Is the feature request related to any of the existing Amplify categories?

Storage

Additional context

No response

[github-actions]

This has been identified as a feature request. If this feature is important to you, we strongly encourage you to give a 👍 reaction on the request. This helps us prioritize new features most important to you. Thank you!

[atierian]

Thanks for opening this feature request @mazhigbee-pb.

Can you please include some details on your use case that requires multiple buckets?

In case you weren't aware of this functionality, a custom defined prefix resolver may help you here.
Here's the Amplify iOS v1 documentation on the subject Storage > File Access Levels > Customization.

[mazhigbee-pb]

@atierian Sorry for the late response!

We have two big features inside our app that have different backend environments, each with its own S3 buckets (inside the same account).

I couldn't figure out how to set up the prefix resolver so that it would work with both buckets. When I made changes in the prefix resolver to add the second bucket prefix whenever I made a request to storage via Amplify, the bucket name would never change to the second one. Both of these buckets have the same permission scope so changing the prefix based on the protection level doesn't solve the issue.

I tried having both buckets defined in my json config, but the resolver never seemed to pickup the second bucket when adding the plugin for storage. It would be great if there was a code example for this in the docs, or maybe this is just not something thats support by Amplify at this time?

[atierian]

Thanks for following up with more information about your use case.

I should have been more clear about the custom prefix resolver suggestion. This won't allow you to use multiple buckets as you've discovered; instead, it could allow you to use a single bucket with a folder structure to delineate the objects you store in two buckets today. For example:

|-- myexamplebucketfoo
|   ├─ a.json (myexamplebucketfoo/a.json)
|   └─ b.txt  (myexamplebucketfoo/b.txt)
|
└-- myexamplebucketbar
    ├─ c.json (myexamplebucketbar/c.json)
    └─ d.txt  (myexamplebucketbar/d.txt)

becomes:

└-- myexamplebucket
    ├─ foo
    |    ├─ a.json (myexamplebucket/foo/a.json)
    |    └─ b.txt  (myexamplebucket/foo/b.txt)
    |
    └─ bar
         ├─ c.json (myexamplebucket/bar/c.json)
         └─ d.txt (myexamplebucket/bar/d.txt)

[mazhigbee-pb]

@atierian Any update on this? Sorry for never responding, but unfortunately it's a must that we use two separate buckets. So creating a nested folder structure isn't going to work for me.

[atierian]

Hey @mazhigbee-pb, thanks for checking in. We are actively working on multi-bucket support, but don't have a timeline to share yet. Stay tuned 😄

[ruisebas]

Hi @mazhigbee-pb, support for multiple buckets has been released in version 2.40.0

You can learn how to configure it in our documentation for setting up Storage, and also on the specific section for each operation (e.g. for uploading into a specific bucket).

I'm closing this feature requests as completed. If you face any problems please open a new issue reporting it.
Thanks!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.