Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 bucket list: max_keys: 0 is ignored #1953

Closed
1 task done
nbargnesi opened this issue Jan 19, 2024 · 2 comments · Fixed by #2328
Closed
1 task done

S3 bucket list: max_keys: 0 is ignored #1953

nbargnesi opened this issue Jan 19, 2024 · 2 comments · Fixed by #2328
Labels
jira

Comments

@nbargnesi
Copy link

Summary

The max_keys parameter of s3_object_module is ignored if set to zero. This causes AWS API defaults to take effect, which for the max-keys URI request parameter is 1000.

AWS API documentation for S3 object listing: ListObjects, ListObjectsV2.

Note by ignoring max_keys: 0, Ansible will retrieve page after page of results, up to the last object in the bucket, possibly leading to considerable costs for the AWS account owner and consuming considerable CPU time and bandwidth.

Using one of the documented examples and setting max_keys: 0:

- name: List keys all options
  amazon.aws.s3_object:
    bucket: mybucket
    mode: list
    prefix: /my/desired/
    marker: /my/desired/0023.txt
    max_keys: 0

The AWS API request line will be:

GET https://mybucket.s3.amazonaws.com/?list-type=2&prefix=%2Fmy%2Fdesired%2F&start-after=%2Fmy%2Fdesired%2F0023.txt&encoding-type=url

Note the lack of the expected &max-keys=0 URI request parameter in the GET line.

This issue seems to be present in the following releases:

  • 6.0.0
  • 6.0.1
  • 6.1.0
  • 6.2.0
  • 6.3.0
  • 6.4.0
  • 6.5.0
  • 6.5.1
  • 7.0.0
  • 7.1.0
  • 7.2.0

Issue Type

Bug Report

Component Name

modules.plugin.s3_object

Ansible Version

$ ansible --version
ansible [core 2.16.2]
  config file = None
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ~/.venvs/ansible-core/lib/python3.12/site-packages/ansible
  ansible collection location = ~/.ansible/collections:/usr/share/ansible/collections
  executable location = ~/.bin/ansible
  python version = 3.12.1 (main, Dec  7 2023, 20:45:44) [Clang 15.0.0 (clang-1500.0.40.1)]
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
Collection                   Version
---------------------------- -------
amazon.aws                   7.2.0

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.34.22
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.12/site-packages
Requires: botocore, s3transfer, jmespath
Required-by:
---
Name: botocore
Version: 1.34.22
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.12/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: s3transfer, boto3, awscli

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = None

OS / Environment

No response

Steps to Reproduce

- name: List keys all options
  amazon.aws.s3_object:
    bucket: mybucket
    mode: list
    prefix: /my/desired/
    marker: /my/desired/0023.txt
    max_keys: 0

Expected Results

I expected max-keys: 0 to be respected in the API request. It was ignored.

Actual Results

Ansible execution hung as the bucket I used never completed full pagination before the process was killed.

Code of Conduct

  • I agree to follow the Ansible Code of Conduct
@nbargnesi nbargnesi mentioned this issue Jan 19, 2024
Closed
@nbargnesi
Copy link
Author

Any other info needed for this issue? The bug has been around for over 6 months and drives up the cost of AWS bills.

@nbargnesi
Copy link
Author

Just in case it's not clear in the links above, there's already a PR for this #1954

@mandar242 mandar242 mentioned this issue Sep 11, 2024
Closed
@mandar242 mandar242 added the jira label Sep 11, 2024
@tremble tremble mentioned this issue Oct 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
jira
Projects
None yet
Milestone
No milestone
3 participants
@nbargnesi @mandar242 @GomathiselviS