Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[unsupported-object-issue-fix part2] Skip unsupported object names in list output #1876

Open
wants to merge 21 commits into
base: master
Choose a base branch
from

Conversation

gargnitingoogle
Copy link
Collaborator

@gargnitingoogle gargnitingoogle commented Apr 29, 2024

Description

Skips unsupported object names during all file operations in GCSFuse mounts (e.g. listings, renames, and deletions). It also add debug logs on both types of above skips.

GCS supports objects of path-type <bucket>/<path1>//<path2>
and treats it different from <bucket>/<path1>/<path2>, while they are both the same in linux filesystem.
GCSFuse being a POSIX-compliant file-system only support the latters and throws error on former. This error can disallow the listing of all directories in the parent directory i.e. <path1>.

The current change ignores the listing/deletion/renaming of prefixes (directory names) which are empty such
as "/" above to ignore the error and logs the above event as a warning.

This is on top of #2561 (utilities) and is followed up in #2011 (which adds e2e tests for this change).

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual -
    • GCS bucket structure: 1. <bucket>//a, 2. <bucket>/b/c, 3. <bucket>/d//e .
    • GCSFuse mount command: gcsfuse --implicit-dirs --log-file=$logfile --debug_fuse --log-format=text $bucket $mountpath
    • Without the fix,
      ls -R $mountpath
      ls: reading directory $mountpath: Input/output error
      cat $logfile
      - No warning/error log -
      rm -rfv $mountpath/*
      - No log -
    • With the fix,
          ls -R $mountpath
          $mountpath:
          total 0
          b
          d
      
          $mountpath/b:
          total 0
          c
      
          $mountpath/d:
          total 0
          
          cat $logfile
          ...
          Ignored following unsupported prefixes: [/]
          Ignored following unsupported prefixes: [d//]
          ...
      
          rm -rfv $mountpath/*
          
          
          ...
  2. Unit tests - NA
  3. Integration tests - With run as presubmit.

@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 3d1d4bf to dfcbf92 Compare April 29, 2024 11:44
@gargnitingoogle gargnitingoogle added the execute-integration-tests Run only integration tests label Apr 29, 2024
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch 2 times, most recently from ed0d51d to 9024dc6 Compare April 30, 2024 09:11
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 9024dc6 to c1195c4 Compare May 4, 2024 20:45
internal/util/util_test.go Outdated Show resolved Hide resolved
internal/util/util_test.go Outdated Show resolved Hide resolved
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch 4 times, most recently from 02488f1 to cca204a Compare May 6, 2024 09:56
@gargnitingoogle gargnitingoogle marked this pull request as ready for review May 6, 2024 10:00
@gargnitingoogle gargnitingoogle requested review from ashmeenkaur and a team as code owners May 6, 2024 10:00
@gargnitingoogle gargnitingoogle changed the title Skip empty-directory names in listed prefixes Skip unsupported names in listed prefixes May 6, 2024
@gargnitingoogle
Copy link
Collaborator Author

Presubmit tests passed: logs

@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 80306c2 to fbbc0b1 Compare May 8, 2024 09:18
Copy link
Collaborator

@ashmeenkaur ashmeenkaur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still need to take a look at the tests

internal/util/util.go Outdated Show resolved Hide resolved
internal/util/util.go Outdated Show resolved Hide resolved
internal/storage/bucket_handle.go Outdated Show resolved Hide resolved
internal/util/util.go Outdated Show resolved Hide resolved
internal/fs/implicit_dirs_test.go Outdated Show resolved Hide resolved
internal/fs/implicit_dirs_test.go Outdated Show resolved Hide resolved
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from fbbc0b1 to dc55c22 Compare May 9, 2024 07:24
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from dc55c22 to 1d76261 Compare May 31, 2024 06:35
@gargnitingoogle gargnitingoogle removed the execute-integration-tests Run only integration tests label Jun 6, 2024
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch 2 times, most recently from 4acab17 to 2a971eb Compare June 11, 2024 04:17
Copy link

codecov bot commented Jun 11, 2024

Codecov Report

Attention: Patch coverage is 80.95238% with 16 lines in your changes missing coverage. Please review.

Project coverage is 77.55%. Comparing base (5ae03ff) to head (97bab99).

Files with missing lines Patch % Lines
internal/fs/inode/dir.go 83.87% 7 Missing and 3 partials ⚠️
internal/fs/fs.go 80.00% 3 Missing and 1 partial ⚠️
internal/fs/inode/base_dir.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1876      +/-   ##
==========================================
+ Coverage   77.47%   77.55%   +0.08%     
==========================================
  Files         110      110              
  Lines       15705    15785      +80     
==========================================
+ Hits        12167    12242      +75     
- Misses       3016     3020       +4     
- Partials      522      523       +1     
Flag Coverage Δ
unittests 77.55% <80.95%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@gargnitingoogle gargnitingoogle changed the base branch from master to gargnitin/add-warning-for-empty-directory-issue November 11, 2024 19:04
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 0d06bb3 to 017da8b Compare November 12, 2024 11:52
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/add-warning-for-empty-directory-issue branch from 232c117 to 872226a Compare November 12, 2024 11:52
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 017da8b to 7f6b0e0 Compare November 12, 2024 11:57
@gargnitingoogle gargnitingoogle changed the base branch from gargnitin/add-warning-for-empty-directory-issue to master November 12, 2024 16:01
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch 2 times, most recently from 14006d7 to 97bab99 Compare November 12, 2024 16:18
@gargnitingoogle gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 97bab99 to 60b8e2e Compare November 12, 2024 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
execute-integration-tests Run only integration tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants