-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial APIs for FileSystem extensions #1470
Draft
swankjesse
wants to merge
8
commits into
master
Choose a base branch
from
jwilson.0415.extensions
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+531
−2
Draft
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
07d0c7c
Initial APIs for FileSystem extensions
squarejesse e054205
Fill in docs and tests
squarejesse b0f4ebb
apiDump and spotlessApply
squarejesse a4f71a1
Experiment with path mapping
squarejesse 9c05d2f
Copyright
squarejesse 0bf63eb
Finish the mapping API and write some tests
squarejesse 7a0425e
apiDump
squarejesse 5325b96
No FS tests on Wasm
squarejesse File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| /* | ||
| * Copyright (C) 2024 Square, Inc. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package okio | ||
|
|
||
| import okio.FileSystemExtension.Mapping | ||
|
|
||
| /** | ||
| * Marks an object that can be attached to a [FileSystem], and that supplements the file system's | ||
| * capabilities. | ||
| * | ||
| * Implementations must support transforms to input and output paths with [Mapping]. To simplify | ||
| * implementation, use [Mapping.NONE] by default and use [Mapping.chain] to combine mappings. | ||
| * | ||
| * ```kotlin | ||
| * class DiskUsageExtension private constructor( | ||
| * private val mapping: Mapping, | ||
| * ) : FileSystemExtension { | ||
| * constructor() : this(Mapping.NONE) | ||
| * | ||
| * override fun map(outer: Mapping): FileSystemExtension { | ||
| * return DiskUsageExtension(mapping.chain(outer)) | ||
| * } | ||
| * | ||
| * fun sizeOnDisk(path: Path): Long { | ||
| * val mappedPath = mapping.mapParameter(path, "sizeOnDisk", "path") | ||
| * return lookUpSizeOnDisk(mappedPath) | ||
| * } | ||
| * | ||
| * fun largestFiles(): Sequence<Path> { | ||
| * val largestFiles: Sequence<Path> = lookUpLargestFiles() | ||
| * return largestFiles.map { | ||
| * mapping.mapResult(it, "largestFiles") | ||
| * } | ||
| * } | ||
| * } | ||
| * ``` | ||
| */ | ||
| interface FileSystemExtension { | ||
| /** Returns a file system of the same type, that applies [outer] to all paths. */ | ||
| fun map(outer: Mapping): FileSystemExtension | ||
|
|
||
| abstract class Mapping { | ||
| abstract fun mapParameter(path: Path, functionName: String, parameterName: String): Path | ||
| abstract fun mapResult(path: Path, functionName: String): Path | ||
|
|
||
| fun chain(outer: Mapping): Mapping { | ||
| val inner = this | ||
| return object : Mapping() { | ||
| override fun mapParameter(path: Path, functionName: String, parameterName: String): Path { | ||
| return inner.mapParameter( | ||
| outer.mapParameter( | ||
| path, | ||
| functionName, | ||
| parameterName, | ||
| ), | ||
| functionName, | ||
| parameterName, | ||
| ) | ||
| } | ||
|
|
||
| override fun mapResult(path: Path, functionName: String): Path { | ||
| return outer.mapResult( | ||
| inner.mapResult( | ||
| path, | ||
| functionName, | ||
| ), | ||
| functionName, | ||
| ) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| companion object { | ||
| val NONE = object : Mapping() { | ||
| override fun mapParameter(path: Path, functionName: String, parameterName: String) = path | ||
| override fun mapResult(path: Path, functionName: String) = path | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
193 changes: 193 additions & 0 deletions
193
okio/src/nonWasmTest/kotlin/okio/FileSystemExtensionMappingTest.kt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,193 @@ | ||
| /* | ||
| * Copyright (C) 2024 Square, Inc. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package okio | ||
|
|
||
| import kotlin.test.Test | ||
| import kotlin.test.assertEquals | ||
| import okio.FileSystemExtension.Mapping | ||
| import okio.Path.Companion.toPath | ||
| import okio.fakefilesystem.FakeFileSystem | ||
|
|
||
| class FileSystemExtensionMappingTest { | ||
| private val rawFileSystem = FakeFileSystem() | ||
|
|
||
| @Test | ||
| fun happyPath() { | ||
| // Create a bunch of files in the /monday directory. | ||
| val mondayExtension = ChefExtension(rawFileSystem) | ||
| val mondayFs = rawFileSystem.extend(mondayExtension) | ||
| mondayFs.createDirectory("/monday".toPath()) | ||
| mondayFs.write("/monday/breakfast.txt".toPath()) { writeUtf8("croissant") } | ||
| mondayFs.write("/monday/lunch.txt".toPath()) { writeUtf8("peanut butter sandwich") } | ||
| mondayFs.write("/monday/dinner.txt".toPath()) { writeUtf8("pizza") } | ||
|
|
||
| // Associate data with these files in the extension. | ||
| mondayExtension.setChef("/monday/dinner.txt".toPath(), "jesse") | ||
| mondayExtension.setChef("/monday/breakfast.txt".toPath(), "benoit") | ||
| mondayExtension.setChef("/monday/lunch.txt".toPath(), "jesse") | ||
|
|
||
| // Confirm we can query the extension. | ||
| assertEquals( | ||
| listOf( | ||
| "/monday/dinner.txt".toPath(), | ||
| "/monday/lunch.txt".toPath(), | ||
| ), | ||
| mondayExtension.listForChef("/monday".toPath(), "jesse"), | ||
| ) | ||
| assertEquals( | ||
| listOf( | ||
| "/monday/breakfast.txt".toPath(), | ||
| ), | ||
| mondayExtension.listForChef("/monday".toPath(), "benoit"), | ||
| ) | ||
|
|
||
| // Apply a path mapping transformation and confirm we can read the metadata. | ||
| val tuesdayFs = TuesdayFileSystem(mondayFs) | ||
| val tuesdayExtension = tuesdayFs.extension<ChefExtension>()!! | ||
| assertEquals( | ||
| listOf( | ||
| "/tuesday/dinner.txt".toPath(), | ||
| "/tuesday/lunch.txt".toPath(), | ||
| ), | ||
| tuesdayExtension.listForChef("/tuesday".toPath(), "jesse"), | ||
| ) | ||
| assertEquals( | ||
| listOf( | ||
| "/tuesday/breakfast.txt".toPath(), | ||
| ), | ||
| tuesdayExtension.listForChef("/tuesday".toPath(), "benoit"), | ||
| ) | ||
|
|
||
| // We should also be able to write through the extension... | ||
| tuesdayFs.write("/tuesday/snack.txt".toPath()) { writeUtf8("doritos") } | ||
| tuesdayExtension.setChef("/tuesday/snack.txt".toPath(), "jake") | ||
| assertEquals( | ||
| listOf( | ||
| "/tuesday/snack.txt".toPath(), | ||
| ), | ||
| tuesdayExtension.listForChef("/tuesday".toPath(), "jake"), | ||
| ) | ||
|
|
||
| // ...And the extension data should map all the way through to the source file system. | ||
| assertEquals( | ||
| listOf( | ||
| "/monday/snack.txt".toPath(), | ||
| ), | ||
| mondayExtension.listForChef("/monday".toPath(), "jake"), | ||
| ) | ||
| } | ||
|
|
||
| @Test | ||
| fun chainTransformations() { | ||
| val mondayExtension = ChefExtension(rawFileSystem) | ||
| val mondayFs = rawFileSystem.extend(mondayExtension) | ||
| mondayFs.createDirectory("/monday".toPath()) | ||
| mondayFs.write("/monday/breakfast.txt".toPath()) { writeUtf8("croissant") } | ||
| mondayExtension.setChef("/monday/breakfast.txt".toPath(), "benoit") | ||
|
|
||
| // Map /monday to /tuesday. | ||
| val tuesdayFs = TuesdayFileSystem(mondayFs) | ||
|
|
||
| // Map / to /menu. | ||
| val menuFs = MenuFileSystem(tuesdayFs) | ||
| val menuExtension = menuFs.extension<ChefExtension>()!! | ||
|
|
||
| // Confirm we can read through the mappings. | ||
| assertEquals( | ||
| listOf( | ||
| "/menu/tuesday/breakfast.txt".toPath(), | ||
| ), | ||
| menuExtension.listForChef("/menu/tuesday".toPath(), "benoit"), | ||
| ) | ||
|
|
||
| // Confirm we can write through also. | ||
| menuFs.write("/menu/tuesday/lunch.txt".toPath()) { writeUtf8("tomato soup") } | ||
| menuExtension.setChef("/menu/tuesday/lunch.txt".toPath(), "jesse") | ||
| assertEquals( | ||
| "tomato soup", | ||
| mondayFs.read("/monday/lunch.txt".toPath()) { readUtf8() }, | ||
| ) | ||
| assertEquals( | ||
| listOf( | ||
| "/monday/lunch.txt".toPath(), | ||
| ), | ||
| mondayExtension.listForChef("/monday".toPath(), "jesse"), | ||
| ) | ||
|
|
||
| // Each extension gets its own mapping. | ||
| assertEquals( | ||
| "tomato soup", | ||
| tuesdayFs.read("/tuesday/lunch.txt".toPath()) { readUtf8() }, | ||
| ) | ||
| assertEquals( | ||
| listOf( | ||
| "/tuesday/lunch.txt".toPath(), | ||
| ), | ||
| tuesdayFs.extension<ChefExtension>()!!.listForChef("/tuesday".toPath(), "jesse"), | ||
| ) | ||
| } | ||
|
|
||
| /** | ||
| * This test extension associates paths with optional metadata: the chef of a file. | ||
| * | ||
| * When tests run there will be multiple instances of this extension, all sharing one [chefs] | ||
| * store, and each with its own [mapping]. The contents of [chefs] uses mapped paths for its keys. | ||
| * | ||
| * Real world extensions will have similar obligations for path mapping, but they'll likely do | ||
| * real things with the paths such as passing them to system APIs. | ||
| */ | ||
| class ChefExtension( | ||
| private val target: FileSystem, | ||
| private val chefs: MutableMap<Path, String> = mutableMapOf(), | ||
| private val mapping: Mapping = Mapping.NONE, | ||
| ) : FileSystemExtension { | ||
| override fun map(outer: Mapping) = ChefExtension(target, chefs, mapping.chain(outer)) | ||
|
|
||
| fun setChef(path: Path, chef: String) { | ||
| val mappedPath = mapping.mapParameter(path, "set", "path") | ||
| chefs[mappedPath] = chef | ||
| } | ||
|
|
||
| fun listForChef(dir: Path, chef: String): List<Path> { | ||
| val mappedDir = mapping.mapParameter(dir, "listForChef", "dir") | ||
| return target.list(mappedDir) | ||
| .filter { chefs[it] == chef } | ||
| .map { mapping.mapResult(it, "listForChef") } | ||
| } | ||
| } | ||
|
|
||
| class TuesdayFileSystem(delegate: FileSystem) : ForwardingFileSystem(delegate) { | ||
| private val monday = "/monday".toPath() | ||
| private val tuesday = "/tuesday".toPath() | ||
|
|
||
| override fun onPathParameter(path: Path, functionName: String, parameterName: String) = | ||
| monday / path.relativeTo(tuesday) | ||
|
|
||
| override fun onPathResult(path: Path, functionName: String) = | ||
| tuesday / path.relativeTo(monday) | ||
| } | ||
|
|
||
| class MenuFileSystem(delegate: FileSystem) : ForwardingFileSystem(delegate) { | ||
| private val root = "/".toPath() | ||
| private val menu = "/menu".toPath() | ||
|
|
||
| override fun onPathParameter(path: Path, functionName: String, parameterName: String) = | ||
| root / path.relativeTo(menu) | ||
|
|
||
| override fun onPathResult(path: Path, functionName: String) = | ||
| menu / path.relativeTo(root) | ||
| } | ||
| } |
101 changes: 101 additions & 0 deletions
101
okio/src/nonWasmTest/kotlin/okio/FileSystemExtensionsTest.kt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| /* | ||
| * Copyright (C) 2024 Square, Inc. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package okio | ||
|
|
||
| import kotlin.test.Test | ||
| import kotlin.test.assertEquals | ||
| import kotlin.test.assertNotEquals | ||
| import kotlin.test.assertNull | ||
| import okio.FileSystemExtension.Mapping | ||
| import okio.fakefilesystem.FakeFileSystem | ||
|
|
||
| class FileSystemExtensionsTest { | ||
| @Test | ||
| fun happyPath() { | ||
| val fakeFileSystem = FakeFileSystem() | ||
| val extension = FooExtension(fakeFileSystem) | ||
| val fileSystemWithExtension = fakeFileSystem.extend(extension) | ||
| assertEquals(fileSystemWithExtension.extension<FooExtension>(), extension) | ||
| } | ||
|
|
||
| @Test | ||
| fun absentExtension() { | ||
| val fakeFileSystem = FakeFileSystem() | ||
| val extension = FooExtension(fakeFileSystem) | ||
| val fileSystemWithExtension = fakeFileSystem.extend(extension) | ||
| assertNull(fileSystemWithExtension.extension<BarExtension>()) | ||
| } | ||
|
|
||
| @Test | ||
| fun overrideExtension() { | ||
| val fakeFileSystem = FakeFileSystem() | ||
| val extension1 = FooExtension(fakeFileSystem) | ||
| val fileSystemWithExtension1 = fakeFileSystem.extend(extension1) | ||
|
|
||
| val extension2 = FooExtension(fakeFileSystem) | ||
| val fileSystemWithExtension2 = fileSystemWithExtension1.extend(extension2) | ||
| assertEquals(fileSystemWithExtension2.extension<FooExtension>(), extension2) | ||
|
|
||
| // Doesn't interfere with any of the wrapped layers. | ||
| assertEquals(fileSystemWithExtension1.extension<FooExtension>(), extension1) | ||
| assertNull(fakeFileSystem.extension<FooExtension>(), null) | ||
| } | ||
|
|
||
| @Test | ||
| fun forwardingFileSystemCoalesced() { | ||
| val fakeFileSystem = FakeFileSystem() | ||
| val fooExtension = FooExtension(fakeFileSystem) | ||
| val fileSystemWithFoo = fakeFileSystem.extend(fooExtension) | ||
|
|
||
| val barExtension = BarExtension(fakeFileSystem) | ||
| val fileSystemWithFooAndBar = fileSystemWithFoo.extend(barExtension) | ||
|
|
||
| assertEquals(fileSystemWithFooAndBar.extension<FooExtension>(), fooExtension) | ||
| assertEquals(fileSystemWithFooAndBar.extension<BarExtension>(), barExtension) | ||
| assertEquals((fileSystemWithFooAndBar as ForwardingFileSystem).delegate, fakeFileSystem) | ||
| } | ||
|
|
||
| @Test | ||
| fun customForwardingFileSystemNotCoalesced() { | ||
| val fakeFileSystem = FakeFileSystem() | ||
| val fooExtension = FooExtension(fakeFileSystem) | ||
| val fileSystemWithFoo = object : ForwardingFileSystem( | ||
| delegate = fakeFileSystem, | ||
| extensions = mapOf(FooExtension::class to fooExtension), | ||
| ) { | ||
| // This extends ForwardingFileSystem. Usually this would be to add new capabilities. | ||
| } | ||
|
|
||
| val barExtension = BarExtension(fakeFileSystem) | ||
| val fileSystemWithFooAndBar = fileSystemWithFoo.extend(barExtension) | ||
|
|
||
| assertEquals(fileSystemWithFooAndBar.extension<FooExtension>(), fooExtension) | ||
| assertEquals(fileSystemWithFooAndBar.extension<BarExtension>(), barExtension) | ||
| assertNotEquals((fileSystemWithFooAndBar as ForwardingFileSystem).delegate, fakeFileSystem) | ||
| } | ||
|
|
||
| class FooExtension( | ||
| val target: FileSystem, | ||
| ) : FileSystemExtension { | ||
| override fun map(outer: Mapping) = this | ||
| } | ||
|
|
||
| class BarExtension( | ||
| val target: FileSystem, | ||
| ) : FileSystemExtension { | ||
| override fun map(outer: Mapping) = this | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So you can shadow extensions ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I sas the
overrideExtensiontest