Skip to content

Commit 990af0f

Browse files
gapra-msftgapra-msft
committed
First commit of blobfuse v2 code
1 parent eb17b3c commit 990af0f

File tree

157 files changed

+53481
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

157 files changed

+53481
-0
lines changed

‎.gitignore

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
.idea/
2+
.vscode/
3+
*.out
4+
config.yaml
5+
testcnf.yaml
6+
blobfuse2.log
7+
blobfuse2.test
8+
azuretest.json
9+
blobfuse2
10+
venv/
11+
*.backup
12+
__debug_bin
13+
.env
14+
test/manual_scripts/create1000.go
15+
test/manual_scripts/cachetest.go

‎CODE_OF_CONDUCT.md

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Microsoft Open Source Code of Conduct
2+
3+
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
4+
5+
Resources:
6+
7+
- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
8+
- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
9+
- Contact [[email protected]](mailto:opencode@microsoft

‎KnownLimitations.txt

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
// Known Limitations of the current implementation :
2+
3+
1. With huge amount of blobs, listing may lead to high mem usage.

‎LICENSE

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
_____ _____ _____ ____ ______ _____ ------
2+
| | | | | | | | | | | | |
3+
| | | | | | | | | | | | |
4+
| --- | | | | |-----| |---- | | |-----| |----- ------
5+
| | | | | | | | | | | | |
6+
| ____| |_____ | ____| | ____| | |_____| _____| |_____ |_____
7+
8+
9+
Licensed under the MIT License <http://opensource.org/licenses/MIT>.
10+
11+
Copyright © 2020-2022 Microsoft Corporation. All rights reserved.
12+
Author : <[email protected]>
13+
14+
Permission is hereby granted, free of charge, to any person obtaining a copy
15+
of this software and associated documentation files (the "Software"), to deal
16+
in the Software without restriction, including without limitation the rights
17+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
18+
copies of the Software, and to permit persons to whom the Software is
19+
furnished to do so, subject to the following conditions:
20+
21+
The above copyright notice and this permission notice shall be included in all
22+
copies or substantial portions of the Software.
23+
24+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
25+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
26+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
27+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
28+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
29+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
30+
SOFTWARE

‎MIGRATION.md

+147
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# Blobfuse Migration Guide from v1.X.X to v2.X.X
2+
3+
In this guide, we list the main changes you need to be aware of when migrating your workloads from blobfuse 1.X.X to blobfuse2 2.X.X.
4+
5+
<!-- Do we need an upgrade story? -->
6+
7+
## Prerequisites
8+
Follow the steps in Installation to get started. <!-- TODO: Add link to installation and copy paste the below to the Installation page -->
9+
10+
blobfuse2 is currently available in the Microsoft product repositories for Ubuntu, SLES, Debian, CentOS/RedHat distros. Packages are only available for x86 platforms. For Kubernetes support, go [here](https://github.com/kubernetes-sigs/blob-csi-driver).
11+
12+
1. Configure the apt repository for Microsoft products following [this](https://docs.microsoft.com/en-us/windows-server/administration/Linux-Package-Repository-for-Microsoft-Software) guideline.
13+
14+
On Ubuntu
15+
```
16+
wget https://packages.microsoft.com/config/ubuntu/<UBUNTU_VERSION>/packages-microsoft-prod.deb
17+
sudo dpkg -i packages-microsoft-prod.deb
18+
sudo apt-get update
19+
```
20+
21+
On RHEL
22+
```
23+
sudo rpm -Uvh https://packages.microsoft.com/rhel/<RHEL_VERSION>/prod/blobfuse2-<BLOBFUSE2_VERSION>-RHEL-<RHEL_VERSION>-x86_64.rpm
24+
```
25+
26+
On Kubernetes, go [here](https://github.com/kubernetes-sigs/blob-csi-driver)
27+
28+
1. Install blobfuse2
29+
30+
On Ubuntu
31+
```
32+
sudo apt-get install blobfuse2 fuse
33+
```
34+
35+
On RedHat/CentOS
36+
```
37+
sudo yum install blobfuse2 fuse
38+
```
39+
40+
Now you're good to go.
41+
42+
## Updated Package Name
43+
The package name has been updated to `blobfuse2` to
44+
45+
- Prevent breaking existing blobfuse workloads
46+
- Allow the two blobfuse versions to co-exist during migration
47+
48+
## Blobfuse2 Config Converter Tool
49+
To make migration easier, we created a tool to mount with v1 configurations to migrate from blobfuse to blobfuse2 seamlessly.
50+
51+
Run the following command with the same CLI parameters as you would pass to blobfuse
52+
53+
```
54+
blobfuse2 mountv1 <MOUNT_PATH> --config-file=<BLOBFUSE_V1_CONFIG_PATH> <BLOBFUSE_V1_CLI_PARAMETERS> --output-file=<OPTIONAL_DESIRED_BLOBFUSE2_CONFIG_PATH>
55+
```
56+
57+
You can also choose to only convert the v1 configuration to v2 without mounting by passing `--convert-config-only=true`
58+
59+
## Mounting
60+
Blobfuse2 can be mounted with the following command
61+
```
62+
blobfuse2 mount <MOUNT_PATH> --config-file=<CONFIG_PATH> <ADDITIONAL_CLI_PARAMETERS>
63+
```
64+
65+
A detailed description of all the mount options offered by blobfuse2 can be see in the README. If you have an existing blobfuse command and config, you may also choose to migrate with the following instructions.
66+
67+
## Mount options
68+
blobfuse2 will read options for a given parameter with the following order of precedence
69+
1. CLI flag
70+
2. environment variable
71+
3. config file
72+
73+
Note: For an exhaustive list of the blobfuse2 config file options and the format of the file, see [here](/setup/baseConfig.yaml).
74+
<!-- TODO: Link that correctly once on github -->
75+
76+
### Blobfuse CLI Flag Options
77+
<!-- Note: When editing this table, please ensure it is formatted neatly -->
78+
| Blobfuse CLI Flag Parameter | Blobfuse2 Replacement CLI Parameter | Blobfuse2 Replacement Config File | Notes |
79+
|-----------------------------------------|-------------------------------------|-----------------------------------|---------------------------------------------------------------------------|
80+
| -o allow_other | --allow_other | allow-other | |
81+
| -o ro | --read-only | read-only | |
82+
| --tmp-path=PATH | --tmp-path=PATH | file_cache.path | |
83+
| --empty-dir-check=false | | file_cache.allow-non-empty-temp | |
84+
| --config-file=PATH | --config-file=PATH | | |
85+
| --container-name=NAME | --container-name=NAME | azstorage.container | |
86+
| --use-https=true | | azstorage.use-http | This parameter has the opposite boolean semantics |
87+
| --file-cache-timeout-in-seconds=120 | | file_cache.timeout-sec | Default changed to 0 |
88+
| --log-level=LOG_WARNING | --log-level=LOG_WARNING | logging.level | |
89+
| --use-attr-cache=true | | attr_cache | Add attr_cache to the components list |
90+
| --use-adls=false | | azstorage.type | Specify either 'block' or 'adls' |
91+
| --no-symlinks=false | --no-symlinks=false | attr_cache.no-symlinks | |
92+
| --cache-on-list=true | | attr_cache.no-cache-on-list | This parameter has the opposite boolean semantics |
93+
| --upload-modified-only=false | | | Default behavior in blobfuse2 |
94+
| --max-concurrency=12 | | azstorage.max-concurrency | |
95+
| --cache-size-mb=0 | | file_cache.max-size-mb | |
96+
| --cancel-list-on-mount-seconds=0 | | azstorage.block-list-on-mount-sec | |
97+
| --high-disk-threshold=90 | | file_cache.high-threshold | |
98+
| --low-disk-threshold=80 | | file_cache.low-threshold | |
99+
| --cache-poll-timeout-msec=1000 | | | Not an option in blobfuse2 |
100+
| ---max-eviction=0 | | file_cache.max-eviction | |
101+
| --set-content-type=false | | | Not an option in blobfuse2, always true |
102+
| --ca-cert-file=/etc/ssl/certs/proxy.pem | | | Store the ca cert file in the default/standard Linux path |
103+
| --https-proxy=http://10.1.22.4:8080/ | | azstorage.https-proxy | |
104+
| --http-proxy=http://10.1.22.4:8080/ | | azstorage.http-proxy | |
105+
| --max-retry=26 | | azstorage.max-retries | |
106+
| --max-retry-interval-in-seconds=60 | | azstorage.max-retry-timeout-sec | |
107+
| --retry-delay-factor=1.2 | | azstorage.retry-backoff-sec | |
108+
| --basic-remount-check=false | | | Not applicable in blobfuse2 |
109+
| --pre-mount-validate=true | | | Always on in Blobfuse2 blobfuse2 |
110+
| --background-download=false | | | Not an option in blobfuse2 |
111+
| --invalidate-on-sync=false | | | Always on in blobfuse2 |
112+
| --streaming=true | | stream | Add stream to the components list |
113+
| --stream-cache-mb=500 | | stream.cache-size-mb | |
114+
| ---max-blocks-per-file=3 | | stream.blocks-per-file | |
115+
| --block-size-mb=16 | | stream.block-size-mb | |
116+
| -o attr_timeout=20 | --attr_timeout=20 | libfuse.attribute-expiration-sec | Default changed to 0 |
117+
| -o entry_timeout=20 | --entry_timeout=20 | libfuse.entry-expiration-sec | Default changed to 0 |
118+
| -o negative_timeout=20 | --negative_timeout=20 | | Default changed to 0 |
119+
| -d | | libfuse.fuse-trace | |
120+
| -o umask | | libfuse.default-permission | |
121+
122+
### Blobfuse Environment Variable Options
123+
Blobfuse2 reads all the environment variables that Blobfuse does, so no changes for environment variables are required during migration.
124+
125+
### Blobfuse Config File Options
126+
<!-- Note: When editing this table, please ensure it is formatted neatly -->
127+
| Blobfuse Config File Parameter | Blobfuse2 Replacement Environment Variable | Blobfuse2 Replacement Config File |
128+
|--------------------------------|--------------------------------------------|-----------------------------------|
129+
| accountName | | azstorage.account-name |
130+
| containerName | | azstorage.container |
131+
| blobEndpoint | | azstorage.endpoint |
132+
| authType | | azstorage.mode |
133+
| accountType | | azstorage.type |
134+
| accountKey | | azstorage.account-key |
135+
| sasToken | | azstorage.sas |
136+
| identityClientId | | azstorage.appid |
137+
| identityObjectId | | azstorage.objid |
138+
| identityResourceId | | azstorage.resid |
139+
| msiEndpoint | MSI_ENDPOINT | |
140+
| servicePrincipalClientId | | azstorage.clientid |
141+
| servicePrincipalTenantId | | azstorage.tenantid |
142+
| aadEndpoint | | azstorage.aadendpoint |
143+
| logLevel | | logging.level |
144+
| httpProxy | | azstorage.http-proxy |
145+
| httpsProxy | | azstorage.https-proxy |
146+
| caCertFile | N/A | N/A |
147+
| dnsType | N/A | N/A |

0 commit comments

Comments
 (0)