blobfuse2-2.0.0-preview.1

First Release

Top Features

• Compatibility with libfuse3, including libfuse2 compatibility for OSes that do not support libfuse3
• Service version upgrade from "2018-11-09" to "2020-04-08" (STG77) through the azure-go-sdk
• Maximum blob size in a single write 64MB -> 5000MB
• Maximum block size 100MB -> 4000MB
• Maximum file size supported 4.77TB -> 196TB
• File creation time and last access time now pulled from service
• Passthrough directly from filesystem calls to Azure Storage APIs
• Read streaming (helpful for large files that cannot fit on disk)
• Logging to syslog or a file
• Mount a subdirectory in a container
• Mount all containers in an account
• Automatic blobfuse2 version checking and prompt for users to upgrade
• Encrypted config support
• Present custom default permissions for files in block blob accounts
• Attribute cache invalidation based on timeout
• Set custom default blob tier while uploading files to container
• Pluggable cache eviction policies for file cache and streaming

User Experience
Blobfuse2 supports a special command that will accept the v1 CLI parameters and v1 config file format, convert it to a v2 config format and mount with v2.

Blobfuse2 exposes all supported options in a clean yaml formatted configuration file in addition to a few common options exposed as CLI parameters. In contrast, Blobfuse exposes most of its options as CLI parameters and a few others in a key-value configuration file.

Validation
Extensive suite of validation run with focus on scale and compatibility.

Various improvements/parity to existing blobfuse validated like

• Listing of >1M files: Parity under the same mount conditions.
• Git clone: Outperforms blobfuse by 20% in git clone (large repo, with over 1 million objects).
• resnet-50: Parity at 32 threads.

blobfuse-1.4.3

Bug Fixes:

#674 : Content-type is not updated for files > 64MB in size
#675 : segfault in streaming when open and close call come in parallel for same file

Other Fixes:

• Default service config to exclude -d option (this was resulting into log flood with libfuse traces)
• Blobfuse deleting files in local system which were out of scope from mount folder
• Documentation : Readme correction for proxy config

blobfuse-1.4.2

Bug Fixes:

#658 : Fixed segfault in streaming when file is 0 byte in length
#660 : Added mp4 to list of supported content types
#665 : SYNC and DIRECT open flags are not supported. New cli param added to supress them in open file call

New Cli Options:

--ignore-open-flags : Supress unsupported flags in open file call.

blobfuse-1.4.1

Bug Fixes:

#638 : Mount blob using Managed Identity or service principal failed in Azure China Cloud

• Gen2 accounts were not showing correct ACL permissions if there are additional service-principals in ACL
• Fix for using MSI Secret from environment variable

Feature addition

• Read Streaming support added for 'read-only' mounts. Instead of downloading entire file only part of it will be downloaded on demand.
• Background download support. Instead of downloading entire file in open call, if configured, open will start download in background and read/write call will wait untill download completes. Thus making 'open' call non-blocking.
• Attribute cache and content cache can be invalidated using 'fsync' comand on file.
• 'fsyncdir' will invalidate attribute-cache of all its children

New Cli Options:

--background-download : Set this true if you want file download to run in the background on open.
--invalidate-on-sync : Set this to true if you want the particular file or directory content and attribute cache to be invalidated on 'fsync' or 'fsyncdir'. Refer README for more details.
--streaming : Enable streaming. Works only for 'read-only' mount. Add '-o ro' in mount cli and set this flag to 'true' to enable streaming.
--stream-cache-mb : Limit total amount of data being cached in memory with streaming.
--max-blocks-per-file : Maximum number of blocks to be cached in memory for read streaming.
--block-size-mb=16 : Size (in MB) of a block to be downloaded during streaming. Refer README for more details.

blobfuse-1.3.8

Bug Fixes:

#604 : Linux file list commands is only showing 5000 files in a folder
#609 : Expose max retry count and retry time out settings
#612 : Segmentation fault when running blobfuse on a pod on AKS version 1.19.11
#613 : blobfuse mount with invalid credentials still returns succeed

New Cli Options:

--max-retry : Maximum retry count if the failure codes are retryable. Default count is 26.
--max-retry-interval-in-seconds : Maximum number of seconds between 2 retries, retry interval is exponentially increased but it can never exceed this value. Default naximum interval is 60 seconds.
--basic-remount-check : Try checking for a remount by reading /etc/mtab instead of calling the syscall setmntent (#612)
--pre-mount-validate : Skip cURL version check and validate storage connection before mount. (#613)

blobfuse-1.3.7-Linux

Changes:

• Features:

  • Blobfuse will block initial on mount list API calls to storage if the duration is configured while mounting, to save billing money.
  • Blobfuse can be configured to mount through Proxy Server
  • Blobfuse will upload only newly created files or modified files to storage, Note: changes to metadata only will not propagate unless there are changes to file contents. It can be configured to turn off this flag.
  • Added Azure CLI script for generating SasToken
  • Content-type update in blob, based on file extension

• Bug Fixes:

  • Corrected available cache calculation for non sudo users
  • #540: MSI and ADLS now working properly with gen-1 and gen-2 accounts
  • #548 :Fixed concurrency issues affecting MSI performance
  • #552 : Made Disk thresholds configurable, check README for CLI options
  • #578 : Reduced the libCurl minimum version post-fork authentication
  • #586: fixed the erroring out of chmod and rename with SAS token on ADLS account
  • #584: Evict read only files as well from cache
  • #590: Added Proxy support with custom client certificate file
  • #593 : Resolved race conflict issue when multiple threads read the same blob

• New Cli Options :

  • --upload-modified-only=true . Default is true so unless there are writes to the file it will not be reuploaded to Storage.
  • --set-content-type=true. Set the 'content-type' property in blob while uploading. This will be based on the file extension.
  • --cancel-list-on-mount-seconds=0. After mount block the list call for give number of seconds. After mounting libfuse automatically fires a list call to list all files and directories which results into unintended calls to container.
  • --cache-poll-timeout-msec=1. Caching polling and eviction check time interval (in milliseconds). Default is 1 milisecond.
  • --high-disk-threshold-in-seconds. The upper limit percentage of disk usage. Disk cache cleanup will be triggered when it reaches this percentage.
  • --low-disk-threshold-in-seconds. The lower limit percentage for used disk space below which GC will not clear disk cache.
  • --ca-cert-file=. Provide client certificate file for proxy environment
  • --httpsproxy= < proxy >. HTTPS proxy through which connection to storage is allowed.
  • --httpproxy= < proxy >. HTTP proxy through which connection to storage is allowed.

blobfuse-1.3.6-Linux

This release has the following fixes and enhancements

1. Parameter file_cache_timeout_in_seconds will be honored based on the time the file was downloaded to the file system as opposed to the original modification date in blob.
2. All retries follow exponential backoff to prevent throttling due to exceeded RPS.
3. Blob names will allow '&', '%', '+' and "="
4. Sticky bits will be allowed in ADLS Gen 2 permission list , this is a fix for the error "Unexpected number of permissions"
5. Blobfuse 1.3.6 is backward compatible with Curl versions < 7.54, this addresses the handshake issue faced by CentOS and RHEL versions < 8

blobfuse-1.3.5-Linux

1. ADLS Gen 2 enhanced to non-public blob endpoints.
2. SAS Token refresh without remount feature implemented.
3. File names with '#' now encoded and supported.
4. Bug fix: Rename operation in ADLS Gen 2
5. Added build support for Debian 10

blobfuse-1.3.4-Linux

Blobfuse 1.3.4 has the following bug fixes only

1. Fix for FileNotFound error when the nested blob directory lexicography overrode the parent directory alone lexicography
2. RPM script modified to compile with RHEL 7.8

blobfuse-1.3.3-Linux

This release has the following bug fixes and perf improvements

1. Perf improvements with file listing when attribute cache is turned on , mount param for the same --use-attr-cache=true
2. Perf improvements with MSI authentication
3. Delete bug fix for ADLS
4. Memory error aka Input/Output error bug fix