You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Write file failed. File not be persisted in remote storage
Have you found a mitigation/solution?
No
Please share logs if available.
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [libfuse_handler.go (792)]: Libfuse::libfuse_write : error writing file iter_0020000/mp_rank_00/model_optim_rng.pt, handle: 162696 [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.]
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [block_cache.go (1491)]: BlockCache::getBlockIDList : Staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [block_cache.go (1445)]: BlockCache::commitBlocks : Failed to get block id list for iter_0020000/mp_rank_00/model_optim_rng.pt [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.]
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [block_cache.go (1076)]: BlockCache::getOrCreateBlock : Failed to commit blocks for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.]
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [block_cache.go (1006)]: BlockCache::WriteFile : Unable to allocate block for iter_0020000/mp_rank_00/model_optim_rng.pt [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.]
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [libfuse_handler.go (792)]: Libfuse::libfuse_write : error writing file iter_0020000/mp_rank_00/model_optim_rng.pt, handle: 162696 [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.]
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [block_cache.go (1491)]: BlockCache::getBlockIDList : Staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [block_cache.go (1445)]: BlockCache::commitBlocks : Failed to get block id list for iter_0020000/mp_rank_00/model_optim_rng.pt [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.]
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [block_cache.go (1076)]: BlockCache::getOrCreateBlock : Failed to commit blocks for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.]
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [block_cache.go (1006)]: BlockCache::WriteFile : Unable to allocate block for iter_0020000/mp_rank_00/model_optim_rng.pt [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
If you see this message, contact [email protected] or create a GitHub issue. We're working on a fix. More details: https://aka.ms/blobfuse2warnings.]
Oct 09 06:03:58 openpai-000003 blobfuse2[3214717]: [/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7457115dfa3ef0cc6da5c97af5af6ebd71d363cd1962130e7ae3f0564f920c21/globalmount] LOG_ERR [libfuse_handler.go (792)]: Libfuse::libfuse_write : error writing file iter_0020000/mp_rank_00/model_optim_rng.pt, handle: 162696 [staged block 52 has less data 3145728 for 162696=>iter_0020000/mp_rank_00/model_optim_rng.pt
Notice: The random write flow using block cache is temporarily blocked due to potential data integrity issues. This is a precautionary measure.
The text was updated successfully, but these errors were encountered:
Hi, we found some issues with random write using block-cache and currently are working on fixing those. Meanwhile, we have blocked random write operations using block-cache. We suggest using file-cache for random writes or wait for our next release. Thanks !
Which version of blobfuse was used?
2.3.2
Which OS distribution and version are you using?
Ubuntu 22.04
If relevant, please share your mount command.
Mount via k8s blob-csi-driver. mount option:
What was the issue encountered?
Write file failed. File not be persisted in remote storage
Have you found a mitigation/solution?
No
Please share logs if available.
The text was updated successfully, but these errors were encountered: