Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting issue while doing a coverity scan on blobfuse2 mounted directory Failed to acquire lock #1584

Open
rahulshrivastava-eaton opened this issue Dec 3, 2024 · 11 comments
Open

Getting issue while doing a coverity scan on blobfuse2 mounted directory Failed to acquire lock #1584

rahulshrivastava-eaton opened this issue Dec 3, 2024 · 11 comments
Assignees
@vibhansa-msft
Labels
triaged V2
Milestone
v2-2.4.1

Comments

@rahulshrivastava-eaton
Copy link

I am trying to run a synopsis coverity scan on a Linux machine , as i wanted to store the coverity scan results to be stored in cloud so i am using a blobfuse2 mounted directory (i checked for file permissions are correct) . I am able so successfully mount the directory , but when i am starting the coverity scan using the command cov-build --<blobfuse2_mounted_directory> then it is not giving me any errors in blobfuse2 logs but coverity scan is getting hang for 30mins and later giving the below error .

Error : Failed to acquire lock "/__w/test-actions/test-actions/idir_new_stream/emit/243b98c66385/emit-db.creation-lock". Unable to protect emit DB creation!

l checked for the file write permissions are there and there is no issues related to authentication

Logs from blobfuse2:

Mon Dec  2 07:10:45 EST 2024 : blobfuse2[32194] : [/home/administrator/blobfuse1] LOG_TRACE [block_blob.go (260)]: BlockBlob::DeleteFile : name emit/QA-GITHUB/emit-db.creation-lock-b72cf4020f6dfad85255871ba3e
Mon Dec  2 07:10:45 EST 2024 : blobfuse2[32194] : [/home/administrator/blobfuse1] LOG_DEBUG [utils.go (156)]: SDK(Retry) : =====> Try=1 for DELETE https://<>.blob.core.windows.net/coveriont1/emit%2QA-GITHUB%2Femit-db.creation-lock-b72cf4020f6dfabb8efca3e
Mon Dec  2 07:10:45 EST 2024 : blobfuse2[32194] : [/home/administrator/blobfuse1] LOG_DEBUG [utils.go (156)]: SDK(Request) : ==> OUTGOING REQUEST (Try=1)
   DELETE https://<>.blob.core.windows.net/coveritytempcont1/emit%2FPSPL-QA-GITHUB%2Femit-db.creation-lock-b72cf4020f6dfad85255871bb8efca3e
   Accept: application/xml
   Authorization: REDACTED
   User-Agent: Azure-Storage-Fuse/2.3.2 (Ubuntu 22.04.5 LTS) azsdk-go-azblob/v1.4.0 (go1.22.4; linux)
   X-Ms-Date: Mon, 02 Dec 2024 12:10:45 GMT
   x-ms-delete-snapshots: include
   x-ms-version: 2024-05-04

Mon Dec  2 07:10:45 EST 2024 : blobfuse2[32194] : [/home/administrator/blobfuse1] LOG_DEBUG [utils.go (156)]: SDK(Response) : ==> REQUEST/RESPONSE (Try=1/368.828002ms, OpTime=369.266698ms) -- RESPONSE RECEIVED
   DELETE https://<>.blob.core.windows.net/coveritytempcont1/emit%2FPSPL-QA-GITHUB%2Femit-db.creation-lock-b72cf4020f6dfad85255871bb8efca3e
   Accept: application/xml
   Authorization: REDACTED
   User-Agent: Azure-Storage-Fuse/2.3.2 (Ubuntu 22.04.5 LTS) azsdk-go-azblob/v1.4.0 (go1.22.4; linux)
   X-Ms-Date: Mon, 02 Dec 2024 12:10:45 GMT
   x-ms-delete-snapshots: include
   x-ms-version: 2024-05-04
   --------------------------------------------------------------------------------
   RESPONSE Status: 202 Accepted
   Content-Length: 0
   Date: Mon, 02 Dec 2024 12:10:46 GMT
   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
   X-Ms-Delete-Type-Permanent: true
   X-Ms-Request-Id: 52707bb0-d01e-000f-01b3-441c92000000
   X-Ms-Version: 2024-05-04

I saw that i am getting the 202 here
@vibhansa-msft
Copy link
Member 

Error : Failed to acquire lock "/__w/test-actions/test-actions/idir_new_stream/emit/243b98c66385/emit-db.creation-lock". Unable to protect emit DB creation!

This error is coming from your application and not directly from Blobfuse2. 202 is a success response only. In your blobfuse log file can you check what kind of error you are getting in blobfuse ?

@vibhansa-msft vibhansa-msft self-assigned this Dec 4, 2024
@vibhansa-msft vibhansa-msft added this to the v2-2.4.1 milestone Dec 4, 2024
@rahulshrivastava-eaton
Copy link
Author

rahulshrivastava-eaton commented Dec 4, 2024 •
edited
Loading

@vibhansa-msft coverity application is running fine if i am using a local directory only issue happens when i use the blobfuse mounted directory , i saw that 202 is ok and success response.
looks like even after a 202 response request to delete /create the log files via coverity (file : emit-db.creation-lock-b72cf4020f6dfad85255871bb8efca3e) files are not getting deleted as i can see them via azure portal . As a azure says 202 response on a delete request means deletion request has been submitted to server successfully but it does not guarantees that resource has been deleted.
Is there some way to drill down more to get to root cause of this , i appreciate your help in understanding the issue .

@vibhansa-msft
Copy link
Member

From the tool side we can only put a request across to backend. Whether backend deletes the file immediately or schedules it for deletion after some time is totally out of our control. I will try to reach out to backend team to better understand the scenario but from blobfuse side we have no control over this.

That being said, how does this impact your flow or failure in Coverity as '202` is still a success for you. Reason for failure might be something else for which we shall investigate more on logs to see which operation actually failed.

@vibhansa-msft
Copy link
Member

Can you confirm couple of things:

  • Is your account HNS enabled?
  • Is soft-delete enabled on your account?
  • Is blob versioning enabled on your account?

When you say delete request was posted and you got 202 but still the blob existed on the portal. Does it get deleted after some time you never see it being deleted?

@rahulshrivastava-eaton
Copy link
Author

@vibhansa-msft
Is your account HNS enabled? - not enabled , I am not very sure though since i can't see HNS settings for me, I saw there a option 'Data Lake Gen 2 updrade' for my storage account, and as per google Hierarchical Namespace (HNS) feature is specifically designed for Azure Data Lake Storage Gen2, so i can assume that this is not enabled

Is soft-delete enabled on your account? - not enabled
Is blob versioning enabled on your account? - not enabled

@vibhansa-msft
Copy link
Member

In portal you can go to overview tab of your storage account and there shall be in option of "Hierarchial Name space" and you need to check whether it says enabled or not.

@vibhansa-msft
Copy link
Member

And how about the other question. File deleted but still visible in portal, does it disappear after some time or just remains for eternity.

@rahulshrivastava-eaton
Copy link
Author

@vibhansa-msft yes Hierarchical namespace is Disabled and yes deleted files are remains for eternity

@jainakanksha-msft
Copy link
Collaborator

@rahulshrivastava-eaton can you please cross-verify the container name/filename, if you are cross-verifying at the right place.

@vibhansa-msft
Copy link
Member

vibhansa-msft commented Dec 9, 2024 •
edited
Loading

@vibhansa-msft yes Hierarchical namespace is Disabled and yes deleted files are remains for eternity

Does that mean backend returned 202 Accepted but files were never deleted? That is highly unlikely to happen, there is something else wrong if that's the observation. From blobfuse end as well we have never seen any such bug report by any of our customers in the past.

@rahulshrivastava-eaton
Copy link
Author

ok @vibhansa-msft i will check again and verify again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vibhansa-msft vibhansa-msft

Labels
triaged V2
Projects
None yet
Milestone
v2-2.4.1
Development

No branches or pull requests
3 participants
@vibhansa-msft @rahulshrivastava-eaton @jainakanksha-msft