HDDS-11784. Allow aborting FSO multipart uploads with missing parent directories #7700
Conversation
There was a problem hiding this comment.
@sokui Thanks for the patch.
Please check the test failures
getMultipartKeyFSO is used by a lot of MPU flows. Furthermore, multipartInfoTable will be accessed two times although. I think we can create another function similar to getMultipartKeyFSO by passing the OmMultipartKeyInfo and use that only for the abort case. This means switching the order in S3MultipartAbortRequest by first getting from multipartInfoTable and then to openFileTable. All other implementations are welcome.
Also let's add a simple test as suggested in #7566 (comment)
For example, there is a directory "/a" and there is a pending MPU key with path "/a/mpu_key" that was initiated but haven't been completed / aborted yet. After the MPU key was initiated, the directory "/a" is deleted, and since mpu_key has not been completed yet and does not exist in fileTable, the DIRECTORY_NOT_EMPTY will not be thrown in OMKeyDeleteRequestWithFSO#validateAndUpdateCache. Therefore mpu_key is orphaned and when it's completed / aborted, it will fail in OMFileRequest#getParentID since the parent directory has been deleted.
ivandika3
changed the title
ivandika3
changed the title
My consideration is that I originally implemented the exact way you describe. But considering the above reasons, I change to directly modify For the interface, I noticed For the tests, I will take a look the failure. For the new test you suggested, do you know if there is similar testing existing so that I can reference it? I am not super familiar with ozone code base. So if there is no such similar code there, could you pls show me some code snippet which I can start with. Really appreciate it! |
There was a problem hiding this comment.
My consideration is that org.apache.hadoop.ozone.om.request.util.OMMultipartUploadUtils#getMultipartOpenKey is used by multiple places, including S3ExpiredMultipartUploadsAbortRequest and S3MultipartUploadAbortRequest. By updating one place here, it benefits for two places (both not work currently). Secondly, if my current implementation of getMultipartKeyFSO is more reliable, there is no reason to only limit this benefit to S3MultipartUploadAbortRequest. All the other places should use this as well.
My worry is that there might be some places where the expectations for OMMultipartUploadUtils#getMultipartKeyFSO is to access the open key/file table or there might some places where the multipartInfoTable entry does not exist yet, which might result in NPE which might crash the OM (we should handle the possible NPE). However, I'm OK as long as there are no test regressions.
For the tests, I will take a look the failure. For the new test you suggested, do you know if there is similar testing existing so that I can reference it? I am not super familiar with ozone code base. So if there is no such similar code there, could you pls show me some code snippet which I can start with. Really appreciate it!
You can start with TestOzoneClientMultipartUploadWithFSO integration test.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
Outdated
Show resolved
Hide resolved
I started with some test with the following code. But It seem I cannot delete the directory. It gave me the following error: I checked the Just wonder if it is deleting a key or a directory? And if it is directory, why I got the above exception? |
@sokui The parentDir should be "a/b/" with trailing slash. Because there was no trailing slash, your key is "a/b<key_name>", so only "a/" parent directory is created. Therefore, deleting "a/b" will return KEY_NOT_FOUND (as it should be). I created the following test, you can adapt it to your test. @Test
public void testAbortMultipartUploadSuccessWithMissingParentDirectories() throws Exception {
String parentDir = "parentDirToDelete/";
keyName = parentDir + UUID.randomUUID();
OzoneManager ozoneManager = cluster.getOzoneManager();
String buckKey = ozoneManager.getMetadataManager()
.getBucketKey(volume.getName(), bucket.getName());
OmBucketInfo buckInfo =
ozoneManager.getMetadataManager().getBucketTable().get(buckKey);
BucketLayout bucketLayout = buckInfo.getBucketLayout();
ozClient.getProxy().createDirectory(volumeName, bucketName, parentDir);
String uploadID = initiateMultipartUpload(bucket, keyName, RATIS,
ONE);
Pair<String, String> partNameAndETag = uploadPart(bucket, keyName, uploadID,
1, "data".getBytes(UTF_8));
OMMetadataManager metadataMgr =
cluster.getOzoneManager().getMetadataManager();
String multipartKey = verifyUploadedPart(uploadID, partNameAndETag.getKey(),
metadataMgr);
// Delete parent directory
ozClient.getProxy().deleteKey(volumeName, bucketName, parentDir, false);
// Abort multipart upload with missing parent directory
bucket.abortMultipartUpload(keyName, uploadID);
String multipartOpenKey =
metadataMgr.getMultipartKeyFSO(volumeName, bucketName, keyName, uploadID);
OmKeyInfo omKeyInfo =
metadataMgr.getOpenKeyTable(bucketLayout).get(multipartOpenKey);
OmMultipartKeyInfo omMultipartKeyInfo =
metadataMgr.getMultipartInfoTable().get(multipartKey);
assertNull(omKeyInfo);
assertNull(omMultipartKeyInfo);
} |
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
Outdated
Show resolved
Hide resolved
oops. I missed the ending |
|
@ivandika3 addressed all the comments. Pls have another look. Thank you! |
There was a problem hiding this comment.
Overall we might want to revisit the multipart open key table key name calculation as a whole. I am not sure if the key should be based on normal FSO keys in the mutipart upload scenario. Maybe, we should just use the path of S3 key as is in the key calculation and not depend on the path/directories created? This fix might be ok for now but something to reconsider as a whole. I will go over the MPU code a few more times to complete the review.
| } catch (final Exception e) { | ||
| // It is possible we miss directories and exception is thrown. | ||
| // see https://issues.apache.org/jira/browse/HDDS-11784 | ||
| LOG.warn("Got exception when finding parent id for {}/{}/{}. Use another way to get it", |
There was a problem hiding this comment.
This might be a common occurance in a concurrent system and might not warrant a WARN log.
There was a problem hiding this comment.
per @ivandika3 's suggestion, we probably need to update the logic of directory deletions for incomplete MPU so that we can prevent missing parent directories case happening (or make it in a very low probability): #7566 (comment)
From high level, I feel if this case happens frequently, it means something is not right (either at code level or design level). How can a key exist but its parent directories already got deleted? Thats why I feel it is good to have this warn log here. Pls let me know your thoughts.
| String fileName = OzoneFSUtils.getFileName(key); | ||
| long parentId; | ||
| try { | ||
| parentId = OMFileRequest.getParentID(volumeId, bucketId, key, this); |
There was a problem hiding this comment.
I am going over the code in a bit more detail but why not use the exception handling way to calculate the parentId as the default way?
There was a problem hiding this comment.
This was my suggestion since there might be some codes that depends on the fact getParentID implementation (i.e. parent directory exists).
I'm fine to use the updated way as long as there are no regressions found.
There was a problem hiding this comment.
Yeah. I just follow @ivandika3 suggestion, which is safer. And I tested if I just use the exception handling way to do it, then OMKeyCreateRequestWithFSO#getDBMultipartOpenKey method will throw exception when the MPU is aborted. And this method is used in multiple places. Of course, we can update all these places. I am just not sure if it is safe. Just try to limit the scope of this PR. Pls let me know your thoughts
There was a problem hiding this comment.
Thanks for the update. LGTM +1.
However, please also address any suggestions from @kerneltime first.
What changes were proposed in this pull request?
HDDS-11784 adding missing parent directories for MPU abort and expired abort request
We observed lots of open key (files) in our FSO enabled ozone cluster. And these are all incomplete MPU keys.
When I tried to abort MPU by using s3 cli as below, I got the exception complaining about the parent directory is not found.
Exceptions in the log
This issue is because of missing the parent directories. The causation and the solution is explained here: #7566 (comment).
There is another PR which we closed. We had some conversation about which approach should be taken. Pls reference here: #7566
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11784
How was this patch tested?
It tested by the CI. Also we validate it in our cluster.