Skip to content

Commit

Permalink
[CELEBORN-1733] Support ordered grouped kv input for Tez
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
1. Add CelebornOrderedGroupedKVInput and CelebornOrderedGroupedInputLegacy
2. CelebornScheduler extends ShuffleScheduler to reduce redundant code

### Why are the changes needed?

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?

Closes #2972 from GH-Gloway/1733.

Authored-by: hongguangwei <[email protected]>
Signed-off-by: mingji <[email protected]>
  • Loading branch information
GH-Gloway authored and FMX committed Dec 6, 2024
1 parent e41ee2d commit 8948df1
Show file tree
Hide file tree
Showing 7 changed files with 1,263 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.celeborn.tez.plugin.util;

import java.util.zip.CRC32;

public class ChecksumUtils {

private static final int LENGTH_PER_CRC = 4 * 1024;

public static long getCrc32(byte[] buf) {
return getCrc32(buf, 0, buf.length);
}

public static long getCrc32(byte[] buf, int offset, int length) {
CRC32 crc32 = new CRC32();

for (int i = 0; i < length; ) {
int len = Math.min(LENGTH_PER_CRC, length - i);
crc32.update(buf, i + offset, len);
i += len;
}

return crc32.getValue();
}
}
Loading

0 comments on commit 8948df1

Please sign in to comment.