Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ORC-1528: Fix readBytes potential overflow in RecordReaderUtils.ChunkReader#create #1662

Closed
wants to merge 8 commits into from
Closed

ORC-1528: Fix readBytes potential overflow in RecordReaderUtils.ChunkReader#create #1662

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ffc6b16
Fix readBytes potential overflow in RecordReaderUtils#create
yebukong Nov 24, 2023
88aecc6
rename variables
yebukong Nov 24, 2023
a9f325f
Revert "rename variables"
yebukong Nov 27, 2023
c6cc177
adjusting code style
yebukong Nov 27, 2023
1869d72
adjust validation for reqBytes and readBytes to MAX_ARRAY_SIZE
yebukong Nov 27, 2023
4f9dfc2
adjust validation conditions
yebukong Nov 27, 2023
26b3501
readBytes and reqBytes validation UT
yebukong Nov 27, 2023
0e6c74a
rename the UT methods
yebukong Nov 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions java/core/src/java/org/apache/orc/impl/RecordReaderUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -770,18 +770,18 @@ void readRanges(FSDataInputStream file, boolean allocateDirect, double extraByte
}

static ChunkReader create(BufferChunk from, BufferChunk to) {
long f = Integer.MAX_VALUE;
long e = Integer.MIN_VALUE;
long f = Long.MAX_VALUE;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: This is already existing so please feel free to ignore.
Since we are touching the code, Can we rename the variables for easy reading?
Couldn't understand the meaning of f, e, cf, ce 😢

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, let me try.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Nov 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I'm opposite to @mystic-lama 's comment.

We don't want to mix a bug fix and a style fix (or refactorying) in the same PR. It's because it increases the review complexity indeed.

If we want to rename this, it should be done before or after this PR.

Please don't change the existing variable name in this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, @dongjoon-hyun has a point. @yebukong we can do that in follow up PR or may be leave it as is :)
Apologies for the noise.

long e = Long.MIN_VALUE;

long cf = Integer.MAX_VALUE;
long ef = Integer.MIN_VALUE;
int reqBytes = 0;
long cf = Long.MAX_VALUE;
long ef = Long.MIN_VALUE;
long reqBytes = 0L;

BufferChunk current = from;
while (current != to.next) {
f = Math.min(f, current.getOffset());
e = Math.max(e, current.getEnd());
if (ef == Integer.MIN_VALUE || current.getOffset() <= ef) {
if (ef == Long.MIN_VALUE || current.getOffset() <= ef) {
cf = Math.min(cf, current.getOffset());
ef = Math.max(ef, current.getEnd());
} else {
Expand All @@ -792,7 +792,14 @@ static ChunkReader create(BufferChunk from, BufferChunk to) {
current = (BufferChunk) current.next;
}
reqBytes += ef - cf;
return new ChunkReader(from, to, (int) (e - f), reqBytes);
if (reqBytes >= Integer.MAX_VALUE) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orc/java/core/src/java/org/apache/orc/impl/StringHashTableDictionary.java

Lines 61 to 65 in 22a06b4

/**
* The maximum size of array to allocate, value being the same as {@link java.util.Hashtable},
* given the fact that the stripe size could be increased to larger value by configuring "orc.stripe.size".
*/
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;

I think it might be more accurate to extract and reuse this MAX_ARRAY_SIZE constant, the java maximum array length is not actually Integer.MAX_VALUE.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I define a MAX_ARRAY_SIZE constant in RecordReaderUtils ? I noticed it is marked as private in StringHashTableDictionary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tend to extract it into a utility class (e.g. IOUtils) and reuse the same constant.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should remove MAX_ARRAY_SIZE from StringHashTableDictionary.java in this pr, the same concept should be kept in a single source (IOUtils).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree "the same concept should be kept in a single source".

Should we include the changes to remove the MAX_ARRAY_SIZE definition from StringHashTableDictionary.java in this PR?

Copy link
Contributor Author

@yebukong yebukong Nov 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guiyanakuang Considering that this PR has been closed, is it still necessary to continue the discussion on the issues mentioned above?

throw new IllegalArgumentException("invalid reqBytes value " + reqBytes + ",out of bounds " + Integer.MAX_VALUE);
}
long readBytes = e - f;
if (readBytes >= Integer.MAX_VALUE) {
throw new IllegalArgumentException("invalid readBytes value " + readBytes + ",out of bounds " + Integer.MAX_VALUE);
}
return new ChunkReader(from, to, (int) readBytes, (int) reqBytes);
}

static ChunkReader create(BufferChunk from, int minSeekSize) {
Expand Down
32 changes: 32 additions & 0 deletions java/core/src/test/org/apache/orc/impl/TestRecordReaderUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.List;
import java.util.Objects;

import static org.junit.jupiter.api.Assertions.assertEquals;
Expand Down Expand Up @@ -177,6 +178,37 @@ public void execute() throws Throwable {
assertTrue(dis.isAllReleased());
}

@Test
public void testBufferChunkOffsetExceedsMaxInt() {
List<long[]> mockData = Arrays.asList(
new long[]{15032282586L, 15032298848L}
, new long[]{15032298848L, 15032299844L}
yebukong marked this conversation as resolved.
Show resolved Hide resolved
, new long[]{15032299844L, 15032377804L}
, new long[]{15058260587L, 15058261632L}
, new long[]{15058261632L, 15058288409L}
, new long[]{15058288409L, 15058288862L}
, new long[]{15058339730L, 15058340775L}
, new long[]{15058340775L, 15058342439L}
, new long[]{15058449794L, 15058449982L}
, new long[]{15058449982L, 15058451700L}
, new long[]{15058451700L, 15058451749L}
, new long[]{15058484358L, 15058484422L}
, new long[]{15058484422L, 15058484862L}
, new long[]{15058484862L, 15058484878L}
);
TestOrcLargeStripe.RangeBuilder rangeBuilder = new TestOrcLargeStripe.RangeBuilder();
mockData.forEach(e -> rangeBuilder.range(e[0], (int) (e[1] - e[0])));
BufferChunkList rangeList = rangeBuilder.build();

RecordReaderUtils.ChunkReader chunkReader =
RecordReaderUtils.ChunkReader.create(rangeList.get(),
134217728);
yebukong marked this conversation as resolved.
Show resolved Hide resolved
long readBytes = mockData.get(mockData.size() - 1)[1] - mockData.get(0)[0];
long reqBytes = mockData.stream().mapToLong(e -> (int) (e[1] - e[0])).sum();
assertEquals(chunkReader.getReadBytes(), readBytes);
assertEquals(chunkReader.getReqBytes(), reqBytes);
}

private static byte[] byteBufferToArray(ByteBuffer buf) {
byte[] resultArray = new byte[buf.remaining()];
ByteBuffer buffer = buf.slice();
Expand Down
Loading