Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Data] Clarify schema validation error #48882

Merged
merged 7 commits into from
Nov 27, 2024
Merged

[Data] Clarify schema validation error #48882

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cb830b0
Initial commit
bveeramani Nov 22, 2024
1d9dd2e
Merge branch 'master' of https://github.com/ray-project/ray
bveeramani Nov 25, 2024
5570cdb
Fix failing test
bveeramani Nov 25, 2024
b1ba618
Appease lint
bveeramani Nov 25, 2024
c3d328f
Fix another test
bveeramani Nov 26, 2024
bed2558
Merge branch 'master' of https://github.com/ray-project/ray
bveeramani Nov 26, 2024
d261426
Merge branch 'master' into improve-error-message
bveeramani Nov 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions python/ray/data/_internal/planner/exchange/sort_task_spec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,9 @@ def validate_schema(self, schema: Optional[Union[type, "pyarrow.lib.Schema"]]):
for column in self._columns:
if column not in schema_names_set:
raise ValueError(
"The column '{}' does not exist in the "
"schema '{}'.".format(column, schema)
Comment on lines -84 to -85
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to print full schema (though fixing the formatting)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexeykudinkin how would I fix the formatting?

I think it still looks a bit weird even if we move the schema to a newline:

ValueError: You specified the column 'does_not_exist', but there's no such column in the dataset. The dataset has schema:
Column  Type
------  ----
id      int64

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, it will look fine when printed but when embedded it's gonna look messy.

Let's keep just the list of cols

f"You specified the column '{column}', but there's no such "
"column in the dataset. The dataset has columns: "
f"{schema_names_set}"
)

@property
Expand Down
15 changes: 4 additions & 11 deletions python/ray/data/tests/test_execution_optimizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1145,9 +1145,7 @@ def test_sort_validate_keys(ray_start_regular_shared):
assert extract_values("id", ds.sort("id").take_all()) == list(range(10))

invalid_col_name = "invalid_column"
with pytest.raises(
ValueError, match=f"The column '{invalid_col_name}' does not exist"
):
with pytest.raises(ValueError, match="there's no such column in the dataset"):
ds.sort(invalid_col_name).take_all()

ds_named = ray.data.from_items(
Expand All @@ -1165,10 +1163,7 @@ def test_sort_validate_keys(ray_start_regular_shared):
assert [d["col1"] for d in r1] == [7, 5, 3, 1]
assert [d["col2"] for d in r2] == [8, 6, 4, 2]

with pytest.raises(
ValueError,
match=f"The column '{invalid_col_name}' does not exist in the schema",
):
with pytest.raises(ValueError, match="there's no such column in the dataset"):
ds_named.sort(invalid_col_name).take_all()


Expand Down Expand Up @@ -1279,9 +1274,7 @@ def test_aggregate_e2e(ray_start_regular_shared, use_push_based_shuffle):
def test_aggregate_validate_keys(ray_start_regular_shared):
ds = ray.data.range(10)
invalid_col_name = "invalid_column"
with pytest.raises(
ValueError, match=f"The column '{invalid_col_name}' does not exist"
):
with pytest.raises(ValueError):
ds.groupby(invalid_col_name).count()

ds_named = ray.data.from_items(
Expand All @@ -1308,7 +1301,7 @@ def test_aggregate_validate_keys(ray_start_regular_shared):

with pytest.raises(
ValueError,
match=f"The column '{invalid_col_name}' does not exist in the schema",
match="there's no such column in the dataset",
):
ds_named.groupby(invalid_col_name).count()

Expand Down