Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: optimise where clause filtering #2076

Merged
merged 62 commits into from
Dec 3, 2024
Merged
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
9cb9f5a
Add filter
robacourt Nov 27, 2024
d1d1adf
Make Shapes.record_in_shape? take a shape
robacourt Nov 27, 2024
f299dc0
Make add_shape public
robacourt Nov 27, 2024
9327bbd
Support other types of change
robacourt Nov 27, 2024
5ca7267
Test id not found
robacourt Nov 27, 2024
285105d
Add Filter struct
robacourt Nov 27, 2024
f4a5e39
Turn other_shapes into a map
robacourt Nov 27, 2024
5012b0c
Support relation changes
robacourt Nov 27, 2024
0586a0e
Support TruncateRelation
robacourt Nov 27, 2024
e97d803
Optimise where clauses
robacourt Nov 27, 2024
e4ccdcf
Optimise where clauses
robacourt Nov 27, 2024
54809ee
Add Filter.remove_shape
robacourt Nov 28, 2024
c697ce8
Update dispatcher
robacourt Nov 28, 2024
319ede5
Add TODOs
robacourt Nov 29, 2024
1e7f050
Support renames
robacourt Nov 29, 2024
1cb01da
Fix remaining tests
robacourt Nov 29, 2024
2881ad2
Remove redundant arg
robacourt Nov 29, 2024
8c3bf5f
Pass inspector to dispatcher
robacourt Nov 29, 2024
9f03c18
Refactor dispatcher so that state is a struct
robacourt Nov 29, 2024
bb2e4fa
Add inspector to dispatcher state
robacourt Nov 29, 2024
55322a6
Add TODO
robacourt Nov 29, 2024
ecf54f9
Parse record rather than converting the const to a string
robacourt Nov 29, 2024
33c8a8b
Create expr on Filter.new
robacourt Nov 29, 2024
684cd47
Move functions into TableFilter
robacourt Nov 30, 2024
04c04e9
Rename TableFilter to Table
robacourt Nov 30, 2024
ac11f0d
Rename public Table functions
robacourt Nov 30, 2024
3db9970
Introduce Table struct
robacourt Nov 30, 2024
7538087
Extract code into Field
robacourt Nov 30, 2024
8a484ab
Rename table_filter to table
robacourt Nov 30, 2024
2d2c3f6
Rename field to index
robacourt Nov 30, 2024
47fc530
Ensure table is first param of Table public functions
robacourt Nov 30, 2024
ba275c5
Neaten up remove_shape
robacourt Nov 30, 2024
f8c2703
Refactor affected_shapes function to improve argument structure and u…
robacourt Nov 30, 2024
c4af878
Refactor: Use Table.empty?/1 for cleaner table emptiness check
robacourt Dec 1, 2024
b604482
Use Index struct
robacourt Dec 1, 2024
001e0d3
Refactor add_shape function to reorder parameters for consistency.
robacourt Dec 1, 2024
f4d3f9b
Remove unused inspector option from Dispatcher and related modules.
robacourt Dec 1, 2024
be41674
Refactor: Rename handle to shape_id
robacourt Dec 1, 2024
7c22ad7
Refactor all_shapes
robacourt Dec 2, 2024
dbf1192
Gracefully handle parsing errors
robacourt Dec 2, 2024
32e75e9
Hide deliberate error in test
robacourt Dec 2, 2024
b150fed
Test error logging for invalid error
robacourt Dec 2, 2024
c660449
Update TODOs
robacourt Dec 2, 2024
6dd4a48
Refactor code into WhereClause module
robacourt Dec 2, 2024
81be707
Remove Filter.new/1
robacourt Dec 2, 2024
e9822d6
Test arrays
robacourt Dec 2, 2024
7f474a1
Test shape with no where clause
robacourt Dec 2, 2024
ab3b970
Update TODOs
robacourt Dec 2, 2024
29fb39d
Update moduledoc
robacourt Dec 2, 2024
c452bc4
Add moduledoc
robacourt Dec 2, 2024
0ac30a6
Add moduledocs
robacourt Dec 2, 2024
efe3613
Remove TODO
robacourt Dec 2, 2024
aa4e7c2
Rescue all errors in affected_shapes
robacourt Dec 2, 2024
f91c830
Remove unnecessary field from dispatcher state
robacourt Dec 2, 2024
c0fba1a
Revert unnecessary change
robacourt Dec 2, 2024
8997ce6
Add typespecs
robacourt Dec 2, 2024
3e40fd7
Add documentation
robacourt Dec 2, 2024
e478fee
Decouple Filter tests from the Filter data structure
robacourt Dec 3, 2024
93ee95e
Add optimisation tests
robacourt Dec 3, 2024
51cd083
Refactor reductions/1
robacourt Dec 3, 2024
0f8ef27
Rename Filter.empty() to Filter.new()
robacourt Dec 3, 2024
12c0644
Add changeset
robacourt Dec 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions packages/sync-service/test/electric/shapes/filter_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,59 @@ defmodule Electric.Shapes.FilterTest do

assert Filter.affected_shapes(filter, truncation) == MapSet.new(["s1", "s2", "s3", "s4"])
end

test "where clause in the form `field = const` is optimised" do
filter =
1..1000
|> Enum.reduce(Filter.empty(), fn i, filter ->
Filter.add_shape(filter, i, Shape.new!("t1", where: "id = #{i}", inspector: @inspector))
end)

reductions =
reductions(fn ->
assert Filter.affected_shapes(filter, change("t1", %{"id" => "7"})) == MapSet.new([7])
end)

assert reductions < 500
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to see this specific number explain. What is the significance of it in this test? What should we do if this assertion fails?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Let me write a comment

end

test "where clause in the form `field = const AND another_condition` is optimised" do
filter =
1..1000
|> Enum.reduce(Filter.empty(), fn i, filter ->
Filter.add_shape(
filter,
i,
Shape.new!("t1", where: "id = #{i} AND id > 6", inspector: @inspector)
)
end)

reductions =
reductions(fn ->
assert Filter.affected_shapes(filter, change("t1", %{"id" => "7"})) == MapSet.new([7])
end)

assert reductions < 500
end

test "where clause in the form `a_condition AND field = const` is optimised" do
filter =
1..1000
|> Enum.reduce(Filter.empty(), fn i, filter ->
Filter.add_shape(
filter,
i,
Shape.new!("t1", where: "id > 6 AND id = #{i}", inspector: @inspector)
)
end)

reductions =
reductions(fn ->
assert Filter.affected_shapes(filter, change("t1", %{"id" => "7"})) == MapSet.new([7])
end)

assert reductions < 500
end
end

test "shape with no where clause is affected by all changes for the same table" do
Expand Down Expand Up @@ -245,4 +298,11 @@ defmodule Electric.Shapes.FilterTest do
]
}
end

defp reductions(fun) do
{:reductions, reductions} = :erlang.process_info(self(), :reductions)
fun.()
{:reductions, new_reductions} = :erlang.process_info(self(), :reductions)
new_reductions - reductions
end
end
Loading