Improve performance of find_in_set function

[tlm365]

Which issue does this PR close?

Closes #.

Rationale for this change

Improve performance of find_in_set function

What changes are included in this PR?

Are these changes tested?

Yes. Added unit tests and benchmarks.

Are there any user-facing changes?

No.

*BENCHMARK RESULT

Compiling datafusion-functions v44.0.0 (/home/tailm/repos/github/datafusion/datafusion/functions)
    Finished `bench` profile [optimized] target(s) in 1m 02s
     Running benches/find_in_set.rs (target/release/deps/find_in_set-c80fb04778cbe610)
Gnuplot not found, using plotters backend
find_in_set/string_len_8
                        time:   [414.89 µs 415.67 µs 416.41 µs]
                        change: [-64.610% -64.526% -64.440%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) low mild
find_in_set/string_view_len_8
                        time:   [440.65 µs 441.02 µs 441.50 µs]
                        change: [-63.746% -63.607% -63.494%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  1 (2.00%) high mild
  1 (2.00%) high severe

find_in_set/string_len_32
                        time:   [474.24 µs 474.97 µs 475.79 µs]
                        change: [-61.751% -61.586% -61.447%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 50 measurements (4.00%)
  2 (4.00%) high mild
find_in_set/string_view_len_32
                        time:   [493.30 µs 494.79 µs 496.58 µs]
                        change: [-61.565% -61.365% -61.171%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 50 measurements (8.00%)
  1 (2.00%) high mild
  3 (6.00%) high severe

find_in_set/string_len_1024
                        time:   [4.8152 ms 4.8243 ms 4.8367 ms]
                        change: [-12.694% -12.465% -12.233%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 50 measurements (6.00%)
  2 (4.00%) high mild
  1 (2.00%) high severe
find_in_set/string_view_len_1024
                        time:   [4.8690 ms 4.8755 ms 4.8825 ms]
                        change: [-13.472% -13.274% -13.072%] (p = 0.00 < 0.05)
                        Performance has improved.

[jayzhan211]

Probably faster if create Vec first and use PrimitiveArray::from()

[tlm365]

@jayzhan211 Thanks for reviewing,

Probably faster if create Vec first and use PrimitiveArray::from()

I have tried implementing it this way, it seems to have similar performance to this PR.

Here is the code I have tested:

pub fn find_in_set_general<'a, T, V>(
    string_array: V,
    str_list_array: V,
) -> Result<ArrayRef>
where
    T: ArrowPrimitiveType,
    T::Native: OffsetSizeTrait,
    V: StringArrayType<'a, Item = &'a str>,
    PrimitiveArray<T>: From<Vec<Option<<T as ArrowPrimitiveType>::Native>>>,
{
    let string_iter = ArrayIter::new(string_array);
    let str_list_iter = ArrayIter::new(str_list_array);

    let mut result: Vec<Option<T::Native>> = Vec::with_capacity(string_iter.len());

    string_iter
        .zip(str_list_iter)
        .for_each(
            |(string_opt, str_list_opt)| match (string_opt, str_list_opt) {
                (Some(string), Some(str_list)) => {
                    let position = str_list
                        .split(',')
                        .position(|s| s == string)
                        .map_or(0, |idx| idx + 1);
                    let value = T::Native::from_usize(position).unwrap();
                    result.push(Some(value));
                }
                _ => result.push(None),
            },
        );
    let array = PrimitiveArray::<T>::from(result);

    Ok(Arc::new(array) as ArrayRef)
}

[comphead]

I think it is a good PR the way it is. One thing comes to my mind which probably relevant for other string functions as well.

So we do split(',') for every search pattern value, but most likely that argument will be literal like SELECT FIND_IN_SET(a, "s,q,l"); so we can do split only once.

I think we need to have a check a literal or column somewhere and we can optimize even further having this info, WDYT @tlm365 ?