map a custom function with two input columns within summarize or mutate

[JP-Solomon]

Hi all, I have a question as to wether this operation in R is possible using tidypolars / polars.

I will preface this discussion by saying it is possible that I have misinterpreted the how to use polar expressions in tidypolars.

Below I have created a reproducable dataset (in R) and part of the function I am looking to translate into python.

I have the function written in python already and it evaluates properly if the input is a two column tibble.

My issue is that I can't see a way to call two columns as an input to one function within .mutate() or .summarize().

Can one do this?

library(tidyverse)
#function definition
get_spec = function(x){
  x$original = resid(lm(x$original~x$time))
  x$original = x$original-median(x$original)
  original = x$original/mad(x$original,center=0)
  return(original)
}

#representative data
time=rep(seq(1, 50, by = 1/10),2)
series=c(rep(1,length(time)/2),rep(0.5,length(time)/2))
original=(sin(time*series)+(time*series))
data=tibble(series=factor(series),time,original) `

#apply function
data=data%>%
  group_by(series)%>%
  mutate(residual=get_spec(tibble(time,original))) %>%
  pivot_longer(c(original,residual)) %>%
  arrange(series,name,time)

#plot
ggplot(data,aes(x=time,y=value,col=series,group=series)) +
  geom_line() +
  facet_grid(~ name)

[markfairbanks]

In a general sense - yes, you can create a custom function that you can input two columns into. There are some specific functions that aren't implemented in polars or tidypolars that you use (like lm()) that limits recreating your exact example.

You're also using a tibble/data frame as an input to your function in your example which isn't how you would do it in polars/tidypolars. You would actually want to input columns. So in R your function would look like this:

library(tidyverse)
#function definition
get_spec = function(time, original){
  original = resid(lm(original~time))
  original = original-median(original)
  original = original/mad(original,center=0)
  return(original)
}

#apply function
data%>%
  group_by(series)%>%
  mutate(residual=get_spec(time,original))

Here's a simple example that shows what you're going for:

import tidypolars as tp
from tidypolars import col

def multiply_cols(col1, col2):
    out = col1 * col2
    return out

df = tp.Tibble(x = range(3), y = range(3))

df.mutate(new = multiply_cols(col('x'), col('y')))

┌─────┬─────┬─────┐
│ x   ┆ y   ┆ new │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ i64 │
╞═════╪═════╪═════╡
│ 0   ┆ 0   ┆ 0   │
├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤
│ 1   ┆ 1   ┆ 1   │
├╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌┤
│ 2   ┆ 2   ┆ 4   │
└─────┴─────┴─────┘

Hope this helps - if you have any questions let me know.

[JP-Solomon]

Thank you for your precise answer! I should have been more specific in my query. Your answer clarified that, unfortunately, polars won't have an implementation for some of the functions I will require in my pipeline.

[mike-lawrence]

@JP-Solomon Just to double check, if my R code were:

library(tidyverse)

# define a user defined function
#  that is reducing (length 1 output)
reducing_f = function(x,y){
	cor(x,y)
}

# define a user defined function
#  that is non-reducing (same length 
#  output as input)
non_reducing_f = function(x,y){
	resid(lm(x~y))
}

#make fake data
dat = tibble(
	a = rep(c(1,2),each=10)
	, b = rnorm(20)
	, c = rnorm(20)
)

(
	dat
	#group by one of the columns
	%>% group_by(a)
	# use the non-reducing user-defined function 
	%>% mutate(
		d = f(b,c)
	)
	# use the reducing user-defined function 
	%>% summarise(
		e = reducing_f(d,b)
	)
)

is my understanding correct that there's no possible translation to tidypolars that can achieve either the non-reducing nor reducing operations that take as input multiple columns?

[markfairbanks]

The issue above isn't that a function is reducing or non-reducing with multiple columns - but rather the fact that lm() isn't implemented in polars/tidypolars.

cor() also isn't implemented in tidypolars (yet), but it's implemented in polars so I'll show how it will be defined in the future.

Here's an example with reducing and non-reducing functions. In this case for the non-reducing function I just use a simple multiplication.

import tidypolars as tp
from tidypolars import col
import polars as pl

# Define cor
def cor(x, y):
    return pl.pearson_corr(x, y)

# Make fake data
df = tp.Tibble(a = [1, 2] * 10,
               b = range(20),
               c = range(20))

# Non-reducing function
def non_reducing_f(x, y):
    return x * y

# Reducing function
def reducing_f(x, y):
    return cor(x, y)

(
    df
    .mutate(d = non_reducing_f(col('b'), col('c')),
            by = 'a')
    .summarize(e = reducing_f(col('d'), col('b')),
               by = 'a')
)

┌─────┬──────────┐
│ a   ┆ e        │
│ --- ┆ ---      │
│ i64 ┆ f64      │
╞═════╪══════════╡
│ 1   ┆ 0.962691 │
├╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┤
│ 2   ┆ 0.969458 │
└─────┴──────────┘

Hope this helps - if you have any questions let me know.