Pathological streaming slowdown with promises 1.3.1

[jcheng5]

With yesterday's {promises} 1.3.1 release, long streaming answers from elmer in shinychat have ground to an absolute halt. Using the example from the README:

app source

library(shiny)
library(shinychat)

ui <- bslib::page_fluid(
  chat_ui("chat")
)

server <- function(input, output, session) {
  chat <- elmer::chat_openai(system_prompt = "You're a trickster who answers in riddles")
  
  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })
}

shinyApp(ui, server)

shinychat-1.mov

The slowness can be addressed by setting options(shiny.deepstacktrace=FALSE), but then you get a stack overflow:

shinychat-2.mov

[jcheng5]

The following test app is an excellent way to see the problem. The only thing it doesn't show is how many promise domains are in effect at the same time--I suspect by the time the slowdown is noticeable, each handler is executing with the burden of hundreds if not thousands of Shiny's call-stack-capturing promise domains being simultaneously active, due to rstudio/shiny#4155.

Run this and type "Tell me a story" as the user prompt.

library(shiny)
library(shinychat)
library(promises)

# options(shiny.deepstacktrace=FALSE)

instrument_async_stream <- function(stream) {
  (coro::async_generator(function() {
    for (i in await_each(stream)) {
      deepStackDepth <- length(shiny:::.globals$deepStack)
      callStackDepth <- length(sys.calls())
      message("deepStackDepth: ", deepStackDepth, ", callStackDepth: ", callStackDepth)
      yield(i)
    }
  }))()
}

ui <- bslib::page_fluid(
  chat_ui("chat")
)

server <- function(input, output, session) {
  chat <- elmer::chat_openai(
    model = "gpt-4o-mini",
    system_prompt = "You're a trickster who answers in riddles"
  )
  
  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input) |>
      instrument_async_stream()
    chat_append("chat", stream)
  })
}

shinyApp(ui, server)

[jcheng5]

Just cause the pingbacks are getting a little confusing, here are the underlying issues along with PRs for their fixes:

shiny

• Recursively adding captureStackTrace domains Stack trace domain explosion rstudio/shiny#4155
• Deep stack traces grow and grow without limit Limit deep stack growth rstudio/shiny#4156

promises

• When re-entering domains, it actually adds the domain to be re-entered on top of any domains that might already exist Fix ever-growing call stacks rstudio/promises#115