Design a way to attach JSON objects to messages

[dlqqq]

Problem

There exists a need to design some way of attaching arbitrary JSON objects to messages.

For Jupyter AI v3, we intend to provide @ commands that allow users to include files & variables from the notebook for context. It would be ideal to retrieve the data of these files/variables directly from the frontend and somehow attach them to the next input for the current user. Defining command handling in the frontend is the most flexible approach, as it allows access to every API provided by Jupyter, not just Python APIs. See this comment for details on the thought process here.

Attachments can also later be used to add support for multimodal models which accept both text & images, and may return text & images.

Proposed Solution

1. Modify the structure of Message to allow a list of attachments, whose contents are arbitrary and JSON-serializable.
2. Design some sort of Attachments API in the frontend.
3. Add a UI to let users attach files manually. This should call the Attachments API.
4. Show attachments as a list of toasts on top of the chat input, and allow users to remove attachments. I've included an example screenshot from GitHub Copilot to use as inspiration.

I'm not sure how the Attachments API should be accessed. I think it would be good to make it accessible from the YChat object in the frontend. This way, the YChat interface provides all of the APIs needed by chat commands.

Additional context

I'm currently quite busy with iterating on my existing designs & planning collaborations with new contributors. @brichet I'm assigning this to you as you have the most context on Jupyter Chat, and can help propose better ideas. Feel free to implement a proof-of-concept in a separate PR.

[brichet]

Thanks @dlqqq for opening this, I can take a look at it.

Proposition

• add an attachments property to Message, containing an array of objects, with this kind of interface (can be extended):

{
  type: 'file' | 'variable' | 'image',
  value: string,
  mimetype?: string
}

The value could be a path, a variable name or an image content.

• add a function in ChatInput to add an attachment to a list, that will be send with the message (can be called several time to add several attachments to one message).

  This function could be called from a UI (button + file selector) or when using a pattern in the message, like @file or @variable.

• show the attachments item on top or bottom of the input, and probably the same for the messages with attachments.

I don't think this need for a specific API, the attachments are part of the message.

Additional questioning

1. In the case of a variable or a file, I don't know how we could handle the value/content, which may change or be deleted after the message has been sent. The response to the message would therefore be irrelevant in the context of a persistent chat.
2. To handle the variables, we'll need to link the chat to a kernel

[brichet]

I don't think this need for a specific API, the attachments are part of the message.

After a very quick test, I agree that we need a sort of API, to be able to add attachments from completion, as expressed at #144 (comment).

One option could be to add the attachments (to be send with the next message) to the model.
That way, the sendMessage() function could add these attachments to the message. In the case of jupyterlab-chat, it should be added there.

[dlqqq]

I agree that implementing this in ChatModel makes more sense. Thanks for the references.