Adding support for transcripts, recording, AI summarization and meeting subscription to channels #377

jespino · 2024-05-15T17:56:42Z

This PR include 3 important changes:

It allows the user to subscribe channels to meetings. That way, whenever that meeting is started outside mattermost it will post a message with a link to the meeting (identical to the one is create when you click the zoom button).
Whenever a meeting is finish, if there is a recording it is going to report the recording, the transcription and the chat messages in replies to the original zoom post (this works for the meetings in the point 1 and for the meetings that are created using the zoom button).
If you have the mattermost AI Copilot plugin installed and configured, whenever you receive the transcription and the chat, you can summarize them using the AI plugin.

Testing

For testing this, you should test the 3 features independently (there is some degree of overlap, but I going to propose independent test cases). Another important thing is that before you test it you need to configure the zoom application in the zoom marketplace with the right webhook events (it has change, so check out the documentation).

I expect all the test proposed here are executed after properly configuring zoom plugin and AI copilot plugin

Test case 1: Subscription

Go to zoom web interface and create a recurrent meeting.
Go to a channel and execute /zoom subscribe [meeting id], where [meeting id] is the ID of the created meeting in the pervious step (without spaces)
Go to the zoom web interface and initiate the recurrent meeting.
The expected behavior is that in some seconds after that, you see the post in the channel that is subscribe.
Stop the meeting.
After some seconds you should see reflected in the post that the meeting has ended
Go back to the channel and execute /zoom unsubscribe [meeting id], with the same meeting id.
Go back to the zoom web interface and initiate the recurrent meeting again.
The expected behavior is you don't see any post created in the channel.

Test case 2: Recordings, Chats, and Transcriptions (Without AI Copilot enabled)

Go to any channel and click the zoom button to start a meeting.
You will see the zoom post and you get automatically connect to the meeting.
Leave the meeting.
The post in some seconds is going to be mark as terminated.
No other messages are published later (you can skip this step because you can verify it later).
Open another zoom meeting using the same process.
Send a chat message
Start the recording
Say some text to get that in the transcription
Send another chat message
Stop the recording
Leave the meeting.
After this, you will see the post message changing to ended (in some seconds).
Then, after some minutes (maybe around 5), you will receive 2 new posts. one attaching the transcription, and one attaching the chat and including a link to the recording with the password.
Check the transcription and see if that is the one.
Check the chat and verify that only the message sent during the recording is there.
Check the recording link and password (maybe you need to use an incognito window to require the password)

Test case 3: AI Summarization (With AI Copilot enabled)

Follow all the steps of the test case 2 with AI copilot enable
In the transcription message and in the chat message you should see a new button that allows you to summarize the content.
Click that button in the chat history message and see the AI copilot summarizing it
Click that button in the transcription message and see the AI copilot summarizing it
If the text in the chat history or the transcription is too short the AI copilot can send you non-sense, if you want to have a better result with the AI copilot use more messages in the chat and speak more to include more text in the transcription.

jespino · 2024-05-15T18:28:51Z

server/webhook.go

+	var response *http.Response
+	for retries > 0 {
+		var err error
+		response, err = http.DefaultClient.Do(request)


This is because we are handling the incoming webhook from zoom as something secure, it should be signed if you configured it properly

jespino · 2024-05-15T18:28:56Z

server/webhook.go

+					return
+				}
+				request.Header.Set("Authorization", "Bearer "+webhook.DownloadToken)
+				response, err := http.DefaultClient.Do(request)


This is because we are handling the incoming webhook from zoom as something secure, it should be signed if you configured it properly

CodeQL code tracing is 👌

Oh yes, I check the "show paths" and I was really impressed, It is pure gold.

mickmister

Excellent feature here 🚀

I gave the PR a review and added some comments for discussion. Let me know what you think 👍

mickmister · 2024-05-15T19:14:40Z

server/webhook.go

+					return
+				}
+				request.Header.Set("Authorization", "Bearer "+webhook.DownloadToken)
+				response, err := http.DefaultClient.Do(request)


CodeQL code tracing is 👌

server/command.go

webapp/src/components/post_type_chat.jsx

server/command.go

server/webhook.go

…ng-and-ai-summarization

jespino · 2024-05-17T08:13:48Z

@mickmister PTAL

…cordings

azigler · 2024-07-15T22:24:32Z

I opened a docs PR to correspond with the changes in this PR.

mattermost/docs#7276

azigler · 2024-07-16T19:57:33Z

@jespino Currently there is no way to know what meetings a channel is subscribed to, which makes it hard to know what state it's in. Should we add a slash command (maybe /zoom list) that shows subscriptions for the channel?

azigler · 2024-09-06T18:53:27Z

Is there a timeline for merging this?

jespino · 2024-09-06T18:59:23Z

@azigler not really, we are still waiting for the security review and there are other things that are higher priority.

wiggin77 · 2024-09-25T21:15:56Z

@jespino Any appetite for getting this merged while we're still using Zoom?

jespino · 2024-09-26T11:29:35Z

@wiggin77 I would love to see this merged, but it requires the security review and it is still pending, I guess V10 release has keep the security team busy for this kind of things. @esarafianou any news on this?

…ng-and-ai-summarization

wiggin77

Leaving this request for change to get this off my review queue until it is ready.

Kshitij-Katiyar · 2024-11-18T08:38:20Z

@jespino Can you sync this PR with master, this fix some of the failing ci

…ng-and-ai-summarization

…dereference

…test

jespino · 2025-01-11T08:15:51Z

@wiggin77 It looks like @enzowritescode is going to start with the security review soon, so feel free to start reviewing the code.

jespino added 6 commits May 9, 2024 18:33

Adding support for transcripts, recording and AI summarization

db76929

WIP

a305b43

Adding chat support

dd47bbf

WIP

12815af

Adding support for subscription of meetings to channels

161b660

Removing debug log messages

467399e

jespino requested a review from mickmister as a code owner May 15, 2024 17:56

jespino added the Do Not Merge Should not be merged until this label is removed label May 15, 2024

github-advanced-security bot found potential problems May 15, 2024

View reviewed changes

jespino added 3 commits May 15, 2024 20:15

Fixing tests

e448589

fixing linter errors

df60c28

Fixing linter errors

bc696fd

mickmister requested a review from ayusht2810 May 15, 2024 19:10

mickmister added 2: Dev Review Requires review by a core committer 3: QA Review Requires review by a QA tester labels May 15, 2024

mickmister reviewed May 15, 2024

View reviewed changes

jespino added 5 commits May 16, 2024 10:39

Addressing PR review comments

513006a

Updating webhooks subscriptions needed in the documentation

8b30a5d

Addressing PR review comments

a810064

Addressing PR review comments

9ae260c

Avoid subscriptions to personal meetings

42bfc3a

mickmister requested a review from AayushChaudhary0001 May 16, 2024 19:31

jespino added 5 commits May 16, 2024 21:45

Merge remote-tracking branch 'origin/master' into transcripts-recordi…

4548448

…ng-and-ai-summarization

Migrating to typescript

4f6233f

Migrating to typescript

0cc74d9

Fixing linter

bf0599d

Adding tests for transcript and chat handlers

7777670

jespino requested a review from mickmister May 17, 2024 08:13

Fixing some linter errors

b67ed82

jespino added 2 commits July 12, 2024 20:29

Fixing a crash on subscription

d75b8a5

Fixing the alteration on the length after receiving transcriptions/re…

5c9a13a

…cordings

This was referenced Jul 15, 2024

Update Zoom plugin configuration docs to reflect settings for new functionality mattermost/docs#7275

Open

Update Zoom plugin configuration docs to reflect settings for new functionality mattermost/docs#7276

Open

Merge remote-tracking branch 'origin/master' into transcripts-recordi…

30d8b15

…ng-and-ai-summarization

wiggin77 requested changes Oct 28, 2024

View reviewed changes

jespino added 13 commits January 11, 2025 07:35

Merge remote-tracking branch 'origin/master' into transcripts-recordi…

0280fc4

…ng-and-ai-summarization

Fixing linter error

d3067af

fix: Add missing mock expectation for KVGet in test case

fc9d542

fix: Add missing mock expectation for GetUser in webhook test

191ef21

fix: Add missing mock expectation for bot user's Zoom token

0af0204

fix: Add missing KVSetWithExpiry mock for meeting post ID test

d3f70c4

fix: Add missing mock expectation for KVSetWithExpiry in test

70f1af5

test: Add mock expectation for PublishWebSocketEvent in test case

5fe64f0

fix: Initialize plugin client in webhook test to resolve nil pointer …

9dc2a3c

…dereference

fix: Initialize plugin client in TestWebhookHandleRecordingCompleted …

c90d235

…test

Fixing tests

fb6ae2f

Fixing linter error

e9ea52a

test: Add tests for handleMeetingStarted webhook functionality

a35591e

jespino requested review from wiggin77 and enzowritescode and removed request for ayusht2810 January 11, 2025 08:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding support for transcripts, recording, AI summarization and meeting subscription to channels #377

Adding support for transcripts, recording, AI summarization and meeting subscription to channels #377

jespino commented May 15, 2024 •

edited

Loading

jespino May 15, 2024

jespino May 15, 2024

mickmister May 15, 2024

jespino May 16, 2024

mickmister left a comment

mickmister May 15, 2024

jespino commented May 17, 2024

azigler commented Jul 15, 2024

azigler commented Jul 16, 2024

azigler commented Sep 6, 2024

jespino commented Sep 6, 2024

wiggin77 commented Sep 25, 2024

jespino commented Sep 26, 2024

wiggin77 left a comment

Kshitij-Katiyar commented Nov 18, 2024

jespino commented Jan 11, 2025

Adding support for transcripts, recording, AI summarization and meeting subscription to channels #377

Are you sure you want to change the base?

Adding support for transcripts, recording, AI summarization and meeting subscription to channels #377

Conversation

jespino commented May 15, 2024 • edited Loading

Testing

Test case 1: Subscription

Test case 2: Recordings, Chats, and Transcriptions (Without AI Copilot enabled)

Test case 3: AI Summarization (With AI Copilot enabled)

jespino May 15, 2024

Choose a reason for hiding this comment

jespino May 15, 2024

Choose a reason for hiding this comment

mickmister May 15, 2024

Choose a reason for hiding this comment

jespino May 16, 2024

Choose a reason for hiding this comment

mickmister left a comment

Choose a reason for hiding this comment

mickmister May 15, 2024

Choose a reason for hiding this comment

jespino commented May 17, 2024

azigler commented Jul 15, 2024

azigler commented Jul 16, 2024

azigler commented Sep 6, 2024

jespino commented Sep 6, 2024

wiggin77 commented Sep 25, 2024

jespino commented Sep 26, 2024

wiggin77 left a comment

Choose a reason for hiding this comment

Kshitij-Katiyar commented Nov 18, 2024

jespino commented Jan 11, 2025

jespino commented May 15, 2024 •

edited

Loading