-
Notifications
You must be signed in to change notification settings - Fork 465
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OVP 2.22.0: recordingStatusChanged webhook event with status [ready] gets triggerred multiple times intermittently which causes erroneous behaviour in our application #747
Comments
Hi, I am afraid we need more details on how to replicate the problem. Without that, we will not be able to fix it. Starting and stopping a recording on OpenVidu > 2.17.0 in a regular deployment only triggers event |
Hello @mallikarjungurav , I've researched your issue and I think I found a possible reason for your duplicated webhook responses. The key here is your phrase:
WebHook retries was a feature added after 2.17.0 to make the WebHookSender more resilient to unexpected situations. The class which implements the WebHook HTTP sender is this: HttpWebhookSender The HTTP Header is implemented in such a way that, if an IOException happen, the webhook is resent to the client: openvidu/openvidu-server/src/main/java/io/openvidu/server/webhook/HttpWebhookSender.java Lines 95 to 100 in f124f0a
Those retries only happen under an IOException, so this basically means, that the server you have which is receiving those HTTP Webhook events is not responding correctly to OpenVidu, the HTTP Connection is closed, or some exception is happening which makes OpenVidu "believe" that the request is not sent correctly. Please verify that the server which is receiving those webhooks responds correctly to OpenVidu webhook events. |
Once we receive webhook we process it and send 200 ok response, irrespective of whether we get any error while processing at out service end. Even if we get any exception we catch it but we send 200 o response back. |
Check logs from OpenVidu, webhooks are resent if a failure happens. |
Is the handle made asynchronously? Take for example the call to the method I recommend you to return the 200 OK before processing the event, or execute |
Where to check whether it is asynchronous? |
It is your code @mallikarjungurav |
In general words @mallikarjungurav, If it takes too much time to return the 200 OK, it will fail and OpenVidu will resend the webhook event. You need to send that 200OK before processing the event. |
If your code is synchronous and you do not want to change too much of your code base, a way to avoid processing repeated webhooks is to have a Concurrent Set to store there which recordings are you processing and avoid handle it if it is being processed. Something like this: private Set<String> processingRecordingSet = ConcurrentHashMap.newKeySet();
...
public void handleOpenViduEvent(uriInfo, webhookEvent) {
...
if (webhookEvent.getType().equals("recordingStatusChanged")) {
String recordingId = webhookEvent.getJson().get("id");
if (this.processingRecordingSet.add(recordingId)) {
// Do your processing here
logger.info("Processing recording [{}]", recordingId);
doing.stuff()
more.stuff()
stuff()
this.processingRecordingSet.remove(recordingId);
} else {
logger.info("Repeated webhook event. Recording [{}] is being processed", recordingId)
}
}
...
} This is an example, I don't know how your classes and methods are in your code, but I hope you get the idea. |
Did you try our proposed solutions? |
I can't use above solution since we have loadbalancer in place so I can't rely on in-memory storage. And I think think that solves the problem since I am giving response back to OVP once I receive the webhook. Currently we are in testing phase, I will let you know if I find any issues there. Thanks for your help. |
Describe the bug
For single vide file [recordingStatusChanged] webhook event with status [ready] gets trigger on provided service endpoint (service API endpoint) multiple times intermittently. We process video file once we receive this webhook. The problem is this webhook gets trigger mulitiple times sometimes twice, 5 times etc.
Expected behavior
We expect for a video file once recording is in ready status, we should receive this webhook only once as that of OVP version 2.17.0
Wrong current behavior
Once recording is ready from OVP, the webhook [recordingStatusChanged] with status [ready] gets trigger mulitiple times. sometimes twice, 5 times etc.
OpenVidu tutorial where to replicate the error
This is an EXTREMELY IMPORTANT STEP. If we are able to replicate the error in any of the official OpenVidu Tutorials or OpenVidu Demos, then we will be able to quickly fix it. If you are getting the error in your own application, please try to add the necessary changes to the most similar tutorial so it fails with the same error (try to keep those changes as contained as possible, so that the original tutorial maintains its integrity). Once you have an application to replicate the error, explain in detail the steps to get it like this:
OpenVidu deployment info
How is your OpenVidu Server instance deployed when you get the bug. A couple of possible examples are listed below:
docker run ...
on macOS Catalina 10.15.1Client device info (if applicable)
Describe the client device(s) or platform(s) where you are able to replicate the error. For example:
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
I think the information provided is enough to understand and replicate the issue highlighted.
The text was updated successfully, but these errors were encountered: