Fix Manual Syncing Tool -

[chrisjtang]

Description

Sometimes, the manual syncing tool would only sync some comments and not others, and the only indication that there was something wrong would be a 500 error. After some digging, it was determined that this was happening when the syncing tool would attempt to sync a child comment before that child's parent comment was synced.

To fix this, we switched the order that we fetch comments from the Disqus API, and we sync parent comments before child comments in order to do our best effort to sync child comments in the right order. To account for child comments with parents outside of the date range that haven't been synced yet, we removed the check for parent comments to allow those child comments to still sync. Later on, if those parent comments get synced, we made a change so that these child comments will update with the parent comment data.

A second issue that was discovered in testing was that comments posted via RSS would have trouble syncing due to missing an identifier in the thread data. This was remedied by removing the error when missing the thread data, which would still allow for syncing of these comments minus a direct link to the comment in the WordPress UI that would otherwise be there in non-RSS comment.

Additionally, these changes include improvements to the logging that we expose in the browser console to make future troubleshooting easier.

And last but not least, per the request of some users, we increased the date range for manual syncing to 5 years to make the tool easier to use for bulk comment syncing.

Motivation and Context

This bug was reported by a few publishers to the Disqus support team and would consistently break the syncing tool.

How Has This Been Tested?

This has been tested on the Disqus WP test site hooked up to the Disqus Blog. Aside from testing the actual syncing, we benchmarked the syncing with staggered parallelism vs without staggered parallelism, and we decided to remove staggered parallelism since the results were about twice as fast with syncing.

• Parameters: 6 months: 6/1/22 - 1/1/23. 2089 comments
• With staggered parallelism: 13 min 38 seconds. All comments synced.
• Without staggered parallelism: 7 min 55 seconds. All comments synced.

Tests for increasing the date range: Was curious as to how the tool would perform without limiting to 1 year, so did additional tests with larger sizes (only without staggered parallelism so it wouldn't take too long):

• Parameters: 3 years (1/1/21 - 12/12/23). 4123 comments

• Without staggered parallelism: 15min 45 seconds. All comments synced.

• Parameters: 5 years (12/12/18 - 12/12/23). 12730 comments

• Without staggered parallelism: 49 minutes. All comments synced.

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly.
• All new and existing tests passed.