[FC-0049] fix: Issues when importing a large taxonomy

[ChrisChV]

Description

• Adds time elapsed to import log.
• Fix issues with memory and time when importing a large taxonomy.

Supporting information

• Github issue: Issues when importing a large taxonomy modular-learning#213
• Related to: [FC-0049] chore: Bump version of openedx-learning for taxonomy import improvements edx-platform#34796
• Internal ticket: https://tasks.opencraft.com/browse/FAL-3721

It has been tested with taxonomies that take too long or could not finish due to a memory backup. These are their average times after fixing the issue:

• WGU Instructional Design: K-12 Collection: 5.95 seconds.
• FlatTaxonomy: 65.55 seconds.
• Lightcast Open Skills Taxonomy: 76.5 seconds.

Testing instructions

• Use this branch of edx-platform: [FC-0049] chore: Bump version of openedx-learning for taxonomy import improvements edx-platform#34796
• Run make requirements on cms-shell
• Ensure you have the testing data: https://github.com/open-craft/taxonomy-sample-data
• Export this taxonomies:
  • WGU Instructional Design
  • FlatTaxonomy
  • Lightcast Open Skills Taxonomy
• Import a new taxonomy using the previous exported files.
• After every import:
  • Ensure that the memory is linear constant and there is not a memory overflow.
  • Go to python cms bash and run the code. Check the elapsed time.

from openedx_tagging.core.tagging.models.import_export import TagImportTask
task = list(TagImportTask.objects.all())[-1]
task.log

[openedx-webhooks]

Thanks for the pull request, @ChrisChV! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[yusuf-musleh]

👍 @ChrisChV Great work on this! Its working well, here are the timings I got when importing the different taxonomies:

• WGU Instructional Design: K-12 Collection: 1.44 seconds.
• FlatTaxonomy: 14.98 seconds.
• Lightcast Open Skills Taxonomy: 18.47 seconds.

I just had a few nit questions below.

• I tested this: I followed the testing instructions in the PR
• I read through the code
• I checked for accessibility issues
• Includes documentation

[yusuf-musleh]

nit: Just curious if this had an impact on improving the performance or is it mainly making the code cleaner?

[ChrisChV]

It improves memory a little because it does not create an instance of Tag, but uses the create function. It was the first thing I tried, but it wasn't the root of the problem. I left it like this because it looks cleaner.

[bradenmacdonald]

Tag.objects.create(...) still creates a Tag instance and calls .save() on it for you, and returns it. It may be garbage-collected slightly better if you don't keep the returned Tag instance here, but I wouldn't expect any major difference. I do like the create() version more though - it is cleaner.

[yusuf-musleh]

nit: similar to the above, if this had any performance improvement or just code style changes?

[ChrisChV]

Same as above

[openedx-webhooks]

@ChrisChV 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.