server(plugins): update without downtime

[kontrollanten]

Description

• Create a new folder each time a new plugin/theme is installed or updated. The folder name is created based on the package.json content hash.
• Removes the need to decache nodes require cache when updating plugins.
• Run install/update/uninstall operations in jobs.

Related issues

closes #4828

Has this been tested?

• 👍 yes, I added tests to the test suite
• 💭 no, because this PR is a draft and still needs work
• 🙅 no, because this PR does not update server code
• 🙋 no, because I need help

[kontrollanten]

I don't know why the test is failing, it seems unrelated to the changes and it works when I test locally.

@Chocobozzz , currently this PR doesn't handle multiple parallell installs. The package.json is copied when install process begins, so if there's multiple parallell installations only the last one will be stored in package.json. I propose to move the plugin install/update/uninstall processes to be handled by a job instead. Cons:

a) More stable server. If the server restarts just after an admin has started a plugin installation it won't be aborted.
b) Better UX. Today the user get an error message saying the installation failed when there's a HTTP timeout.

[kontrollanten]

@Chocobozzz Let me know what you think of this draft. If you're positive to the solution I'll fix the broken tests.

[kontrollanten]

@Chocobozzz Ping

[Chocobozzz]

@Chocobozzz Ping

I'm sorry but I unfortunately don't have time at the moment to review your work

[Chocobozzz]

Hi @kontrollanten

This method seems interesting since we have many "issues" with the decache system, but I fear an unavoidable increase of the PeerTube process memory no?

[kontrollanten]

Hi,

I don't remember the details how this is implemented, but I can't remember it should increase the process memory. Why do you fear it?

[Chocobozzz]

I don't remember the details how this is implemented, but I can't remember it should increase the process memory. Why do you fear it?

Because every time we update a plugin we re-load all of them, and since we can't control the ESM cache I wonder if the RAM would not increase indefinitely

[kontrollanten]

We only reload the updated plugin, but yes the old plugin version will still be loaded in the cache. So the memory usage will increase until PT is restarted. It should be possible determine how much this will affect with tests done with clinic.js or similar.

To avoid increasing memory usage the server code be restarted either on every or update or depending on memory usage/size. A more robust solution could be to run plugins as spawned microservices that communicates with PT over HTTP.