Random runtime crashes in pkg on Windows

[mikiher]

What version of pkg are you using?

5.12.0

What version of Node.js are you using?

18.5.0

What operating system are you using?

Windows

What CPU architecture are you using?

x86_64

What Node versions, OSs and CPU architectures are you building for?

node18-win-x64

Describe the Bug

This was reported by serveral audiobookshelf-windows users (I'm maintaining this app) in this bug.

In a nutshell, audiobookshelf-windows packs the Audiobookshelf server into a windows executable using yao-pkg and runs it from tray application. The pkg command and config can be found here. The server also uses a native addon (sqlite3) and pkg config puts the relevant .node files in the assets section.

Users have reported random crashes inside pkg runtime code, which I have not been able to reproduce, but I wanted to see if you've seen this before or have some advice.

The crashes seems to happen at random points, and all look roughly like this:

[2024-08-06 08:13:55.840] FATAL: [Server] Uncaught exception origin: uncaughtException, error: Error: ENOENT: no such file or directory, open 'C:\Users\shaun\AppData\Local\Temp\pkg-NGVozZ\88e6d330c6f684cce2af952a76696bfc3ba3d62a88e46fd9b599d7b54591a743'
    at Object.openSync (node:fs:599:3)
    at Object.openSync (pkg/prelude/bootstrap.js:793:32)
    at uncompressExternallyAndOpen (pkg/prelude/bootstrap.js:748:19)
    at pkg/prelude/bootstrap.js:763:32
    at FSReqCallback.oncomplete (node:fs:201:23) {
  errno: -4058,
  syscall: 'open',
  code: 'ENOENT',
  path: 'C:\\Users\\shaun\\AppData\\Local\\Temp\\pkg-NGVozZ\\88e6d330c6f684cce2af952a76696bfc3ba3d62a88e46fd9b599d7b54591a743'
} (    at process.<anonymous> (C:\snapshot\audiobookshelf\server\Server.js))
node:internal/fs/utils:347
    throw err;
    ^

Error: ENOENT: no such file or directory, open 'C:\Users\shaun\AppData\Local\Temp\pkg-NGVozZ\88e6d330c6f684cce2af952a76696bfc3ba3d62a88e46fd9b599d7b54591a743'
    at Object.openSync (node:fs:599:3)
    at Object.openSync (pkg/prelude/bootstrap.js:793:32)
    at uncompressExternallyAndOpen (pkg/prelude/bootstrap.js:748:19)
    at pkg/prelude/bootstrap.js:763:32
    at FSReqCallback.oncomplete (node:fs:201:23) {
  errno: -4058,
  syscall: 'open',
  code: 'ENOENT',
  path: 'C:\\Users\\shaun\\AppData\\Local\\Temp\\pkg-NGVozZ\\88e6d330c6f684cce2af952a76696bfc3ba3d62a88e46fd9b599d7b54591a743'
}

Expected Behavior

Obviously, not crash :)

To Reproduce

Unfortunately, I have not been able to repro this - I'm not quite sure how to trigger this as it seems to happen randomly and quite rarely.

[robertsLando]

Hi @mikiher , that C:\Users\shaun\AppData\Local\Temp\pkg-NGVozZ\ is where pkg copies .node and .dll files when them are required from the code, reason is that them cannot be loaded from snapshot like it happens for all other files. My feel is that for some reason that folder is automatically removed by Windows for some reason? Maybe the anti-virus detect that file as a virus and removes it from there? Unfortunately with the informations provided I cannot say much more.

The code that handles that is here:

https://github.com/yao-pkg/pkg/blob/main/prelude/bootstrap.js#L2199-L2255

[robertsLando]

The strange thing is the folder should be different. Starting from 5.12.0 we moved the cache folder to homedir: #55

Could you try to switch to pkg latest first?

I see you are not specifing a version here: https://github.com/mikiher/audiobookshelf-windows/blob/master/.github/workflows/build-installer.yml#L27

I suggest you to put a fixed version instead when installing it

[mikiher]

The strange thing is the folder should be different. Starting from 5.12.0 we moved the cache folder to homedir: #55

Ah, sorry for the confusion - some of the user reports may date back to when we used previous versions.

I suggest you to put a fixed version instead when installing it

OK, thanks for the suggestion, will do. It looks like I unintentionally used the latest version anyway when building the executable, but as I said above, some of the reports may have used older versions.

At which point in my code would dlopen() be called? if this has to do with loading the sqlite native addon, I expect this would happen at server startup, but it looks like the crashes occur at random times. Is the dll sometimes unloaded and needs to be reloaded?

[robertsLando]

At which point in my code would dlopen() be called

It is called when a dll is required in code somewhere (so in theory only on startup). Problem is that if for some reason sometihng (like an anti virus) or else cleans the dir where we store this files the error could happen also during runtime. I suggest you to try replicate the issue with latest version as we changed the location where we load those files and so the issue may not happen anymore

[marcinstrzelczyk]

Hello,

I have a user that experienced similar issue, the application crashes with Error: ENOENT: no such file or directory, open 'C:\WINDOWS\TEMP\pkg-P5JELD\a82b8f8e47c556fab0bed2a245fd1b023a61e754838dcf9cba504296a207c930' in the log.

In my case the application tries to read a simple .txt file that is built into the .exe.

The application was built using pkg 6.2.0.

The code that handles that is here:
https://github.com/yao-pkg/pkg/blob/main/prelude/bootstrap.js#L2199-L2255

I've looked at that code and I think it's not the right place - it creates the tmp path in the /pkg/<hash> format, whereas both @mikiher and I have the /pkg-XXXXXX/<hash> path in logs.

We should probably be looking at
https://github.com/yao-pkg/pkg/blob/main/prelude/bootstrap.js#L712-L737

that's the only place I've found that uses the /pkg-XXXXXX/<hash> format - that XXXXXX seems to be added by mkdtempSync (and it still uses tmpdir, not the homedir).

@robertsLando could you reopen this issue (I don't have permissions to do that)?