GSC Failed to load entrypoint (missing shebang support in execve())

[dzygann]

Hi,

as you can see in issue gramineproject/graphene#2632 we're trying to run Postgres in Graphene. OS is Ubuntu 20.04.

Contrary to issue gramineproject/graphene#2632 we didn't use our own dockerfile, but instead, we used one from Dockerhub https://hub.docker.com/_/postgres

Then again we build and signed it using gsc.
When trying to run this image however we encountered the following problem:

[P9:T1:docker-entrypoint.sh] error: Failed to load /docker-entrypoint.sh. This may be caused by the binary being non-PIE, in which case Graphene requires a specially-crafted memory layout. You can enable it by adding 'sgx.nonpie_binary = 1' to the manifest.
[P9:T1:docker-entrypoint.sh] error: Error during shim_init() in init_loader (-22)

If we follow the instruction to add the sgx.nonpie_binary = 1 we get the following error:

Parsing /entrypoint.manifest as TOML failed: Duplicate keys! (line 26 column 1 char 1073)

We have also used the -L option during the gsc build, but there are no more debug information available.

Do you have any ideas to fix this issue?

Thanks in advance.

[mkow]

[P9:T1:docker-entrypoint.sh] error: Failed to load /docker-entrypoint.sh. This may be caused by the binary being non-PIE, in which case Graphene requires a specially-crafted memory layout. You can enable it by adding 'sgx.nonpie_binary = 1' to the manifest.

Graphene doesn't support running scripts as entrypoints. You need to pass an ELF binary as the executable argument to Graphene (but it can be e.g. Bash, which could start the script you want).

Parsing /entrypoint.manifest as TOML failed: Duplicate keys! (line 26 column 1 char 1073)

Exactly as the error says, you already have this entry in your manifest and you added another, duplicated one.

Anyways, this error message seems to be confusing users, I'll try to fix it (there's even an old TODO there).

[dimakuv]

@dzygann If this answers your question, please close the issue. Otherwise feel free to ask further.

[dzygann]

Hi @dimakuv,
sorry for the delay. We are struggling with the task to start the script. The current failing approach is:

COPY docker-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["docker-entrypoint.sh"]

We have tried e.g.

ENTRYPOINT ["/bin/bash", "-c", "docker-entrypoint.sh"]

but it doesn't work. The container stops quietly.

What are we doing wrong?

[dimakuv]

Hard to say... Did you try to run with debug logs of Graphene? Did you try this without GSC (bare metal, no Dockers)?

[dzygann]

Hi,

we could run the shell script by adding the user as a parameter

ENTRYPOINT ["/bin/bash","-c","docker-entrypoint.sh postgres"]

The postgres docker runs for a short time (~1min) and terminates with the following log entries:

[P53:T33:bash] error: Fail: Trying to drop reference count below 0
[P53:T33:bash] error: BUG() ../include/shim_internal.h:320
error: Unknown or illegal instruction at RIP 0x00000000fb32f167
[P53:T33:bash] error: Illegal instruction during Graphene internal execution at 0xfb32f167 (IP = +0x1d167, VMID = 53, TID = 33)
find: 'chown' terminated by signal 30

Do you have an idea what the illegal instruction is?
Why does it terminate with the chown command? We didn't use this command explicitly.

[mkow]

Which commit of Graphene do you use? (internally, inside GSC)

[dzygann]

Hi @mkow,
we really did use an older version, thanks for pointing it out. Now we pulled the current version. We have seen that the behaviour changed a little bit after the commit gramineproject/graphene@7059cd4

Now we are back at the starting point:

[P1:T1:bash] error: /usr/local/bin/postgres-docker-entrypoint.sh is not an ELF executable. Please note that Graphene doesn't support executing scripts as executables.
: /usr/local/bin/postgres-docker-entrypoint.sh: /usr/bin/env: bad interpreter: Invalid argument

The workaround to use the /bin/bash as entrypoint does not work anymore for this case.

[mkow]

Ok, seems this time someone used raw exec on a script from another Bash script, which resulted in execve() on a script.

Dmitrii said you wanted to start contributing to Graphene, right? If that's true, then this may be a good candidate for your first contribution :) I.e.: adding support for shebang scripts in execve() syscall implementation + entrypoint loading code. You probably should start with reading our loading code and then looking up how Linux implements this, to correctly handle all corner cases. And then some LTP tests may be useful, there should be some with execve+shebangs.

[Time0o]

@mkow It seems like shebang support was built into execve up until commit c8fc2a1106e28ceb81fa308721209162afbc33b2. It's not quite clear to me why that commit removed it again.

Also, if a new implementation were to be based on Linux, wouldn't that cause license issues? LGPLv3 is not compatible with Linux's implementation of execve.

[pwmarcz]

@mkow It seems like shebang support was built into execve up until commit c8fc2a1. It's not quite clear to me why that commit removed it again.

We removed the old shebang code because it was not covered by any tests, we were unaware of any real use cases (IIRC it was actually broken at that point, and nobody complained), and the support was only partial (only execve, not entrypoint).

We would definitely welcome a patch adding the support again, as long as these problems are solved (at the very least, it would be good to have two tests in LibOS/shim/test/regression: one for running a script with shebang as libos.entrypoint; another for a program invoking execve() on such script).

Also, if a new implementation were to be based on Linux, wouldn't that cause license issues? LGPLv3 is not compatible with Linux's implementation of execve.

Linux is GPLv2, so yes, we cannot accept any code copied from Linux. That said, it should be OK if you study the Linux sources in order to come up with your own implementation of the same behavior. I haven't seen it, but I would also recommend looking at *BSD sources: they're often simpler.

[Michael-M-Zhang]

Is this issue still exist now?

[mkow]

@Michael-M-Zhang The implementation of this feature is WIP at the moment: #459