Converting from FuncX endpoint to Globus Compute Endpoint does not work

[astro-friedel]

Describe the bug
I have not used my FuncX endpoint for about a year, so when I needed to use it I updated my codebase to use Globus Compute. When I tried to activate my endpoint I got the message that the configuration needed to be updated, which I allowed. That part went fine, but when I went to start the endpoint I got error messages with statements like:
400, error: invalid grant

or when I try to get a Client()

AuthAPIError: ('POST', 'https://auth.globus.org/v2/oauth2/token', None, 400, 'Error', 'Bad Request')

The first I was able to trace as originating from the globus_sdk. I spent significant time trying to fix the bug. I finally found the solution: delete the .funcx directory from my home directory. Then everything works fine. So it looks like the existence of old endpoint configs in .funcx interferes with the operation of globus compute.

To Reproduce
Steps to reproduce the behavior, for e.g:

1. Have an existing FuncX config on your system
2. Install globus-compute-sdk==2.4.0 and globus-compute-endpoint==2.4.0 with Python 3.10 any linux machine
3. Activate the endpoint
4. Allow globus compute to convert the endpoint
5. Start the endpoint to produce the error, or create a Client()

Expected behavior
I expect the endpoint to start without issue or I get a Client object without error.

Environment

• OS: Ubuntu 22.04 and RHEL 8.8

• OS & Container technology: identical conda environments on both machines with the following config
  channels:
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=2_gnu
- bzip2=1.0.8=h7f98852_4
- ca-certificates=2023.7.22=hbcca054_0
- ld_impl_linux-64=2.40=h41732ed_0
- libexpat=2.5.0=hcb278e6_1
- libffi=3.4.2=h7f98852_5
- libgcc-ng=13.2.0=h807b86a_2
- libgomp=13.2.0=h807b86a_2
- libnsl=2.0.0=hd590300_1
- libsqlite=3.43.0=h2797004_0
- libuuid=2.38.1=h0b41bf4_0
- libzlib=1.2.13=hd590300_5
- ncurses=6.4=hcb278e6_0
- openssl=3.1.3=hd590300_0
- pip=23.2.1=pyhd8ed1ab_0
- python=3.11.6=hab00c5b_0_cpython
- readline=8.2=h8228510_1
- setuptools=68.2.2=pyhd8ed1ab_0
- tk=8.6.13=h2797004_0
- tzdata=2023c=h71feb2d_0
- wheel=0.41.2=pyhd8ed1ab_0
- xz=5.2.6=h166bdaf_0
- pip:
- attrs==23.1.0
- bcrypt==4.0.1
- certifi==2023.7.22
- cffi==1.16.0
- charset-normalizer==3.3.0
- click==8.1.7
- cryptography==41.0.4
- dill==0.3.6
- docutils==0.20.1
- globus-compute-common==0.3.0a2
- globus-compute-endpoint==2.4.0
- globus-compute-sdk==2.4.0
- globus-sdk==3.28.0
- idna==3.4
- jinja2==3.1.2
- jsonschema==4.19.1
- jsonschema-specifications==2023.7.1
- lockfile==0.12.2
- markupsafe==2.1.3
- packaging==23.2
- paramiko==3.3.1
- parsl==2023.7.3
- pika==1.3.2
- psutil==5.9.5
- pycparser==2.21
- pydantic==1.10.13
- pyjwt==2.8.0
- pynacl==1.5.0
- pyprctl==0.1.3
- python-daemon==2.3.2
- pyyaml==6.0.1
- pyzmq==23.2.0
- referencing==0.30.2
- requests==2.31.0
- rpds-py==0.10.4
- setproctitle==1.3.3
- six==1.16.0
- tblib==1.7.0
- texttable==1.7.0
- typeguard==2.13.3
- types-paramiko==3.3.0.0
- types-requests==2.31.0.8
- types-six==1.16.21.9
- typing-extensions==4.8.0
- urllib3==2.0.6
- websockets==10.3

• Python version @ 3.11.6

• Python version @ endpoint 3.11.6

• globus-compute-sdk version @ client 2.4.0

• globus-compute-endpoint version @ endpoint 2.4.0

Distributed Environment

• Where are you running the funcX script from? Laptop
• Where does the endpoint run? Login Node
• What is your endpoint-uuid? 40272761-4b82-47cb-9599-277fea474d76

[LeiGlobus]

If you haven't used the endpoint for a while (> 6 months) most likely the credentials expired and the refresh token no longer works.

If you are using a late-ish version of globus-compute-endpoint, there is a convenient globus-compute-endpoint logout command that will clear your cached credentials. The next time you start an endpoint, it will ask you to go to auth.globus.org to get another token again, and that should fix it.

Let us know if this doesn't resolve the issue!

[astro-friedel]

I had recently been using a globusconnectpersonal endpoint (within a few days) so I had recent credentials. Additionally, the error messages were beyond obscure and did not indicate a specific issue with credentials. When I did get the endpoint to start I did not need to create it again (because it already existed in the .globus_compute directory), this indicated to me that even though the endpoint had been converted, the existence of the FuncX version was superceding the globus_compute version, causing issues.

[LeiGlobus]

Yes, I agree that the "Invalid Grant" error message isn't very clear on what the issue/solution might be. We will change/address it as soon as we can, as others have also encountered it and been puzzled.

The crux is that Globus Compute/FuncX uses a separate set of credentials than Transfer/Globus Connect Personal, and the FuncX->Compute package conversion only renames/symlinks the .globus_compute directory and does not delete/reset the (1-yr old) Globus Compute credentials, if existent (in case they were recently generated in FuncX). It also doesn't touch or use the GCP/Globus-general credentials at this time.

So the globus-compute-endpoint logout should reset the (old) Compute specific credentials to refresh them. globus login and globus logout and globus session are the commands one would use to manage GCP/Transfer ones.

[LeiGlobus]

The latest globus-compute-endpoint (and -sdk) 2.6.0 now prints a more appropriate message about the credentials expiring with a suggested how-to-fix command using logout.

Closing this issue.

[khk-globus]

For reference, this was implemented in PR #1334.