unable to refresh token

[martenson]

this is likely related to #45

I'm seeing this on QA2, we now set up the offline_access scope, so we get the initial refresh token, but there seems to be another issue present:

Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]: requests_oauthlib.oauth2_session DEBUG 2025-01-14 11:48:41,074 [pN:main.1,p:939522,tN:WSGI_0] Invoking 0 token response hooks.
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]: galaxy.authnz.managers ERROR 2025-01-14 11:48:41,074 [pN:main.1,p:939522,tN:WSGI_0] An error occurred when refreshing user token
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]: Traceback (most recent call last):
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:   File "/srv/galaxy/server/lib/galaxy/authnz/managers.py", line 288, in refresh_expiring_oidc_tokens_for_provider
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:     refreshed = backend.refresh(trans, auth)
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:   File "/srv/galaxy/server/lib/galaxy/authnz/custos_authnz.py", line 142, in refresh
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:     token = oauth2_session.refresh_token(token_endpoint, **params)
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:   File "/srv/galaxy/venv/lib/python3.11/site-packages/requests_oauthlib/oauth2_session.py", line 496, in refresh_token
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:     self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:   File "/srv/galaxy/venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 427, in parse_request_body_response
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:     self.token = parse_token_response(body, scope=scope)
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:   File "/srv/galaxy/venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 441, in parse_token_response
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:     validate_token_parameters(params)
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:   File "/srv/galaxy/venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 448, in validate_token_parameters
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:     raise_from_error(params.get('error'), params)
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:   File "/srv/galaxy/venv/lib/python3.11/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 399, in raise_from_error
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]:     raise cls(**kwargs)
Jan 14 11:48:41 galaxy-qa-nd-2 galaxyctl[939522]: oauthlib.oauth2.rfc6749.errors.InvalidTokenError: (invalid_token) Invalid refresh token: <PRESENT_BUT_REDACTED>

[martenson]

it may be expiring too fast: 2025-01-14 08:30:47.476807

[martenson]

galaxyproject/galaxy#19411