[Feature] Add SSL Certificate Context And Setting For Async/Sync HTTP Requests #6976

deeleeramone · 2024-11-29T23:09:42Z

Why?:
- Resolves [BUG][FR] Impossible to specify SSL certification #6939

The Requests library accepts an environment variable, REQUESTS_CA_BUNDLE, and AIOHTTP does not. This creates inconsistencies between synchronous and async HTTP requests. Furthermore, defining this variable limits the Requests library to that particular file. If you want to use a self-signed certificate for only some things, it is not straightforward.

This solution combines the specified certificate with the certifi defaults.

What?:
- Allows a CA certificate file to be specified for verifying HTTPS requests.
  - Self-signed certificates do not need to be part of the system's trust store.
- Allows "python_settings" in system_settings.json to accept extras.
  - Uses keys http as a nested dictionary.
- Applies the configuration to the internal helpers:
  - make_request
  - amake_request
  - amake_requests
- Handles the scenarios where libraries are using the Requests library directly. This is done by manipulating environment variables before and after making the request.
  - yFinance
  - Finviz
  - Posthog
  - OpenBB Hub
- Allows keyword arguments to be passed to uvicorn.run by storing them as a dictionary to: system_settings.python_settings.uvicorn
Impact:
- Should have absolutely no impact unless these items are specified.
- The behavior of Requests environment variable, REQUESTS_CA_BUNDLE, ports directly to the async requests.
- Allows it to be defined in system_settings.json instead of environment variables.
- cafile is an equivalent to REQUESTS_CA_BUNDLE, when pointing to a file.
- When "verify_ssl": false, SSL certificate verification is disabled within all OpenBB functions.
- Keyword arguments supplied to the "python_settings["uvicorn"]" dictionary are passed directly to uvicorn.run when launching the API as:
  - python -m openbb_core.api.rest_api
  - openbb-api

These items, in system_settings.json, will take precedence over environment variables:

{
    "python_settings": {
        "http": {
            "cafile": "/full/path/to/certificate/localhost.crt",
            "verify_ssl": null,
            "proxy": null,
            "certfile": null,
            "keyfile": null,
        }
    }
}

certfile and keyfile are intended more for future use, the most important key is cafile.

Note: Keyword arguments added to the command line from openbb-api take precedence over the system_settings.json file.

Testing Done:
- Create a self-signed certificate and start an OpenBB API over HTTPS
- Add the full path to the .crt file as shown above.
- Import both requests and openbb_core.provider.utils.helpers.make_request
  - requests.get should fail while make_request succeeds.
  - Set as an environment variable instead, "cafile" is null and add REQUESTS_CA_BUNDLE='/full/path/to/certificate/localhost.crt' to the .env file.
    - Both requests.get and make_request should succeed.
    - Making an outside call using requests.get("https://google.com") will fail, make_request should succeed.

With the environment variable defined, and not system_settings.json, the same A/B can be applied to yfinance.download() vs. `obb.equity.price.historical(provider="yfinance")

This fails because yFinance is only verifying against the self-signed certificate for localhost. openbb_yfinance.utils.helpers.yf_download applies the environment configuration for the duration of the request.

Update to the behavior: the implementation now impacts the yFinance singleton.

montezdesousa

I have a few concerns with this solution.

We are modifying environment variables from within the application and the logic to so in get_certificates and restore_certs is complex to follow.
We are introduction a state inside Env(). I assume that's why we have some lines with Env() to refresh the state (?). This class should be stateless and readonly for consistency and predictability.
The implementation requires us to constantly get and restore certificates, which is not obvious when implementing new requests.

I suggest using a single session that picks configurations and is shared across the app as proposed here #6939.

deeleeramone · 2024-12-05T02:14:30Z

I suggest using a single session that picks configurations and is shared across the app as proposed here #6939.

OK, try it out now, @montezdesousa. I found a way to hack the session into the Posthog handler. The implementation now goes the other way where trust_env is disabled for Requests - which is the default for AIOHTTP - and cut everything down into two helper functions:

get_requests_session()
get_async_requests_session()

Using any of the utility HTTP request functions will read the system_settings.json and take precedence over any environment variable. Binding session objects to the obb app doesn't accomplish much because it needs to happen at the provider level, and we also don't want singleton behavior because it would mess with threading and concurrency. But, a session of the appropriate type can be passed into the functions as a parameter.

make_request
amake_request
amake_requests

Any unclosed async session object will be closed on exit.

…ure/ssl-context

This reverts commit 714df64.

…kwargs

openbb_platform/core/openbb_core/api/rest_api.py

montezdesousa · 2025-01-07T12:22:12Z

When trying to test I get the same error both with requests and make_request. Also tried to provide the path to the certificate through verify and get the same error. Uvicorn is running over HTTPS and I get access it through the browser fine.

SSLError: HTTPSConnectionPool(host='0.0.0.0', port=8000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1133)')))

deeleeramone · 2025-01-07T17:42:08Z

When trying to test I get the same error both with requests and make_request. Also tried to provide the path to the certificate through verify and get the same error. Uvicorn is running over HTTPS and I get access it through the browser fine.
SSLError: HTTPSConnectionPool(host='0.0.0.0', port=8000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1133)')))

This error seems to indicate that the self-signed certificate is not being found at all. I know there's lots of ways to make it not work, so rooting out the edge cases would be nice.

What are your exact deployment steps here, is this a network deployment? Is the same .crt file accessible to both uvicorn and the Python environment? Something like:

    "python_settings": {
        "http": {
            "cafile": "/Users/darrenlee/OpenBBUserData/localhost.crt"
        },
        "uvicorn": {
            "ssl_certfile": "/Users/darrenlee/OpenBBUserData/localhost.crt",
            "ssl_keyfile": "/Users/darrenlee/OpenBBUserData/localhost.key"
        }
    }

Was the certificate created by the same version of Python and openssl used in the deployment and environment?

…to feature/ssl-context

…ce/OpenBB into feature/ssl-context

montezdesousa

I tested and it works as expected, just left some comments and suggestions mostly on code readability.

montezdesousa · 2025-01-08T15:43:46Z

openbb_platform/core/openbb_core/provider/utils/helpers.py

+def get_python_request_settings() -> dict:
+    """Get the python settings."""
+    # pylint: disable=import-outside-toplevel
+    from openbb_core.app.service.system_service import SystemService
+
+    python_settings = SystemService().system_settings.python_settings.model_dump()
+    http_settings = python_settings.get("http", {})
+    allowed_keys = [
+        "cafile",
+        "certfile",
+        "keyfile",
+        "password",
+        "verify_ssl",
+        "fingerprint",
+        "proxy",
+        "proxy_auth",
+        "proxy_headers",
+        "timeout",
+        "auth",
+        "headers",
+        "cookies",
+    ]
+
+    return {
+        k: v for k, v in http_settings.items() if v is not None and k in allowed_keys
+    }


http_settings should be defined explicitly in openbb_platform/core/openbb_core/app/model/python_settings.py as a dict, this becomes hidden if just mentioned here. That way we also don't need extra_allow=True in PythonSettings

I'll add these to documentation so they are not hidden.

extra_allow makes it extendable for other futures uses. Adding extra keys to the JSON file won't actually do anything because those keys would not be reached by any code.

I don't see any real benefit to not allowing extras here. Think, VS Code settings file. All the VS Code extensions define specific behaviors and preferences in the same settings file as the main program.

openbb_platform/core/openbb_core/provider/utils/helpers.py

montezdesousa · 2025-01-08T16:14:29Z

openbb_platform/core/openbb_core/provider/utils/helpers.py

+    if kwargs:
+        for key, value in kwargs.items():
+            try:
+                if hasattr(_session, key):
+                    if hasattr(getattr(_session, key, None), "update"):
+                        getattr(_session, key, {}).update(value)
+                    else:
+                        setattr(_session, key, value)
+            except AttributeError:
+                continue


I don't see this being used anywhere, what's the purpose of updating the keys of kwargs?

In the sync/async functions the kwargs are applied in the function, but if you were to use this as a standalone function to create a session object, you might want to set something not defined by python_settings.

This was the intention at least.

openbb_platform/core/openbb_core/provider/utils/helpers.py

montezdesousa · 2025-01-08T16:21:50Z

When trying to test I get the same error both with requests and make_request. Also tried to provide the path to the certificate through verify and get the same error. Uvicorn is running over HTTPS and I get access it through the browser fine.
SSLError: HTTPSConnectionPool(host='0.0.0.0', port=8000): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1133)')))
This error seems to indicate that the self-signed certificate is not being found at all. I know there's lots of ways to make it not work, so rooting out the edge cases would be nice.

What are your exact deployment steps here, is this a network deployment? Is the same .crt file accessible to both uvicorn and the Python environment? Something like:
    "python_settings": {
        "http": {
            "cafile": "/Users/darrenlee/OpenBBUserData/localhost.crt"
        },
        "uvicorn": {
            "ssl_certfile": "/Users/darrenlee/OpenBBUserData/localhost.crt",
            "ssl_keyfile": "/Users/darrenlee/OpenBBUserData/localhost.key"
        }
    }
Was the certificate created by the same version of Python and openssl used in the deployment and environment?

Thanks for the help. I'm using mkcert and it turns out the cafile needs to come from /Users<username>/Library/Application Support/mkcert/rootCA.pem and not the self signed certificate itself.

add SSL certificate context and setting

6634dd0

deeleeramone added enhancement Enhancement platform OpenBB Platform v4 PRs for v4 labels Nov 29, 2024

codespell

ea65641

deeleeramone added the do not merge Label to prevent pull request merge label Nov 29, 2024

deeleeramone added 3 commits November 29, 2024 15:40

linting

1a0a679

cleanup section

1d36365

allow defining uvicorn.run kwargs in system_settings

f6159ab

deeleeramone removed the do not merge Label to prevent pull request merge label Dec 1, 2024

Merge branch 'develop' into feature/ssl-context

44da668

deeleeramone mentioned this pull request Dec 1, 2024

[BUG][FR] Impossible to specify SSL certification #6939

Open

Merge branch 'develop' into feature/ssl-context

a38770a

deeleeramone requested review from piiq, IgorWounds, jmaslek, andrewkenreich, DidierRLopes, hjoaquim and montezdesousa December 3, 2024 23:06

montezdesousa reviewed Dec 4, 2024

View reviewed changes

deeleeramone added 4 commits December 4, 2024 17:17

refactor to return session objects

dec94bc

hub service rename it as _request_session

46362fa

typing

43b753a

more linting

1754813

deeleeramone added 4 commits December 4, 2024 18:16

pylint

2297169

missing __init__ files

cc46123

fix test_hub_service

1a9e2fe

missing init files..?

8dad0f9

piiq and others added 10 commits December 5, 2024 21:13

Update lock files

714df64

undo another test check thing

be3446a

Merge remote-tracking branch 'upstream/feature/ssl-context' into feat…

2a0b354

…ure/ssl-context

Revert "Update lock files"

cfa1b59

This reverts commit 714df64.

try with just the platform lock

60f6768

unexpected argument

c77af38

do a dummy check to return the session object if it is passed in the …

cb43eed

…kwargs

also dummy check for TCPConnector instance

91e5c64

bit more cleanup

c690661

no implementation of server_hostname

228518a

deeleeramone requested a review from tehcoderer December 6, 2024 16:20

deeleeramone added 7 commits December 6, 2024 18:24

Merge branch 'develop' into feature/ssl-context

50d737b

Move setting session to PosthogHandler init

808d93b

Merge branch 'develop' into feature/ssl-context

4f9c026

Merge branch 'develop' into feature/ssl-context

5a65612

Merge branch 'develop' into feature/ssl-context

1d50c89

Merge branch 'develop' into feature/ssl-context

a8da6dd

Merge branch 'develop' into feature/ssl-context

54c1deb

montezdesousa reviewed Jan 6, 2025

View reviewed changes

openbb_platform/core/openbb_core/api/rest_api.py Show resolved Hide resolved

deeleeramone added 2 commits January 7, 2025 22:37

Merge branch 'develop' into feature/ssl-context

d548cb7

Merge branch 'develop' into feature/ssl-context

5edc443

montezdesousa self-requested a review January 8, 2025 15:04

deeleeramone added 3 commits January 8, 2025 08:04

Merge branch 'develop' of https://github.com/OpenBB-finance/OpenBB in…

0597722

…to feature/ssl-context

add comments

86ce65f

Merge branch 'feature/ssl-context' of https://github.com/OpenBB-finan…

1a693a4

…ce/OpenBB into feature/ssl-context

montezdesousa reviewed Jan 8, 2025

View reviewed changes

review items

01ab7bb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add SSL Certificate Context And Setting For Async/Sync HTTP Requests #6976

[Feature] Add SSL Certificate Context And Setting For Async/Sync HTTP Requests #6976

deeleeramone commented Nov 29, 2024 •

edited

Loading

montezdesousa left a comment

deeleeramone commented Dec 5, 2024

montezdesousa commented Jan 7, 2025

deeleeramone commented Jan 7, 2025 •

edited

Loading

montezdesousa left a comment •

edited

Loading

montezdesousa Jan 8, 2025 •

edited

Loading

deeleeramone Jan 8, 2025

montezdesousa Jan 8, 2025

deeleeramone Jan 8, 2025

montezdesousa commented Jan 8, 2025

[Feature] Add SSL Certificate Context And Setting For Async/Sync HTTP Requests #6976

Are you sure you want to change the base?

[Feature] Add SSL Certificate Context And Setting For Async/Sync HTTP Requests #6976

Conversation

deeleeramone commented Nov 29, 2024 • edited Loading

montezdesousa left a comment

Choose a reason for hiding this comment

deeleeramone commented Dec 5, 2024

montezdesousa commented Jan 7, 2025

deeleeramone commented Jan 7, 2025 • edited Loading

montezdesousa left a comment • edited Loading

Choose a reason for hiding this comment

montezdesousa Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

deeleeramone Jan 8, 2025

Choose a reason for hiding this comment

montezdesousa Jan 8, 2025

Choose a reason for hiding this comment

deeleeramone Jan 8, 2025

Choose a reason for hiding this comment

montezdesousa commented Jan 8, 2025

deeleeramone commented Nov 29, 2024 •

edited

Loading

deeleeramone commented Jan 7, 2025 •

edited

Loading

montezdesousa left a comment •

edited

Loading

montezdesousa Jan 8, 2025 •

edited

Loading