Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting an exception while capturing an existing #3508

Closed
jainankit opened this issue Sep 7, 2024 · 3 comments · Fixed by #3511
Closed

Getting an exception while capturing an existing #3508

jainankit opened this issue Sep 7, 2024 · 3 comments · Fixed by #3511

Comments

@jainankit
Copy link

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.13.0

Steps to Reproduce

We are using sentry-sdk 2.13.0 for python. We also alongside use "structlog-sentry ~= 2.1.0". We are seeing a large number of Sentry errors while capturing the exception stack-trace. It seems like there's an issue with capturing breadcrumbs, in what is being captured in timestamp:

...
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 106, in __call__
    response = self.app(
               ^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 86, in <lambda>
    lambda *a, **kw: old_app(self, *a, **kw),
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1498, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/opentelemetry/instrumentation/flask/__init__.py", line 390, in _wrapped_app
    result = wsgi_app(wrapped_app_environ, _start_response)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__
    return self.app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1476, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 803, in handle_exception
    got_request_exception.send(self, _async_wrapper=self.ensure_sync, exception=e)
  File "/usr/local/lib/python3.12/site-packages/blinker/base.py", line 279, in send
    result = receiver(sender, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/utils.py", line 1730, in runner
    return sentry_patched_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 207, in _capture_exception
    sentry_sdk.capture_event(event, hint=hint)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/api.py", line 162, in capture_event
    return get_current_scope().capture_event(event, hint, scope=scope, **scope_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1139, in capture_event
    event_id = self.get_client().capture_event(event=event, hint=hint, scope=scope)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 745, in capture_event
    event_opt = self._prepare_event(event, hint, scope)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 453, in _prepare_event
    event_ = scope.apply_to_event(event, hint, self.options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 149, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1452, in apply_to_event
    self._apply_breadcrumbs_to_event(event, hint, options)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1309, in _apply_breadcrumbs_to_event
    event["breadcrumbs"]["values"].sort(key=lambda crumb: crumb["timestamp"])
TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 134, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.12/site-packages/gunicorn/workers/sync.py", line 177, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/flask.py", line 89, in sentry_patched_wsgi_app
    return middleware(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 113, in __call__
    reraise(*_capture_exception())
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/integrations/wsgi.py", line 195, in _capture_exception
    sentry_sdk.capture_event(event, hint=hint)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/api.py", line 162, in capture_event
    return get_current_scope().capture_event(event, hint, scope=scope, **scope_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1139, in capture_event
    event_id = self.get_client().capture_event(event=event, hint=hint, scope=scope)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 745, in capture_event
    event_opt = self._prepare_event(event, hint, scope)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/client.py", line 453, in _prepare_event
    event_ = scope.apply_to_event(event, hint, self.options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 149, in wrapper
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1452, in apply_to_event
    self._apply_breadcrumbs_to_event(event, hint, options)
  File "/usr/local/lib/python3.12/site-packages/sentry_sdk/scope.py", line 1309, in _apply_breadcrumbs_to_event
    event["breadcrumbs"]["values"].sort(key=lambda crumb: crumb["timestamp"])
TypeError: '<' not supported between instances of 'str' and 'datetime.datetime'

Expected Result

No error is thrown while capturing the sentry exception

Actual Result

When this issue happens, the error is not captured in Sentry dashboard, so we are dropping events.

@getsantry getsantry bot moved this to Waiting for: Product Owner in GitHub Issues with 👀 3 Sep 7, 2024
@jonas-debeuk
Copy link

Also seeing this, same versions of sentry and sentry-struct-logs, and error 🙈

@szokeasaurusrex
Copy link
Member

@jainankit or @jonas-debeuk, could one of you please share a minimal reproduction? I am not very familiar with structlog or the structlog-sentry integration, since it is not an official Sentry integration.

@sentrivana
Copy link
Contributor

Hey folks, the breadcrumb sorting code that the error originates from was added recently. If you revert to SDK version 2.10.0, you shouldn't see this anymore.

FWIW the structlog-sentry issue is here: kiwicom/structlog-sentry#159 But we should patch this in the SDK as well. I'll look into it.

arthurdarcet added a commit to arthurdarcet/structlog-sentry that referenced this issue Oct 22, 2024
- get_isolation_scope needs 2.12 (not 2.0)
- this breadcrumb sorting issue/crash is fixed in 2.15 getsentry/sentry-python#3508 (comment)
arthurdarcet added a commit to arthurdarcet/structlog-sentry that referenced this issue Oct 22, 2024
- get_isolation_scope needs 2.12 (not 2.0)
- this breadcrumb sorting issue/crash is fixed in 2.15 getsentry/sentry-python#3508 (comment)
arthurdarcet added a commit to arthurdarcet/structlog-sentry that referenced this issue Oct 22, 2024
- get_isolation_scope needs 2.12 (not 2.0)
- this breadcrumb sorting issue/crash is fixed in 2.15 getsentry/sentry-python#3508 (comment)
arthurdarcet added a commit to arthurdarcet/structlog-sentry that referenced this issue Oct 24, 2024
- get_isolation_scope needs 2.12 (not 2.0)
- this breadcrumb sorting issue/crash is fixed in 2.15 getsentry/sentry-python#3508 (comment)
paveldedik pushed a commit to kiwicom/structlog-sentry that referenced this issue Nov 12, 2024
- get_isolation_scope needs 2.12 (not 2.0)
- this breadcrumb sorting issue/crash is fixed in 2.15 getsentry/sentry-python#3508 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants