Remove distracting DataDog errors on bibdata-staging

[maxkadel]

When working on the servers (both staging and production), running rake tasks or in the rails console, DataDog raises a long error and info. It does not appear to have an impact on the code you're running, but it makes debugging more difficult, because you can't scroll through the history easily, and it makes actual errors much less visible.

Expected behavior

When you ssh onto a server, you can:

1. go onto a rails console using bundle exec rails c and then work within the console without raising a DataDog error
2. run a rake task without raising a DataDog error

Actual behavior

Outputs a very very long error message, which can mask if an actual error occurs

Loading production environment (Rails 6.1.7)
irb(main):001:0> u = User.find_by(uid: 'mk8066')
I, [2023-02-28T17:43:40.096149 #29782]  INFO -- ddtrace: [ddtrace] DATADOG TRACER CONFIGURATION - {"date":"2023-02-28T17:43:40-05:00","os_name":"x86_64-pc-linux-gnu","version":"0.54.2","lang":"ruby","lang_version":"3.1.0","enabled":true,"service":"rails","agent_url":"http://127.0.0.1:8126?timeout=30","debug":false,"analytics_enabled":false,"runtime_metrics_enabled":false,"integrations_loaded":"[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]","vm":"ruby-3.1.0","partial_flushing_enabled":false,"priority_sampling_enabled":false,"health_metrics_enabled":false,"integration_rails_analytics_enabled":"","integration_rails_analytics_sample_rate":"1.0","integration_rails_enabled":"true","integration_rails_service_name":"marc_liberation","integration_rails_cache_service":"marc_liberation-cache","integration_rails_controller_service":"marc_liberation","integration_rails_database_service":"marc_liberation-postgres","integration_rails_distributed_tracing":"true","integration_rails_exception_controller":"","integration_rails_job_service":"marc_liberation-active_job","integration_rails_middleware":"true","integration_rails_middleware_names":"false","integration_rails_template_base_path":"views/","integration_rails_log_injection":"false","integration_redis_analytics_enabled":"false","integration_redis_analytics_sample_rate":"1.0","integration_redis_enabled":"true","integration_redis_service_name":"redis","integration_redis_command_args":"true","integration_http_analytics_enabled":"false","integration_http_analytics_sample_rate":"1.0","integration_http_enabled":"true","integration_http_service_name":"net/http","integration_http_distributed_tracing":"true","integration_http_split_by_domain":"false","integration_sidekiq_analytics_enabled":"false","integration_sidekiq_analytics_sample_rate":"1.0","integration_sidekiq_enabled":"true","integration_sidekiq_service_name":"sidekiq","integration_sidekiq_tag_args":"false","integration_sidekiq_client_service_name":"sidekiq-client","integration_sidekiq_error_handler":"#\u003cProc:0x00007f1756d7b0d0 /opt/bibdata/shared/bundle/ruby/3.1.0/gems/ddtrace-0.54.2/lib/ddtrace/tracer.rb:33\u003e","integration_faraday_analytics_enabled":"false","integration_faraday_analytics_sample_rate":"1.0","integration_faraday_enabled":"true","integration_faraday_service_name":"faraday","integration_faraday_distributed_tracing":"true","integration_faraday_error_handler":"#\u003cProc:0x00007f17556ef7d0 /opt/bibdata/shared/bundle/ruby/3.1.0/gems/ddtrace-0.54.2/lib/ddtrace/contrib/faraday/configuration/settings.rb:12 (lambda)\u003e","integration_faraday_split_by_domain":"false","integration_rack_analytics_enabled":"","integration_rack_analytics_sample_rate":"1.0","integration_rack_enabled":"true","integration_rack_service_name":"marc_liberation","integration_rack_application":"#\u003cMarcLiberation::Application:0x00007f174febbac8\u003e","integration_rack_distributed_tracing":"true","integration_rack_headers":"{:response=\u003e[\"Content-Type\", \"X-Request-ID\"]}","integration_rack_middleware_names":"false","integration_rack_quantize":"{}","integration_rack_request_queuing":"false","integration_rack_web_service_name":"web-server","integration_action_cable_analytics_enabled":"false","integration_action_cable_analytics_sample_rate":"1.0","integration_action_cable_enabled":"true","integration_action_cable_service_name":"marc_liberation-action_cable","integration_action_mailer_analytics_enabled":"false","integration_action_mailer_analytics_sample_rate":"1.0","integration_action_mailer_enabled":"true","integration_action_mailer_service_name":"marc_liberation-action_mailer","integration_action_mailer_email_data":"false","integration_active_support_analytics_enabled":"false","integration_active_support_analytics_sample_rate":"1.0","integration_active_support_enabled":"true","integration_active_support_service_name":"","integration_active_support_cache_service":"marc_liberation-cache","integration_action_pack_analytics_enabled":"","integration_action_pack_analytics_sample_rate":"1.0","integration_action_pack_enabled":"true","integration_action_pack_service_name":"marc_liberation","integration_action_pack_controller_service":"marc_liberation","integration_action_pack_exception_controller":"","integration_action_view_analytics_enabled":"false","integration_action_view_analytics_sample_rate":"1.0","integration_action_view_enabled":"true","integration_action_view_service_name":"marc_liberation","integration_action_view_template_base_path":"views/","integration_active_job_analytics_enabled":"false","integration_active_job_analytics_sample_rate":"1.0","integration_active_job_enabled":"true","integration_active_job_service_name":"marc_liberation-active_job","integration_active_job_error_handler":"#\u003cProc:0x00007f1756d7b0d0 /opt/bibdata/shared/bundle/ruby/3.1.0/gems/ddtrace-0.54.2/lib/ddtrace/tracer.rb:33\u003e","integration_active_job_log_injection":"false","integration_active_record_analytics_enabled":"false","integration_active_record_analytics_sample_rate":"1.0","integration_active_record_enabled":"true","integration_active_record_service_name":"marc_liberation-postgres","integration_active_record_orm_service_name":""}
=> #<User id: 24, email: "[email protected]", created_at: "2022-06-16 17:06:54.714818000 +0000", updated_at: "2023-01-24 18:38:15.481179000 +0000", provid...
irb(main):002:0> exit

Steps to replicate

1. ssh onto the staging server ssh deploy@bibdata-alma-worker-staging1
2. go into the active app cd /opt/bibdata/current
3. go into a rails console bundle exec rails c
4. run some code, e.g. `u = User.find_by(uid: 'mk8066')
5. Note long error that is not related to the User.find_by task

Impact of this bug

This makes seeing and debugging issues while running rake tasks difficult. This has a particular impact on the re-indexing rake tasks, since they are run by humans on the servers, and knowing whether or not a true error has occurred is important.

Implementation notes

• See Tigerdata commit as an example of fixing this issue

[kevinreiss]

@maxkadel I am not seeing these datadog errors any longer. Can we close this?

[maxkadel]

@kevinreiss - this ticket didn't have enough replication info - it's still happening if I try to run anything in the console. I'll update the ticket.

[maxkadel]

Closed by pulibrary/princeton_ansible#4862 and #1921