Custom instrumentation for rake tasks?

[mkilling]

We do have quite a few (long-running) rake tasks in our Rails application. Runtimes vary from a few minutes up to 12 hours. For several reasons, it’s not trivial to convert these to Resque jobs. Still, we need proper instrumentation with Scout for these tasks, as right now our picture of the database queries in Scout is incomplete, and these rake tasks cause quite a bit of database load.

I looked at the Sneakers example (https://gist.github.com/itsderek23/685c7485a3bd020b6cdd9b1d61cb847f) and tried to roll my own instrumentation for rake tasks. However it seems that I’m missing something important, as the traces don’t show up on our Scout dashboard.

My code gets executed and the logs show up, but it’s almost as if the traces never get posted to Scout?

# Add your own tasks in files placed in lib/tasks ending in .rake,
# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.

require 'rake/hooks'
require File.expand_path('../config/application', __FILE__)
require 'resque/tasks'
require 'resque/scheduler/tasks'

Ptcshell::Application.load_tasks

module InstrumentWithScoutApm
  def execute(args = nil)
    # Make sure that only application-specific tasks get instrumented.
    # We can identify them but checking whether they have the `environment` task as a prerequisite
    return super(args) unless all_prerequisite_tasks.map(&:name).include?('environment')
    # Don’t instrument rake tasks, as Scout already does this for us
    return super(args) if name.start_with?('resque:')

    # ensure that ScoutAPM is running
    ScoutApm::Agent.instance.logger.debug "Instrumenting rake #{name}"
    unless ScoutApm::Agent.instance.background_worker_running?
      ScoutApm::Agent.instance.logger.debug "Starting Scout Agent..."
      ScoutApm::Agent.instance.start(skip_app_server_check: true)
      ScoutApm::Agent.instance.start_background_worker
    end

    # start Queue and Job Layers
    req = ScoutApm::RequestManager.lookup
    req.start_layer(ScoutApm::Layer.new('Queue', 'rake'))
    req.start_layer(ScoutApm::Layer.new('Job', name))

    # run the rake task and make sure we properly stop the layers on error
    begin
      return super(args)
    rescue
      req.error!
      raise
    ensure
      ScoutApm::Agent.instance.logger.debug "Stopping Job layer..."
      req.stop_layer
      ScoutApm::Agent.instance.logger.debug "Stopping Queue layer..."
      req.stop_layer
    end
  end
end

class Rake::Task
  prepend InstrumentWithScoutApm
end

[mkilling]

From how I read the code of this gem, only metrics recorded more than 2 minutes ago will be reported to the backend. In my development environment, it seems I could trigger the logs indicating that my metrics were properly submitted by adding a sleep(180) to the end of rake tasks.

I haven’t yet found a way to force reporting of all pending metrics. Is there a mechanism for that?

[seth-macpherson]

@mkilling Did you ever get this working? I'd really like to see something like this incorporated into Scout.

[mkilling]

@seth-macpherson we actually ended up switching to NewRelic, which supports this out of the box and is actually cheaper than Scout for our workload.

[seth-macpherson]

Thanks for the update @mkilling! 👍