Campaign testing approach

[joshuap]

According to the Rails testing guide:

The goals of testing your mailer classes are to ensure that:

• emails are being processed (created and sent)
• the email content is correct (subject, sender, body, etc)
• the right emails are being sent at the right times

12.1.1 From All Sides

There are two aspects of testing your mailer, the unit tests and the functional tests. In the unit tests, you run the mailer in isolation with tightly controlled inputs and compare the output to a known value (a fixture.) In the functional tests you don't so much test the minute details produced by the mailer; instead, we test that our controllers and models are using the mailer in the right way. You test to prove that the right email was sent at the right time.

There likewise could be two aspects of testing Heya campaigns: unit and functional.

Unit Testing

I think I have a good approach for unit testing emails, using a similar API to Rails (this is currently working):

RSpec.describe SignupFirstRunCampaign, type: :campaign do
  describe "welcome" do
    it do
      user = FactoryBot.create(:user)
      email = SignupFirstRunCampaign.welcome(user)

      assert_emails 1 do
        email.deliver_now
      end

      assert_equal ["[email protected]"], email.from
      assert_equal [user.email], email.to
      assert_equal "Welcome to Honeybadger! 👋", email.subject
      assert_ican email
    end
  end
end

Functional Testing

Unlike ActionMailer, Hey campaign emails are sent from Heya's scheduler rather than arbitrary places like Rails controllers. It would be nice to be able to test that the correct emails are sent at the correct times, similar to how we do in the internal scheduler tests. Something along these lines (this is pseudo-code):

    test "it sends campaign emails on time" do
      user = users(:first)
      OnboardingCampaign.add(user)

      assert_step OnboardingCampaign, :step_one, user

      refute_step OnboardingCampaign, :step_two, user

      TimeCop.travel(1.day.from_now)

      assert_step OnboardingCampaign, :step_two, user
    end

I'm not actually sure if this falls under functional testing or unit testing, since it's not really testing the integration between Heya and application components. In any case, I'm looking for feedback on it--would something like this be a good addition?

[chalmagean]

Hi @joshuap

I've been thinking about testable workflows/funnels for a while now. While I don't have a solution in mind yet, it would be nice to have a way to make sure that our marketing funnels work as expected and ideally have a high-level overview of what the funnel looks like (how the different campaigns work together).

Here's some pseudo-code:

context "when the user first subscribes to our newsletter" do
  it "receives the onboarding campaign"

  context "when he clicks one of the three segmentation links" do
	context "when the user clicks the 'beginner' link" do
	  it "receives campaign_A"
	  it "receives campaign_B"
	end

	context "when the user clicks the 'intermediate' link" do
	  it "receives campaign_C"
	  it "receives campaign_D"
	end

	context "when the user clicks the 'advanced' link" do
	  it "receives campaign_E"
	  it "receives campaign_F"
	end
    end
  end
end

[joshuap]

Hi @joshuap

I've been thinking about testable workflows/funnels for a while now. While I don't have a solution in mind yet, it would be nice to have a way to make sure that our marketing funnels work as expected and ideally have a high-level overview of what the funnel looks like (how the different campaigns work together).

Here's some pseudo-code:

context "when the user first subscribes to our newsletter" do
  it "receives the onboarding campaign"

  context "when he clicks one of the three segmentation links" do
	context "when the user clicks the 'beginner' link" do
	  it "receives campaign_A"
	  it "receives campaign_B"
	end

	context "when the user clicks the 'intermediate' link" do
	  it "receives campaign_C"
	  it "receives campaign_D"
	end

	context "when the user clicks the 'advanced' link" do
	  it "receives campaign_E"
	  it "receives campaign_F"
	end
    end
  end
end

Nice! I think these would be fairly simple to implement since you could basically test that the user is in the campaign, right? Something like:

expect(
  Heya::CampaignMembership.where(
    user: user,
    campaign_gid: OnboardingCampaign.gid
  ).exists?
).to eq(true)

[chalmagean]

Well, sure, but I would do more of an integration test, like ensuring that the user receives the email it's supposed to receive.

Now what I think it's most valuable here is to have a high-level overview of what your funnel looks like. So you can identify holes in it. I haven't tried it yet, but that's what I hope my tests can reveal.