Messaging: Weekly Digest

[joethreepwood]

Messaging

Blockers

• Need to confirm with @raquelmsmith that all the events are correctly set up
• The design (especially on the Hogstradamus section) could do with some @corywatilo love. May be a nice opportunity to move to the proposed new email designs? Feel strongly that this would be a big and impactful change for a lot of users.

Pitch

Continuing from the hackathon: @raquelmsmith's idea is to have a weekly digest of certain events sent to all users. We've set up the intrumentation for this, and the messaging is broadly done. I've also added this to the new product notifications subscription tag, so users can unsubscribe from this if they don't want it.

Here's the trigger:

The goal, broadly, is to encourage more feature usage and log ins. So, I'd propose adding it to a hold-out test where 50% of users would get the email and 50% won't. We'll then monitor as a success metric if users log in more and activate with more products.

What I was trying to do with the Hogstradamus section was mockup something like this, but it... is shit.

Here's a peek of what the code looks like for each section:

Last week your team made {% if event.new_dashboards_in_last_7_days %}{% if event.new_dashboards_in_last_7_days > 1 %}<a href="https://us.posthog.com/dashboard" target="_blank" style="color: #f54e00;">{{ event.new_dashboards_in_last_7_days }}</a> dashboards{% elsif event.new_dashboards_in_last_7_days == 1 %}<a href="https://us.posthog.com/dashboard" target="_blank" style="color: #f54e00;">1</a> dashboard{% elsif event.new_dashboards_in_last_7_days < 1 %}no new dashboards{% endif %}{% else %}<a href="https://us.posthog.com/dashboard" target="_blank" style="color: #f54e00;">some new dashboards</a>{% endif %} and {% if event.new_event_definitions_in_last_7_days %}{% if event.new_event_definitions_in_last_7_days > 1 %}<a href="https://us.posthog.com/data-management/events" target="_blank" style="color: #f54e00;">{{ event.new_event_definitions_in_last_7_days }} new events</a> {% elsif event.new_event_definitions_in_last_7_days == 1 %}<a href="https://us.posthog.com/data-management/events" target="_blank" style="color: #f54e00;">1 new event</a> {% elsif event.new_event_definitions_in_last_7_days < 1 %}no new events{% endif %}{% else %}<a href="https://us.posthog.com/data-management/events" target="_blank" style="color: #f54e00;">some new events</a>{% endif %}. Check out our templates for more ideas.

And here's how it looks with the fallback error copy:

[raquelmsmith]

@joethreepwood I think we should include more info about the items. Basically list out the experiments that were started, the flags that were made, etc. Something like:

• New experiments
  • Green vs Blue Button Color started on Nov 14
  • Move onboarding buttons started on Nov 16
  • (one more)
  • and 3 other experiments (link to experiments list)

The hogs are fairly overwhelming IMO. I would remove most of them.

@corywatilo can you help us with some design here?

[joethreepwood]

Yep, def agree on the hogs. TBH I had a much grander idea for the design originally (wanted it to look like an actual newspaper) but...hackathon 🤷

Let me look at modifying the liquid for the other stuff tomorrow.

[jamesefhawkins]

a bunch of customers will have poor usage - this will probably cause them to churn. do you think we should consider this scenario?

[joethreepwood]

Raquel and I discussed this briefly at the offsite, as I agree it's a risk. One idea I've since had is basically setting up a filter on the campaign that means it only triggers if users have usage above a certain amount. This would mean we'd probably want to change this to a monthly digest, however.

We could then set it so that it only sends if a user has created more than 3 dashboards and either 1 playlist, experiment, or warehouse source (for example).

I also think a monthly cadence would be less annoying for users here.

[joethreepwood]

@corywatilo can you help us with some design here?

TBH, if @corywatilo has time to spend on emails I'd rather have that focused on the new email templates that work in dark mode. I'll want to move this to that template as soon as they're available anyway.

I've tinkered together some improvements in the mean time and am working on adding in the new liquid. See below.

a bunch of customers will have poor usage - this will probably cause them to churn. do you think we should consider this scenario?

I agree with this btw, @raquelmsmith -- I've changed the copy to be monthly, which I think will work better. Easy to change to something else if you want though.

[joethreepwood]

ok, new code is all in place

[raquelmsmith]

@joethreepwood I'd like this to be weekly because I'm planning on adding a lot more to it.

We should just not show a section if they aren't using it or there is no content for it.

If there are fewer than 2 sections with content we can just not send it.

I think it should actually list out the data like I mentioned here.

Just saying "some new experiments were launched" isn't actionable at all. But saying "an experiment called change homepage background color was launched 3 days ago -> check results" is much more actionable.

[joethreepwood]

@joethreepwood I'd like this to be weekly because I'm planning on adding a lot more to it.

I do feel somewhat strongly this is too often, but for now I'll change the copy back to weekly and we can watch the unsubscribe rate.

Just saying "some new experiments were launched" isn't actionable at all. But saying "an experiment called change homepage background color was launched 3 days ago -> check results" is much more actionable.

Yeah, what you're seeing now is just the fall-back error code that I put in for testing. This only displays if the event is completely not present. This can't actually occur IRL because the event itself is the campaign trigger - so this is just for testing until we have your PR merged and real data sent.

The issue with formatting the data the way you cited is length. If someone launches a lot of experiments, has a lot of dashboards, etc then that bullet list you suggested will get very long and unwieldy - especially if they have complex names. Both of these feel like likely situations.

I'm deliberately omitting info about time at the moment -- it just gets too much to say A launched X days ago, B launched Y days ago, and C launched Z days ago. Also, D completed W days ago, E completed V days ago. etc. The important info is the state (launched/completed) and the name (which is descriptive), not the period.

The actual outputs from the current Liquid code (see bottom) compromise on this. If a user has multiple launched and completed experiments:

You launched 3 experiments last month, including Experiment A, Experiment B, Experiment C. You also completed 2 experiments, including Experiment X, Experiment Y. [Check the results here!](https://us.posthog.com/experiments)

If no experiments launch/complete, we omit that sentence:

You completed 1 experiment last month, called Experiment X. [Check the results here!](https://us.posthog.com/experiments) or You launched 1 experiment last month, called Experiment A. [Check the progress here!](https://us.posthog.com/experiments)

And if the data is present but has no values (e.g. an error), we say:

It looks like we couldn’t retrieve your experiments data. If you think this is an error, please [open a support ticket](http://app.posthog.com/home#supportModal) for assistance!

We should just not show a section if they aren't using it or there is no content for it.

Customer.io doesn't have a reliable way to remove the entire row if nothing launched or completed. We could not display text, but that'd leave the rest of the space, the image, and available heading. We can remove these elements to accomodate a text only solution, but I think that also becomes less actionable and is something we should iterate on once we have data.

Right now, if there's no data on the event then we display the below. I decided to link to the docs because they're more actively updated and therefore less repetitive than any guides we have -- but once this is out I can definitely iterate variations into this.

You didn’t launch or complete any new experiments last month. Maybe our [experiments guide](https://posthog.com/docs/experiments) can help!

If there are fewer than 2 sections with content we can just not send it.

I presume you mean not send the email, rather than the event? I'll have to give some thought to how I'd build that trigger.

---

Here's the code I wrote to power the above, if you want to check it. There's similar code for each section.

{% if event is not none %}
    {% assign launched_count = event.new_experiments_launched_in_last_7_days %}
    {% assign completed_count = event.new_experiments_completed_in_last_7_days %}
    {% if launched_count or completed_count %}
        {% if launched_count %}
            You launched {{ launched_count | default: 0 }} experiment{% if launched_count > 1 %}s{% endif %} last month
            {% if launched_count > 1 %}, including {% for experiment in launched_count %}
                {{ experiment.name }}{% if forloop.last == false %}, {% else %}.{% endif %}
            {% endfor %}
            {% else %}, called {{ launched_count.0.name }}.{% endif %}
        {% endif %}
        {% if completed_count %}
            {% if launched_count %} You also completed {% else %} You completed {% endif %}
            {{ completed_count | default: 0 }} experiment{% if completed_count > 1 %}s{% endif %}
            {% if completed_count > 1 %}, including {% for experiment in completed_count %}
                {{ experiment.name }}{% if forloop.last == false %}, {% else %}.{% endif %}
            {% endfor %}
            {% else %}, called {{ completed_count.0.name }}.{% endif %}
            <a href="https://us.posthog.com/experiments" target="_blank" style="color: #f54e00;">Check the results here!</a>
        {% else %}
            <a href="https://us.posthog.com/experiments" target="_blank" style="color: #f54e00;">Check the progress here!</a>
        {% endif %}
    {% else %}
        You didn’t launch or complete any new experiments last month. Maybe our <a href="https://posthog.com/docs/experiments" target="_blank" style="color: #f54e00;">experiments guide</a> can help!
    {% endif %}
{% else %}
    It looks like we couldn’t retrieve your experiments data. If you think this is an error, please <a href="http://app.posthog.com/home#supportModal" target="_blank" style="color: #f54e00;">open a support ticket</a> for assistance!
{% endif %}

[joethreepwood]

Changed back to weekly ✅

If there are fewer than 2 sections with content we can just not send it.

This is actually proving incredibly complex to do because of how I believe the event is currently setup.

Currently, this campaign is triggered by the transactional_email event. Customer.io allows us to filter down on that event to only trigger based on certain properties but only using an ALL condition. We therefore can't use this to filter for At Least N

I've looked into solutions using TRUE/FALSE branches in the workflow, but the most workable solution in Customer.io is a segment filter that checks every possible variation of 2 event attributes existing - e.g. new_dashboards_in_last_7_days + new_sources_in_last_7_days, new_dashboards_in_last_7_days + new_surveys_in_last_7_days.

That's doable, but it gives us 21 possible variations right now - and would be a hassle to scale if you plan to add more here.

So, before we fall back to that: is there a solution we could work into your PR where we ensure weekly_digest only sends for users with 2 or more sections, or that there is a single additional eligibility value I can check in the trigger conditions?

[raquelmsmith]

Yeah I can put a single var like number_of_product_with_updates that just sums them up? Can you filter on that, so that it just has to be greater than 2?

[joethreepwood]

Yep, that's doable!

[raquelmsmith]

The issue with formatting the data the way you cited is length. If someone launches a lot of experiments, has a lot of dashboards, etc then that bullet list you suggested will get very long and unwieldy - especially if they have complex names. Both of these feel like likely situations.

We should list the first 3 and then say "and 8 more" that just links to the relevant page (eg the experiments list)

I don't think this should have a lot of "english" in it. It should be a concise list that's easy to scan and click what you want. Paragraphs are not the best solution here imo.

If they are just list items, we can show the time. It's also easier to deal with formatting and sentence structure.

Customer.io doesn't have a reliable way to remove the entire row if nothing launched or completed. We could not display text, but that'd leave the rest of the space, the image, and available heading. We can remove these elements to accomodate a text only solution, but I think that also becomes less actionable and is something we should iterate on once we have data.

I really think we should figure out how to not show entire sections. Having sections that are unused makes this whole email difficult to read and not very actionable, and presents the churn problem that james mentioned. "huh my team isn't doing X, maybe we should stop paying for it"

[raquelmsmith]

Here's a very rough mockup of what I was thinking:

Cory said he can pick this up next week!

[raquelmsmith]

@joethreepwood I've updated the template in c.io to fix a bug with the dashboards section not showing up, and also reformat the experiments and flags sections (the flags have to use the key as the primary thing since a name/description isn't required.

I've merged a PR to exclude any generated dashboards, and will go make a fix for experiments that were launched and completed showing up twice (they should only show up in the completed list).

[raquelmsmith]

I've also edited this filter to only send if there are 2+ sections with items. Now that we don't show sections without items I don't know if it's necessary.. what do you think?

Regardless, customer.io says that no one has been seen with that prop, but it's in the events when I do the preview thing, and then it also says 53k people have recently performed it, so idk what its deal is.

[raquelmsmith]

@joethreepwood I also have this PR up now that will les users manage their notification settings in the app UI. This will let them turn it on/off for specific projects, instead of just disabling them wholesale (which is what the c.io implementation would have had them do).

[posthog-contributions-bot]

This issue has 2001 words at 16 comments. Issues this long are hard to read or contribute to, and tend to take very long to reach a conclusion. Instead, why not:

1. Write some code and submit a pull request! Code wins arguments
2. Have a sync meeting to reach a conclusion
3. Create a Request for Comments and submit a PR with it to the meta repo or product internal repo

Is this issue intended to be sprawling? Consider adding label epic or sprint to indicate this.

[raquelmsmith]

Damn, I think I added one too many words 😱

[andyvan-ph]

Fwiw, @joethreepwood / @raquelmsmith, I think these digest emails get very noisy if you're on multiple projects. I got four emails at the same time, including one that was empty.

[joethreepwood]

Fwiw, @joethreepwood / @raquelmsmith, I think these digest emails get very noisy if you're on multiple projects. I got four emails at the same time, including one that was empty.

Yeah, I agree and it's something we're discussing. I think we'd be better off doing these either monthly or fortnightly, still.
The empty issue is something we're already working on if you follow the thread.

I've also edited this filter to only send if there are 2+ sections with items. Now that we don't show sections without items I don't know if it's necessary.. what do you think?

I've modified this to be 1 instead of 2 since the filter variable is Greater than rather than Equal to or greater than, but I think this looks good.

@joethreepwood I also have this PR up now that will les users manage their notification settings in the app UI. This will let them turn it on/off for specific projects, instead of just disabling them wholesale (which is what the c.io implementation would have had them do).

Cool, and thanks. I've updated the Unsub option in the email to now direct to the settings and have relevant language. ✅

Regardless, customer.io says that no one has been seen with that prop, but it's in the events when I do the preview thing, and then it also says 53k people have recently performed it, so idk what its deal is.

No idea what the issue is here, tbh. I'm going to reach out to their support about it.

@joethreepwood I've updated the template in c.io to fix a bug with the dashboards section not showing up, and also reformat the experiments and flags sections (the flags have to use the key as the primary thing since a name/description isn't required.

FYI this unfortunately introduced a bug: undefined variable: event.new_experiments_completed, line:17, col:2160 - I've updated the code to correct this. If anyone else updates the code, please check the error review in C.io!

[joethreepwood]

Spoke with C.io regarding the trigger issue:

OK, so this I think is come confusion due to our UI. When we check for instances of the event being performed with the matching event data, we only check the last 50 instances of the event being performed (with the last 30 days). This is why it shows "We couldn't find any instances of your trigger event with this data within the last 30 days. Do you want to send one?" - it is only looking at the last 50 instances of the event being performed.

This also extends to the email previewer.

I've had them check it out themselves and they've confirmed that the campaign will trigger correctly, and I've given them feedback to update this element of their UI because last 30 days and last 50 events are not the same thing at all.

[raquelmsmith]

@andyvan-ph I agree and I'm trying to figure out what to do about the multiple projects thing.

I have a PR up that will let people turn these off for specific projects (or off entirely) (it's a user setting, so only impacts that one person). So they will be able to leave them on for only the projects they care about.

Most users don't have lots of projects. But even 3-4 active projects can be overwhelming to get that many emails. But shoving all the content into a single email could make the email super long. I'm not really sure what to do about that. Do you have any ideas?

Sending less often won't resolve this problem and would only make the combined email longer.

Another thing is that most users won't use all the products, so that will theoretically make the emails shorter. What we receive for the PostHog org projects will be the worst case scenario..

[joethreepwood]

Sending less often won't resolve this problem and would only make the combined email longer.

I think the issue is that sending more often compounds the problem. Getting 3-4 emails every week is worse than getting 1 email every week, but getting 1 email every week is still bad for most users. Especially if we auto-opt them in and they get it around other, existing emails.

I'm less concerned about email length on the project front, tbh. If the info is useful, length is fine. If it's not, we shouldn't be sending it. And the most frequent friction I see on this topic from users is them being frustrated at sends at all. If you do want this info then you won't care how much of it there is, and if conversely if you don't want it you won't care how little there is either -- so, again, it's sends that are the biggest issue for users.

I'm not really sure what to do about that. Do you have any ideas?

The three most obvious suggestions to me are:

1. We apply a message limit to only send for one project per week, dropping all others. We add copy to clarify.
2. We make digest emails so that you have to opt-in, giving users a prompt about it when they invite their first new team member to a project (since they'll already be seeing everything done before this point) and giving new users who join an org to enable it for specific projects.
3. We apply logic to the campaign so it only sends for the most popular project that week, dropping others. We add copy to clarify.

Of these, 1 and 3 would either be tricky or would be very difficult. I think 2 is a good solution personally, especially as it waits until the point of impact before offering it to users. The hard part there is it's more overhead for users.