Skip to content

Commit

Permalink
AB test - replace some content on UTR page
Browse files Browse the repository at this point in the history
HMRC would like to see whether a link to a video performs better than
some descriptive help text. We've agreed to run an AB test to compare
these variants.

There's no elegant way to AB test bits of content in GOV.UK, so we've
had to resort to the fragile and ugly approach of find / replacing the
content.

As per the comments in the code, this will break if the content in the
content item changes, and we'll have to update the hardcoded replacement
if the content in the B variant needs to change.

Nevertheless, we think this is a pragmatic thing to do as a one off, as
it keeps an immportant stakeholder happy, helps us learn about video
content (which is a strategic priority), and helps us learn about the
use case for content AB tests in mainstream content.
  • Loading branch information
richardTowers committed Oct 6, 2023
1 parent aea97f6 commit 10139fe
Show file tree
Hide file tree
Showing 2 changed files with 44 additions and 0 deletions.
29 changes: 29 additions & 0 deletions app/controllers/content_items_controller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,35 @@ class ContentItemsController < ApplicationController
def show
load_content_item

# TEMPORARY (author: richard.towers, expected end date: November 30 2023)
# Content specific AB test for the Find your UTR number page
if @content_item.base_path == "/find-utr-number"
ab_test = GovukAbTesting::AbTest.new(
"find_utr_number_video_links",
dimension: 300, # TODO: which dimension should we use?
allowed_variants: %w[HelpText VideoLink],
control_variant: "HelpText",
)
@requested_variant = ab_test.requested_variant(request.headers)
@requested_variant.configure_response(response)

if @requested_variant.variant? "VideoLink"
# NOTE: this is a fragile way of doing an AB test on content.
#
# Any change to the base content, or to the way the content is rendered
# could potentially break the B variant of the test, and result in both
# variants being the same.
# We're aware of this risk, and we're going to be careful in this one off
# situation. This is not a sustainable way of doing AB tests in the
# future.
@content_item.body.sub!(
/<li>\s*in\ the\s+<a\ href="[^"]*"><abbr\ title="[^"]+">HMRC<\/abbr>\s+app<\/a>/,
'<li>in the <a href="https://www.gov.uk/guidance/download-the-hmrc-app"><abbr title="HM Revenue and Customs">HMRC</abbr> app</a> - watch a <a href="https://www.youtube.com/watch?v=LXw9ily9rTo">video about finding your UTR number in the app</a>',
)
end
end
# /TEMPORARY

set_expiry

if is_service_manual?
Expand Down
15 changes: 15 additions & 0 deletions test/controllers/content_items_controller_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -364,6 +364,21 @@ class ContentItemsControllerTest < ActionController::TestCase
assert_equal "true", @response.headers[Slimmer::Headers::REMOVE_SEARCH_HEADER]
end

test "AB test replaces content on the find-utr-number page" do
content_item = content_store_has_schema_example("answer", "answer")
content_item["base_path"] = "/find-utr-number"
content_item["details"]["body"] = '<li>in the <a href=""><abbr title="HM Revenue and Customs">HMRC</abbr> app</a>'
content_item["locale"] = "en"

stub_content_store_has_item(content_item["base_path"], content_item)

request.headers["HTTP_GOVUK_ABTEST_FIND_UTR_NUMBER_VIDEO_LINKS"] = "VideoLink"

get :show, params: { path: path_for(content_item) }
assert_response :success
assert_match 'watch a <a href="https://www.youtube.com/watch?v=LXw9ily9rTo">video about finding your UTR number in the app</a>', response.body
end

def path_for(content_item, locale = nil)
base_path = content_item["base_path"].sub(/^\//, "")
base_path.gsub!(/\.#{locale}$/, "") if locale
Expand Down

0 comments on commit 10139fe

Please sign in to comment.