Add "day zero" experiment feature & metrics #22195

DJAndries · 2024-02-21T01:46:37Z

Resolves brave/brave-browser#36243

Submitter Checklist:

I confirm that no security/privacy review is needed and no other type of reviews are needed, or that I have requested them
There is a ticket for my issue
Used Github auto-closing keywords in the PR description above
Wrote a good PR/commit description
Squashed any review feedback or "fixup" commits before merge, so that history is a record of what happened in the repo, not your PR
Added appropriate labels (QA/Yes or QA/No; release-notes/include or release-notes/exclude; OS/...) to the associated issue
Checked the PR locally:
- npm run test -- brave_browser_tests, npm run test -- brave_unit_tests wiki
- npm run presubmit wiki, npm run gn_check, npm run tslint
Ran git rebase master (if needed)

Reviewer Checklist:

A security review is not needed, or a link to one is included in the PR description
New files have MPL-2.0 license header
Adequate test coverage exists to prevent regressions
Major classes, functions and non-trivial code blocks are well-commented
Changes in component dependencies are properly reflected in gn
Code follows the style guide
Test plan is specified in PR before merging

After-merge Checklist:

The associated issue milestone is set to the smallest version that the
changes has landed on
All relevant documentation has been updated, for instance:

Test Plan:

Only test for Windows and Android.

Start with fresh profile, ensure metrics do not exist.
Start again with fresh profile using the switch --enable-features="BraveDayZeroExperiment:variant/a". Ensure Brave.DayZero.A.InstallTime has value of 0.
Close browser, advance system time by 7 days, start browser, ensure metric has value of 7.
Close browser, advance system time by 14 days, start browser with --enable-features="BraveDayZeroExperiment:variant/b", ensure metric has value of 21, and Brave.DayZero.B.InstallTime does NOT exist
Close browser advance system time by 14 days, start browser, ensure that the metric still has a value of 21 (to ensure that the metric was not updated internally).
Start with fresh profile with --enable-features="BraveDayZeroExperiment:variant/b" switch
Ensure Brave.DayZero.B.InstallTime has a value of 0.
Close browser, advance system time by 7 days, start browser, ensure metric has value of 7.
Close browser, advance system time by 14 days, start browser without --enable-features="BraveDayZeroExperiment:variant/b", ensure metric has value of 21, and Brave.DayZero.A.InstallTime does NOT exist
Close browser advance system time by 14 days, start browser, ensure that the metric still has a value of 21 (to ensure that the metric was not updated internally).

Ensure the metrics only exist in the logs_constellation_prep_express pref.

DJAndries · 2024-02-21T01:47:13Z

Will change base to master once #21583 lands.

chrismore · 2024-02-21T15:33:55Z

android/java/org/chromium/base/BraveFeatureList.java

@@ -31,6 +31,6 @@ public abstract class BraveFeatureList {
    public static final String AI_CHAT = "AIChat";
    public static final String BRAVE_SHOW_STRICT_FINGERPRINTING_MODE =
            "BraveShowStrictFingerprintingMode";
-    public static final String BRAVE_ZERO_DAY_FLAG_ANDROID = "BraveZeroDayFlagAndroid";
+    public static final String BRAVE_DAY_ZERO_EXPERIMENT = "BraveDayZeroExperiment";


@DJAndries the BRAVE_DAY_ZERO_EXPERIMENTvariable is the same variable cross platforms and as a replacement for the previous Android-specific one?

yes, correct

chrismore · 2024-02-21T15:37:25Z

components/misc_metrics/general_browser_usage.cc

+#if BUILDFLAG(IS_ANDROID) || BUILDFLAG(IS_WIN)
+void GeneralBrowserUsage::ReportInstallTime() {
+  int days_since_install = (base::Time::Now() - first_run_time_).InDays();
+  if (days_since_install < 0 || days_since_install > 30) {


@DJAndries this condition is to just keep the histogram putting users into 0-30 day buckets and anyone outside of this range, gets returned to the previous function, right? e.g. negative or >=31 will not be included.

yes, that's right. users outside of the 0-30 day range will not report either metric.

chrismore · 2024-02-21T20:31:17Z

@DJAndries If and when we want to bring this idea to Mac, will any of this approach work on Mac?

DJAndries · 2024-02-21T20:39:01Z

@DJAndries If and when we want to bring this idea to Mac, will any of this approach work on Mac?

Yes, once we add support for loading variations upon first run for Mac, it will work. We'll just need to make a couple tweaks.

mkarolin

chromium_src++

kjozwiak · 2024-04-11T04:57:18Z

Verification PASSED on Win 11 x64 using the following build(s):

Brave | 1.67.1 Chromium: 124.0.6367.29 (Official Build) nightly (64-bit)
-- | --
Revision | f4edcfc3cafa0d8609db03da09eb8582d83d1a27
OS | Windows 11 Version 23H2 (Build 22631.3374)

`Test Case #1` - `--enable-features=BraveDayZeroExperiment:variant/a`

launched 1.67.1 Chromium: 124.0.6367.29 using --enable-features=BraveDayZeroExperiment:variant/a
re-launched once again using --enable-features=BraveDayZeroExperiment:variant/a
ensured that StudyBraveDayZeroExperiment:GroupBraveDayZeroExperiment via brave://version

ensured that Brave.DayZero.A.InstallTime was 0 via brave://local-state

            "Brave.DayZero.A.InstallTime": {
               "sent": false,
               "value": "0"

closed the browser and moved the machines time/clock forwards by 7 days
re-launched once again using --enable-features=BraveDayZeroExperiment:variant/a
ensured that Brave.DayZero.A.InstallTime was set as 7 via brave://local-state

            "Brave.DayZero.A.InstallTime": {
               "sent": false,
               "value": "7"
            },

closed the browser and moved the machines time/clock forwards by 14 days
re-launched once again using --enable-features=BraveDayZeroExperiment:variant/b
ensured that Brave.DayZero.A.InstallTime was set as 21 via brave://local-state
ensured that Brave.DayZero.B.InstallTim doesn't exist under brave://local-state

            "Brave.DayZero.A.InstallTime": {
               "sent": false,
               "value": "21"
            },

`Example`	`Example`

closed the browser and moved the machines time/clock forwards by 14 days
re-launched once again using --enable-features=BraveDayZeroExperiment:variant/b
ensured that Brave.DayZero.A.InstallTime was set as 21 via brave://local-state
ensured that Brave.DayZero.B.InstallTim doesn't exist under brave://local-state

            "Brave.DayZero.A.InstallTime": {
               "sent": false,
               "value": "21"
            },

`Test Case #2` - `--enable-features=BraveDayZeroExperiment:variant/b`

launched 1.67.1 Chromium: 124.0.6367.29 using --enable-features=BraveDayZeroExperiment:variant/b
re-launched once again using --enable-features=BraveDayZeroExperiment:variant/b
ensured that StudyBraveDayZeroExperiment:GroupBraveDayZeroExperiment via brave://version

ensured that Brave.DayZero.B.InstallTime was 0 via brave://local-state

            "Brave.DayZero.B.InstallTime": {
               "sent": false,
               "value": "0"
            },

closed the browser and moved the machines time/clock forwards by 7 days
re-launched once again using --enable-features=BraveDayZeroExperiment:variant/b
ensured that Brave.DayZero.B.InstallTime was set as 7 via brave://local-state

            "Brave.DayZero.B.InstallTime": {
               "sent": false,
               "value": "7"
            },

closed the browser and moved the machines time/clock forwards by 14 days
re-launched once again using --enable-features=BraveDayZeroExperiment:variant/a
ensured that Brave.DayZero.B.InstallTime was set as 21 via brave://local-state
ensured that Brave.DayZero.A.InstallTim doesn't exist under brave://local-state

            "Brave.DayZero.B.InstallTime": {
               "sent": false,
               "value": "21"
            },

`Example`	`Example`

closed the browser and moved the machines time/clock forwards by 14 days
re-launched once again using --enable-features=BraveDayZeroExperiment:variant/a
ensured that Brave.DayZero.B.InstallTime was set as 21 via brave://local-state
ensured that Brave.DayZero.A.InstallTim doesn't exist under brave://local-state

            "Brave.DayZero.B.InstallTime": {
               "sent": false,
               "value": "21"
            },

Uplift of #22195 and #22892 (squashed) to beta

DJAndries requested review from a team February 21, 2024 01:46

github-actions bot assigned DJAndries Feb 21, 2024

github-actions bot added the CI/run-upstream-tests Run upstream unit and browser tests on Linux and Windows (otherwise only on Linux) label Feb 21, 2024

chrismore reviewed Feb 21, 2024

View reviewed changes

mkarolin approved these changes Feb 21, 2024

View reviewed changes

DJAndries force-pushed the p3a-refcode branch 2 times, most recently from b0ff7c6 to b5cc650 Compare February 26, 2024 08:09

Base automatically changed from p3a-refcode to master February 26, 2024 22:36

DJAndries force-pushed the day-zero-p3a branch from f80c790 to 7d564d6 Compare February 27, 2024 00:06

DJAndries force-pushed the day-zero-p3a branch from 7d564d6 to 147048d Compare March 18, 2024 21:21

iefremov approved these changes Mar 20, 2024

View reviewed changes

Add "day zero" feature, add metrics for "day zero"

cffd773

DJAndries force-pushed the day-zero-p3a branch 2 times, most recently from d9b634d to a242dfd Compare March 21, 2024 22:02

Add "day zero" metrics to refcode inclusion list

8eddd3a

DJAndries force-pushed the day-zero-p3a branch from a242dfd to 8eddd3a Compare March 22, 2024 19:33

DJAndries merged commit 6f30c85 into master Mar 23, 2024
19 checks passed

DJAndries deleted the day-zero-p3a branch March 23, 2024 01:00

github-actions bot added this to the 1.66.x - Nightly milestone Mar 23, 2024

brave-builds added a commit that referenced this pull request Mar 26, 2024

Uplift of #22195 (squashed) to beta

0d10372

brave-builds mentioned this pull request Mar 26, 2024

Add "day zero" experiment feature & metrics (uplift to 1.65.x) #22768

Merged

7 tasks

DJAndries mentioned this pull request Apr 2, 2024

Add variant feature param to "day zero" feature, only report metric for feature participants #22892

Merged

24 tasks

DJAndries pushed a commit that referenced this pull request Apr 3, 2024

Uplift of #22195 and #22892 (squashed) to beta

311b269

DJAndries added a commit that referenced this pull request Apr 3, 2024

Uplift of #22892 and #22195 to 1.64.x

115ed2d

DJAndries mentioned this pull request Apr 3, 2024

Add "day zero" experiment daily retention P3A metric to 1.64.x #22920

Closed

24 tasks

kjozwiak pushed a commit that referenced this pull request Apr 11, 2024

Add "day zero" experiment feature & metrics (uplift to 1.65.x) (#22768)

81afde3

Uplift of #22195 and #22892 (squashed) to beta

GeetaSarvadnya mentioned this pull request Apr 12, 2024

Add "day zero" experiment daily retention P3A metric brave/brave-browser#36243

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add "day zero" experiment feature & metrics #22195

Add "day zero" experiment feature & metrics #22195

DJAndries commented Feb 21, 2024 •

edited

Loading

DJAndries commented Feb 21, 2024

chrismore Feb 21, 2024

DJAndries Feb 21, 2024

chrismore Feb 21, 2024

DJAndries Feb 21, 2024

chrismore commented Feb 21, 2024

DJAndries commented Feb 21, 2024

mkarolin left a comment

kjozwiak commented Apr 11, 2024 •

edited

Loading

Add "day zero" experiment feature & metrics #22195

Add "day zero" experiment feature & metrics #22195

Conversation

DJAndries commented Feb 21, 2024 • edited Loading

Submitter Checklist:

Reviewer Checklist:

After-merge Checklist:

Test Plan:

DJAndries commented Feb 21, 2024

chrismore Feb 21, 2024

Choose a reason for hiding this comment

DJAndries Feb 21, 2024

Choose a reason for hiding this comment

chrismore Feb 21, 2024

Choose a reason for hiding this comment

DJAndries Feb 21, 2024

Choose a reason for hiding this comment

chrismore commented Feb 21, 2024

DJAndries commented Feb 21, 2024

mkarolin left a comment

Choose a reason for hiding this comment

kjozwiak commented Apr 11, 2024 • edited Loading

Test Case #1 - --enable-features=BraveDayZeroExperiment:variant/a

Test Case #2 - --enable-features=BraveDayZeroExperiment:variant/b

DJAndries commented Feb 21, 2024 •

edited

Loading

kjozwiak commented Apr 11, 2024 •

edited

Loading

`Test Case #1` - `--enable-features=BraveDayZeroExperiment:variant/a`

`Test Case #2` - `--enable-features=BraveDayZeroExperiment:variant/b`