Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "day zero" experiment daily retention P3A metric #36243

Closed
DJAndries opened this issue Feb 21, 2024 · 6 comments · Fixed by brave/brave-core#22195
Closed

Add "day zero" experiment daily retention P3A metric #36243

DJAndries opened this issue Feb 21, 2024 · 6 comments · Fixed by brave/brave-core#22195
Assignees
Labels
feature/new-metric Proposing a new metric for Privacy Preserving Product Analytics (P3A) features/P3A OS/Android Fixes related to Android browser functionality OS/Desktop OS/Windows QA Pass-Win64 QA/Yes release-notes/exclude

Comments

@DJAndries
Copy link
Collaborator

DJAndries commented Feb 21, 2024

"Day zero" experiments will be performed in the future in an attempt to improve user retention. The experiments will involve small changes in the UI. Add daily retention metrics for measuring the results of the experiment.

If you are included in the A variant of the day zero experiment, how long ago did you install the browser?
Brave.DayZero.A.InstallTime

If you are not included in the B variant of the day zero experiment, how long ago did you install the browser?
Brave.DayZero.B.InstallTime

Values for the metric range from 0 to 30 days. The refcode will be included with the report.

cc @chrismore @timchilds

@kjozwiak
Copy link
Member

kjozwiak commented Apr 11, 2024

The above requires 1.65.110 or higher for 1.65.x verification 👍 The above will also be verified/checked using brave/brave-variations#979 and will be pushed into production via brave/brave-variations#980 once 1.65.x is released.

@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Apr 12, 2024

Verification PASSED on

Brave | 1.65.110 Chromium: 124.0.6367.29 (Official Build) (64-bit)
-- | --
Revision | 6271fdd3c799c0db71c1e6ca8c83888a7bbbfbd2
OS | Windows 10 Version 22H2 (Build 19045.4046)
  • Verified the test plan from Add "day zero" experiment feature & metrics brave-core#22195

  • Open brave and open brave://local-state file and look for the metrics Brave.DayZero.A.InstallTime and ensured that the metrics response value is 0 as expected
    image

  • Clean profile, launch brave using brave.exe --enable-features="BraveDayZeroExperiment:variant/a" and open brave://local-state file and look for the metrics Brave.DayZero.A.InstallTime and ensured that the metrics response value is 0 as expected
    image

  • Close the browser and set the sys date ahead by 7 days, relaunch brave and open brave://local-state file and look for the metrics Brave.DayZero.A.InstallTime and ensured that the metrics response value is 7 as expected'

  • step 3, Close browser, advance system time by 14 days, start browser with --enable-features="BraveDayZeroExperiment:variant/b", ensure metric has value of 21, and Brave.DayZero.B.InstallTime does NOT exist

Example Example
image image (3)
  • step 4, Close browser advance system time by 14 days, start browser, ensure that the metric still has a value of 21 (to ensure that the metric was not updated internally).
    image

  • steps 5, 6- Start with fresh profile with --enable-features="BraveDayZeroExperiment:variant/b" switch and Ensure Brave.DayZero.B.InstallTime has a value of 0.
    image

  • step 7- Close browser, advance system time by 7 days, start browser, ensure metric has value of 7.
    image

  • step 8, Close browser, advance system time by 14 days, start browser without --enable-features="BraveDayZeroExperiment:variant/b", ensure metric has value of 21, and Brave.DayZero.A.InstallTime does NOT exist

Example Example
image image
  • step 9, Close browser advance system time by 14 days, start browser, ensure that the metric still has a value of 21 (to ensure that the metric was not updated internally).
    image

  • Confirmed that the metrics only exist in the logs_constellation_prep_express pref.

@hffvld hffvld added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Apr 15, 2024
@GeetaSarvadnya
Copy link

GeetaSarvadnya commented Apr 17, 2024

Verification INPROGRESS on Samsung Galaxy version 12 running Bravemonoarm64.apk_1.65.114

  • Verified the test plan from Add "day zero" experiment feature & metrics brave-core#22195
  • Open brave and open brave://local-state file and look for the metrics Brave.DayZero.A.InstallTime and ensured that the metrics isn't shown
    image
  • Clean profile, launch brave using brave.exe --enable-features="BraveDayZeroExperiment:variant/a" and open brave://local-state file and look for the metrics Brave.DayZero.A.InstallTime and ensured that the metrics response value is 0 as expected
    image
  • Close the browser and set the sys date ahead by 7 days, relaunch brave and open brave://local-state file and look for the metrics Brave.DayZero.A.InstallTime and ensured that the metrics response value is 7 as expected'
  • step 3, Close browser, advance system time by 14 days, start browser with --enable-features="BraveDayZeroExperiment:variant/b", ensure metric has value of 21, and Brave.DayZero.B.InstallTime does NOT exist
Example Example
image image (3)
  • step 4, Close browser advance system time by 14 days, start browser, ensure that the metric still has a value of 21 (to ensure that the metric was not updated internally).
    image

  • steps 5, 6- Start with fresh profile with --enable-features="BraveDayZeroExperiment:variant/b" switch and Ensure Brave.DayZero.B.InstallTime has a value of 0.
    image

  • step 7- Close browser, advance system time by 7 days, start browser, ensure metric has value of 7.
    image

  • step 8, Close browser, advance system time by 14 days, start browser without --enable-features="BraveDayZeroExperiment:variant/b", ensure metric has value of 21, and Brave.DayZero.A.InstallTime does NOT exist

Example Example
image image
  • step 9, Close browser advance system time by 14 days, start browser, ensure that the metric still has a value of 21 (to ensure that the metric was not updated internally).
    image

  • Confirmed that the metrics only exist in the logs_constellation_prep_express pref.

@hffvld hffvld removed the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Apr 17, 2024
@GeetaSarvadnya
Copy link

Verification INPROGRESS on Vivo X70 Pro version 12 running Bravemonoarm64.apk_1.65.114

@GeetaSarvadnya
Copy link

@DJAndries I have verified the test plan from brave/brave-core#22195 and noticed that the metrics Brave.DayZero.A.InstallTime isn't shown on the local state file on Android. Steps followed are as below:

  1. Clean profile 1.65.114
  2. Enable developer options
  3. Open QA preferences and enter the flag --enable-features="BraveDayZeroExperiment:variant/a" on Command Line String field via Developer options->QA preferences
  4. Relaunch the browser
  5. Open brave://version
  6. Ensured that the StudyBraveDayZeroExperiment:GroupBraveDayZeroExperiment is shown under Active Variations via brave://version
  7. Open brave://local-state file
  8. Look for the metrics Brave.DayZero.A.InstallTime
  9. Metrics is not shown
  10. Close the browser and set the sys date ahead by 7 days
  11. Relaunch the browser
  12. Reload brave://local-state file
  13. Look for the metrics Brave.DayZero.A.InstallTime
  14. Metrics is not shown

cc: @hffvld

@GeetaSarvadnya GeetaSarvadnya added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Apr 17, 2024
@DJAndries
Copy link
Collaborator Author

This issue cannot be QA'd effectively on Android, since the feature needs to be enabled upon first run. There is no easy way to do this on Android, currently.

@GeetaSarvadnya GeetaSarvadnya removed the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/new-metric Proposing a new metric for Privacy Preserving Product Analytics (P3A) features/P3A OS/Android Fixes related to Android browser functionality OS/Desktop OS/Windows QA Pass-Win64 QA/Yes release-notes/exclude
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants