Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Goals first: Use actual experiment checks within useGoalsFirstExperiment #97699

Open
wants to merge 4 commits into
base: trunk
Choose a base branch
from
Open

Goals first: Use actual experiment checks within useGoalsFirstExperiment #97699

wants to merge 4 commits into from
+36 −42

Conversation

zaguiini
Copy link
Contributor

@zaguiini zaguiini commented Dec 20, 2024 •
edited by p-jackson
Loading

Closes #96179.
Closes #98001

Proposed Changes

Remove the flag onboarding/goals-first and use the actual experiment name and variant names.

@zaguiini zaguiini requested a review from a team December 20, 2024 13:31
@zaguiini zaguiini self-assigned this Dec 20, 2024
@matticbot matticbot added the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Dec 20, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 20, 2024 •
edited by matticbot
Loading

Calypso Live (direct link)
https://calypso.live?image=registry.a8c.com/calypso/app:build-129204
Jetpack Cloud live (direct link)
https://calypso.live?image=registry.a8c.com/calypso/app:build-129204&env=jetpack
Automattic for Agencies live (direct link)
https://calypso.live?image=registry.a8c.com/calypso/app:build-129204&env=a8c-for-agencies

@matticbot
Copy link
Contributor

matticbot commented Dec 20, 2024 •
edited
Loading

This PR modifies the release build for the following Calypso Apps:

For info about this notification, see here: PCYsg-OT6-p2

  • notifications

To test WordPress.com changes, run install-plugin.sh $pluginSlug goals-first/use-experiment on your sandbox.

@zaguiini zaguiini force-pushed the goals-first/use-experiment branch from fb6fc54 to a19a04c Compare December 20, 2024 13:40
@matticbot
Copy link
Contributor

matticbot commented Dec 20, 2024 •
edited
Loading

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

App Entrypoints (~2804 bytes added 📈 [gzipped])

name                   parsed_size           gzip_size
entry-stepper              +6785 B  (+0.2%)    +2588 B  (+0.3%)
entry-main                  +886 B  (+0.0%)     +335 B  (+0.1%)
entry-login                 -189 B  (-0.0%)     +100 B  (+0.0%)
entry-domains-landing       -167 B  (-0.0%)     +108 B  (+0.1%)
entry-browsehappy           -167 B  (-0.1%)     +108 B  (+0.2%)
entry-subscriptions         -154 B  (-0.0%)      +97 B  (+0.0%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~22895 bytes removed 📉 [gzipped])

name                                parsed_size           gzip_size
link-in-bio-tld-flow                    -7416 B  (-0.5%)    -2237 B  (-0.5%)
site-setup-wg                           -6831 B  (-5.0%)    -2656 B  (-6.1%)
site-setup-flow                         -6831 B  (-5.0%)    -2656 B  (-6.2%)
update-design-flow                      -6437 B  (-0.7%)    -1844 B  (-0.7%)
stats                                   -1121 B  (-0.1%)     +752 B  (+0.2%)
copy-site-flow                           -626 B  (-0.1%)      -91 B  (-0.1%)
onboarding-flow                          -624 B  (-2.8%)     -157 B  (-2.1%)
import-hosted-site-flow                  -531 B  (-0.1%)      +96 B  (+0.0%)
google-my-business                       +264 B  (+0.1%)     +526 B  (+0.3%)
themes                                   -248 B  (-0.0%)    -1672 B  (-0.5%)
site-purchases                           +196 B  (+0.0%)    +1704 B  (+0.4%)
settings-writing                         +196 B  (+0.0%)    +1117 B  (+0.7%)
settings-newsletter                      +196 B  (+0.0%)    +1108 B  (+0.7%)
purchases                                +196 B  (+0.0%)    +1704 B  (+0.3%)
jetpack-cloud-partner-portal             +196 B  (+0.0%)     +906 B  (+0.3%)
jetpack-cloud-agency-sites-v2            +196 B  (+0.0%)     +906 B  (+0.2%)
theme                                    -194 B  (-0.0%)    -1948 B  (-0.8%)
staging-site                             -187 B  (-0.0%)     -806 B  (-0.2%)
site-tools                               -187 B  (-0.0%)     -804 B  (-0.2%)
settings-performance                     -187 B  (-0.0%)     -818 B  (-0.5%)
hosting                                  -187 B  (-0.0%)     -812 B  (-0.1%)
site-migration-flow                      -172 B  (-0.3%)     -574 B  (-3.6%)
migration-flow                           -172 B  (-0.5%)     -574 B  (-4.9%)
hosted-site-migration-flow               -172 B  (-0.3%)     -574 B  (-3.5%)
plugins                                  -168 B  (-0.0%)    +1681 B  (+0.2%)
plans                                    -160 B  (-0.0%)     -683 B  (-0.2%)
earn                                     -157 B  (-0.0%)      -45 B  (-0.0%)
settings-podcast                         +142 B  (+0.0%)    +1099 B  (+0.7%)
jetpack-app                              +134 B  (+0.0%)    +1060 B  (+0.8%)
devdocs                                  +108 B  (+0.1%)      +32 B  (+0.1%)
checkout                                 +108 B  (+0.0%)    -1242 B  (-0.2%)
tailored-ecommerce-flow                  -102 B  (-1.6%)      -24 B  (-1.1%)
hundred-year-plan                        -101 B  (-0.2%)      -25 B  (-0.2%)
site-settings                             -87 B  (-0.0%)     -807 B  (-0.2%)
reblogging-flow                           -83 B  (-1.4%)      -19 B  (-1.0%)
newsletter-flow                           -83 B  (-0.3%)      -19 B  (-0.2%)
new-hosted-site-flow-user-included        -83 B  (-1.4%)      -11 B  (-0.6%)
new-hosted-site-flow                      -83 B  (-0.9%)      -22 B  (-0.8%)
connect-domain                            -83 B  (-0.3%)      -10 B  (-0.1%)
home                                      +71 B  (+0.0%)     -260 B  (-0.1%)
hosting-features                          -68 B  (-0.0%)     -770 B  (-0.2%)
signup                                    +64 B  (+0.0%)      +14 B  (+0.0%)
marketplace                               -61 B  (-0.0%)    -1128 B  (-0.4%)
write-flow                                -54 B  (-0.0%)       -2 B  (-0.0%)
site-marketing                            -54 B  (-0.0%)      -34 B  (-0.0%)
settings-security                         -54 B  (-0.0%)      -24 B  (-0.0%)
posts-custom                              -54 B  (-0.0%)      -33 B  (-0.0%)
posts                                     -54 B  (-0.0%)      -32 B  (-0.0%)
marketing                                 -54 B  (-0.0%)      -34 B  (-0.0%)
domains                                   -54 B  (-0.0%)      +25 B  (+0.0%)
build-flow                                -54 B  (-0.0%)       +0 B
activity                                  -54 B  (-0.0%)       -8 B  (-0.0%)
jetpack-cloud-plugin-management           -53 B  (-0.0%)    +1557 B  (+0.2%)
people                                    +48 B  (+0.0%)       +9 B  (+0.0%)
woocommerce                               -40 B  (-0.0%)      +16 B  (+0.0%)
start-writing-flow                        +24 B  (+0.1%)       -4 B  (-0.0%)
design-first-flow                         +24 B  (+0.1%)       +1 B  (+0.0%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~5274 bytes removed 📉 [gzipped])

name                                                                              parsed_size           gzip_size
async-load-signup-steps-page-picker                                                   -6831 B  (-1.7%)    -2656 B  (-2.0%)
async-load-purchase-modal-wrapper                                                     -6831 B  (-1.7%)    -2656 B  (-2.1%)
async-load-my-sites-checkout-purchase-modal-is-eligible-for-one-click-checkou...      -6831 B  (-1.7%)    -2656 B  (-2.2%)
async-load-calypso-my-sites-checkout-modal                                            -6738 B  (-0.5%)    -2378 B  (-0.6%)
async-load-signup-steps-plans-theme-preselected                                       -6697 B  (-1.4%)    -1600 B  (-1.1%)
async-load-signup-steps-plans                                                         -6697 B  (-1.4%)    -1600 B  (-1.1%)
async-load-design                                                                      +301 B  (+0.0%)    +1123 B  (+0.3%)
async-load-design-blocks                                                               -250 B  (-0.0%)    -1270 B  (-0.2%)
async-load-store-app-store-stats                                                       -128 B  (-0.1%)    -1102 B  (-1.6%)
async-load-calypso-reader-sidebar                                                       -54 B  (-0.1%)       -9 B  (-0.0%)
async-load-calypso-my-sites-current-site-notice                                         -54 B  (-0.1%)       -9 B  (-0.0%)
async-load-calypso-components-web-preview-component                                     -54 B  (-0.0%)      -13 B  (-0.0%)
async-load-calypso-blocks-jitm-templates-sidebar-banner                                 -54 B  (-0.1%)       -9 B  (-0.0%)
async-load-calypso-blocks-jitm-templates-notice                                         -54 B  (-0.1%)      -10 B  (-0.1%)
async-load-calypso-blocks-jitm-templates-default                                        -54 B  (-0.1%)      -10 B  (-0.1%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

paulopmt1
paulopmt1 approved these changes Dec 20, 2024
View reviewed changes
Copy link
Contributor

@paulopmt1 paulopmt1 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working as expected!

When connecting it to the running experiment 22021-explat-experiment (for testing purposes since we'll create a new experiment), it will redirect control users to the new flow and holdout to the old one.

I still recommend a new check once the oficial experiment is in place.

@zaguiini zaguiini force-pushed the goals-first/use-experiment branch from a19a04c to 39d0758 Compare December 23, 2024 20:21
@zaguiini zaguiini marked this pull request as ready for review December 23, 2024 20:22
@zaguiini zaguiini requested a review from a team as a code owner December 23, 2024 20:22
candy02058912
candy02058912 approved these changes Dec 24, 2024
View reviewed changes
Copy link
Contributor

@candy02058912 candy02058912 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes LGTM 👍

@alshakero
Copy link
Member

I'd test this thoroughly because I have seen issues happen with Stepper's useSteps hook when it returns dynamic results. Consider that Stepper redirects to the first step of the flow when you visit /flowName. So if the experiment returns control, the first step will be different than when it does treatment. And the experiment may load well after Stepper had already redirected.

@zaguiini
Copy link
Contributor Author

zaguiini commented Dec 26, 2024 •
edited
Loading

@alshakero, I see what you're saying. The thing is, we're already using the useAssertConditions hook to prevent any action from the flow while the experiment is loading. Does that have no effect?

@alshakero
Copy link
Member

Ah I missed that. It should be good then 👍

@candy02058912 candy02058912 requested review from a team and necronet and removed request for a team December 27, 2024 10:03
@zaguiini zaguiini requested a review from a team December 27, 2024 18:30
@zaguiini zaguiini requested a review from p-jackson January 4, 2025 01:50
@autumnfjeld autumnfjeld mentioned this pull request Jan 6, 2025
Merged
8 tasks
@p-jackson p-jackson force-pushed the goals-first/use-experiment branch from 39d0758 to a08b379 Compare January 6, 2025 23:14
@p-jackson p-jackson mentioned this pull request Jan 6, 2025
Open
@p-jackson p-jackson force-pushed the goals-first/use-experiment branch from a08b379 to f2580fd Compare January 8, 2025 07:26
@p-jackson
Copy link
Member

I've made a change that loads the experiment assignment much earlier. f2580fd

We've run into the exact issue you were talking here pdDR7T-22t-p2 @alshakero. Sorry I haven't commented there yet, but we need something like bootFlow (although I'd call it init or something).

I thought by implementing our dummy useGoalsFirstExperiment() hook with a fake timeout that we were already handling the async nature of experiment assignment. However the dummy implementation always returned the correct assignment as the second tuple element. However in the real world, the hook returns control for treatment users while we're still loading. And that doesn't work with useSteps. And even those the assertion hook returns loading, it still breaks.

My first attempt to fix was to default to returning all the steps during loading. Maybe removing unneeded steps later works better than adding steps later.

It actually sort of worked, except for logged out users. Users would get taken to the account step, then after signup (and assuming they were assigned to control) would be redirected to the curious /setup/onboarding/domains/goals URL.

So I've moved the experiment assignment further up the React tree. This means it's high enough that the <RedirectToStep> component isn't rendered. But it might need to go even higher. I need to do more testing with the tools provided by ExPlat.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@candy02058912 candy02058912 candy02058912 approved these changes

@paulopmt1 paulopmt1 paulopmt1 approved these changes

@necronet necronet Awaiting requested review from necronet necronet was automatically assigned from Automattic/experiment-code-reviewers

@p-jackson p-jackson Awaiting requested review from p-jackson

Assignees

@zaguiini zaguiini

Labels
DO NOT MERGE [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update e2e tests for new onboarding flow order Set up experiment holdout for new goals step position
6 participants
@zaguiini @matticbot @alshakero @p-jackson @paulopmt1 @candy02058912