[2016 Velocity NY] There has been a lot of historical work that looks at the relationship between performance and conversions, but most of it has been after the fact or relied on linear models. Google partnered with SOASTA to train a machine-learning model on a large sample of real-world performance, conversion, and bounce data. Patrick Meenan and Tammy Everts offer an overview of the resulting model, able to predict the impact of performance work and other site metrics on conversion and bounce rates. The code used to generate the model is freely available.
8. Vectorizing the data
• Everything needs to be numeric
• Strings converted to several inputs as yes/no
(1/0)
• i.e. Device manufacturer
• “Apple” would be a discrete input
• Watch out for input explosion (UA String)
9. Balancing the data
• 3% conversion rate
• 97% accurate by always guessing no
• Subsample the data for 50/50 mix
10. Smoothing the data
ML works best on normally distributed data
scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)
x_val = scaler.transform(x_val)
11. Validation data
• Train on 80% of the data
• Validate on 20% to prevent overfitting
–Training accuracy from validation set
12. Input/output relationships
• SSL highly correlated with conversions
• Long sessions highly correlated with
not bouncing
• Remove correlated features from
training
17. Brute force FTW
• 93 input “features”
• Train 93 models with 1 input
– Measuring the prediction accuracy of each
• Train 92 models with 2 inputs
– Top feature from first round
– Measure combined prediction accuracy
• Lather, rinse, repeat…
18. Visualizing the model
• Take trained model (X inputs)
• Vary inputs
–100ms to 20 seconds in 100ms intervals
• Apply the data smoothing from training
set
• model.predict_proba
40. 1. YMMV
2. Do try this at home
3. Gather your RUM data (lots of it)
4. Run the machine learning against it
5. If you get unexpected results, keep
digging
mPulse is built above the boomerang JavaScript library that collects web performance data from a user’s web browser and sends that back to the mPulse servers on a beacon. The simple definition of a beacon is that it is an HTTP(S) request with a ton of data included either as HTTP headers or as part of the Request’s Query String.
“DOM ready” refers to the amount of time it takes for the page’s HTML to be received and parsed by the browser. Actual page elements, such as images, haven’t appeared yet. (It’s kind of like getting ready to cook. Your cookbook is open, your recipe is in front of you, and your ingredients are on standby.)
“DOM ready” refers to the amount of time it takes for the page’s HTML to be received and parsed by the browser. Actual page elements, such as images, haven’t appeared yet. (It’s kind of like getting ready to cook. Your cookbook is open, your recipe is in front of you, and your ingredients are on standby.)
It is the same as the DOM Content Loaded event in nav timing but the polyfill version that works across all browsers.
DOM_ready + load time gets us up to 89.5% accuracy on the predictions
Takeaway: External blocking scripts (such as third-party ads, analytics, and social widgets) and styles (such as externally hosted CSS and fonts) have the greatest impact on DOM ready times. Site owners should measure the impact that these external elements have on their pages and conduct ongoing monitoring to ensure that scripts and styles are available and fast. Whenever possible, scripts should be served asynchronously (in parallel with the rest of the page) or in a non-blocking fashion.
DOM_ready + load time gets us up to 89.5% accuracy on predictions
Sessions that converted contained 48% more scripts (including third-party scripts, such as ads, analytics beacons, and social buttons) than sessions that didn’t.
Before around 300 scripts, it's possible that it learned the patterns of what some checkout flows looked like.
Scripts are one of those things that may be more fixed than timings so it might be easier for deep learning to just learn what all sites checkout flows look like.
max_params_dom_In (number of DOM elements) -- the more complex pages after (1000?) elements starts to fall off.
Median_bandwidth_kbps was 44
User_agent_device_type was 79
Mobile_connection_type was 89
Shoppers who used low-bandwidth or mobile connections didn’t convert significantly less than shoppers on faster connections. This is interesting because it confirms that we’ve entered a “mobile everywhere” phase.
Takeaway: Internet users don’t behave especially differently depending on what device they’re using. Site owners need to ensure they’re delivering consistent user experiences across device types.
Start render tells you when content begins to display in the user’s browser.
Of the 1M records, 720k did not have render start times included (because the browser didn't support it) which is why it ended up being a not-important feature.
Pat re-ran the deep learning version of the importances on a filtered dataset that only includes records that also included a render time to see how it looked relative to the other times.
Filtered down to just records that also include a start render, start render is basically the same importance as dom ready. There is a similar pattern to the others where it plateaus though it looks like the plateau starts pretty early (around 3 seconds) which generally makes sense since usually render < dom ready < onload. In all cases, there doesn't seem to be a point where it isn't worth making it faster. If anything, the gains become more significant as you get closer to zero.