Dikla Cohen on web.dev

Luxury retailer Farfetch sees higher conversion rates for better Core Web Vitals

2022-07-12T00:00:00Z

Inside many companies, website speed performance and Core Web Vitals are still mostly seen as the responsibility of the engineering teams. Site speed can become invisible to other areas of the business if the business and customer experience worth is not understood. This can cause performance to be overlooked when key decisions are made and roadmaps are defined.

To improve the performance culture across teams and drive significant improvements to their web experience, luxury e-commerce fashion retailer Farfetch launched a project to define and use true customer-centric performance metrics. They aimed to correlate these to business metrics as a way of illuminating how performance affects the company's KPIs.

Their ambitions didn't end there, though. Ultimately, the project's objective was cultural change at scale—breaking down silos within the organization and introducing a new business-oriented language to give everyone a shared way to talk about what used to be considered technical topics. Farfetch wanted to have website speed performance as a shared responsibility, facilitate informed decision making and establish it as a main pillar of what makes a good web experience.

To kick off, Farfetch realized a single department couldn't achieve this as before and assembled a core team of experts from various areas in the company—engineering, infrastructure, architecture, and product—that put in place a step-by-step strategy to revamp how the company looked at this topic.

Step 1: Defining, measuring and monitoring metrics #

First of all, Farfetch needed to have the right monitoring tools in place to understand current state and deviations across journey touchpoints and applications.

They used both lab data and real user monitoring (field data) to track Core Web Vitals and additional user-centric performance metrics to analyze the current state of speed performance. They used JavaScript and the web-vitals.js library to capture the data, which allowed the product analytics team to gain visibility of performance metrics alongside business metrics in the same session and thereby start examining how one affects the other.

The multidisciplinary group set out to understand which metrics mattered most for the business. To do so, they looked at the critical journey path of Farfetch users and tried to link that journey with performance markers. In addition to the Core Web Vitals metrics that Google has outlined, each representing a distinct facet of the user experience, they also used custom JavaScript to track Time to First Byte (TTFB), First Contentful Paint (FCP), First Paint and Time to Interactive (TTI).

The metrics are collected using several methods of the Performance API, Long Tasks API and Google's polyfills. More details can be found in this mid 2020 Farfetch Tech Blog post by Manuel Garcia, Senior Principal Engineer for the Web.

On the data analytics side, Farfetch have their own multi-channel tracking solution used by front end applications called Omnitracking. It tracks the events generated by page views, user actions and system actions. The Omnitracking Data Model is Farfetch's solution for analytical, data exploration and reporting cases, made on the top of the events generated by trackers. The data model's objective is to help and support anyone who needs to understand:

User behavior
User experience on Farfetch applications
Applications usage
Macro and micro conversions
Cross channel and funnel analysis

The idea was then to add the performance data of each pageview on farfetch.com, captured by JavaScript, to this data layer. Following this model guaranteed a match between performance data and the main conversion funnel metrics for each session and the base for an analytical exploration on the topic.

Finally, Farfetch established time-based performance budgets for each metric across the main journey pages and a governance process to handle budget breaches. They've also started incorporating performance metrics on CI pipelines to understand budget deviations as soon as possible in the development flow.

Step 2: Communicating via business language #

With performance data now available in Farfetch's internal business intelligence datasets, the analytics team started exploring mathematical models and patterns in the data that could indicate a correlation between performance metrics and business KPIs (for example, conversion rate and percentage of single-page visits), allowing a new view of the financial impact of site speed and UX for the business. This enabled performance to be discussed in a common language with the business decision makers. The analysis included all Core Web Vitals and the other metrics Farfetch deemed valuable. It revealed truly impactful insights.

Having in mind that Google recommends Largest Contentful Paint (LCP) to stay below 2.5 seconds to provide an optimal user experience, Farfetch studied this threshold carefully and found meaningful results.

Farfetch statistical correlation analysis showed that beyond that point, the conversion rate starts to drop, as well as the exit rate rises. This shows that users really start to feel the downside of a slow page load perception and conversion rate decreases on average -1.3% with each 100ms more of LCP.

Farfetch also verified an exit rate decrease of -3.1% for each 0.01 less on the Cumulative Layout Shift (CLS) score, reaffirming the impact of page stability to keep users on a website.

Regarding page interactivity and fluidity, although First Input Delay (FID) is tracked and analyzed continuously, Farfetch also measures TTI, which proved to be a considerably impactful metric for the Farfetch business conversion funnel.

For this, they injected Google's TTI polyfill into the website to store this metric. Using the Long Tasks API to report long tasks (tasks taking longer than 50 milliseconds on the browser's main thread).

The analytics team was then able to find that the conversion rate increased 2.8% for each second reduction on TTI, making a strong case for better code efficiency and unclogging the browser's main thread.

In the end, this analysis was also able to show that some metrics didn't have a significant impact on business KPIs, or some would be more relevant at different stages of the user journey. This allowed for a complete understanding of the opportunities at hand on each point of the conversion funnel.

Step 3: Embedding cultural change #

Showcasing the insights above alongside qualitative user research on user perception of site speed was paramount to establish alignment with company goals, secure executive-level awareness and buy-in for performance based decision making across product roadmaps. It was now possible to prove how much performance was worth for Farfetch.

To streamline prioritization, Farfetch created a self-service tool they named the Site Speed Business Case Calculator, taking inspiration from Google's Speed Impact Calculator. It allows any product manager to create a business case out of performance improvements by calculating the business impact on the fly. With a data model using the correlation between conversion rate and user experience metrics, it's flexible to adapt to different product scopes, devices and user journey touch points.

Meanwhile, a set of self-service analytical dashboards have created business-wide visibility of real-time performance indicators and their impact on the business. Performance is now becoming fully embedded in product development, and product teams enjoy easy access to metrics, auditing tools, and performance budget monitoring. Furthermore, because of the data layer integration, performance metrics are also available on Farfetch A/B testing tooling, giving product managers yet another powerful vector of insights.

In recent months, the core team is also on track to establish this culture not only on front-end development teams but also within the platform domain, using similar methodologies to monitor and prove the impact of the main micro services and transactions.

There have been a number of Farfetch driven presentations on this topic but also external mentions. For example, a mention on a 2021 Google I/O talk on the business impact of Core Web Vitals. This also contributed to giving continuous relevance to the theme and solidified the team's strategy on culture.

Step 4: Improving metrics #

In the end, all of this work needed to contribute to Farfetch objectively improving their website speed metrics, and guaranteeing that their teams would follow best in class practices and pursue enhancement opportunities.

One of the main opportunities detected in 2021 was the need to improve LCP on the two main page types of Farfetch—product pages and product listing pages.

The teams addressed how they were loading the main content of those pages. Armed with a business case that showed the impact of chasing this opportunity, they were able to:

Adapt the product image loading component from a JavaScript-based solution to a native implementation.
Define the priority of the images and divide them into critical and non-critical assets.
Load critical images early, with the source already inline in the HTML and using <link rel="preload"> so they download as soon as possible.
Use the <img loading="lazy"> attribute for non-critical images, with a polyfill using Intersection Observer on unsupported browsers, such as Safari.

With this, they were able to move the needle and prove through A/B testing the hypothesis and business impact. On product pages for example, this effort shaved off more than 600ms, and the A/B test showed an uplift on conversion rate in the range of 1-5% with the company's defined confidence level.

Below are the improvements the team were able to make in terms of the percentage of pageviews that are considered "good", "needs improvement" and "poor", based on Google's definition for LCP score.

The rewards of a faster site and better working practices #

Building a culture around performance and tools like the business case calculator allowed everyone to begin speaking a shared language that can be understood by product managers, stakeholders, and engineers alike. This has precipitated ongoing discussions around how to prioritize new initiatives and performance enhancements.

"We wanted to break the cycle of performance being a tech-only concern, something owned only by the engineering team to deal with and fix," explains Farfetch's Web Channels Senior Principal Product Manager, Rui Santos. "Connecting performance metrics with business metrics was surprisingly effective to pass the message across very, very quickly. Business drives the company, and connecting its success to speed metrics encouraged a broader set of stakeholders to understand and navigate the trade-off decisions."

In the luxury e-commerce segment, whether your site is fast or slow can dictate how consumers see your brand and your service quality as a whole. For users, quality equates to luxury—and this applies to every aspect of their experience, including how your website performs. With site speed producing a proven effect on conversion rate, performance now occupies a safe position in forward planning at Farfetch.

Building a PWA at Google, part 1

2020-07-29T00:00:00Z

This is the first of a series of blog posts about the lessons that the Google Bulletin team learned while building an external-facing PWA. In these posts we'll share some of the challenges we faced, the approaches we took to overcome them, and general advice for avoiding pitfalls. This is by no means a complete overview of PWAs. The aim is to share learnings from our team's experience.

For this first post we'll cover a little background information first and then dive into all the stuff we learned about service workers.

Background #

Bulletin was in active development from mid-2017 to mid-2019.

Why we chose to build a PWA #

Before we delve into the development process, let's examine why building a PWA was an attractive option for this project:

Ability to iterate quickly. Especially valuable since Bulletin would be piloted in multiple markets.
Single code base. Our users were roughly evenly split between Android and iOS. A PWA meant we could build a single web app that would work on both platforms. This increased the velocity and impact of the team.
Updated quickly and independent of user behavior. PWAs can automatically update which reduces the amount of out-of-date clients in the wild. We were able to push out breaking backend changes with a very short amount of migration time for clients.
Easily integrated with first- and third-party apps. Such integrations were a requirement for the app. With a PWA it often meant simply opening a URL.
Removed the friction of installing an app.

Our framework #

For Bulletin, we used Polymer, but any modern, well-supported framework will work.

What we learned about service workers #

You can't have a PWA without a service worker. Service workers give you a lot of power, such as advanced caching strategies, offline capabilities, background sync, etc. While service workers do add some complexity, we found that their benefits outweighed the added complexity.

Generate it if you can #

Avoid writing a service worker script by hand. Writing service workers by hand requires manually managing cached resources and rewriting logic that is common to most service workers libraries, such as Workbox.

Having said that, due to our internal tech stack we could not use a library to generate and manage our service worker. Our learnings below will at times reflect that. Go to Pitfalls for non-generated service workers to read more.

Not all libraries are service-worker-compatible #

Some JS libraries make assumptions that don't work as expected when run by a service worker. For instance, assuming window or document are available, or using an API not available to service workers (XMLHttpRequest, local storage, etc). Make sure any critical libraries you need for your application are service-worker compatible. For this particular PWA, we wanted to use gapi.js for authentication, but were unable to because it did not support service workers. Library authors should also reduce or remove unnecessary assumptions about the JavaScript context where possible to support service worker use cases, such as by avoiding service worker-incompatible APIs and avoiding global state.

Avoid accessing IndexedDB during initialization #

Don't read IndexedDB when initializing your service worker script, or else you can get into this undesired situation:

User has web app with IndexedDB (IDB) version N
New web app is pushed with IDB version N+1
User visits PWA, which triggers download of new service worker
New service worker reads from IDB before registering install event handler, triggering an IDB upgrade cycle to go from N to N+1
Since user has old client with version N, service worker upgrade process hangs as active connections are still open to the old version of the database
Service worker hangs, and never installs

In our case, the cache was invalidated on service worker install, so if the service worker never installed, users never received the updated app.

Make it resilient #

Though service worker scripts run in the background, they can also be terminated at any time, even when in the middle of I/O operations (network, IDB, etc). Any long-running process should be resumable at any point.

In the case of a sync process that uploaded large files to the server and saved to IDB, our solution for interrupted partial uploads was to take advantage of our internal upload library's resumable system, saving the resumable upload URL to IDB prior to uploading, and using that URL to resume an upload if it did not complete the first time around. Also before any long running I/O operation, the state was saved to IDB to indicate where in the process we were for each record.

Don't depend on global state #

Because service workers exist in a different context, many symbols you might expect to exist are not present. A lot of our code ran in both a window context, as well as a service worker context (such as logging, flags, syncing, etc.). Code needs to be defensive about the services it uses, such as local storage or cookies. You can use globalThis to refer to the global object in a manner which will work across all contexts. Also use data stored in global variables sparingly, as there is no guarantee as to when the script will be terminated and the state evicted.

Local development #

A major component of service workers is caching resources locally. However, during development this is the exact opposite of what you want, particularly when updates are done lazily. You still want the server worker installed so you can debug problems with it or work with other APIs like background sync, or notifications. On Chrome you can achieve this through Chrome DevTools by enabling the Bypass for network checkbox (Application panel > Service workers pane) in addition to enabling the Disable cache checkbox in the Network panel in order to also disable the memory cache. In order to cover more browsers, we opted for a different solution by including a flag to disable caching in our service worker which is enabled by default on developer builds. This ensures that devs always get their most recent changes without any caching issues. It's important to include the Cache-Control: no-cache header as well to prevent the browser from caching any assets.

Lighthouse #

Lighthouse provides a number of debugging tools useful for PWAs. It scans a site and generates reports covering PWAs, performance, accessibility, SEO, and other best practices. We recommend running Lighthouse on continuous integration to alert you if you break one of the criteria to be a PWA. This actually happened to us once, where the service worker wasn't installing and we didn't realize it before a production push. Having Lighthouse as part of our CI would have prevented that.

Embrace continuous delivery #

Because service workers can automatically update, users lack the ability to limit upgrades. This significantly reduces the amount of out-of-date clients in the wild. When the user opened our app, the service worker would serve the old client while it lazily downloaded the new client. Once the new client downloaded, it would prompt the user to refresh the page to access new features. Even if the user ignored this request, the next time they refreshed the page they would receive the new version of the client. As a result, it is quite difficult for a user to refuse updates in the same way they can for iOS/Android apps.

We were able to push out breaking backend changes with a very short amount of migration time for clients. Typically, we would give a month for users to update to newer clients before making breaking changes. Since the app would serve while stale, it was actually possible for older clients to exist in the wild if the user had not opened the app for a long time. On iOS, service workers are evicted after a couple weeks so this case does not happen. For Android, this problem could be mitigated by not serving while stale, or manually expiring the content after a few weeks. In practice, we never encountered problems from stale clients. How strict a given team wants to be here is up to their specific use case, but PWAs provide significantly more flexibility than iOS/Android apps.

Getting cookie values in a service worker #

Sometimes it is necessary to access cookie values in a service worker context. In our case, we needed to access cookie values to generate a token to authenticate first-party API requests. In a service worker, synchronous APIs such as document.cookies are not available. You can always send a message to active (windowed) clients from the service worker to request the cookie values, though it's possible for the service worker to run in the background without any windowed clients available, such as during a background sync. To work around this, we created an endpoint on our frontend server that simply echoed the cookie value back to the client. The service worker made a network request to this endpoint and read the response to get the cookie values.

With the release of the Cookie Store API, this workaround should no longer be necessary for browsers that support it, as it provides asynchronous access to browser cookies and can be used directly by the service worker.

Pitfalls for non-generated service workers #

Ensure service worker script changes if any static cached file changes #

A common PWA pattern is for a service worker to install all static application files during its install phase, which enables clients to hit the Cache Storage API cache directly for all subsequent visits . Service workers are only installed when the browser detects that the service worker script has changed in some way, so we had to ensure the service worker script file itself changed in some way when a cached file changed. We did this manually by embedding a hash of the static resource fileset within our service worker script, so every release produced a distinct service worker JavaScript file. Service worker libraries like Workbox automate this process for you.

Unit testing #

Service worker APIs function by adding event listeners to the global object. For example:

self.addEventListener('fetch', (evt) => evt.respondWith(fetch('/foo')));

This can be a pain to test because you need to mock the event trigger, the event object, wait for the respondWith() callback, and then await the promise, before finally asserting on the result. An easier way to structure this is to delegate all implementation to another file, which is more easily tested.

import fetchHandler from './fetch_handler.js';
self.addEventListener('fetch', (evt) => evt.respondWith(fetchHandler(evt)));

Due to the difficulties of unit testing a service worker script, we kept the core service worker script as bare-bones as possible, splitting most of the implementation into other modules. Since those files were just standard JS modules, they could be more easily unit tested with standard test libraries.

Stay tuned for parts 2 and 3 #

In parts 2 and 3 of this series we'll talk about media management and iOS-specific issues. If you want to ask us more about building a PWA at Google, visit our author profiles to find out how to contact us:

Relating site speed and business metrics

2020-07-17T00:00:00Z

Over the last few years it has been well established that site speed performance is a significant part of the user experience and that improving it benefits different business metrics such as conversion rates and bounce rates. Multiple articles and case studies have been published to back this up, including Cloudflare's How Website Performance Affects Conversion Rates, Deloitte's Milliseconds Make Millions, and Shopping for Speed on eBay.com, to name a few.

Even though the case for speed is clear, many companies still struggle with prioritizing work that will improve their site speed since they do not know exactly how it affects their users and as a result their business.

In the absence of data, it is easy to delay site speed work and focus on other tasks. A common scenario is having some people in the company recognize the importance of site speed and yet not be able to build a case for it and convince multiple stakeholders to invest accordingly.

This article provides high-level guidance on how to leverage A/B testing to evaluate the impact of site speed on business metrics therefore enabling more effective decision-making on the matter.

Step 1: Pick a page to A/B test #

Your aim is to test the hypothesis that page speed relates to your business metrics. For simplicity's sake, you can initially limit yourself to identifying a single page for analysis. Future work can expand to multiple pages of the same type to verify findings, and then to other areas of the site. Some suggestions for where to start are provided at the bottom of this step. Several requirements drive the page selection process:

The A/B test should only be run on the devices of mobile users. Globally, the partner sites we assist see an average of more than 50% (and growing!) of their traffic coming from mobile, but this can increase significantly based on region and vertical. Mobile devices are more sensitive to slower websites due to processing and memory constraints and less stable networks. Also, on-the-go usage patterns mean expectations for speed are higher.
The page you choose for testing should be an important part of your conversion funnel. Every site has a different purpose, so every site tracks different success metrics. These metrics are usually related to a user journey which is analyzed using a funnel. For example, users on an e-commerce website will have to navigate through a home page, category pages, product pages, and a checkout page to complete a purchase. If you're optimizing for conversions, one of those pages would be a good candidate.
The page should have a singular purpose. Unless your site has a very specific mission, it's usually best to avoid using the homepage for your test. Many commercial sites' homepages are portals to a wide variety of functionality that will make your analysis noisy. For example, if you're optimizing for pageviews per session on a news site, it might be best to exclude the non-commercial portions of the site and focus on monetized sections and articles.
The page chosen should get enough traffic so that you don't have to wait long before getting a statistically significant result.
The selected page should be relatively slow. In fact, the slower the better. This not only means that you will likely have an easier time improving the page, it also means that the data should be much clearer. You can do a quick scan through your Google Analytics Speed Report or Search Console Core Web Vitals report to see which of your pages are slowest.
The page should be relatively stable. Don't update pages (anything that would impact business metrics) until the test is complete. The fewer external factors there are to consider, the cleaner the analysis will be.

Using the above as a guide it should be a bit more clear which pages are good candidates for your test. Ad landing pages are also a good choice, because you're likely to have built-in business metrics, A/B testing, and analysis at your disposal. In case you're still unsure, here are some ideas per vertical:

Content Websites: sections, articles
Storefronts: category pages, product pages
Media Players: video discovery/search pages, video play page
Services & Discovery (travel, rental cars, service registration, etc.): initial form-entry page

Step 2: Measure performance #

There are two general ways to measure metrics: in the lab and in the field. We personally have found measuring metrics in the field (also known as Real User Monitoring (RUM)) to be more useful because it reflects the experience of real users. Examples of libraries and services that can help you report RUM data include Perfume, Firebase Performance Monitoring, and Google Analytics Events.

There are many metrics to choose from because they aim to capture different aspects of the user experience. Remember that your goal is to eventually determine if there's a direct correlation between your speed and business metrics, so it might be useful to track a few speed metrics in order to determine which one has the strongest correlation with your business success. In general we recommend starting with the Core Web Vitals. The web-vitals.js library can help you measure some of the Core Web Vitals in the field, although note that browser support isn't 100%. Beyond the Core Web Vitals, the Other Web Vitals are also worth checking out. You can also define custom metrics, such as "Time To First Ad Click".

Step 3: Create speed performance variants #

In this stage you will implement changes to create a faster version of the page to be tested against the current version.

A couple of things to keep in mind:

Avoid making any changes to the UI or UX. Aside from one page being faster than the other, changes must be invisible to users.
Measuring is also a key aspect of this stage. While developing, lab testing tools such as Lighthouse should be used to indicate the effect your changes have on performance. Keep in mind that changes to one metric can often influence another. Once the pages are live, stick to RUM for a more accurate evaluation.

Creating performance variants can be done in different ways. For the purpose of the test, you would want to do so as simply as possible. Below are a few options.

Create a faster page

Use a tool like Squoosh to manually optimize the images on your test page
Use DevTools code coverage to manually eliminate unused JavaScript or CSS just for that one page
Efficiently load third-party scripts
Use a tool like Critical to break out and inline critical CSS
Remove non-critical JavaScript code that does not impact the user experience and which you can do without for the purpose of the test (for example, certain third-party libraries)
Implement browser-level lazy loading which isn't supported by all browsers but may still improve performance significantly where supported
Remove non-critical analytics tags or load them asynchronously

Additional optimizations to consider can be found at Fast load times and Frontend Performance Checklist. You can also use PageSpeed Insights to run Lighthouse, which identifies opportunities to improve performance.

Slow down the page

This may be the easiest way to create variants and can be achieved by adding a simple script, slowing down server response times, loading larger images, etc. The Financial Times opted for this option when testing how performance impacted their business metrics: see A faster FT.com.

Speed up page load

For cases where the test page (let's say a product detail page) is mostly linked out from a different page (let's say the homepage), prefetching or prerendering the product page directly from the homepage for the test group will speed up the subsequent load of the page. Note that in this case the A/B test split (step 4) is done on the homepage. In addition, all of this may slow down the first page so be sure to measure that and take it under consideration when analyzing the test results (step 5).

Step 4: Create an A/B test #

Once you have two versions of the same page where one is faster than the other, the next step is to split traffic to measure impact. In general there are many techniques and tools to perform A/B tests, but be aware that not all methods are well suited to measure speed performance impact.

If you're using an A/B testing tool like Optimizely or Optimize we strongly recommend setting up a server-side test instead of a client-side test, as client side A/B testing often works by hiding page content until the experiment is loaded, which would mean the A/B testing itself would skew the metrics you wanted to measure. If you can only do client-side testing, consider setting up the experiment on a different page and changing the link outs to your test page to split the traffic. This way the test page itself isn't dragged down by the client-side test.

Example

Example of AB testing performance changes on a given product detail page (PDP) via server-side testing:

The request goes to the backend, which distributes the users to the two different versions of the page. While this is in general a good setup, it often needs IT resources to set up the server-side split.

Here is an example of a client side testing setup, using the prior page (the homepage in the diagram below) to run the testing JavaScript:

The testing JavaScript manipulates the outgoing link to give the two test groups of users links to the two versions of the PDP in question. This is easy to setup and maintain through common A/B testing tools like Optimizely or Optimize and does not influence the performance test because the testing JavaScript runs on a different page.

Alternatively, you can pick two pages which behave and perform very similarly (for example, for two very similar products). Apply your changes to one of them and then compare the difference in metrics over time. This will mean you're not conducting a proper A/B test, yet it can still be quite insightful.

In case your test page is used as a landing page for ad campaigns, it can be handy to use your ad network's built-in A/B testing tools, like Facebook Ads Split Test or Google Ads Drafts & Experiments. If that's not an option, you can use two campaigns with the same setup, and set different landing pages as targets.

Step 5: Analyze A/B test #

After you've run your test for long enough and have enough data to feel confident about the results, it's time to put it all together and run an analysis. How you do that really depends on how the test has run, so let's go through the options.

If your test was run on ad landing pages using the above-mentioned tools, analysis should be as straightforward as reading a result. If you're using Google's Drafts & Experiments, take a look at the comparison using the ScoreCard.

Platforms like Optimizely or Optimize also offer easy ways to interpret the results and determine how much of an impact speed has on your pages.

If you're using Google Analytics or a similar tool, you will have to pull together the report yourself. Luckily, Google Analytics makes it pretty easy to build custom reports, so that's where you should start. If you've sent speed data to Google Analytics using a custom dimension, check out the Reporting guide to learn how to set those up and include them in your custom reports. Make sure your report covers the date of the experiment and is configured to display both variants. What should go in this report?

Primarily, you need to include the business metrics that you most care about: conversions, page views, ads viewed, conversion rate, e-commerce metrics, click-through rate, etc.
Additionally, other standard page metrics that also make a good case for improving site speed are bounce rate, average session duration, and exit percentage.

You may also need to filter for mobile and make sure to exclude bots and other non-user traffic. More advanced analysis would also filter by region, networks, devices, traffic source, and user profiles and types, such as new users versus repeat visitors. Each group of users may be more or less sensitive to slower speeds and identifying these is also quite helpful.

Looker Studio (formerly Data Studio) or other data visualization tools make it easy to integrate various data sources including Google Analytics. This makes it easy to conduct analysis, and also create dashboards that are shareable with the many stakeholders involved in running a modern website for further buy-in. For example, the Guardian created an automated alert system that warned the editorial team when recently published content crossed their page size or speed thresholds and was likely to result in unsatisfied users.

Step 6: Draw conclusions and decide on next steps #

Once you have data that connects performance and business metrics, you can examine the results and start to draw conclusions.

If you can clearly see a correlation between improving performance and improving business metrics, summarize the results and report them across the company. Now that you can talk about speed performance in a "business language" you are more likely to grab the attention of different stakeholders and place site speed performance on everyone's radar. The next step is to set performance budgets based on the results, and plan out work to meet those budgets. Since you know the value such work will provide, you'll be able to prioritize accordingly.

If you can't identify a correlation, take a look at the caveats below and assess whether similar tests should be run elsewhere on the site (for example, through the entire purchase funnel or on a different type of page).

Caveats

There might be several reasons for not finding significant correlation between site speed metrics and business metrics:

The page chosen does not have enough influence on the business metrics you're examining. For example, a faster product page may not have a big effect on conversion rates if the checkout page is very unfriendly or slow. Consider looking at more relevant metrics such as bounce rates, add-to-basket rates or any other metric that is more directly connected to the page you're testing.
The difference in speed is not significant enough between the two versions. This should be evaluated according to the RUM metrics you're measuring.
There's a fault with the A/B testing mechanism. Traffic might not be distributed properly or analytics not reported correctly. In order to rule this out, consider running an A/A test where you test the same version of a page using the same testing mechanism and ensure there's no difference in results when doing so.
Site speed really doesn't influence your business metrics. This is rare but can occur in cases where your target market is less sensitive to speed (e.g. the site is mostly accessed from powerful devices on a strong network) or user demand is very high and choice is limited (e.g. a ticketing service that exclusively sells tickets for high-demand shows). Note that this does not mean that a faster site will not improve user experience and as a result affect brand reputation.

Conclusion #

While it is tempting to launch speed optimizations across the entire site, it is usually more beneficial in the long run to first understand what it means to your users and your company to have a faster website. It is the difference between saying "we improved FCP by 1.5 seconds" and "we improved FCP by 1.5 seconds and that improved our conversion rates by 5%". This will allow you to prioritize further work, get buy-in from different stakeholders and make site speed performance a company-wide effort.