Martin Schierle on web.dev

How committing to Core Web Vitals increased Netzwelt's advertising revenues by 18%

2021-07-19T00:00:00Z

When Google announced the Core Web Vitals initiative, German publisher Netzwelt quickly realized the potential of these new metrics towards a great page experience and improved advertising-based monetization. They went on a journey to relaunch their website, applying common performance best practices while optimizing ad tags and placements in parallel. Committing to great UX and fast pages proved to be a path for optimizing engagement and ad revenues hand in hand, with page views up by 27%, ad viewability over 75%, and advertising revenues improving by 18%.

27_%

Increase in page views

18_%

Increase in ad revenue

75_%

Ad viewability

The challenge #

Like many other news publishers, Netzwelt struggled to find the right balance between optimizing user experience and page speed while maintaining high ad revenues. The main challenges they encountered were:

High Cumulative Layout Shift (CLS) due to layout shifts triggered by ads, especially from multisize slots and top banners.
Largest Contentful Paint (LCP) coming in late due to ads being the largest paint or by taking bandwidth from hero image loading.
High First Input Delay (FID) caused by third-party JavaScript needed for advertising, header bidding, and other purposes.
Side effects on Core Web Vitals from consent dialogs controlled by third-party vendors, which also added to layout shifts and might be detected as late largest paints.

Optimizing a news website for Core Web Vitals #

When Netzwelt started working on core web vitals, they quickly noticed that optimizing Core Web Vitals doesn't need to affect advertisement negatively but can be used as an opportunity to improve ad viewability. Therefore, the Netzwelt team optimized Core Web Vitals to lift ad viewability above 75% to attract higher-value advertising (the global average for display ads is less than 50%).

Optimizing CLS #

Advertisements often load late (sometimes consciously through lazy loading), and their real size is often not known in advance due to multisize and fluid ad placements.

The problem can be divided into two—ads above and below the fold.

Ads above the fold can cause massive layout jumps due to loading late with unknown sizes. Blocking the largest space they might need can lead to bad UX, while asking advertisers for fixed sizes may reduce ad income. Netzwelt solved this problem by making the top ad sticky and removing some of the bigger allowed banner sizes. The overlaid ad avoids triggering layout jumps for ads of different sizes. Although the reduced eligible sizes do impact ad sales, the better viewability offsets this easily.

Historical data and A/B tests helped Netzwelt find the right size and positioning for different devices and screen sizes, and CSS media rules used to reserve space.

Ads below the fold offer room for significant improvement:

A banner that is not seen but loaded creates poor ad viewability and worsens the page experience.
A banner that is loaded at the time a user scrolls over it can introduce additional content jumps.

To maintain a good page experience and high ad viewability, Netzwelt implemented lazy loading and reserved space for the size of the lowest common denominator. In a multisize zone the banners requested in the sizes 300x250 to 300x600, reserved 250px in height. This eliminated layout shifts for the smaller sizes; and reduced them massively for larger banners. This approach is not optimal, but it is a start and a good compromise.

To optimize further, Netzwelt used Intersection Observer and the Network Information API to control ad placements and reduce layout shifts. Different ad positions and lazy loading strategies are used depending on scroll position and network connection quality, and ads may be changed from multiple to fixed sizes. The aim of the algorithm is always to maximize ad viewability while minimizing layout shifts. Browsers not supporting the NetworkInfo API use a proxy value based on a combination of device and IP derived network type. This adaptive loading strategy especially reduces CLS for users with slow internet connections.

Optimizing FID #

First Input Delay might appear to be a problem for news publishers, as advertisement often comes with many additional JavaScript libraries. And there does seem to be a negative impact when looking at lab data like Lighthouse. However, looking at field data in the 2020 Web Almanac, many websites have sufficiently fast response. Part of this is due to advertising JavaScript loading late (after first input), caching well (for example getting the treatment of v8 code caching), or being optimized well by the ad vendors. Vendor viewability trackers make sure to avoid long tasks—so even when the sum of the runtime is long Total Blocking Time (TBT) and FID are not affected. While FID was not a huge problem for Netzwelt, there were still some optimizations to make:

Reducing ad scripts and providers, concentrating on a single stack if possible.
Loading all scripts with defer or async.
Using webpack or similar technologies for treeshaking and unbundling.
Using simple BEM-like CSS rules.
Avoiding long-running tasks, and using the idle-until-urgent pattern.
Working with RequestAnimationFrame wherever layout is affected.

Optimizing LCP #

The Largest Contentful Paint can be influenced by advertisements in two ways—explicitly by recognizing an ad as the largest paint, and indirectly by congesting network bandwidth or affecting critical path so that the actual largest paint (for example a hero image) can't load in fast enough.

Netzwelt had already removed medium rectangle ads from the top ad slots while optimizing for CLS. This had the additional benefit of ads not becoming the largest element.

Netzwelt follows a strict policy to prioritize content above ads. Netzwelt prioritized the hero images and fonts used above the fold and optimized the critical path to take precedence over advertising scripts and adverts. This prioritization helped LCP to be unaffected by the ads.

Besides these optimizations, Netzwelt followed other best practices:

Separated CSS for critical rendering path and put it in the header.
Preloaded the most important fonts, scripts, and images.
Avoided lazy loading images above the fold.
Used font-display="swap".

Consent dialogs often negatively impact Core Web Vitals. As with advertisements, there are two ways in which a consent dialog can influence CWV:

Explicitly if it is detected as the largest paint, or causes layout shifts.
Implicitly, by stealing bandwidth from the actual largest paint, blocking the critical path to the largest paint, or delaying ads until consent is in, which can, in turn, trigger layout shifts.

Netzwelt works with a third-party consent provider, which also implemented optimizations. First Netzwelt made sure to avoid the straightforward pitfalls:

Consent scripts are loaded async, so as not to block critical resources.
Consent is formatted so that large elements are not eligible as the largest element within LCP.
Consent overlay uses position: fixed to avoid shifts.
While ads are shown only after consent is given, adequate whitespace is preserved beforehand to avoid layout shifts when ads load in.
Making sure the display and positioning of the consent dialog does not trigger layout shifts. One problem found and fixed here was the scroll-lock option of the service provider, which disabled scrolling while consent shows by moving the main content of the article on scroll, causing layout shifts.

After this work, there were still large discrepancies between field and lab CLS. While lab CLS was close to zero, field values were significantly above thresholds. After investigation, the culprit was layout shifts within the consent iframe, which currently are only correctly captured in field data. Netzwelt continues to work with the third-party consent provider to improve this issue.

News Subscription Models and Core Web Vitals #

News publishers are moving to subscription models to fund journalism. This model is relevant for Core Web Vitals as people will not subscribe to websites with poor user experience. In addition, subscribers do not expect to see ads in the paid content, but hiding ads may cause layout shifts. The website needs to check if users are entitled to view the content and whether to display ads. These checks may cause additional latencies, leading to content shifts or a poor user experience.

Netzwelt is using a model where content is always free, but subscribers will not see ads. They investigated different ways to query and store entitlements to reduce latencies and layout shifts.

An initial query of entitlement always caused latencies, and therefore if querying the entitlements takes too long, the site will display the last cached state. For new subscribers, this means a small risk of a paying user seeing ads once. Users just ending a subscription might see one load without ads before the locally stored entitlements update. Once entitlements are known, they are stored locally using the LocalStorage API, to avoid additional latencies and layout shifts during future navigation.

Optimization Results #

The results of Netzwelt's optimizations speak for themselves. Their PageSpeed Insights score is unrivaled within news publishers worldwide:

The optimizations improved the page experience but were made with the business in mind and enhanced ad experience, ad viewability, and revenue. After relaunching the optimized page, Netzwelt saw CPM increases of 20-30%, with an ad viewability above 75%. However, Netzwelt assumes the overall revenue uplift to be even higher. Traffic increased since the relaunch, probably driven by higher user engagement and lower bounce rates due to improved UX. These effects are hard to quantify and set in causal relation to the relaunch, as traffic naturally fluctuates, and there are also effects from the global pandemic. These indirect effects are one of the reasons why Netzwelt always looks at all numbers when reviewing their site, not just CPM, which may be misleading. Core Web Vitals and UX metrics, in combination with all ad-related metrics, provide the real picture.

Looking into the future #

Netzwelt believes that in a future without third-party cookies, contextual targeting via the content, combined with good UX and page experience (including ad viewability), is the key to success as a news publisher.

Therefore Netzwelt doesn't stop with Core Web Vitals but goes the extra mile by implementing many modern web capabilities such as Progressive Web Apps (PWA), offline content, dark mode, the Web Share API, and Trusted Web Activities (TWA) using the new Digital Goods API.

Measuring offline usage

2020-10-28T00:00:00Z

This article shows you how to track offline usage of your site to help you make a case for why your site needs a better offline mode. It also explains pitfalls and problems to avoid when implementing offline usage analytics.

The pitfalls of the online and offline browser events #

The obvious solution for tracking offline usage is to create event listeners for the online and offline events (which many browsers support) and to put your analytics tracking logic in those listeners. Unfortunately, there are several problems and limitations with this approach:

In general tracking every network connection status event might be excessive, and is counter-productive in a privacy-centric world where as little data as possible should be collected. Additionally the online and offline events can fire for just a split second of network loss, which a user probably wouldn't even see or notice.
The analytics tracking of offline activity would never reach the analytics server because the user is… well, offline.
Tracking a timestamp locally when a user goes offline and sending the offline activity to the analytics server when the user goes back online depends on the user revisiting your site. If the user drops off your site due to a lack of an offline mode and never revisits, you have no way to track that. The ability to track offline drop-offs is critical data for building a case about why your site needs a better offline mode.
The online event is not very reliable as it only knows about network access, not internet access. Therefore a user might still be offline, and sending the tracking ping can still fail.
Even if the user still stays on the current page while being offline, none of the other analytics events (e.g. scroll events, clicks, etc.) are tracked either, which might be the more relevant and useful information.
Being offline in itself is also not too meaningful in general. As a website developer it may be more important to know what kinds of resources failed to load. This is especially relevant in the context of SPAs, where a dropped network connection might not lead to a browser offline error page (which users understand) but more likely to random dynamic parts of the page failing silently.

You can still use this solution to gain a basic understanding of offline usage, but the many drawbacks and limitations need to be considered carefully.

A better approach: the service worker #

The solution that enables offline mode turns out to be the better solution for tracking offline usage. The basic idea is to store analytics pings into IndexedDB as long as the user is offline, and just resend them when the user goes online again. For Google Analytics this is already available off-the-shelf through a Workbox module, but keep in mind that hits sent more than four hours deferred may not be processed. In its simplest form, it can be activated within a Workbox-based service worker with these two lines:

import * as googleAnalytics from 'workbox-google-analytics';

googleAnalytics.initialize();

This tracks all existing events and pageview pings while being offline, but you wouldn't know that they happened offline (as they are just replayed as-is). For this you can manipulate tracking requests with Workbox by adding an offline flag to the analytics ping, using a custom dimension (cd1 in the code sample below):

import * as googleAnalytics from 'workbox-google-analytics';

googleAnalytics.initialize({
  parameterOverrides: {
    cd1: 'offline',
  },
});

What if the user drops out of the page due to being offline, before an internet connection comes back? Even though this normally puts the service worker to sleep (because it's unable to send the data when the connection comes back), the Workbox Google Analytics module uses the Background Sync API, which sends the analytics data later when the connection comes back, even if the user closes the tab or browser.

There is still a drawback: while this makes existing tracking offline-capable, you would most likely not see much relevant data coming in until you implement a basic offline mode. Users would still drop off your site quickly when the connection breaks away. But now you can at least measure and quantify this, by comparing average session length and user engagement for users with the offline dimension applied versus your regular users.

SPAs and lazy loading #

If users visiting a page built as a multi-page website go offline and try to navigate, the browser's default offline page shows up, helping users understand what is happening. However, pages built as single-page applications work differently. The user stays on the same page, and new content is loaded dynamically through AJAX without any browser navigation. Users do not see the browser error page when going offline. Instead, the dynamic parts of the page render with errors, go into undefined states, or just stop being dynamic.

Similar effects can happen within multi-page websites due to lazy loading. For example, maybe the initial load happened online, but the user went offline before scrolling. All lazy loaded content below the fold will silently fail and be missing.

As these cases are really irritating to users, it makes sense to track them. Service workers are the perfect spot to catch network errors, and eventually track them using analytics. With Workbox, a global catch handler can be configured to inform the page about failed requests by sending a message event:

import { setCatchHandler } from 'workbox-routing';

setCatchHandler(({ event }) => {
  // https://developer.mozilla.org/docs/Web/API/Client/postMessage
  event.waitUntil(async function () {
    // Exit early if we don't have access to the client.
    // Eg, if it's cross-origin.
    if (!event.clientId) return;

    // Get the client.
    const client = await clients.get(event.clientId);
    // Exit early if we don't get the client.
    // Eg, if it closed.
    if (!client) return;

    // Send a message to the client.
    client.postMessage({
      action: "network_fail",
      url: event.request.url,
      destination: event.request.destination
    });

    return Response.error();

  }());
});

Rather than listening to all failed requests, another way is to catch errors on specific routes only. As an example, if we want to report errors happening on routes to /products/* only, we can add a check in setCatchHandler which filters the URI with a regular expression. A cleaner solution is to implement registerRoute with a custom handler. This encapsulates the business logic into a separate route, with better maintainability in more complex service workers:

import { registerRoute } from 'workbox-routing';
import { NetworkOnly } from 'workbox-strategies';

const networkOnly = new NetworkOnly();
registerRoute(
  new RegExp('https:\/\/example\.com\/products\/.+'),
  async (params) => {
    try {
      // Attempt a network request.
      return await networkOnly.handle(params);
    } catch (error) {
      // If it fails, report the error.
      const event = params.event;
      if (!event.clientId) return;
      const client = await clients.get(event.clientId);
      if (!client) return;

      client.postMessage({
        action: "network_fail",
        url: event.request.url,
        destination: "products"
      });

      return Response.error();
    }
  }
);

As a final step, the page needs to listen to the message event, and send out the analytics ping. Again, make sure to buffer analytics requests that happen offline within the service worker. As described before, initialize the workbox-google-analytics plugin for built-in Google Analytics support.

The following example uses Google Analytics, but can be applied in the same way for other analytics vendors.

if ("serviceWorker" in navigator) {
  // ... SW registration here

  // track offline error events
  navigator.serviceWorker.addEventListener("message", event => {
    if (gtag && event.data && event.data.action === "network_fail") {
      gtag("event", "network_fail", {
        event_category: event.data.destination,
        // event_label: event.data.url,
        // value: event.data.value
      });
    }
  });
}

This will track failed resource loads in Google Analytics, where they can be analyzed with reporting. The derived insight can be used to improve service worker caching and error handling in general, to make the page more robust and reliable under unstable network conditions.

Next steps #

This article showed different ways of tracking offline usage with their advantages and shortcomings. While this can help to quantify how many of your users go offline and run into problems due to it, it's still just a start. As long as your website does not offer a well-built offline mode, you obviously won't see much offline usage in analytics.

We recommend to get the full tracking in place, and then extend your offline capabilities in iterations with an eye on tracking numbers. Start with a simple offline error page first–with Workbox it's trivial to do–and should be considered a UX best practice similar to custom 404 pages anyway. Then work your way towards more advanced offline fallbacks and finally towards real offline content. Make sure you advertise and explain this to your users well, and you will see increasing usage. After all, everyone goes offline every once in a while.

Check out How to report metrics and build a performance culture and Fixing website speed cross-functionally for tips on persuading cross-functional stakeholders to invest more in your website. Although those posts are focused on performance, they should help you get general ideas about how to engage stakeholders.

Hero photo by JC Gellidon on Unsplash.

How ZDF created a video PWA with offline and dark mode

2020-10-07T00:00:00Z

When broadcaster ZDF was considering redesigning their frontend technology stack, they decided to take a closer look at Progressive Web Apps for their streaming site ZDFmediathek. Development agency Cellular took on the challenge to build a web-based experience that is on par with ZDF's platform-specific iOS and Android apps. The PWA offers installability, offline video playback, transition animations, and a dark mode.

Adding a service worker #

A key feature of a PWA is offline support. For ZDF most of the heavy lifting is done by Workbox, a set of libraries and Node modules that make it easy to support different caching strategies. The ZDF PWA is built with TypeScript and React, so it uses the Workbox library already built into create-react-app to precache static assets. This lets the application focus on making the dynamic content available offline, in this case the videos and their metadata.

The basic idea is quite simple: fetch the video and store it as a blob in IndexedDB. Then during playback, listen for online/offline events, and switch to the downloaded version when the device goes offline.

Unfortunately things turned out to be a little more complex. One of the project requirements was to use the official ZDF web player which doesn't provide any offline support. The player takes a content ID as input, talks to the ZDF API, and plays back the associated video.

This is where one of the web's most powerful features comes to the rescue: service workers.

The service worker can intercept the various requests done by the player and respond with the data from IndexedDB. This transparently adds offline capabilities without having to change a single line of the player's code.

Since offline videos tend to be quite large, a big question is how many of them can actually be stored on a device. With the help of the StorageManager API the app can estimate the available space and inform the user when there is insufficient space before even starting the download. Unfortunately Safari isn't on the list of browsers implementing this API and at the time of writing there wasn't much up-to-date information about how other browsers applied quotas. Therefore, the team wrote a small utility to test the behavior on various devices. By now a comprehensive article exists that sums up all the details.

Adding a custom install prompt #

The ZDF PWA offers a custom in-app installation flow and prompts users to install the app as soon as they want to download their first video. This is a good point in time to prompt for install because the user has expressed a clear intention to use the app offline.

Custom install prompt being triggered when downloading a video for offline consumption.

Building an offline page to access downloads #

When the device is not connected to the internet and the user navigates to a page that is not available in offline mode, a special page is shown instead that lists all videos that have previously been downloaded or (in case no content has been downloaded yet) a short explanation of the offline feature.

Offline page showing all content available for watching offline.

Using frame loading rate for adaptive features #

To offer a rich user experience the ZDF PWA includes some subtle transitions that happen when the user scrolls or navigates. On low-end devices such animations usually have the opposite effect and make the app feel sluggish and less responsive if they don't run at 60 frames per second. To take this into account the app measures the actual frame rate via requestAnimationFrame() while the application loads and disables all animations when the value drops below a certain threshold.

const frameRate = new Promise(resolve => {
  let lastTick;
  const samples = [];

  function measure() {
    const tick = Date.now();
    if (lastTick) samples.push(tick - lastTick);
    lastTick = tick;
    if (samples.length < 20) requestAnimationFrame(measure);
    else {
      const avg = samples.reduce((a, b) => a + b) / samples.length;
      const fps = 1000 / avg;
      resolve(fps);
    }
  }
  measure();
});

Even if this measurement provides only a rough indication of the device's performance and varies on each load, it was still a good basis for decision-making. It's worth mentioning that depending on the use case there are other techniques for adaptive loading that developers can implement. One great advantage of this approach is that it is available on all platforms.

Dark mode #

A popular feature for modern mobile experiences is dark mode. Especially when watching videos in low ambient light many people prefer a dimmed UI. The ZDF PWA not only provides a switch that allows users to toggle between a light and a dark theme, it also reacts to changes of the OS-wide color preferences. This way the app will automatically change its appearance on devices that have set up a schedule to change the theme base on the time of day.

Results #

The new progressive web app was silently launched as a public beta in March 2020 and has received a lot of positive feedback since then. While the beta phase continues, the PWA still runs under its own temporary domain. Even though the PWA wasn't publicly promoted there is a steadily growing number of users. Many of these are from the Microsoft Store which allows Windows 10 users to discover PWAs and install them like platform-specific apps.

What's next? #

ZDF plans to continue adding features to their PWA, including login for personalization, cross-device and platform viewing, and push notifications.

Relating site speed and business metrics

2020-07-17T00:00:00Z

Over the last few years it has been well established that site speed performance is a significant part of the user experience and that improving it benefits different business metrics such as conversion rates and bounce rates. Multiple articles and case studies have been published to back this up, including Cloudflare's How Website Performance Affects Conversion Rates, Deloitte's Milliseconds Make Millions, and Shopping for Speed on eBay.com, to name a few.

Even though the case for speed is clear, many companies still struggle with prioritizing work that will improve their site speed since they do not know exactly how it affects their users and as a result their business.

In the absence of data, it is easy to delay site speed work and focus on other tasks. A common scenario is having some people in the company recognize the importance of site speed and yet not be able to build a case for it and convince multiple stakeholders to invest accordingly.

This article provides high-level guidance on how to leverage A/B testing to evaluate the impact of site speed on business metrics therefore enabling more effective decision-making on the matter.

Step 1: Pick a page to A/B test #

Your aim is to test the hypothesis that page speed relates to your business metrics. For simplicity's sake, you can initially limit yourself to identifying a single page for analysis. Future work can expand to multiple pages of the same type to verify findings, and then to other areas of the site. Some suggestions for where to start are provided at the bottom of this step. Several requirements drive the page selection process:

The A/B test should only be run on the devices of mobile users. Globally, the partner sites we assist see an average of more than 50% (and growing!) of their traffic coming from mobile, but this can increase significantly based on region and vertical. Mobile devices are more sensitive to slower websites due to processing and memory constraints and less stable networks. Also, on-the-go usage patterns mean expectations for speed are higher.
The page you choose for testing should be an important part of your conversion funnel. Every site has a different purpose, so every site tracks different success metrics. These metrics are usually related to a user journey which is analyzed using a funnel. For example, users on an e-commerce website will have to navigate through a home page, category pages, product pages, and a checkout page to complete a purchase. If you're optimizing for conversions, one of those pages would be a good candidate.
The page should have a singular purpose. Unless your site has a very specific mission, it's usually best to avoid using the homepage for your test. Many commercial sites' homepages are portals to a wide variety of functionality that will make your analysis noisy. For example, if you're optimizing for pageviews per session on a news site, it might be best to exclude the non-commercial portions of the site and focus on monetized sections and articles.
The page chosen should get enough traffic so that you don't have to wait long before getting a statistically significant result.
The selected page should be relatively slow. In fact, the slower the better. This not only means that you will likely have an easier time improving the page, it also means that the data should be much clearer. You can do a quick scan through your Google Analytics Speed Report or Search Console Core Web Vitals report to see which of your pages are slowest.
The page should be relatively stable. Don't update pages (anything that would impact business metrics) until the test is complete. The fewer external factors there are to consider, the cleaner the analysis will be.

Using the above as a guide it should be a bit more clear which pages are good candidates for your test. Ad landing pages are also a good choice, because you're likely to have built-in business metrics, A/B testing, and analysis at your disposal. In case you're still unsure, here are some ideas per vertical:

Content Websites: sections, articles
Storefronts: category pages, product pages
Media Players: video discovery/search pages, video play page
Services & Discovery (travel, rental cars, service registration, etc.): initial form-entry page

Step 2: Measure performance #

There are two general ways to measure metrics: in the lab and in the field. We personally have found measuring metrics in the field (also known as Real User Monitoring (RUM)) to be more useful because it reflects the experience of real users. Examples of libraries and services that can help you report RUM data include Perfume, Firebase Performance Monitoring, and Google Analytics Events.

There are many metrics to choose from because they aim to capture different aspects of the user experience. Remember that your goal is to eventually determine if there's a direct correlation between your speed and business metrics, so it might be useful to track a few speed metrics in order to determine which one has the strongest correlation with your business success. In general we recommend starting with the Core Web Vitals. The web-vitals.js library can help you measure some of the Core Web Vitals in the field, although note that browser support isn't 100%. Beyond the Core Web Vitals, the Other Web Vitals are also worth checking out. You can also define custom metrics, such as "Time To First Ad Click".

Step 3: Create speed performance variants #

In this stage you will implement changes to create a faster version of the page to be tested against the current version.

A couple of things to keep in mind:

Avoid making any changes to the UI or UX. Aside from one page being faster than the other, changes must be invisible to users.
Measuring is also a key aspect of this stage. While developing, lab testing tools such as Lighthouse should be used to indicate the effect your changes have on performance. Keep in mind that changes to one metric can often influence another. Once the pages are live, stick to RUM for a more accurate evaluation.

Creating performance variants can be done in different ways. For the purpose of the test, you would want to do so as simply as possible. Below are a few options.

Create a faster page

Use a tool like Squoosh to manually optimize the images on your test page
Use DevTools code coverage to manually eliminate unused JavaScript or CSS just for that one page
Efficiently load third-party scripts
Use a tool like Critical to break out and inline critical CSS
Remove non-critical JavaScript code that does not impact the user experience and which you can do without for the purpose of the test (for example, certain third-party libraries)
Implement browser-level lazy loading which isn't supported by all browsers but may still improve performance significantly where supported
Remove non-critical analytics tags or load them asynchronously

Additional optimizations to consider can be found at Fast load times and Frontend Performance Checklist. You can also use PageSpeed Insights to run Lighthouse, which identifies opportunities to improve performance.

Slow down the page

This may be the easiest way to create variants and can be achieved by adding a simple script, slowing down server response times, loading larger images, etc. The Financial Times opted for this option when testing how performance impacted their business metrics: see A faster FT.com.

Speed up page load

For cases where the test page (let's say a product detail page) is mostly linked out from a different page (let's say the homepage), prefetching or prerendering the product page directly from the homepage for the test group will speed up the subsequent load of the page. Note that in this case the A/B test split (step 4) is done on the homepage. In addition, all of this may slow down the first page so be sure to measure that and take it under consideration when analyzing the test results (step 5).

Step 4: Create an A/B test #

Once you have two versions of the same page where one is faster than the other, the next step is to split traffic to measure impact. In general there are many techniques and tools to perform A/B tests, but be aware that not all methods are well suited to measure speed performance impact.

If you're using an A/B testing tool like Optimizely or Optimize we strongly recommend setting up a server-side test instead of a client-side test, as client side A/B testing often works by hiding page content until the experiment is loaded, which would mean the A/B testing itself would skew the metrics you wanted to measure. If you can only do client-side testing, consider setting up the experiment on a different page and changing the link outs to your test page to split the traffic. This way the test page itself isn't dragged down by the client-side test.

Example

Example of AB testing performance changes on a given product detail page (PDP) via server-side testing:

The request goes to the backend, which distributes the users to the two different versions of the page. While this is in general a good setup, it often needs IT resources to set up the server-side split.

Here is an example of a client side testing setup, using the prior page (the homepage in the diagram below) to run the testing JavaScript:

The testing JavaScript manipulates the outgoing link to give the two test groups of users links to the two versions of the PDP in question. This is easy to setup and maintain through common A/B testing tools like Optimizely or Optimize and does not influence the performance test because the testing JavaScript runs on a different page.

Alternatively, you can pick two pages which behave and perform very similarly (for example, for two very similar products). Apply your changes to one of them and then compare the difference in metrics over time. This will mean you're not conducting a proper A/B test, yet it can still be quite insightful.

In case your test page is used as a landing page for ad campaigns, it can be handy to use your ad network's built-in A/B testing tools, like Facebook Ads Split Test or Google Ads Drafts & Experiments. If that's not an option, you can use two campaigns with the same setup, and set different landing pages as targets.

Step 5: Analyze A/B test #

After you've run your test for long enough and have enough data to feel confident about the results, it's time to put it all together and run an analysis. How you do that really depends on how the test has run, so let's go through the options.

If your test was run on ad landing pages using the above-mentioned tools, analysis should be as straightforward as reading a result. If you're using Google's Drafts & Experiments, take a look at the comparison using the ScoreCard.

Platforms like Optimizely or Optimize also offer easy ways to interpret the results and determine how much of an impact speed has on your pages.

If you're using Google Analytics or a similar tool, you will have to pull together the report yourself. Luckily, Google Analytics makes it pretty easy to build custom reports, so that's where you should start. If you've sent speed data to Google Analytics using a custom dimension, check out the Reporting guide to learn how to set those up and include them in your custom reports. Make sure your report covers the date of the experiment and is configured to display both variants. What should go in this report?

Primarily, you need to include the business metrics that you most care about: conversions, page views, ads viewed, conversion rate, e-commerce metrics, click-through rate, etc.
Additionally, other standard page metrics that also make a good case for improving site speed are bounce rate, average session duration, and exit percentage.

You may also need to filter for mobile and make sure to exclude bots and other non-user traffic. More advanced analysis would also filter by region, networks, devices, traffic source, and user profiles and types, such as new users versus repeat visitors. Each group of users may be more or less sensitive to slower speeds and identifying these is also quite helpful.

Looker Studio (formerly Data Studio) or other data visualization tools make it easy to integrate various data sources including Google Analytics. This makes it easy to conduct analysis, and also create dashboards that are shareable with the many stakeholders involved in running a modern website for further buy-in. For example, the Guardian created an automated alert system that warned the editorial team when recently published content crossed their page size or speed thresholds and was likely to result in unsatisfied users.

Step 6: Draw conclusions and decide on next steps #

Once you have data that connects performance and business metrics, you can examine the results and start to draw conclusions.

If you can clearly see a correlation between improving performance and improving business metrics, summarize the results and report them across the company. Now that you can talk about speed performance in a "business language" you are more likely to grab the attention of different stakeholders and place site speed performance on everyone's radar. The next step is to set performance budgets based on the results, and plan out work to meet those budgets. Since you know the value such work will provide, you'll be able to prioritize accordingly.

If you can't identify a correlation, take a look at the caveats below and assess whether similar tests should be run elsewhere on the site (for example, through the entire purchase funnel or on a different type of page).

Caveats

There might be several reasons for not finding significant correlation between site speed metrics and business metrics:

The page chosen does not have enough influence on the business metrics you're examining. For example, a faster product page may not have a big effect on conversion rates if the checkout page is very unfriendly or slow. Consider looking at more relevant metrics such as bounce rates, add-to-basket rates or any other metric that is more directly connected to the page you're testing.
The difference in speed is not significant enough between the two versions. This should be evaluated according to the RUM metrics you're measuring.
There's a fault with the A/B testing mechanism. Traffic might not be distributed properly or analytics not reported correctly. In order to rule this out, consider running an A/A test where you test the same version of a page using the same testing mechanism and ensure there's no difference in results when doing so.
Site speed really doesn't influence your business metrics. This is rare but can occur in cases where your target market is less sensitive to speed (e.g. the site is mostly accessed from powerful devices on a strong network) or user demand is very high and choice is limited (e.g. a ticketing service that exclusively sells tickets for high-demand shows). Note that this does not mean that a faster site will not improve user experience and as a result affect brand reputation.

Conclusion #

While it is tempting to launch speed optimizations across the entire site, it is usually more beneficial in the long run to first understand what it means to your users and your company to have a faster website. It is the difference between saying "we improved FCP by 1.5 seconds" and "we improved FCP by 1.5 seconds and that improved our conversion rates by 5%". This will allow you to prioritize further work, get buy-in from different stakeholders and make site speed performance a company-wide effort.

How can performance improve conversion?

2019-06-11T00:00:00Z

In our other e-commerce guides you have learned about what you should measure to improve performance, and how to measure and report metrics to build a performance culture.

A conversion funnel.

In this guide we'll address the different ways in which a website should be optimized for performance to yield maximum conversions at the end of the funnel.

Discovery #

New users discover a website in most cases through organic search, social sharing, website links or paid campaigns. Some important discovery mechanisms are directly affected by website performance. Website crawlers may have difficulty indexing sites that are slow to load or have extensive client side rendering and JavaScript.

Speed can also be a direct ranking factor, for example on web search, ad campaigns or social networks. Keep in mind that new users who discover your website will get an uncached first load, so basically the worst possible experience. This can be especially frustrating if good money was spent to get the user to the website, just to see them dropping off due to a long first load.

Make sure to use appropriate tools as described in Fast Load Times to optimize towards a first load, because first impressions matter—if the first load is too slow, the user may never see the optimized second load or stay around to look at your products. In general loading times of a website map very well to bounce rates, which in turn often correlate well with conversions.

Engagement #

After getting users to your site, you need to keep them engaged with your content, which you can verify in your analytics of choice by looking at session length, time-on-page, pages-per-session and general user flows.

A user flow through the funnel as seen by Google Analytics.

Besides various UX best practices, a smooth, fast and responsive experience is key here. While optimizing a website for discovery means optimizing for first load, optimizing for engagement means fast navigations and fast repeat loads. Analyze at which steps of the flow users drop out, and then relate back to speed metrics for these navigations. This can be analyzed for example via WebPageTest, Puppeteer or via the Chrome DevTools Record feature. We will show you more examples of those in the following guides.

Conversion #

While website conversions are often bound to good discovery paired with great engagement, there are a couple of additional points to remember. Users expect hero images to load fast, call-to-action buttons should be rendered and labeled quickly, page should be responsive and layout jumps should be avoided. A user won't buy anything if they can't click the Buy Now button due to busy CPU or a jumping or unlabeled button. In general it's best to measure and track the time-to-action towards a conversion or a subgoal for it, for example the median time it takes shoppers to get from landing on your site, to viewing a product, to completing payment .

Re-Engagement #

It turns out that only 2% of users convert on first visit, so it is important to get the other 98% to come back, and re-engage them with your content. Modern websites have different ways to do this, for example via mail, tailored display ads in remarketing or notifications. This works best if the flow from reengagement to website is as smooth as possible. Unfortunately this is not always the case, as for example mail apps often open links in their in-app webview, slowing down page load and complicating logins through different cache and cookie storage. Make sure to optimize for fast repeat loads and smooth UX flows to increase chances of reengagement.

Recap #

E-commerce sites always strive for conversions, which are at the end of a purchase funnel. Every step along the funnel needs to be optimized for website speed to minimize bounce rates and drop-offs, and for each step there are different things to optimize, different pitfalls and culprits:

An e-commerce funnel showing which metric to optimize in which step.

To learn more, be sure to check out the other posts in this series on measuring to improve performance, and how report metrics to build a performance culture.

Hero image by Campaign Creators on Unsplash.

What should you measure to improve performance?

2019-05-31T00:00:00Z

The different steps of a purchase funnel are prone to performance issues in different ways, and therefore need different measurement and optimizations:

A conversion funnel.

In this guide we'll address how the different steps can be measured. For this we recommend you to look at lab as well as field data.

Lab data is gathered by running tests locally, for example by using Lighthouse and other tools. This can make it possible to compare website performance over time and with competitors through a controlled, stable environment, but it might not be representative of the performance users experience in real life.

Field data is gathered via analytics from real users, and is therefore representative of their experience. However, it can't easily be compared over time or with competitors. Network connections and smartphone hardware evolve over time, and different target audiences might have different devices, thereby making comparisons with field data tough. See also Measure Performance In The Field.

To get a complete picture, both data sources are needed. The following sections show how to gather lab and field data for the different relevant performance marks across the funnel.

Discovery #

Optimizing for discovery means optimizing for the first load, which is what new users will get, but also search and social crawlers. Lab data for a first load can be easily acquired through Lighthouse, while field data (at least for Chrome) is readily available through Chrome UX reports. A convenient combination of both can be found in PageSpeed Insights. You should also track relevant metrics from the field yourself: Measuring these metrics on real users' devices provides a good overview.

From a user perspective the most important metrics are:

First Contentful Paint (FCP): The time the user stares at a blank screen. This is when most users bounce, as they don't see progress.
First Meaningful Paint (FMP): When the user begins to see the main content they came for. This is often the hero image, but for a landing page it may even be a call to action such as a Buy button, since the user may have arrived with a clear intent (for example, through a targeted ad campaign).
First Input Delay (FID): The time the website needs to react to the user's first input. Excessive JavaScript and other asset loading problems can block this, leading to failed taps or clicks, erroneous inputs and page abandonment.

There are more metrics you can look at, but these are a good baseline. Additionally also make sure to capture bounce rates, conversions and user engagement so that you can set these in relation.

Engagement And Conversion #

After the first load of a landing page, a user will proceed through your site, hopefully towards a successful conversion.

In this phase it is important to have fast and responsive navigations and interactions. Unfortunately it is not trivial to measure the complete flow of navigation and interaction events in the field, as every user takes different paths through the page. We therefore recommend measuring the time needed towards conversion or conversion subgoals ("Time-to-Action") by scripting and measuring the flow in a lab test, to compare performance over time or with competitors.

There are two convenient ways of doing this:

WebPageTest #

WebPageTest offers a very flexible scripting solution. The basic idea is to:

Tell WebPageTest to navigate through the pages of the flow with the navigate command.
If needed script the clicking of buttons via clickAndWait commands and fill text fields via setValue. For testing of Single Page Applications use clickAndWait rather than navigate commands for all steps after the first, as navigate will do a full load instead of the lightweight virtual page load.
Make sure to combine the different steps of the flow in the analysis via combineSteps to produce a single overall result report for the complete flow.

Such a script could look like this:

combineSteps
navigate	https://www.store.google.com/landingpage
navigate	https://www.store.google.com/productpage
clickAndWait	innerText=Buy Now
navigate	https://www.store.google.com/basket
navigate	https://www.store.google.com/checkout

With a script like this in place you can easily measure and compare performance over time. This can even be automated through the WebPageTest API.

Puppeteer #

Another great option to script testing is via headless Chrome, which can be controlled through the Node API Puppeteer. The general idea is to start the browser through Puppeteer, navigate to the landing page through the goto function, inject JavaScript to fill fields or click buttons and proceed through the funnel through further goto calls as needed.

As a metric the duration of the flow can be measured directly, but you could also sum up the FCP, FMP or TTI values of the individual loads of the flow. Test website performance with Puppeteer provides an overview of how to get performance metrics via Puppeteer. A very simplified example Node script could look like this:

const puppeteer = require('puppeteer');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  const start = performance.now();
  await page.goto('https://www.store.google.com/landingpage');
  await page.goto('https://www.store.google.com/productpage');
  // click the buy button, which triggers overlay basket
  await page.click('#buy_btn');
  // wait until basket overlay is shown
  await page.waitFor('#close_btn');
  await page.goto('https://www.store.google.com/basket');
  await page.goto('https://www.store.google.com/checkout');
  console.log('Flow took ' + parseInt((performance.now() - start)/1000) + ' seconds');
  await browser.close();
})();

This script can easily be automated, and even be made part of the build process or perf budgets, and be monitored regularly.

Re-Engagement #

Users will return to your site in different time intervals. Depending on time passed, the browser may have less of the website's resources cached, needing more network requests. This makes it difficult to estimate performance differences across repeat visits in lab tests. It is still recommended to keep an eye on this though, and a great lab test tool for repeat visits is WebPageTest, which has a dedicated option for a direct repeat visit:

Webpagetest offers options to test first load and repeat load as well

To get a better feeling for repeat visit performance in the field use your analytics package of choice to segment your performance metrics by user type. Here is an example of such a report in Google Analytics:

A Google Analytics custom report can be used to report speed metrics for new and returning users.

A report like this will give you page load times for new and returning users as well.

Recap #

This guide showed you how to measure first load, flow and repeat load via field and lab tests. Make sure to optimize the different steps of the funnel accordingly to maximize discovery (first load), engagement (navigations and flow) and re-engagement (repeat load).

How to report metrics and build a performance culture

2019-05-05T00:00:00Z

Introduction #

Website performance is a key goal towards e-commerce conversions, so it must be a core priority for business stakeholders and leadership, not just development teams.

For that reason, it's crucial to make performance metrics visible and tangible for all stakeholders, and to build reporting and monitoring into your workflow.

This guide describes what to be aware of and what to avoid to achieve this successfully.

A word about metrics #

There are countless metrics to evaluate website performance, and while it may seem useful to collect all of them, too many metrics can be confusing and misleading. There are several ways to deal with this:

Gather multiple metrics and try to narrow and filter afterwards on what might be relevant for the task at hand.
Abstract metrics into an overall score, as for example Lighthouse does. This can be especially useful for non-technical staff and other stakeholders, but is probably insufficient for deeper technical analysis.
Try to find the one metric which is most relevant as a predictor for your conversions and then optimize towards this.

In reality, a pragmatic combination makes the most sense here. Gather more rather than less to begin with, report an abstract score to management, and optimize towards the one metric which best predicts your conversions.

Finding this metric can be done by using your analytics tool of choice to map performance metrics to user engagement, conversions and transaction values. For example, a custom report to do so looks like this in Google Analytics:

Fig 1: Custom report in Google Analytics to analyze the impact of speed on conversions and engagement.

Increasing bounce rate #

Problem #

Often it is assumed that bounce rate will drop when page load speed increases. While this normally does hold true, measurements sometimes show the opposite. This is because analytics can only measure a bounce after the analytics library is loaded. A faster page load means analytics code also loads faster, so analytics may see more bounces even if there aren't more happening.

Solution #

This can be eased by measuring real page abandonment instead.

Decreasing relative conversions #

Problem #

Relative conversions may sometimes seem to drop for faster sites. This is because faster pages reach a bigger audience who might be less engaged or committed. While incremental traffic and conversions increase with faster pages, relative conversions (the ratio of conversions to page views or visitors) might still drop.

Solution #

This effect can be eased by looking out for absolute conversions instead, and even calculate Cost Per Sales (conversions divided with investment level) or ROI.

Dropping engagement #

Problem #

Page engagement may seem to drop for a faster page.

Solution #

This can actually be a positive signal! If the page is faster users can reach their goal more quickly and might just have shorter sessions and less time on page.

Using averages or medians #

Problem #

In general, it can be deceiving to look at averages and medians of performance metrics (see this nicely described in Everything You Know About Latency Is Wrong). Similar to a chain being just as strong as the weakest link, the performance of a funnel is only as good as its slowest load. A single slow load may be enough to lose the user. Therefore averages and medians are more likely to hide the real performance issues, than to reveal them.

Solution #

It's best to analyze complete distributions, but as this is not always easily possible we recommend using 90th percentiles—this is the value for which 90% of loads were faster. Even then a user hitting ten pages will still have one load slower than this, and might drop out there.

While performance measurement is highly important, make sure to keep an open mind and question unexpected results—and make sure to not report misleading numbers to stakeholders and management. If unsure on what to pick and report, we'd advise as a minimum for 90th percentile First Contentful Paint, which is also what we use across our public tooling.

Third party content #

Website performance is especially prone to being dragged down by third party content (see Eliminate render-blocking resources). This is a particular problem for e-commerce, often due to trackers and widgets.

Some ways to handle third party content with respect to performance:

Always keep third party content out of the critical rendering path. If the third party has server problems and times out, it will impact your website heavily. You can test and simulate this via WebPageTest Single-Point-of-Failure tests.
Continually measure and report the ratio of third party content, for example via WebPageTest domain breakdown. Make sure to use performance budgets for third party content.
If you suspect that a particular third party component is having a detrimental effect on performance, do a performance comparison with it included and excluded. Find out how.
Try to stay on one stack from one vendor if possible. For example, if you have a tag manager and analytics on one stack, you may only need a single script, and may be able to take advantage of HTTP2 synergies as there is only one host involved.
Make sure not to use the same functionality from two different vendors. You shouldn't need two tag managers or two analytics platforms.
Routinely audit and clean out redundant third party scripts, trackers and widgets. This can easily be done via Ghostery Extension or tools like WhatRuns:

A Ghostery report showing all loaded trackers

Learn more in Loading Third-Party JavaScript.

Performance Culture #

Unfortunately performance is often seen as a one-off optimization task, and then regresses over time as stakeholders raise new feature requests or insist on adding new trackers and widgets.

Performance must be a continuous goal to improve acquisition, discovery, and conversion rates as well as safeguarding the reputation of your brand. This can be achieved with performance budgets like Tinder did, and by establishing and fostering a performance culture, where all employees and especially decision makers recognize speed as a core feature of the website.

Make metrics tangible #

Reports on metrics are often abstract and easy to challenge or dismiss. It's best to make your performance tangible and visible. There are several good ways to do this:

Facebook and Google do this by providing slow networks across the company for testing.
Make average, low-spec devices with low bandwidth or high latencies available to management and other stakeholders.
Consider adding overlays showing performance metrics on your development or staging servers. Connections to these servers from mobile can potentially be throttled by default on corporate networks.
Monitors can be placed strategically across the company showing videos or timestrips of your website's loading behavior, preferably in comparison to competitors. WebPageTest can create these very easily and automatically.
Performance can also be fun—maybe a game based on an individual Lighthouse report can reach audiences which can't be reached by pure reports and metrics?

Recap #

This guide explains why the selection of metrics, and how they are reported and handled, is as important as the measurement and optimization itself—or even more. Make sure to prefer percentiles or distributions over averages, be cautious of using just bounce rate or relative conversion rate as impact measures, and make sure that metrics are easy to understand and tangible for stakeholders across the company. Establishing a performance culture is also an important step towards a well performing e-commerce site.

Martin Schierle on web.dev

How committing to Core Web Vitals increased Netzwelt's advertising revenues by 18%

The challenge #

Optimizing a news website for Core Web Vitals #

Optimizing CLS #

Optimizing FID #

Optimizing LCP #

GDPR Consent and Core Web Vitals #

News Subscription Models and Core Web Vitals #

Optimization Results #

Looking into the future #

Measuring offline usage

The pitfalls of the online and offline browser events #

A better approach: the service worker #

SPAs and lazy loading #

Next steps #

How ZDF created a video PWA with offline and dark mode

Adding a service worker #

Adding a custom install prompt #

Building an offline page to access downloads #

Using frame loading rate for adaptive features #

Dark mode #

Results #

What's next? #

Relating site speed and business metrics

Step 1: Pick a page to A/B test #

Step 2: Measure performance #

Step 3: Create speed performance variants #

Step 4: Create an A/B test #

Step 5: Analyze A/B test #

Step 6: Draw conclusions and decide on next steps #

Conclusion #

How can performance improve conversion?

Discovery #

Engagement #

Conversion #

Re-Engagement #

Recap #

What should you measure to improve performance?

Discovery #

Engagement And Conversion #

WebPageTest #

Puppeteer #

Re-Engagement #

Recap #

How to report metrics and build a performance culture

Introduction #

A word about metrics #

Increasing bounce rate #

Problem #

Solution #

Decreasing relative conversions #

Problem #

Solution #

Dropping engagement #

Problem #

Solution #

Using averages or medians #

Problem #

Solution #

Third party content #

Performance Culture #

Make metrics tangible #

Recap #