Jonathan Chen on web.dev

Automating audits with AutoWebPerf

2020-12-09T00:00:00Z

What is AutoWebPerf (AWP)? #

AutoWebPerf (AWP) is a modular tool that enables automatic gathering of performance data from multiple sources. Currently there are many tools available to measure website performance for different scopes (lab and field), such as Chrome UX Report, PageSpeed Insights, or WebPageTest. AWP offers integration with various audit tools with a simple setup so you can continuously monitor the site performance in one place.

The release of Web Vitals guidance means that close and active monitoring of web pages is becoming increasingly important. The engineers behind this tool have been doing performance audits for years and they created AWP to automate a manual, recurring, and time consuming part of their daily activities. Today, AWP has reached a level of maturity and it's ready to be shared broadly so anyone can benefit from the automation it brings.

The tool is accessible on the AutoWebPerf public repository on GitHub.

What is AWP for? #

Although several tools and APIs are available to monitor the performance of web pages, most of them expose data measured at a specific time. To adequately monitor a website and maintain good performance of key pages, it's recommended to continuously take measurements of Core Web Vitals over time and observe trends.

AWP makes that easier by providing an engine and pre-built API integrations which can be programmatically configured to automate recurrent queries to various performance monitoring APIs.

For example, with AWP, you can set a daily test on your home page to capture the field data from CrUX API and lab data from a Lighthouse report from PageSpeed Insights. This data can be written and stored over time, for example, in Google Sheets and then visualised in the Data Studio dashboard. AWP automates the heavy-lifting part of the entire process, making it a great solution to follow lab and field trends over time. See Visualising audit results in Data Studio below for more details).

Architecture overview #

AWP is a modular-based library with three different types of modules:

the engine
connector modules
gatherer modules

The engine takes a list of tests from a connector (for example, from a local CSV file), runs performance audits through selected gatherers (such as PageSpeed Insights), and writes results to the output connector (for example, Google Sheets).

AWP comes with a number of pre-implemented gatherers and connectors:

Pre-implemented gatherers:
Pre-implemented connectors:
- Google Sheets
- JSON
- CSV

Automating audits with AWP #

AWP automates the performance audits via your preferred audit platforms such as PageSpeed Insights, WebPageTest, or CrUX API. AWP offers the flexibility to choose where to load the list of tests, and where to write the results to.

For example, you can run audits for a list of tests stored in a Google Sheet, and write the results to a CSV file, with the command below:

PSI_APIKEY=<YOUR_KEY> SHEETS_APIKEY=<YOUR_KEY> ./awp run sheets:<SheetID> csv:output.csv

Recurring audits #

You can run recurring audits in daily, weekly, or monthly frequency. For example, you can run daily audits for a list of tests defined in a local JSON like below:

{
  "tests": [
    {
      "label": "web.dev",
      "url": "https://web.dev",
      "gatherer": "psi"
    }
  ]
}

The command below reads the list of audit tests from the local JSON file, runs audits on a local machine, then outputs results to a local CSV file:

PSI_APIKEY=<YOUR_KEY> ./awp run json:tests.json csv:output.csv

To run audits every day as a background service continuously, you can use the command below instead:

PSI_APIKEY=<YOUR_KEY> ./awp continue json:tests.json csv:output.csv

Alternatively, you can set up the crontab in a Unix-like environment to run AWP as a daily cron job:

0 0 * * * PSI_APIKEY=<YOUR_KEY> ./awp run json:tests.json csv:output.csv

You can find more ways to automate daily audits and result collection in the AWP GitHub repository.

Visualising audit results in Data Studio #

Along with continuously measuring Core Web Vitals, it is important to be able to evaluate the trends and discover potential regressions with real user metrics (RUM) or the Chrome UX Report (CrUX) data collected by AWP. Note that Chrome UX Report (CrUX) is a 28-day moving aggregation, hence it is recommended to also use your own RUM data along with CrUX so you can spot regressions sooner.

Data Studio is a free visualization tool that you can easily load performance metrics into and draw trends as charts. For example, the time series charts below show Core Web Vitals based on Chrome UX Report data. One of the charts shows increasing Cumulative Layout Shift in recent weeks, which means regressions in the layout stability for certain pages. In this scenario, you would want to prioritize the efforts to analyze the underlying issues of these pages.

To simplify the end-to-end process from data collection to visualization, you can run AWP with a list of URLs to automatically export results to Google Sheets with the following command:

PSI_APIKEY=<YOUR_KEY> SHEETS_APIKEY=<YOUR_KEY> ./awp run sheets:<SheetID> csv:output.csv

After collecting daily metrics in a spreadsheet, you can create a Data Studio dashboard that loads the data directly from the spreadsheet, and plots the trends into a time series chart. Check out Google Spreadsheets API Connector for detailed steps about how to set up AWP with spreadsheets as a data source to visualize on Data Studio.

What's next? #

AWP provides a simple and integrated way to minimize the efforts to set up a continuous monitoring pipeline to measure Core Web Vitals and other performance metrics. As for now, AWP covers the most common use cases and will continue to provide more features to address other use cases in the future.

Learn more in the AutoWebPerf repository.

Service worker caching and HTTP caching

2020-07-17T00:00:00Z

While service workers and PWAs are becoming standards of modern web applications, resource caching has become more complex than ever. This article covers the big picture of browser caching, including:

The use cases of and differences between service worker caching and HTTP caching.
The pros and cons of different service worker caching expiry strategies compared to regular HTTP caching strategies.

Overview of caching flow #

At a high-level, a browser follows the caching order below when it requests a resource:

Service worker cache: The service worker checks if the resource is in its cache and decides whether to return the resource itself based on its programmed caching strategies. Note that this does not happen automatically. You need to create a fetch event handler in your service worker and intercept network requests so that the requests are served from the service worker's cache rather than the network.
HTTP cache (also known as the browser cache): If the resource is found in the HTTP Cache and has not yet expired, the browser automatically uses the resource from the HTTP cache.
Server-side: If nothing is found in the service worker cache or the HTTP cache, the browser goes to the network to request the resource. If the resource isn't cached in a CDN, the request must go all the way back to the origin server.

Caching layers #

Service worker caching #

A service worker intercepts network-type HTTP requests and uses a caching strategy to determine what resources should be returned to the browser. The service worker cache and the HTTP cache serve the same general purpose, but the service worker cache offers more caching capabilities, such as fine-grained control over exactly what is cached and how caching is done.

Controlling the service worker cache #

A service worker intercepts HTTP requests with event listeners (usually the fetch event). This code snippet demonstrates the logic of a Cache-First caching strategy.

It's highly recommended to use Workbox to avoid reinventing the wheel. For example, you can register resource URL paths with a single line of regex code.

import {registerRoute} from 'workbox-routing';

registerRoute(new RegExp('styles/.*\\.css'), callbackHandler);

Service worker caching strategies and use cases #

The next table outlines common service worker caching strategies and when each strategy is useful.

Strategies	Freshness rationale	Use cases
Network only	The content has to be up-to-date at all times.	Payments and checkouts Balance statements
Network falling back to cache	It's preferable to serve the fresh content. However if the network fails or is unstable, it's acceptable to serve slightly old content.	Timely data Prices and rates (requires disclaimers) Order statuses
Stale-while-revalidate	It's okay to serve cached content right away, but updated cached content should be used in the future.	News feeds Product listing pages Messages
Cache first, fall back to network	The content is non-critical and can be served from the cache for performance gains, but the service worker should occasionally check for updates.	App shells Common resources
Cache only	The content rarely changes.	Static content

Additional benefits of service worker caching #

In addition to fine-grained control of caching logic, service worker caching also provides:

More memory and storage space for your origin: The browser allocates HTTP cache resources on a per-origin basis. In other words, if you have multiple subdomains, they all share the same HTTP cache. There is no guarantee that the content of your origin/domain stays in the HTTP cache for a long time. For example, a user may purge the cache by manually cleaning up from a browser's settings UI, or triggering a hard-reload on a page. With a service worker cache you have a much higher likelihood that your cached content stays cached. See Persistent storage to learn more.
Higher flexibility with flaky networks or offline experiences: With the HTTP cache you only have a binary choice: either the resource is cached, or not. With service worker caching you can mitigate little "hiccups" much easier (with the "stale-while-revalidate" strategy), offer a complete offline experience (with the "Cache only" strategy) or even something in between, like customized UIs with parts of the page coming from the service worker cache and some parts excluded (with the "Set catch handler" strategy) where appropriate.

HTTP caching #

The first time a browser loads a web page and related resources, it stores these resources in its HTTP cache. The HTTP cache is usually enabled automatically by browsers, unless it has been explicitly disabled by the end user.

Using HTTP caching means relying on the server to determine when to cache a resource and for how long.

Control HTTP cache expiry with HTTP response headers #

When a server responds to a browser request for a resource, the server uses HTTP response headers to tell a browser how long it should cache the resource. See Response headers: configure your web server to learn more.

HTTP caching strategies and use cases #

HTTP caching is much simpler than service worker caching, because HTTP caching only deals with time-based (TTL) resource expiration logic. See Which response header values should you use? and Summary to learn more about HTTP caching strategies.

Designing your cache expiry logic #

This section explains the pros and cons of using consistent expiry logic across the service worker cache and HTTP cache layers, as well as the pros and cons of separate expiry logic across these layers.

The Glitch below demonstrates how service worker caching and HTTP caching work in action across different scenarios:

Consistent expiry logic for all cache layers #

To demonstrate the pros and cons, we'll look at 3 scenarios: long-term, medium-term, and short-term.

Scenarios	Long-term caching	Medium-term caching	Short-term caching
Service worker caching strategy	Cache, falling back to network	Stale-while-revalidate	Network falling back to cache
Service worker cache TTL	30 days	1 day	10 mins
HTTP cache max-age	30 days	1 day	10 mins

Scenario: Long-term caching (Cache, falling back to network) #

When a cached resource is valid (<= 30 days): The service worker returns the cached resource immediately without going to the network.
When a cached resource is expired (> 30 days): The service worker goes to the network to fetch the resource. The browser doesn't have a copy of the resource in its HTTP cache, so it goes server-side for the resource.

Con: In this scenario, HTTP caching provides less value because the browser will always pass the request to the server-side when the cache expires in the service worker.

Scenario: Medium-term caching (Stale-while-revalidate) #

When a cached resource is valid (<= 1 day): The service worker returns the cached resource immediately, and goes to the network to fetch the resource. The browser has a copy of the resource in its HTTP cache, so it returns that copy to the service worker.
When a cached resource is expired (> 1 day): The service worker returns the cached resource immediately, and goes to the network to fetch the resource. The browser doesn't have a copy of the resource in its HTTP cache, so it goes server-side to fetch the resource.

Con: The service worker requires additional cache-busting to override the HTTP cache in order to make the most of the "revalidate" step.

Scenario: Short-term caching (Network falling back to cache) #

When a cached resource is valid (<= 10 mins): The service worker goes to the network to fetch the resource. The browser has a copy of the resource in its HTTP cache so it returns that to the service worker without going server-side.
When a cached resource is expired (> 10 mins): The service worker returns the cached resource immediately, and goes to the network to fetch the resource. The browser doesn't have a copy of the resource in its HTTP cache, so it goes server-side to fetch the resource.

Con: Similar to the medium-term caching scenario, the service worker requires additional cache-busting logic to override the HTTP cache in order to fetch the latest resource from the server-side.

Service worker in all scenarios #

In all scenarios, the service worker cache can still return cached resources when the network is unstable. On the other hand, the HTTP cache is not reliable when the network is unstable or down.

Different cache expiry logic at the service worker cache and HTTP layers #

To demonstrate the pros and cons, we'll again look at long-term, medium-term, and short-term scenarios.

Scenarios	Long-term caching	Medium-term caching	Short-term caching
Service worker caching strategy	Cache, falling back to network	Stale-while-revalidate	Network falling back to cache
Service worker cache TTL	90 days	30 days	1 day
HTTP cache max-age	30 days	1 day	10 mins

Scenario: Long-term caching (Cache, falling back to network) #

When a cached resource is valid in the service worker cache (<= 90 days): The service worker returns the cached resource immediately.
When a cached resource is expired in the service worker cache (> 90 days): The service worker goes to the network to fetch the resource. The browser doesn't have a copy of the resource in its HTTP cache, so it goes server-side.

Pros and cons:

Pro: Users experience instant response as the service worker returns cached resources immediately.
Pro: The service worker has more fine-grained control of when to use its cache and when to request new versions of resources.
Con: A well-defined service worker caching strategy is required.

Scenario: Mid-term caching (Stale-while-revalidate) #

When a cached resource is valid in the service worker cache (<= 30 days): The service worker returns the cached resource immediately.
When a cached resource is expired in the service worker cache (> 30 days): The service worker goes to the network for the resource. The browser doesn't have a copy of the resource in its HTTP cache, so it goes server-side.

Pros and cons:

Pro: Users experience instant response as the service worker returns cached resources immediately.
Pro: The service worker can ensure that the next request for a given URL uses a fresh response from the network, thanks to the revalidation that happens "in the background."
Con: A well-defined service worker caching strategy is required.

Scenario: Short-term caching (Network falling back to cache) #

When a cached resource is valid in the service worker cache (<= 1 day): The service worker goes to the network for the resource. The browser returns the resource from the HTTP cache if it's there. If the network is down, the service worker returns the resource from the service worker cache
When a cached resource is expired in the service worker cache (> 1 day): The service worker goes to the network to fetch the resource. The browser fetches the resources over the network as the cached version in its HTTP cache is expired.

Pros and cons:

Pro: When the network is unstable or down, the service worker returns cached resources immediately.
Con: The service worker requires additional cache-busting to override the HTTP Cache and make "Network first" requests.

Conclusion #

Given the complexity of the combination of caching scenarios, it's not possible to design one rule that covers all cases. However, based on the findings in the previous sections, there are a few suggestions to look at when designing your cache strategies:

Service worker caching logic doesn't need to be consistent with HTTP caching expiry logic. If possible, use longer expiry logic in the service worker to grant the service worker more control.
HTTP caching still plays an important role, but it's not reliable when the network is unstable or down.
Revisit your caching strategies for each resource to make sure your service worker caching strategy provides its value, without conflicting with the HTTP cache.

Jonathan Chen on web.dev

Automating audits with AutoWebPerf

What is AutoWebPerf (AWP)? #

What is AWP for? #

Architecture overview #

Automating audits with AWP #

Recurring audits #

Visualising audit results in Data Studio #

What's next? #

Service worker caching and HTTP caching

Overview of caching flow #

Caching layers #

Service worker caching #

Controlling the service worker cache #

Service worker caching strategies and use cases #

Additional benefits of service worker caching #

HTTP caching #

Control HTTP cache expiry with HTTP response headers #

HTTP caching strategies and use cases #

Designing your cache expiry logic #

Consistent expiry logic for all cache layers #

Scenario: Long-term caching (Cache, falling back to network) #

Scenario: Medium-term caching (Stale-while-revalidate) #

Scenario: Short-term caching (Network falling back to cache) #

Service worker in all scenarios #

Different cache expiry logic at the service worker cache and HTTP layers #

Scenario: Long-term caching (Cache, falling back to network) #

Scenario: Mid-term caching (Stale-while-revalidate) #

Scenario: Short-term caching (Network falling back to cache) #

Conclusion #

Learn more #