Lizzi Harvey on web.dev

Making JavaScript and Google Search work together

2019-06-05T00:00:00Z

Great things are happening with Google Search, and we were excited to share them at Google I/O 2019!

In this post we'll focus on best practices for making JavaScript web apps discoverable in Google Search, including:

The new evergreen Googlebot
Googlebot's pipeline for crawling, rendering and indexing
Feature detection and error handling
Rendering strategies
Testing tools for your website in Google Search
Common challenges and possible solutions
Best practices for SEO in JavaScript web apps

Meet the evergreen Googlebot #

This year we announced the much-awaited new evergreen Googlebot.

Googlebot is now running a modern Chromium rendering engine.

Googlebot now uses a modern Chromium engine to render websites for Google Search. On top of that, we will test newer versions of Chromium to keep Googlebot updated, usually within a few weeks of each stable Chrome release. This announcement is big news for web developers and SEOs because it marks the arrival of 1000+ new features—such as ES6+, IntersectionObserver, and Web Components v1—in Googlebot.

Learn how Googlebot works #

Googlebot is a pipeline with several components. Let's take a look to understand how Googlebot indexes pages for Google Search.

Googlebot's pipeline for crawling, rendering, and indexing a page.

The process works like this:

Googlebot queues URLs for crawling.
It then fetches the URLs with an HTTP request based on the crawl budget.
Googlebot scans the HTML for links and queues the discovered links for crawling.
Googlebot then queues the page for rendering.
As soon as possible, a headless Chromium instance renders the page, which includes JavaScript execution.
Googlebot uses the rendered HTML to index the page.

Your technical setup can influence the process of crawling, rendering, and indexing. For example, slow response times or server errors can impact the crawl budget. Another example would be requiring JavaScript to render the links can lead to a slower discovery of these links.

Use feature detection and handle errors #

The evergreen Googlebot has lots of new features, but some features are still not supported. Relying on unsupported features or not handling errors properly can mean Googlebot can't render or index your content.

Let's look at an example:

 <body>
   <script>
     navigator.geolocation.getCurrentPosition(function onSuccess(position) {
       loadLocalContent(position);
     });
   </script>
 </body>

This page might not show any content in some cases because the code doesn't handle when the user declines the permission or when getCurrentPosition call returns an error. Googlebot declines permission requests like this by default.

This is a better solution:

 <body>
   <script>
     if (navigator.geolocation) {
       // this browser supports the Geolocation API, request location!
       navigator.geolocation.getCurrentPosition(
         function onSuccess(position) {
           // we successfully got the location, show local content
           loadLocalContent(position);
         }, function onError() {
           // we failed to get the location, show fallback content
           loadGlobalContent();
         });
     } else {
       // this browser does not support the Geolocation API, show fallback content
       loadGlobalContent();
     }
   </script>
 </body>

If you have problems with getting your JavaScript site indexed, walk through our troubleshooting guide to find solutions.

Choose the right rendering strategy for your web app #

The default rendering strategy for single-page apps today is client-side rendering. The HTML loads the JavaScript, which then generates the content in the browser as it executes.

Let's look at a web app that shows a collection of cat images and uses JavaScript to render entirely in the browser.

The rendering strategy influences the performance and robustness of your web apps.

If you're free to choose your rendering strategy, consider server-side rendering or pre-rendering. They execute JavaScript on the server to generate the initial HTML content, which can improve performance for both users and crawlers. These strategies allow the browser to start rendering HTML as it arrives over the network, making the page load faster. The rendering session at I/O or the blog post about rendering on the web shows how server-side rendering and hydration can improve the performance and user experience of web apps and provides more code examples for these strategies.

If you're looking for a workaround to help crawlers that don't execute JavaScript—or if you can't make changes to your frontend codebase—consider dynamic rendering, which you can try out in this codelab. Note, though, that you won't get the user experience or performance benefits that you would with server-side rendering or pre-rendering because dynamic rendering only serves static HTML to crawlers. That makes it a stop-gap rather than a long-term strategy.

Test your pages #

While most pages generally work fine with Googlebot, we recommend testing your pages regularly to make sure your content is available to Googlebot and there are no problems. There are several great tools to help you do that.

The easiest way to do a quick check of a page is the Mobile-Friendly Test. Besides showing you issues with mobile-friendliness, it also gives you a screenshot of the above-the-fold content and the rendered HTML as Googlebot sees it.

The mobile-friendly test shows you the rendered HTML Googlebot uses.

You can also find out if there are resource loading issues or JavaScript errors.

It's recommended to verify your domain in Google Search Console so you can use the URL inspection tool to find out more about the crawling and indexing state of a URL, receive messages when Search Console detects issues and find out more details of how your site performs in Google Search.

The URL Inspection Tool in Search Console shows the status of a page in crawling, rendering, and indexing.

For general SEO tips and guidance, you can use the SEO audits in Lighthouse. To integrate SEO audits into your testing suite, use the Lighthouse CLI or the Lighthouse CI bot.

Lighthouse shows general SEO recommendations for a page.

These tools help you identify, debug, and fix issues with pages in Google Search and should be part of your development routine.

Stay up to date and get in touch #

To stay up to date with announcements and changes to Google Search, keep an eye on our Webmasters Blog, the Google Webmasters Youtube channel, and our Twitter account. Also check out our developer guide to Google Search and our JavaScript SEO video series to learn more about SEO and JavaScript.

How search works

2018-11-05T00:00:00Z

What does a search engine do? #

Search engines are the digital version of a librarian. They use a comprehensive index to help find the right information for a query. Understanding the basics of search prepares you to make your content discoverable for users.

How crawlers browse the web #

Crawling is like reading through all the books in the library. Before search engines can bring any search results, they need to have as much information from the web as possible. To do this, search engines use a crawler—a program that travels from site to site and acts like a browser.

If a book or document is missing or damaged, the crawler can't read it. Crawlers try to fetch each URL to determine the state of the document. If a document returns an error status code, crawlers cannot use any of its content, and might retry the URL at a later time. This ensures only publicly accessible documents get into the index.

If crawlers discover a redirection status code (like 301 or 302), they follow the redirection to a new URL and continue there. Once they get a successful response, meaning they've found a document accessible to users, they check if it's allowed to be crawled and then download the content.

This check includes the HTML and all content mentioned in the HTML, such as images, videos, or JavaScript. Crawlers also extract the links from HTML documents so that the crawler can visit the linked URLs as well. Following links is how crawlers find new pages on the web.

Crawlers don't actively click links or buttons, but instead send URLs to a queue to crawl them later. When accessing a new URL, no cookies, service workers or local storage (like IndexedDB) are available.

Building an index #

After retrieving a document, the crawler hands the content to the search engine to add it to the index. The search engine now renders and analyzes the content to understand it. Rendering means displaying the page as a browser would (with some limitations).

Search engines look at keywords, the title, links, headings, text, and many other things. These are called signals which describe the content and context of the page. Signals allow search engines to answer any given query with the best possible page.

Search engines might find the same content at different URLs. For example a recipe for "apple pie" might live under /recipes/apple-pie and under /recipes/1234. To avoid indexing and showing the recipe twice, search engines determine what the main URL should be and discard the alternative URLs showing the same content.

Serving the most useful results #

Search engines do more work than just matching the query to keywords in the index. To give useful results, they might consider context, alternative wording, location of the user, and more. For example, "silicon valley" might refer to the geographic region or the TV show. But if the query is "silicon valley cast", results on the region aren't very helpful.

Some queries can be indirect, like "the song from pulp fiction", and search engines need to interpret that and show results for the music in the film. When a user searches for something, search engines determine the most useful results and then show them to the user. Ranking, or ordering, the pages happens based on the query. The order can often change over time if better information becomes available.

Next steps: how to optimize for search engines #

Now that you understand the basics of how search engines work, you may see the value in optimizing for search engines. This is called SEO, or 'Search Engine Optimization.' By making sure search engines can find and automatically understand your content, you are improving the visibility of your site for relevant searches. This can result in more interested users coming to your site. Audit your site with Lighthouse and check the SEO results to see how well search engines can make users aware of your content.

How to measure search engine optimization with Lighthouse

2018-11-05T00:00:00Z

Why does this matter? #

The Lighthouse search engine optimization (SEO) audit scans your page, tests for things that matter to search engines, and gives you a score so you can see specific areas for improvement. SEO matters because it's how you get more relevant users viewing your content. If a search engine has trouble seeing your page, you're possibly missing out on traffic sources.

Audit your page with Lighthouse #

Run Lighthouse on a page that's representative of the content that you want search engines to see:

Press `Control+Shift+J` (or `Command+Option+J` on Mac) to open DevTools.
Click the Lighthouse tab.
Make sure the SEO checkbox is selected in the Categories list.
Click the Generate report button.

Lighthouse generates a report for your page so you can see areas where you can improve SEO for your site.

To learn how to fix issues flagged by Lighthouse, see the SEO audits collection.

Next steps #

It's worth noting that the audit doesn't cover everything you could do to increase visibility in search engines. You'll have to check the guidelines for individual search engines because they may have different requirements.

Here are some guides for specific search engines:

There are also non-technical aspects that can improve visibility of your content in search engines, such as writing in the style of your target audience. Bottom line: make great content for the people you want to attract.

To learn more, check out Google's I/O talks: