Martin Splitt on web.dev

Web developer tools for debugging JavaScript issues in Google Search

2020-06-25T00:00:00Z

Google provides a lot of tools to help you debug JavaScript SEO issues in Google Search. This guide gives you an overview of the available tools and suggestions on when to use each tool.

Find basic SEO issues with Lighthouse #

Use Lighthouse for your first investigation. It comes with a bunch of SEO audits.

The Lighthouse SEO audits are very basic first checks for a single page of your website. They catch the most common mistakes and give you a first impression on how your website is doing in terms of search engine discoverability. Note that Lighthouse runs in your browser, which is not an accurate representation of how Googlebot might see a web page. For instance, browsers (and Lighthouse) don't use robots.txt to decide if they can fetch resources from the network, while Googlebot does. So when Lighthouse identifies potential problems, you should fix them, but you may have to use other tools to debug issues further.

Validate pages with Google Search testing tools #

Google Search provides a set of tools for testing how Googlebot sees your web content.
Some of these tools are particularly useful when testing from your development environment:

The Mobile-Friendly Test ensures that a page is mobile-friendly, which has been a Google Search ranking signal since 2015
The Rich Results Test validates that a page is eligible for rich results based on the structured data that it provides
The AMP Test validates your AMP HTML

In combination with tools like local-tunnel or ngrok you can create a temporary public URL from your local development environment and iterate quickly while you are testing with Google's testing tools.

These testing tools provide you with multiple helpful pieces of information, like:

The rendered HTML that Googlebot will use for indexing
An overview of the resources loaded and explanations of why resources can't be loaded
Console log messages and JavaScript errors with stack traces

The Google Search Console URL Inspection Tool can also give you detailed information about the status of a page.

Here you can find out:

If the URL is in the Google Search index or can be indexed in the future
What the rendered HTML from the most recent crawl looks like
What the rendered HTML looks like for a fresh crawl of the page
Information about page resources
JavaScript log messages and errors with stack traces
A screenshot
Mobile usability issues
What structured data was detected on the page and if it's valid

Using these tools you can identify most issues and resolve them. Google Search also provides documentation for fixing Google Search-related JavaScript problems for more guidance on what to do once you identified the cause of a problem.

Investigate site health with Google Search Console #

The tools from the last section are great at resolving specific issues on a single page of your website, but if you want to get a better overview of your entire website, the Google Search Console is where you need to go.

Coverage report #

The Coverage report shows you which pages of your website are indexed and which ones have problems.

Core Web Vitals report #

The Core Web Vitals report helps you get an overview of how the pages of your website are performing in terms of the Core Web Vitals.

Make these tools part of your developer tooling #

In this article, we've seen a series of tools for various purposes from testing a page before publishing it to monitoring the pages on a live website that give you transparency on how your website does in terms of discoverability for Google Search. Some of these tools might become useful parts of your development toolkit, others might be more like ad-hoc tools to identify the cause of a problem and fix affected pages. To learn more about Google Search for developers or JavaScript SEO, check out the official Search for developers documentation.

Making JavaScript and Google Search work together

2019-06-05T00:00:00Z

Great things are happening with Google Search, and we were excited to share them at Google I/O 2019!

In this post we'll focus on best practices for making JavaScript web apps discoverable in Google Search, including:

The new evergreen Googlebot
Googlebot's pipeline for crawling, rendering and indexing
Feature detection and error handling
Rendering strategies
Testing tools for your website in Google Search
Common challenges and possible solutions
Best practices for SEO in JavaScript web apps

Meet the evergreen Googlebot #

This year we announced the much-awaited new evergreen Googlebot.

Googlebot is now running a modern Chromium rendering engine.

Googlebot now uses a modern Chromium engine to render websites for Google Search. On top of that, we will test newer versions of Chromium to keep Googlebot updated, usually within a few weeks of each stable Chrome release. This announcement is big news for web developers and SEOs because it marks the arrival of 1000+ new features—such as ES6+, IntersectionObserver, and Web Components v1—in Googlebot.

Learn how Googlebot works #

Googlebot is a pipeline with several components. Let's take a look to understand how Googlebot indexes pages for Google Search.

Googlebot's pipeline for crawling, rendering, and indexing a page.

The process works like this:

Googlebot queues URLs for crawling.
It then fetches the URLs with an HTTP request based on the crawl budget.
Googlebot scans the HTML for links and queues the discovered links for crawling.
Googlebot then queues the page for rendering.
As soon as possible, a headless Chromium instance renders the page, which includes JavaScript execution.
Googlebot uses the rendered HTML to index the page.

Your technical setup can influence the process of crawling, rendering, and indexing. For example, slow response times or server errors can impact the crawl budget. Another example would be requiring JavaScript to render the links can lead to a slower discovery of these links.

Use feature detection and handle errors #

The evergreen Googlebot has lots of new features, but some features are still not supported. Relying on unsupported features or not handling errors properly can mean Googlebot can't render or index your content.

Let's look at an example:

 <body>
   <script>
     navigator.geolocation.getCurrentPosition(function onSuccess(position) {
       loadLocalContent(position);
     });
   </script>
 </body>

This page might not show any content in some cases because the code doesn't handle when the user declines the permission or when getCurrentPosition call returns an error. Googlebot declines permission requests like this by default.

This is a better solution:

 <body>
   <script>
     if (navigator.geolocation) {
       // this browser supports the Geolocation API, request location!
       navigator.geolocation.getCurrentPosition(
         function onSuccess(position) {
           // we successfully got the location, show local content
           loadLocalContent(position);
         }, function onError() {
           // we failed to get the location, show fallback content
           loadGlobalContent();
         });
     } else {
       // this browser does not support the Geolocation API, show fallback content
       loadGlobalContent();
     }
   </script>
 </body>

If you have problems with getting your JavaScript site indexed, walk through our troubleshooting guide to find solutions.

Choose the right rendering strategy for your web app #

The default rendering strategy for single-page apps today is client-side rendering. The HTML loads the JavaScript, which then generates the content in the browser as it executes.

Let's look at a web app that shows a collection of cat images and uses JavaScript to render entirely in the browser.

The rendering strategy influences the performance and robustness of your web apps.

If you're free to choose your rendering strategy, consider server-side rendering or pre-rendering. They execute JavaScript on the server to generate the initial HTML content, which can improve performance for both users and crawlers. These strategies allow the browser to start rendering HTML as it arrives over the network, making the page load faster. The rendering session at I/O or the blog post about rendering on the web shows how server-side rendering and hydration can improve the performance and user experience of web apps and provides more code examples for these strategies.

If you're looking for a workaround to help crawlers that don't execute JavaScript—or if you can't make changes to your frontend codebase—consider dynamic rendering, which you can try out in this codelab. Note, though, that you won't get the user experience or performance benefits that you would with server-side rendering or pre-rendering because dynamic rendering only serves static HTML to crawlers. That makes it a stop-gap rather than a long-term strategy.

Test your pages #

While most pages generally work fine with Googlebot, we recommend testing your pages regularly to make sure your content is available to Googlebot and there are no problems. There are several great tools to help you do that.

The easiest way to do a quick check of a page is the Mobile-Friendly Test. Besides showing you issues with mobile-friendliness, it also gives you a screenshot of the above-the-fold content and the rendered HTML as Googlebot sees it.

The mobile-friendly test shows you the rendered HTML Googlebot uses.

You can also find out if there are resource loading issues or JavaScript errors.

It's recommended to verify your domain in Google Search Console so you can use the URL inspection tool to find out more about the crawling and indexing state of a URL, receive messages when Search Console detects issues and find out more details of how your site performs in Google Search.

The URL Inspection Tool in Search Console shows the status of a page in crawling, rendering, and indexing.

For general SEO tips and guidance, you can use the SEO audits in Lighthouse. To integrate SEO audits into your testing suite, use the Lighthouse CLI or the Lighthouse CI bot.

Lighthouse shows general SEO recommendations for a page.

These tools help you identify, debug, and fix issues with pages in Google Search and should be part of your development routine.

Stay up to date and get in touch #

To stay up to date with announcements and changes to Google Search, keep an eye on our Webmasters Blog, the Google Webmasters Youtube channel, and our Twitter account. Also check out our developer guide to Google Search and our JavaScript SEO video series to learn more about SEO and JavaScript.

Fix sneaky 404s

2018-11-05T00:00:00Z

Single Page Apps can show different content without loading a new page. To do so, they use click handlers on links and the History API. The History API allows to manipulate the browser session history. This way we can update the URL when showing a different page (usually called a "view" in Single Page Apps). It also makes sure the browser's back button still works as expected.

Take a look at the Single Page App in this codelab. It shows either cat or dog images and provides links to toggle between the two animals. It seems to work fine!

Uncovering the sneaky 404 #

Unfortunately there is a subtle bug in the app. Let's take a look!

To preview the site, press View App. Then press Fullscreen .
Click the Doggos link. Notice how the URL changed.
Reload the app.

You get a page with "Cannot GET /doggos" on it—a sneaky 404. It is "sneaky", because the web app seems to work fine as long as you only click on links within it. It breaks when using the URLs in a new browser window or when refreshing the page. The issue is that the server does not know how to respond to a request for these URLs. The JavaScript code in our web app is using the History API to navigate between them, but the server does not know what to do with them. Whenever the server does not know what to do for a requested URL, it responds with the HTTP status code 404. With this code the server says it hasn't found anything for the requested URL.

Search engines will not index the URLs in this case, because a user would click on a search result and find the error message, but not the content they were looking for, such as the dog pictures.

Fixing the server #

This project uses an express.js server written in JavaScript. Let's fix the server, so it responds with index.html and the single page app will take care of the rest.

Click Remix to Edit to make the project editable.
Select the server.js file.

This file contains the server code. It sets up an express.js server and sends the content of index.html. The route setup in line 15 only serves the web app when requests go to the URL /. The server should also serve the other URLs we have created. Let's change this to serve all URLS, so it also works with additional URLs in the future.

To do so, we can change the code starting at line 15 to this:

app.get('/*', function(request, response) {
  response.sendFile(__dirname + '/views/index.html');
});

The /* matches any URL and the server now responds with the web app in index.html for any given URL.

To preview the site, press View App. Then press Fullscreen .

Refreshing and opening the links in a new incognito window should now work as expected.