How Google handles JavaScript throughout the indexing process – Vercel

🌈 Abstract

The article discusses the evolution of Google's ability to crawl, render, and index web pages, and how this impacts search engine optimization (SEO) for modern web applications. It addresses common myths and misconceptions about JavaScript-heavy websites and provides insights from a research study conducted by the author.

🙋 Q&A

[01] Understanding Google's Rendering Capabilities

1. What are the key aspects of Google's current system for rendering web pages?

• Google uses an up-to-date version of Chrome for rendering, keeping pace with the latest web technologies.
• Google's Web Rendering Service employs its own internal heuristics to determine when cached assets are still fresh and when they need to be downloaded again.
• Google no longer relies on the User-Agent to determine rendering capabilities, and instead optimizes for stateless rendering and implements personalization through stateful methods.

2. How has Google's ability to crawl and index web content evolved over the years?

• In the early days, Google primarily indexed static HTML content, and JavaScript-generated content was largely invisible to search engines.
• Google introduced the AJAX crawling scheme, allowing websites to provide HTML snapshots of dynamically generated content.
• Google then began rendering pages using a headless Chrome browser, which had limitations in processing modern JavaScript features.
• Today, Google uses an up-to-date version of Chrome for rendering, keeping pace with the latest web technologies.

[02] Debunking Common Myths about JavaScript and SEO

1. What were the key findings from the research study regarding Google's ability to render JavaScript-heavy content?

• The research found that 100% of HTML pages on nextjs.org, excluding status code errors and non-indexable pages, resulted in full-page renders, including pages with complex JavaScript interactions.
• The study also found no correlation between JavaScript complexity and rendering delay on nextjs.org, though more complex JavaScript on a much larger site can impact crawl efficiency.

2. How did the research address the myth that Google has a separate process or criteria for JavaScript-heavy pages?

• The researchers conducted targeted tests, such as the @import test and using noindex meta tags, to verify that Google's renderer processes CSS and JavaScript-heavy pages in the same way.
• The findings showed that Google successfully renders pages with or without JavaScript, and that the noindex meta tag in the initial HTML response is not rendered, regardless of JavaScript content.

3. What did the research reveal about the impact of rendering queue and timing on SEO?

• The research found that the 25th percentile of pages on nextjs.org were rendered within 4 seconds of the initial crawl, challenging the notion of a long "queue".
• While some pages faced significant delays (up to ~18 hours at the 99th percentile), these were the exception and not the rule.
• The researchers also observed that Google treats URLs differently if they have query strings that don't affect the content, with URLs containing query strings experiencing longer rendering delays.

4. How did the research address the myth that JavaScript-heavy sites, especially those using client-side rendering (CSR), suffer from slower page discovery by Google?

• The researchers tested the discoverability of new, previously undiscovered pages by embedding links in the JavaScript data on the /showcase page of nextjs.org.
• They found that the location of the link (whether in an tag or embedded in a JSON payload) did not impact how Google prioritized its crawl, and that the use of a sitemap.xml significantly reduces, if not eliminates, the time-to-discovery differences between different rendering patterns.

[03] Recommendations for Optimizing Web Applications for SEO

1. What are the key recommendations for optimizing web applications for SEO based on the research findings?

• Ensure that critical information (such as noindex tags) is added to the page in the initial HTML response, as Google may not process client-side changes.
• Use a robots.txt file to control how search engines crawl and index your site.
• Prioritize using standard HTML navigation (with links) rather than JavaScript-based navigation, as this aids both user navigation and search engine crawling efficiency.
• Utilize XML sitemaps and update the tag only when significant content updates occur.

2. How do the different rendering strategies (SSG, ISR, SSR, CSR) compare in terms of their impact on SEO?

• The article provides a comparison table that highlights the differences between the rendering strategies in terms of crawl efficiency, discovery, rendering completeness, rendering time, link structure evaluation, and indexing.
• While there are some differences, the article emphasizes that Google will quickly discover and index your site regardless of the rendering strategy, and that the focus should be on creating performant web applications that benefit users.