How Does JS Work?
Client-side and server-side rendering
Traditionally, as is the case with static HTML pages, the code is rendered on the server (server-side rendering). When visiting a page, Googlebot gets the complete and "ready" content, so it doesn't need to do anything but download the CSS files and show the information in your browser.
Now let's look at how you can avoid these said problems.
How to check if your website is rendered properly
Getting Googlebot to correctly render your site requires you to focus on properly rendering both the links and content on your website. If you don't render links, Googlebot will have difficulty finding its way around your site. And if you don't properly render the content on your website, Googlebot won't be able to see it.
1. The 'site:' command
First of all, the site: command will show you how many pages of your website are currently indexed by Google. If many of them aren't in the index yet, it might be a sign of rendering problems with internal links.
Note that this won't work with the cache: command, since the cached versions of your pages contain only the original, not yet rendered code.
2. The Evergreen GoogleBot
Googlebot is the crawler that visits web pages to include them within Google Search index. The number one question Google got from the community at events and social media was if they could make Googlebot evergreen with the latest Chromium. At the time of writing, GoogleBot runs on the latest version of Chrome, 74.
This is a game changer for SEO, because now you can check how Google actually renders a page instead of guessing and hoping for the best.
3. Chrome DevTools
- Open your website in Chrome
- Open the Elements tab in DevTools
- See how your page was rendered by looking through the actual page's DOM built by the browser – make sure all the crucial navigational and content elements are already present there.
I recommend checking it in Chrome 41. This will provide you with confidence that the element can technically be seen by Googlebot.
You can also, of course, check it in your current Chrome version and make some comparisons.
4. Google Search Console's Fetch and Render
Another useful tool, which can provide us with an idea of how Google renders our website, is the Fetch and Render function in Google Search Console.
First, you have to copy and paste your URL. Then click 'Fetch and Render' and wait a while. This will allow you to see if Googlebot can render your page and see any related articles, copy or links.
5. Server logs analysis
The last thing you can resort to in order to verify how Googlebot crawls your website is the server logs analysis. By taking a hard look at the server logs, you can check if specific URLs were visited by Googlebot and which sections were or were not crawled by Googlebot.
There are many elements you can analyze thanks to your server side logs. For example, you can check if Googlebot visits your older articles — if not, there is probably a problem with the links, which could be caused by problems with JS.
You can also check if Googlebot sees every section of your website — if not, this could also be caused by rendering problems.
Possible problems in rendering your website
Even if your page is rendered correctly by Fetch and Render in the Search Console, this does not mean you can sit back and relax. There are still other problems you need to keep an eye out for.
Although exact timeouts are not specified, it's said that Google can't wait for a script longer than 5 seconds. Our experiments at Elephate confirm this rule. If the script loads for more than 5 seconds, Google generally will not index the content it takes to generate.
Fetch and Render will show you if the page can be rendered by Google, but it will not include timeouts. It's important to remember that Fetch and Render is much more forgiving than a standard Googlebot, so you'll need to take the extra step to make sure that the scripts you are serving are able to load under 5 seconds.
2. Browser limitations
Therefore, the best solution is to download the Chrome 41 browser (at the time of writing, the version that Google uses for web rendering) and familiarize yourself with it. Check the console log to see where any errors occur, and have your developers fix them.
3. Content requiring user interaction to load
I know I said this already, but it really needs repeating: Googlebot does not act like a user. Googlebot does not click on buttons, does not expand "read more", does not enter tabs, does not fill any forms... it only reads and follows.
This means that the entire content you want to be rendered should be loaded immediately to the DOM and not after the action has been performed.
This is particularly important with regard to "read more" content and menus, for example, in a hamburger menu.
What can we do to help Googlebot render websites better?
Avoid 'OnClick' links
Search engines do not treat 'onclick="window.location="' as ordinary links, which means that in most cases they won't follow this type of navigation—and search engines most definitely won't treat them as internal link signals.
It is paramount that the links are in the DOM before the click is made. You can check this by opening the Developers Tools in Chrome 41, and check if the important links have already been loaded — without any extra user actions.
Avoid # in URLs
The # fragment identifier is not supported and ignored by Googlebot. So instead of using the example.com/#url URL structure, try to stick to the clean URL formatting — example.com/url.
Unique URLs per a unique piece of content
Each piece of your content must be located "somewhere" for a search engine to index it. This is why it's important to remember that if you dynamically change your content without changing a URL, you're preventing search engines from accessing it.
Avoid JS errors
If your website contains errors in JS scripts, they will simply not be executed, which may lead to your website content not being displayed at all. One error in the script can cause a domino effect resulting in a lot of additional errors.
Don't block JS in robots.txt
Blocking JS files is a very old practice, but it still happens quite often. It even sometimes occurs as a default in some CMSs. And while the goal is to optimize the crawl budget, blocking JS files (and CSS style sheets) is considered a very bad practice. Don't take it from me, here's what Google has to say about the topic:
"We recommend making sure Googlebot can access any embedded resource that meaningfully contributes to your site's visible content or its layout..."
So don't even dream about blocking search engine crawlers with something like this:
The most popular solution is to use external services for prerendering, like Prerender.io, which is compatible with all of the most important JS frameworks.
The 21st century way
Why not give it a try and download it for free.
Bartosz Goralewicz is the co-founder of Elephate, which was recently awarded "Best Small SEO Agency" at the 2018 European Search Awards. Elephate believes in paving new ways in the SEO industry through bold, precise SEO campaigns for clients, SEO experiments and case studies. Bartosz leads a highly specialized team of technical SEO experts who work on the deep technical optimization of large, international structures. Technical SEO is not only Bartosz's job, but one of his biggest passions, which is why he enjoys traveling around the world to share this enthusiasm with like-minded SEO folks.