Your browser does not support JavaScript!
Home of the best Internet Marketing tools

The Battle of Web Crawlers:
WebSite Auditor vs. Screaming Frog

By: Valerie Niechai
March 13th, 2018

People are coming to us and ask: "Is WebSite Auditor any better than Screaming Frog?"

First of all, it hurts to hear such things. Second of all, yes, we know that lots of people use Screaming Frog with pure delight and even do not have any idea about WebSite Auditor. And then it hurts even more.

In order to make people see, we have put up a comparison review of the two tools. It ended up to be the mother of comprehensiveness, but we have immediately realized that no comparison table will reflect all those killer features and design that WebSite Auditor has.

So, I decided to devote a post to what these two tools can do for your optimization process. This way you can make your own judgment on which of the two is more convenient, or if you like, better.

1. See to the tech part of your site.

While the keyword research and link-building business is a bit tricky and demands constant creativity and competitor-peeking, the technical part of your website is fully under your control. With the right tools you will be able to see all the issues that need your interference, like:


  • Crawlability and indexing — check robots instructions and HTTP response codes to make sure different resources are accessible to search engines and users;
  • Redirects — check for 302 redirects and meta refresh, mixed content, rel="canonical", etc. to make sure all the important pages of your site will be found by search engines and users;
  • Code — check for the presence of frames (that can decrease the relevance of your page in the eyes of search engines) and W3C issues (that can lead to poor user experience);
  • Internal linking — check for broken links that can negatively affect user experience and confuse search engines, and keep an eye on outgoing links, as pages with too many links may be considered spammy by search engines;
  • Page speed — check for uncompressed resources and images, server response time, and the number of redirects to avoid negative user experience and reduce bounce rates;
  • Images — check for broken images and empty alt texts to avoid negative user experience and give search engines one more sign of what your page is about;
  • Mobile-friendliness — 2018 should become the year when Google finally shifts to mobile-first indexing so you have to check your mobile version for all the possible errors and problems.

Now that you see a scope of work to do, you have to be happy that there are some tools in existence that will do all these things for you.


Both WebSite Auditor and Screaming Frog are fully apt for this task. However, the obvious advantage of WebSite Auditor is that all the data for your analysis will be logically clustered and visually rich. With Screaming Frog, you will have to visit quite a few tabs and columns to collect all the necessary info, and then analyze the scraped data in spreadsheets.


Screaming Frog:

You know, when it comes to working with Screaming Frog, people compare a site audit to a treasure hunt (where treasures are those errors that tank your rankings). While you can really find almost anything with this tool, you should actually hunt for it: you have to know what to look for and where to go to check it.

I mean, to check the info for one block, like, page speed, for example, you have to visit quite a few tabs and columns to collect all you need for a thorough analysis.

Plus, some data is extracted as reports in CSV. It means that to do a site audit, you basically have to manage spreadsheets and fish out the necessary info. Could be scary for users who are not tech-savvy.

WebSite Auditor:

At the same time, WebSite Auditor has the same functions, it provides the same data, but this data is logically clustered (indexing issues, redirects, encoding issues, page speed, etc.). So you do not have to go places to gather all the necessary info, it is already collected for you!

What's more, in case you have any issues or errors, WebSite Auditor will give you useful recommendations on how to fix if anything is wrong, what to pay attention to, and why this factor is important. Basically, in the course of this analysis, you can find out some problems even if you did not have any intentions to look for them. WebSite Auditor will simply point out if you have any.

And that's not it!

The tool also allows its users to run a mobile audit:

While WebSite Auditor is checking for mobile-friendliness according to the Google mobile guidelines, Screaming Frog only allows its users to crawl a site as a mobile bot.

2. Optimize your content for better rankings.

Content is king, and this saying is so true, it is irritating. However, even if you are sure that your content is unique and relevant and your keyword research is semantic-based and exhaustive, there can be dozens of sites that also can boast of their perfect content.

In a situation like this, you have to make sure that your site has all those signposts that will tell search engines and users: this is the destination you have been looking for, come and stay as long as you need, and enjoy what you see and experience.

To provide this state of things, you will need the following:


  • Title tag optimization — check titles (the most important element of a page and SERP snippet) for their length, relevancy, and keywords.
  • Meta description optimization — check meta descriptions (one more important element of a page and SERP snippet) for their length, relevancy, and keywords.
  • Headings optimization — headings are strong relevancy signals for search engines, plus their logical structure can help win a featured snippet box in SERPs. H1 is the most important one, thus, it is advisable to include your keywords to it as well as to H2-6.
  • Alt text optimization — check images for the presence of alt texts (with keywords) to make sure search engines will understand your images during crawling.

Screaming Frog is not able to optimize content. It can only scrape the necessary data for a further analysis and optimization with additional tools.

However, WebSite Auditor is an all-in-one tool, and it can do both technical and on-page audits with equal quality and without additional tools.


WebSite Auditor:

WebSite Auditor has a special module which is exclusively dedicated to content analysis where it is possible to check how you are doing about keywords in titles, meta descriptions, body, and images.

Plus, all the keyword-based calculations are built on the advanced TF-IDF (term frequency - inverse document frequency) algorithm. It has nothing to do with old-school keyword density metrics. Instead, it accurately determines how well your page or any other element is optimized for target keywords.

At the same place, you can spot whether any technical factors truncate your rankings.

Your analysis will be followed by relevant recommendations in case anything is not right or requires minor changes. Plus, you will be provided with the keywords that your competitors are using:

The best thing is that it is possible to start applying corrections right away in the corresponding module of the app and see the changes as they will look on your page. Alternatively, you can make corrections in the source code. All the changes are then saved in HTML.

It is also possible to change your SERP snippet and see how it will look in the search listings:

One more killer feature of the tool is that it has a whole dashboard dedicated to a TF-IDF analysis. It helps to improve your page's topical relevance by analyzing terms and concepts at the pages of your top ranking competitors. It shows you whether any terms are relevant to the topic, whether they are used properly, whether they should be used less or more. It is a great source of inspiration for semantic keyword research.

Screaming Frog:

Though Screaming Frog is more of a crawler and technical audit tool, it scrapes all your headings, meta descriptions, headings (H1&H2), and images' information. By means of filtering, it is possible to find errors and soft spots, though you will have to see to keyword optimization all by yourself or with the help of other tools.

The tool also allows its users to make changes to a SERP snippet and see how it will look in search listings:

Unfortunately, when it comes to TF-IDF, Screaming Frog does not have any alternatives to it. It is clear that this tool is not quite designated for an on-page audit, but we think that such a module as TF-IDF is essential to the whole process of optimization. You will anyway need such an analysis to rank higher. Thus, you will have to find an additional app to do it.

3. Visualize your site structure.

A picture is worth a thousand words. It has suddenly become true for SEO when some apps acquired a new function — site structure visualizations. Why?

You can scroll an endless list of your internal links, go through redirects and orphans, or you can glance at a picture and immediately see everything in full view. It is clear that some link patterns can be not so obvious in a spreadsheet but immediately identified by means of a visualization.


  • Problems in a site structure: isolated successive strings of links, redirects, broken links, orphan pages
  • Problems with internal link juice distribution

WebSite Auditor will delight you and your clients with interactive editable visualizations of your site's structure.

Screaming Frog will give you all the necessary data on your site structure in spreadsheets. You can either use your imagination to visualize everything or use an additional tool to make the data interactive.


WebSite Auditor:

WebSite Auditor has been proud to introduce interactive visualizations that show your site structure by click depth where you will immediately see orphan pages (shown in grey), isolated parts of your site (chains of successive nodes), redirects (shown in blue), broken links (shown in red). Or you can filter by internal PageRank to see how your link juice is distributed across your pages.

This is truly a great way to show your clients and teammates the problems in a site structure. Instead of unimpressive spreadsheets, they will get intuitive site visualizations that highlight these problems graphically.

What's more, the app allows you to edit your site structure by adding or removing links and coloring the visualization graphs by tags so you can improve internal linking and PageRank flow, get rid of broken links and unnecessary redirects, and so on — until your graph is clean and perfectly structured.

Screaming Frog:

Screaming Frog is not able to delight its users with interactive visualizations of a site structure. However, it still has a crawl path report as well as reports on redirect chains and orphan pages.

Plus, it is possible to use the data from Screaming Frog in integration with a visualization tool, like Gephi, for example. Beware, it will be a sweaty piece of work. First of all, you will have to do an elaborate extraction of links from Screaming Frog.

Then, once your spreadsheets are loaded into Gephi, you will see the data represented as a random group of connected dots. After that, you will have to run a number of layout algorithms to get something close to an interactive picture of your site structure.

4. Create a report.

Unless you are a small business or your business team consists of you and your cat, you will need to make reports: for your clients, your boss, or your team. Creating beautiful reports that will speak to your clients without any additional explanations from your side may become a real pain in the neck. Imagine a level of relief when your auditing tool is able to create workable reports for you.


  • Possibility to create reports
  • Possibility to share reports (download, publish, print)
  • Possibility to customize reports
  • Possibility to make reports beautiful and vocal

If you dig spreadsheets, then both WebSite Auditor and Screaming Frog will serve you excellent CSV files with the necessary data.

However, if you are more into visually rich and vocal reports (and customizable ones!), then it is better to go with WebSite Auditor. These are the reports to impress clients and teammates.


WebSite Auditor:

WebSite Auditor allows its users to extract practically any chunk of data as a CSV file, but it is also no stranger to creating beautiful reports.

One of the most exciting things is that the report function allows users to customize how a future report will look and which data it will show. And even to preset color and font settings (swaaaag)!

And finally, it is possible to save a report to your computer, share it with your clients or teammates via a shareable link, email to clients, or upload to a website.

Screaming Frog:

Screaming Frog allows its users to export reports only in CSV. And though they will surely contain comprehensive data, it will be a challenge to impress the clients with it.


So, is WebSite Auditor any better than Screaming Frog?

It is obvious that WebSite Auditor wins on:

  • data representation (it's more logical!)
  • visual richness (it's just beautiful!)
  • set of functions (it's wider!)
  • cutting-edge features (TF-IDF & Visualizations)

It is a well enough reason to try this tool if you have never dealt with it before. Or use it on a regular basis to save time and enjoy the whole process without resorting to a number of tools, spreadsheets, and data sets.

And By The Way

We have tried to study both tools from top to toe and find out what they can and cannot do. When it comes to what WebSite Auditor lacks when comparing to Screaming Frog, we have this honest list:

  • Integrations with Google Search Console/ Majestic/ Ahrefs
  • Hreflang reports (but coming soon in WebSite Auditor as well!)
  • Crawl path reports
  • Response times of each page

If you know more features that we lack, feel free to share the information with us.

If you would like WebSite Auditor to have those features that we lack comparing to Screaming Frog (or any other similar tool), welcome to urge us! We will try to make our tool better first of all for you.

By: Valerie Niechai