Screaming Frog – what is it?

Screaming Frog is nothing more than an advanced crawler that scans websites for a comprehensive analysis of their structures and elements affecting on-site SEO. The tool facilitates the work of auditors because it detects numerous irregularities in connection with meta tags, duplicate content, headers (H1-H6), 404 errors or even incorrectly implemented redirects.

Free version of Screaming Frog
The free version of Screaming Frog allows you to e.g. generating XML sitemaps, analyzing 500 random URLs from a selected website, as well as visualizing specific domains.

The paid version of Screaming Frog
The paid version of Screaming Frog offers much more and costs around $ 150 a year. At this price, we can, among others analyze any number of URLs, render JavaScript, integrate the tool with Google Analytics / Google Search Console, and even check spelling and grammar on the page.

Screaming Frog – how to use?

To start working with Screaming Frog, enter any URL that you want to be scanned by robots (Mode -> Spider / Mode -> List). Then, it is worth learning how to use the basic functionalities of this tool to facilitate the collection of information on the subject of the website, which are presented below.

Title tag and meta description, H1 / H2 headers
The tool provides us with information about duplicate or missing parameters, possibly not matching the Google guidelines. We can analyze this type of data by entering the target URL. In the SERP Snippet tab, we will check how individual tags are displayed in the search engine.

Verifying broken or redirected sites
This functionality is especially appreciated by Paraphrase-Online.com on-site specialists. Well, it allows you to find all broken pages within the site. That’s not all, because if we click a specific address – we will verify where the links to pages returning the 404 code came from.

Checking the loading time of domains
The third, very useful option of Screaming Frog is checking the loading time of domains. Just go to the Internal tab, then HTML, and then click “Response Time”. Then we will be able to sort the data based on the loading time and of course compare it with the GA (Google Analytics) report on the bounce rate.

Remember!
Pages that load much slower tend to have a higher bounce rate than sites that load faster.

Information on internal links that lead to specific pages
An important point for every SEO specialist specializing in link building. Well, thanks to the Screaming Frog tool, we will verify each URL address based on internal links leading to it, as well as text anchors. Just select the “Inlinks” tab at the bottom of the program to find out, among other things, what pages are the links coming from, are they dofollow / nofollow links and what anchor texts have been redirected to the site.

Additional “Tree” view facilitating the analysis of the website structure
ccording to Paraphrase-Online experts, the analysis of the website structure, using the additional “Tree” view, is very useful because it allows you to verify what the URL of each subpage looks like. Thanks to this, we will be able to eliminate incorrect navigation elements and implement the necessary structural changes.

Checking the correctness of the implementation of Google Analytics and Google Search Console
Screaming Frog also allows you to verify the implementation of GSC and GA tags on each of the subpages of the selected website. It is also worth knowing that the columns available in the program can be rotated and placed in the correct order. As we mentioned before, all information can be easily exported to Excel.

Create an XML Map
The functionalities of Screaming Frog allow you to instantly create a graphic sitemap and XML sitemap. If we want additional configuration – we will determine the frequency of visits to the site by the robot, which will be based on the place in the domain structure of a specific subpage.

Remember!
We can open the sitemap in Excel. This is very useful because we will verify that the specific addresses do not contain duplicate parameters. That’s not all, because for key URL links we can set the most important guidelines manually (e.g. frequency of changes, priority, and other values). After you finish working with the sitemap, save the file as XML.

It is also worth mentioning the possibility of verifying information in the context of website graphics (eg images exceeding 100 Kb, with a missing “alt” attribute or exceeding 100 characters). Screaming Frog SEO spider also allows you to check the correctness of URLs and list errors, e.g. URLs exceeding 115 characters, including underscores or uppercase letters.

How to speed up the analysis in Screaming Frog?

To speed up the analysis in Screaming Frog, you should divide your work into stages – then we will be able to initially verify selected information about a given website, and then use the additional functions of this tool. First of all, we recommend paying special attention to, among others:

– page size,
– the number of dofollow and nofollow links,
– number of internal and outbound links,
– page description (meta description),
– page title (meta title),
– keywords,
– status (there is 200 or 404), – page encoding (UTF),
– domain address.

After a thorough analysis of the above factors, let’s check the side functions of the program, that is: all internal and external links, meta robots, duplicate domains, etc. Below are the other Screaming Frog options:

– page depth level,
– file size,
– inlinks,
– outlinks,
– XML Sitemap Generator,
– pictures,
– cannonical,
– 301 or 302 redirects,
– User-Agent Switcher,
– flaws with urls,
– Anchor text.

It is also worth mentioning the possibility of analyzing file sizes, errors (e.g. 4xx, 5xx), headers, as well as problems with URLs, which include, for example, underlines, capital letters or even the lack of ASCII characters.

We can significantly speed up the work in Screaming Frog by performing actions in the right order, taking into account individual assumptions specified in the initial SEO strategy. If we want a comprehensive on-site website optimization – this tool will turn out to be perfect, however, we have to buy the premium version, because the free version is too limited to perform a meticulous domain analysis.

How to crawl problematic sites with Screaming Frog?

Crawling problematic sites doesn’t have to be difficult. All you have to do is go to the “Advanced” tab, and then change the settings of several important options, including “Always follow redirects”, “response timeout” as well as “max redirects to follow”.

Always follow redirects
After selecting this option, Screaming Frog will pay attention to 3XX redirect directives, and what’s more – he will ignore the so-called crawl depth. The above functionality is very useful when we intend to strongly change the structure of the website or migrate to other URLs.

Remember!
Thanks to the “always follow redirects” option, we will verify multiple redirects and the redirect loop much more accurately than in the default settings. This is because SF often does not identify all “deep redirects”.

If, however, we want to check incorrect redirections within the website – we recommend downloading the “Redirect Chains” report from the Reports tab. We will then check the URLs redirected more than twice (single redirects are not visible here).

Response timeout
In the event that we have a free domain, but we want to obtain as much relevant data as possible about it, we should increase the waiting time by Screaming Frog for an HTTP response. We recommend that you change the value beyond the default of 20 seconds.

Max redirects to follow
An interesting option that allows you to direct Screaming Frog robots to determine the length of the redirect path that they are to present in the summary. As a standard, crawlers follow 5 consecutive redirections.

This tab offers one more additional function – extracting images with the srcset attribute. The use of this option is very useful when we want to place the same image in different resolutions – for example in responsive websites.

Summary

Screaming Frog, developed in 2010 (founder: Dan Sharp), is used all over the world by the vast majority of SEO specialists. It has many useful functions, and its main operation is based on reliable scanning of websites, so that you can later verify their dysfunctions and effectively fix them. At Paraphrase-Online, we use Screaming Frog primarily to improve on-site SEO audits and we believe that in this aspect it is an absolute leader!

FAQ

Here are the answers to the most frequently asked questions about Screaming Frog SEO spider!

When is it worth analyzing the website?
The analysis of the website is very useful when we see that it is not working properly, its position does not increase according to the assumptions made, and the traffic from SERPs has definitely decreased. It is then worth verifying meta tags, headers, internal and external links, as well as other on-site factors available in Screaming Frog, which may be a potential cause of irregularities.

What is the price of Screaming Frog?
Screaming Frog is priced at $ 149 annually. The premium version is free, but it has many limitations (e.g. no possibility of integration with Google Search Console and Google Analytics).