Screaming Frog Crawl Process
The Screaming Frog SEO Spider is a fast and advanced SEO site audit tool. It can be used to crawl both small and very large websites, where manually checking every page would be extremely labour intensive, and where you can easily miss a redirect, meta refresh or duplicate page issue. Using the Screaming Frog SEO Spider can be an efficient way to go through these items without having to click through everything on the website yourself.
Load up Screaming Frog (from your computer’s Applications).
Paste the website’s URL that you want to crawl in the top bar and click Start.
Screaming Frog will slowly populate as it crawls the website.
Wait until the bar to the right is fully green (Crawl: 100%).
On the right, click SEO Elements > Internal > All.
Click export at the top-left.
On the right of Screaming Frog, click SEO Elements > Internal > Images
In the spreadsheet area of Screaming Frog, Shift+Click to select all of the images
Every image should now be selected.
Click the Image Details tab underneath.
Click export at the top-left.
Click on the Response Codes.
Click export at the top-left.
Make a copy of the Screaming Frog Crawl Template and save it to the client folder.
Title This “[Client] Screaming Frog SEO Results”.
On the first tab “HTML”.
Export/Copy the information needed on the “HTML” tab or you may use the VLOOKUP to export the exported data from the Screaming Frog.
Sort Column B by A-Z. Remove any rows that do not contain “text/html” in Column B, “Content”. Remove text/ccs, image/jpeg, Application/Javascript & etc.
Go through the columns “Title 1”, “Meta Description“, and “H1-1” and mark any empty cells and duplicates in red.
On the second tab “Image”.
Export/Copy and paste the data from the exported file of images.
Sort Column C by A-Z.
Mark any rows that do not contain any alt-text in Column C, in red and orange for duplicates.
Open the third tab “Response Codes”.
Export/Copy and paste the data from the exported Screaming Frog Response Codes.
Remove any rows that don’t have text/html or text/html;charset=utf-8 in the second column (Column B – Content Type)
For any rows with “Status Code” 2XX, highlight it green.
For any rows with “Status Code” 3XX, highlight it yellow
For any rows with “Status Code” 4XX, highlight it red