We can change this by changing the user agent within Screaming Frog. You can do that if you go to the configuration tab, go to user-agents. Here you have plenty of user agents that you could click on, for example Firefox or Chrome or Google itself that you can click to crawl the website in a different way than with the standard Screaming Frog bot. If you use that setting, it will undoubtedly succeed or in many cases succeed in crawling the website. It is good to know that your crawling process may demand a bit more from your computer, so the crawling can also take a bit longer than when you use the standard Screaming Frog bot.
Adjust crawl settings There are also many other crawl settings that you can set to include multiple or other aspects of the website in the crawl. Screaming Frog naturally takes a things Singapore phone number list to make lot of things with it as standard, such as the meta descriptions and the images, but you can also look specifically at other parts and you can discover exactly what that is here in the menu, in the spider menu under configuration. You then go to extraction, the tab there. If you look further down, you can also see that, for example, in addition to the standard components, you can also view microdata or JSON-LD.
This can then take Screaming Frog and discover pages that may have schema implementation errors. If you press 'okay' and you start crawling the website, it may also take a little longer because it will of course retrieve more elements, but eventually we will go to the structured data tab – and if you do not see it below, you can also turn off some tabs here so that it comes into view - and eventually you will see in this section which structured data has been implemented, is being detected and whether there are any errors.