Crawling
Crawling (or “a crawl”) is when you try to get all available information about a website with a tool. It’s all about the acquisition of data about a website or domain. On the one hand search engines crawl websites with crawlers/spiders/bots to gather details about every single URL they can find. On the other hand, there are crawlers out there that you can use to crawl any website. Crawling comes first, indexing second and serving is what follows as long as we are talking about search engines.
“The crawling process begins with a list of web addresses from past crawls and sitemaps provided by website owners. As our crawlers visit these websites, they use links on those sites to discover other pages.”
https://support.google.com/webmasters/answer/70897?hl=en
Software to be used when you need to crawl a website
Why you need a crawling software
Even if you are only halfway serious about SEO, you need some kind of crawling software. If you are low on a budget, take the free Xenu´s Link Sleuth because it’s free and open source. If you can afford a £149.00 per year, go for the full version of Screaming Frog, my preferred tool of choice.
Why care about crawling?
Each and every search engine will start crawling your website sooner or later. The bigger your site gets, the more important the crawling budget gets for you. If you follow a few rules, you will not get into trouble with that. It would be wise to care about this from the very beginning of starting a new website, blog or webshop. It’ll save you time later in the process.