Also known as spiders or bots
Search engine crawlers (spiders, bots, and robots) are automated programs that crawl around websites searching for data. Google uses a search engine crawler called Googlebot.
A search engine crawler is a bit like a holidaymaker on the first day of a holiday in a new place. To follow this analogy we would like you to put yourself in the shoes of someone who is on holiday. It’s a nice place to be isn’t it?
On the first day of your holiday, you might know roughly what attractions you want to visit, but on your first day, you might just fancy a bit of an aimless wander.
As you leave your hotel, you’ll invariably find the opportunity to make a left or a right turn. You spot a signpost encouraging you to turn left, so that’s what you do. A few minutes later, you spot a giant flashing neon sign flashing above a pizza restaurant. You head in and gorge yourself on pizza.
While you’re inhaling pizza, you get chatting with the owner, and you ask him for a recommendation for somewhere to visit once enough of the pizza you have consumed has been digested, and you have regained the use of your lower body.
The pizza shop owner (let’s call him Ruttiger) tells you his brother owns a great bar just a few blocks away. Rutiger scribbles down some rudimentary directions on the back of an empty cigarette packet, and you cheerfully waddle off in search of the bar.
After a few beverages in the bar, you realise you have no idea how to get back to your hotel, so you call a taxi. The taxi driver asks what you plan to do tomorrow because, believe it or not, he’s got some suggestions for you. ‘What a friendly town’, you think as you doze off to sleep, loaded with pizza and local ales.
When you eventually wake up the next morning, you know exactly what local attractions you want to visit. Thanks to your (initially aimless) wandering the day before, you know exactly how to find the aforementioned local attractions.
When you have been on holiday for a few days, your wanderings have given you a pretty decent mental map of the surrounding area. You know the place well enough to start giving recommendations and directions to newly arrived guests in your hotel. You have become a search engine.
Googlebot – the Littlest Hobo of the Internet
The Googlebot, that wee scamp of a search engine crawler, has been wandering around the Internet for well over two decades. Oh, the places it has been, oh, the horror of the things it has seen.
Search engine crawlers theoretically start their journey on a single website. They follow all the internal links on that website to get an idea of what the website is about.
They follow the links from the website to other websites. They follow those links to etc etc etc etc.
Search engine crawlers are like the curtain-twitchers of neighbourhood watch schemes.
Why are Search Engine Crawlers important to SEO?
Without search engine crawlers, search engines wouldn’t be able to gather information to index websites and place them in SERPs.
Crawlers like Googlebot don’t just assess the content on your website, they also take note of who is linking to your website and which websites you are linking to.
As web 3.0 looms up on the horizon, we might find that search engines will adopt other technologies to create massive databases, but for now, crawlers are here to stay.
SEO professionals keep a close eye on how many times a crawler visits a website. The easy way to get a clear idea of how the Google search engine crawler perceives your website you can sign up for a free account at Google Search Console.
Search Engine Crawlers – explain them to me like I’m five years old, please!
Search engines create software that travels around the web gathering data on what websites contain and how they link to each other. This data is used to build a huge database that is referenced when you conduct a search on a search engine.
