You’re probably wondering what your ecommerce store has to do with mischievous infants and early learning development.
Fret less, crawling is directly related to SEO and we will explain.
Contrary to popular belief (or at least what I used to think), Google doesn’t have a hidden community of speed readers holed up in a big old warehouse sorting through millions of webpages to decide the #1 ranking page every time you type something into your search box.
Google has ‘crawlers’ or Googlebots to understand whether your content relates to a search engine query. Think giant robot arachnids with creepy computer search arms. If the subject of the content closely matches the phrase used by the searcher, the Googlebot spiders know your work might be of interest to the searcher – or your potential customer.
For this to happen, you need to make sure Google can ‘crawl’ all of the pages you want to rank. But if the bots can’t crawl your content, it doesn’t matter how well your site is optimised – you’ll get no search engine love.
4 Common Ecommerce SEO Crawling Blunders
Here’s 4 common problems we come across that hinder search engines from crawling website content, along with a step-by-step guide to help you check and fix each issue.
We’ve spilled out enough detail to make it super quick and easy for any skilled marketing pro to implement these changes. You don’t have to be a techie code nerd to fix the majority of these errors (we let you know when it’s time to tap out and call in the experts).
#1 Crawling Error: You Have Site Errors
As the web bots crawl your website it is crucial to have active working pages for your website visitors. Apart from Google being blind to your content, your potential customers will miss out too.
Remember, Customer First, Google Second. Always.
Although there are different types of site errors, the main one we are looking for is the dreaded 404 (file not found). You probably know the one. Even if you had no idea what it meant, chances are you’ve been infuriated by one before.
How to check for ecommerce site errors
The best and most reliable way to check for 404 errors is to use Webmaster tools. If you haven’t got this already installed, jump on it:
https://www.google.com/webmasters/tools/
At the site, head to Crawl > Crawl Errors
On this page you will see the (hopefully small) list of 404 errors your site has.
How to fix your ecommerce site errors
If you’re selling online, your problem will most likely fit within one of these scenarios. Look through your site errors to find out which course of action to take based on the 4 suggestions we have here:
- Discontinued products – Your error might be down to products you no longer offer. If this is your problem, you have two options. You can just leave the 404 error page as it is, or if a suitable product replacement exists, use a 301 redirect. A 301 redirect is the best webpage redirection option for search engine friendliness. It’s relatively easy to implement and it usually allows you to keep your existing search engine rankings for the page in question. If you have to change file names or move pages around, it’s the safest option. Search engines read the code “301” as “moved permanently”.
- Out of Stock products – If you notice these happening to products you still intend to sell, it’s best to consult your CMS to see if these pages can display “out of stock” error messages instead of deleting your product pages altogether. That way, you keep your rankings juice, and you can let your customers know exactly when they can check back in to be first in/on line.
- Category/Sub-Category pages – Occasionally your important category pages might experience errors. More often than not a 301 redirect will be the best course of action here. Use 301’s to direct your visitors to the next best suitable page.
- Non-shop/catalogue pages – For all other non-shop related pages do your best to find the next best alternative and use a 301 redirect for your visitors. If nothing relevant exits, use a trusty 404 error page and explain yourself!
If you still need a 404 page, make it awesome
Don’t worry, we all have 404 errors from time to time that we can’t fix straight away. Still, that gives you no excuse to bore, infuriate and confuse your potential customers. Take the opportunity to ease the frustration of your visitor with your 404 page. Show some of your brand’s personality, help the user find related info and have a little fun…
Or show a GIF of a screaming goat. That’s what one of our favourite 404 exponents did. You have to visit our friends over at Bluegg for the full 404 sensory experience. The image alone doesn’t do it justice.
#2 Crawling Error: You’ve Got Sitemap Issues
Sitemaps make it easy for Google to find, crawl and index webpages on your site and we don’t need to tell you how important it is for Google to list your pages. So stay sharp and listen up to make sure you’re not guilty of this SEO no no.
How to check your sitemap
The best format option (when considering search engines) for your sitemap is an XML file. The quickest way to check if you have this sorted already is to type in “sitemap.xml” after your website in your internet browser’s address bar;
mydomain.com.au/sitemap.xml
If you can’t find anything here, things get a little more technical. You can search in your root directory folder in FTP and look for any file with an XML extension.
(If you thought we dropped into another language for a sentence here, you’re right. That was techie web nerd speak. Your IT team or web developer will be able to decode in a jiffy.)
If you find your sitemap, you need to make sure you have uploaded and processed the XML file through Webmaster tools. If it hasn’t been done already, Google has the answer wrapped up in a neat little Q&A package.
How to generate your sitemap
Most Content Management Systems automatically generate sitemaps, so it’s best to have a look around in the settings or configurations of your platform to see if you can find anything related to sitemaps. Before you launch into the more time consuming process of generating a sitemap, check out the support section of your CMS provider’s site for an easy answer.
If your store has no XML sitemap file, and you’d like some more customers coming your way from search engines, you’ll need to build one.
Head to one of the following XML sitemap generators (keep in mind that you may need to pay a small fee if your site is above the maximum page threshold for these providers).
Once you have successfully created your XML sitemap, follow these instructions and submit it through Webmaster tools.
#3 Crawling Error: Your Robots Are Creating Mischief
The SEO type of robot is quite a bit more boring than your regular sci-fi jobs. However, the ‘robots.txt’ file is an extremely important file for any site, particularly one selling products online.
The file gives instructions to web bots on which sections of the site they can and can’t visit.
If you get the robots.txt file setup wrong, you prevent web crawlers (like our friend the Google bot) from crawling and indexing your site. For an e-commerce site this means lost sales and online marketing sadness.
How to check if your robot file is set up correctly
Enter “yourdomain.com.au/robots.txt” into your browser’s address bar
If you have set things up correctly, your robots code will probably look like this:
User-agent:
*Disallow: /login
Disallow: /account
This tells the search bots to ignore the “/login” and “/account” directories of your site – typically pages you wouldn’t want displayed in search results.
If you have ‘disallows’ in your robots, make sure you have a look through them. You need to double check that important pages of your site are not being ignored. Chances are you don’t want ‘disallows’ stopping Google from finding your store’s category or product pages.
If you find anything you aren’t sure about, or if you don’t seem to have a robots file at all, you’re getting to the point where you might need some expert help.
Brave the lottery and call IT, or try your web developer. Otherwise, get an SEO pro involved. You’ll get to the bottom of things quickly, with your blood pressure and hairline left intact. This one’s best left to the nerds technical experts.
This is a hidden ecommerce site killer, lurking in the shadows and attaching a ball and chain to your rankings. Not many marketers are aware of the impact the structure of an ecommerce site can have on Google’s crawling ability.
#4 Crawling Error: Your Site Architecture Is Too Deep
This is a hidden ecommerce site killer, lurking in the shadows and attaching a ball and chain to your rankings. Not many marketers are aware of the impact the structure of an ecommerce site can have on Google’s crawling ability.
We often we come across site’s with layers and layers of unnecessary categories and sub-categories. Put simply, the deeper your site’s architecture, the more difficulty Google bots will have finding and indexing your deep pages (like product pages). We can’t have the crawlers missing these moneymakers.
By reducing the hierarchy of your site’s architecture (getting closer to the page structure on the left rather than the right below), you can make it easier for the crawlers to find your content. In addition, it’s easier for you to squeeze the rankings juice from your higher traffic pages (like your home page and category pages) down to those super important product pages.
You might not even know you have a problem. But time and time again we’ve seen ecommerce sites make significant rankings gains by flattening their site architecture.
Your Guide to Ecommerce Site Architecture
Lucky for you, we’ve written a whole post all about it. To find out how to check, fix and optimize your site architecture for customer attracting SEO glory, you’ll want to have a good read.
Crawl Before You Rank
If you’re having rankings problems – remember you have to get your ecommerce website to crawl before your rank.
Rather than jump in to onpage SEO troubleshooting straight away, it’s a good idea to check these 4 errors to make sure the Google bot can crawl and index your website.
Of course, there can be a myriad of other issues related to crawling and indexation – not just the 4 common issues I’ve covered in detail. If you come across a specific issue I haven’t mentioned, give me a yell on Twitter and I’ll get in touch with some advice for you.
If you’re confident that search engines can find your content, you can start to focus your SEO energy elsewhere. Now is a good time to start optimising your content to help you improve your rankings.
Get amongst our DIY Online Store Audit and help yourself to over 30 plus pages of specific ecommerce wisdom to help you and your team improve your rankings without paying for outsourced expertise.
We’ll have you crawling, walking and dominating your online niche in no time. Read up and thank me later.
