You’ve spent countless hours designing and developing your website, and now you’re wondering, are people actually finding it on Google? Or is your hard work going to waste?
One of the most important factors in determining whether or not your website shows up in search engine results is whether or not your pages are indexed.
In this article, we’re going to dive deep into the definition of indexed pages and explain everything you need to know to make sure your website is reaching your audience.
What are indexed pages?
First things first, let’s define what we mean by “indexed pages.”
When a search engine like Google, Bing, or DuckDuckGo crawls the Internet, it’s constantly looking for new websites and pages (for example, blog posts) to add to its index. This index is essentially a giant library or database of all the pages on the Internet that the search engine is aware of.
When a page is added to the index, it means that the search engine knows about it and will consider it when determining which pages to show in search results before deciding where to rank them.
So, in a nutshell, an indexed page is one that search engines know and recognise.
Why you should index pages
Now that you know the answer to “what are indexed pages?”, you might be wondering why it’s so important to make sure your pages are indexed.
It’s simple, really; if your pages aren’t indexed, they’re not going to show up in search results. In other words, all the time and effort you put into creating your website will be for nought if people can’t find it.
How to index pages
Alright, so you’re convinced that indexing your pages is important. But how do you actually go about doing it?
We got you. Here are a few methods you can use:
Submit a sitemap to search engines
As the name indicates, a sitemap is a map of your website. Technically, it’s a file that lists all the pages on your website and tells search engines where to find them.
By submitting your sitemap to search engines, you’re making it easy for them to find and index all the pages on your site.
First, generate a sitemap
There are several methods to generate a website sitemap, including:
Manually creating a sitemap
This method involves manually creating a sitemap document in XML format and listing all the URLs of your website in it. The process includes creating an XML file, adding the required tags, and listing all the URLs.
To be clear, we’re including this method just for reference. We don’t actually recommend following it because it’s time-consuming, and the process can be easily done following one of the other methods below.
Using a sitemap generator tool
There are several online sitemap generator tools that you can use. Those include XML Sitemaps, Screaming Frog, and SiteMap X, all of which can automatically generate a sitemap for any website. All you have to do is enter your website’s URL, and any of those tools will generate the sitemap in seconds.
Using a WordPress plugin
If you built your website using a content management system (CMS), such as WordPress, you can use a plugin to generate a sitemap directly from your website’s dashboard.
Here are some of the WordPress plugins capable of doing that:
- All-In-One SEO
- Yoast
- RankMath
- All-In-One SEO Pack
- Sitemap by BestWebSoft
- Jetpack
All you have to do is install and activate the plugin, then generate the sitemap from the settings page.
For example, here’s how to do it in Yoast:
1. First, open Yoast SEO
2. Head over to the Features tab
3. Scroll down and click on “See the XML sitemap”.
Using website builder tool
Popular website-building tools like Wix, Weebly, and Squarespace have built-in options to create a sitemap. That said, when using a website builder, you usually have the option to connect Google Search Console or Google Analytics to have any new pages you add submitted directly for indexing.
Now, submit it
To submit a sitemap to Google, for example, you can use the Google Search Console. Here’s how to do it:
- Go to Google Search Console
- Add your website if you haven’t done it already. Here’s a guide from Google on how to do it
- Select your website
- Click on ‘Sitemaps’ on the left sidebar
- Click on the red ‘Submit a Sitemap’ button
- Enter the sitemap URL (usually something like “yoursite.com/sitemap.xml”) and click ‘Submit’
Create and submit a robots.txt file
A robots.txt file is another way to tell search engines which pages they should and shouldn’t index.
What is robots.txt?
Robots.txt is a simple text file that instructs web robots (often referred to as “bots”), such as those of Google, Ahrefs, etc., on which pages or sections of a website they shouldn’t crawl and index.
It’s important to note that the robots.txt file is only a request, and not all web robots will obey the instructions it contains.
What does it do?
Basically, instead of telling Google, “here, index these pages”, like the other methods in this guide, robots.txt tells Google, “hey, leave these pages alone and don’t add them to your index.”
Wait, so there are pages that shouldn’t be indexed by Google?
Well, yes! Some of the pages that you don’t want to be indexed include the following:
- Login pages
- Admin pages
- Author archives on a one-author blog
- Internal search results
Here’s how to create this file
- Create a text file and name it ‘robots.txt’
- Add the following code to the file: User-agent: * Disallow: /path-to-page/
- Replace ‘/path-to-page/’ with the specific path of the page you don’t want to be indexed
- Upload the file to the root directory of your website
- Open ‘robots.txt Tester’ from Google. Here’s the link
- Add your robots.txt file
- Click on ‘Submit
Submit individual pages on Google Search Console
One of the coolest features of Google Search Console is that it allows you to submit individual pages for indexing. This method allows you to index pages and check if they’re on Google at the same time. You can follow it whenever you add a new post or page on your website.
Here’s how to do it :
- Go to Google Search Console
- Select your website
- Click on ‘URL inspection’ on the left sidebar
- Enter the URL of the page you want to submit for indexing
- Click on ‘Inspect’
- If the page is not indexed, click on ‘Request indexing.’
- Voila!
Which method to follow?
After creating a new website, you should submit the sitemap to Google and de-index the pages mentioned in the robots.txt section. Then, Google will regularly crawl your website to check if you’ve added new pages or posts. That said, the process may take time, so if you want a page or post to be indexed faster you should submit it to Google using the Search Console method.
What are indexed pages? How to check if a page is indexed
Now that you know how to get your pages indexed, you’ll want to make sure that the process is actually working. Here’s how to do that:
Using Google Search Console
You can check if a page is indexed by using the “coverage” report in Google Search Console.
- Go to Google Search Console
- Select your website
- Click on “Pages” on the left sidebar
- The report will show you the indexed and non-indexed pages on your website, along with the reasons for any indexing problems
Use a search engine query
You can also check if a page is indexed by searching for the page in question on the search engine and seeing if it shows up in the results. The problem with this method is that it doesn’t tell you the reason why a page doesn’t show up.
Use third-party index-checking tools
There are also third-party tools available that can check if a page is indexed, such as “Google Index Checker” or “Google Index Checker Tool.” These tools will check the index status of your website and pages by pinging the search engine and then give you a report of the indexed and non-indexed pages.
Keep in mind that we’re only including this for reference. We don’t actually recommend using such a tool since Google does allow you to check the index status of your pages using GSC.
Reasons your pages may not be indexed
There are several reasons that can explain why a page on your website may be missing from the search engine index. According to Search Engine Journal, those reasons include:
The page is blocked by robots.txt
If a website’s robots.txt file specifically disallows search engine crawlers from accessing a page (we’ve explained how to do that for certain pages above), search engines won’t index it.
The page has low-quality content
Search engines may not index pages that they consider to have low-quality content, such as pages thin or duplicate articles. To fix this, create high-quality content by following our content creation guides:
The page is not mobile-friendly
With mobile-first indexing, Google prioritises mobile-friendly pages for indexing and ranking over pages that are not mobile-friendly. We covered how to make sure your content is mobile-friendly in this guide.
The page has a canonical issue
If a page has a canonical tag pointing to a different URL, search engines may index the canonical URL instead of the original URL.
The page has slow loading times
Search engines may not index pages that take too long to load. Check these guides to learn how to make your website load faster:
The page has been penalised by the search engine
If a page has been penalised by a search engine for violating its guidelines, it may not be indexed.
If you post an article or a page and it doesn’t get picked up by Google for long, consider checking if the page in question has any of those issues.
Need some help?
Now that you understand what are indexed pages, the world is your oyster, so go out there and make sure your website is being seen by all the right people!
That said, it’s important to note that indexing your pages is just one aspect of SEO. Other important factors, such as keyword research, meta tags, and high-quality content, will also play a key role in improving your website’s visibility and performance. If you feel that the process is overwhelming, worry not, as we can help. Just book a discovery call with us today!
0 Comments