A few months ago, I analyzed a website having to deal with an indexing problem on Google.
The site in question contained some older pages that were properly indexed. However, every time a new article was published, it was rarely added to Google's index, except in biblical times.
This case study may interest you if you find yourself in one or more of the following situations:
• You already have content indexed.
• New posts (or pages) are not indexed.
• Contents are indexed only after a few weeks or months.
If you have a search engine indexing problem, read on to find out how I solved it.
The situation is very different if the site does not appear on Google at all, not even on the home page. In this case, there may be a block, such as the "noindex" meta tag or the "disallow" in the "robots.txt" file, which prevents spiders from accessing the site.
Here's a standard robots.txt file. Please replace "yourURL" with your website's URL and ensure that it's not blocked by robots.txt.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://yourURL.com/sitemap.xml
Ensure you have the correct robots.txt configuration to prevent blocking the crawler.
Indexing vs. Ranking
Before starting, I make a small disclaimer.
The case study I will present to you concerns indexing and not ranking—two completely different aspects.
1. It all starts with crawling, the phase in which crawlers scan the pages of your website.
2. If the crawling goes smoothly and there are no errors, The pages can be added to the Google index.
3. Ranking on Google, on the other hand, concerns the position in which a page appears in search results. This depends on several factors, such as the relevance of the content, the authority and popularity of the website, and the structure of the site itself.
Before thinking about ranking in the first search results, you need to understand why there are pages that Google is not indexing.
How To Check If Pages Are Indexed
Returning to my case study, after a careful analysis of the website, I noticed that the new articles, even if optimized for search engines, were not included in the Google index.
To check if a page on your site has been indexed, you can use two different methodologies:
The first method involves a simple search in the Google search bar, typing something like "site:websitename.com keyword" or using the command "info:https://websitename.com/url-article".
Secondly, you can access Google's Search Console and click on the "Pages" item in the left menu. In this section, you can check which pages have been indexed and how many have been excluded from the index.
You can also individually check each URL using the inspect tool section.
Why Is Google Not Indexing My Pages?
What are the reasons why Google might decide to exclude a site's pages from indexing? Let's look at the most frequent causes:
2. Duplicate Content: The pages contain duplicate content or compete with a keyword very similar to other pages already indexed by Google.
3. Navigation: The menu or general navigation of the site is unclear.
4. Site Speed: The site is slow or has problems loading pages.
5. Low-Quality Content: The pages have content that is not very relevant for search purposes and/or is of low quality.
6. Backlinks: You have obtained backlinks of dubious quality or attempted to manipulate the SERP.
These are the technical reasons that could affect correct indexing.
Here are other possible reasons why Google hasn't indexed your site/post:
• Your website is new and hasn't been crawled by Google yet.
• The content on your posts is very similar or duplicate from other existing pages.
• You haven't submitted your sitemap to Google, making it unaware of your site structure.
• Your website's SEO is not optimized,
• You have used a "noindex" meta tag or directive, preventing search engines from indexing the posts.
• Not a Mobile friendly website
• You website don't have any backlinks
• Low value content
Now, let's look at specific examples where the entire site has already been indexed, but new articles are not included in the index even after several weeks.
• The site stopped publishing articles for a long period of time, so Google needs to be given time to understand that the site will publish regularly from now on.
• The site has changed topics or introduced topics that are unrelated to those previously covered. In this case, it is important to find a way to create a connection between the new topics and the previous ones.
• Pages (or even the entire site) are blocked by the noindex tag or the robots.txt file. In this case, you need to remove the restrictions to allow Google to index your site correctly.
The first case is exactly the problem I had posited for the website I performed the analysis on.
In the website I analyzed, I posited that the first reason why the new articles were not indexed was that they dealt with a topic completely distant from the previous ones, without any connection between them.
Imagine being an expert on a certain topic, as if Google knows you as such, and suddenly you start talking about something completely different.
If you also find yourself in this condition, ask yourself some questions:
• Is the new topic relevant to my business?
• How can it help me achieve my goals and sell my service?
How to Index an Article on Google?
Let's again take the case study as an example, where there was already a website with several pages indexed and ranked on Google, but the new pages were not indexed.
What were the first approaches I took to get new pages indexed on Google?
The first step was to submit the URLs of the new pages using the “URL Inspection” tool in Search Console.
Since the first point did not give the desired results, I used Rank Math's Instant Indexing plugin, which allows you to communicate new updates to Google via API.
To speed up indexing, I added the “Latest articles” section on the homepage and sidebar of the blog(and it worked for me). Furthermore, I worked on internal linking, creating links to new articles.
After a long period of time, article indexing resumed in full swing, and new pages were added to Google's index within a few days.
Was it enough to report the new content to Google and optimize internal linking?
No, I had to do other work to optimize the website for search engines. First, I ensured that the new content was related and linked to previously published content; then, I improved the site's performance.
Google Indexing Times
By analyzing different websites, particularly blogs that publish or have started publishing constantly, I have been able to verify that indexing times vary greatly. To give you clearer examples:
• A blog with a very specific editorial plan that has consistently published every week for many years: Google indexes new articles in a short time, even after a few hours.
• A website that publishes new content once a month and publishes articles that are extremely related (and linked) to articles already published: gets indexed within a couple of days.
• A blog that has not published articles for a few months needs to get back on track, and it may take a few days or weeks before the new content is taken into consideration and included in the search engine index.
When it comes to SEO optimization, results aren't immediate; it can take weeks or even months to see the desired outcomes.
Conclusion
Why is Google not indexing my pages? If your website is new, then probably that's the main reason why Google is not indexing your site. If this is your case, my advice is to build high-quality backlinks. It will take time, and after a period, Google understands that, "Oh, this website is consistent," and will index each and every post you upload unless the posts are duplicates and low value. We will provide more information regarding indexing in the future, so don't forget to follow us.
