Google — The All‐Knowing Decider of Indexing
Conflicts Within Site Signals Muddy The Process
Crawled Not Indexed
The “Noindex,Follow” Way to Wealth, Fame and Lost Value
But Wait — There’s More to It!
URL Parameters, Sitemap XML Files, and Inbound Links
Whenever discussing the crawl and indexation decision process, it’s important to also mention the fact that URL paramters, when set to “representative URL”, or “Let Googlebot Decide” can also muck up that decision process. The same is true for inclusion in sitemap XML files and when enough high value inbound links exist. All of these can influence, to varying degrees, how and what Google crawls and ultimately indexes. In spite of robots settings or canonical tags.
The SEO Indexing Bottom Line — Consistency
Okay that wasn’t the bottom line. It was just the last section label in this post. The best recommendation I have is one I repeat often in my audit work — Never leave it to Google to “figure it all out” when you have the ability to control, through consistency of signals, what you want their systems to do and how you want their systems to behave regarding your site.