.Google.com's John Mueller addressed a question regarding why Google marks webpages that are actually prohibited coming from crawling by robots.txt and why the it is actually risk-free to dismiss the relevant Look Console reports about those creeps.Robot Website Traffic To Inquiry Criterion URLs.The person talking to the question documented that crawlers were generating web links to non-existent question criterion Links (? q= xyz) to webpages with noindex meta tags that are actually likewise blocked in robots.txt. What cued the inquiry is actually that Google is actually creeping the hyperlinks to those webpages, obtaining shut out through robots.txt (without noticing a noindex robots meta tag) then getting shown up in Google.com Look Console as "Indexed, though obstructed through robots.txt.".The person inquired the complying with question:." But below's the significant concern: why will Google mark pages when they can not also see the web content? What's the benefit because?".Google.com's John Mueller confirmed that if they can not crawl the page they can not see the noindex meta tag. He also makes an appealing mention of the site: hunt driver, urging to dismiss the outcomes since the "ordinary" users will not find those results.He wrote:." Yes, you are actually right: if we can't crawl the webpage, our company can't find the noindex. That mentioned, if our experts can not creep the webpages, at that point there's not a whole lot for us to index. Therefore while you might observe a few of those web pages with a targeted website:- query, the typical consumer won't view all of them, so I wouldn't bother it. Noindex is likewise great (without robots.txt disallow), it only means the URLs will definitely find yourself being actually crept (and also find yourself in the Search Console report for crawled/not indexed-- neither of these standings result in problems to the rest of the internet site). The fundamental part is actually that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the restrictions in operation the Site: hunt accelerated search driver for analysis factors. One of those causes is actually given that it's certainly not linked to the regular search index, it is actually a distinct trait completely.Google's John Mueller talked about the web site search operator in 2021:." The quick response is actually that a site: question is certainly not suggested to become comprehensive, neither utilized for diagnostics purposes.A site question is a certain type of search that confines the results to a certain site. It is actually basically just words website, a digestive tract, and after that the internet site's domain.This question restricts the outcomes to a particular website. It is actually certainly not meant to be a complete assortment of all the pages from that site.".2. Noindex tag without utilizing a robots.txt is alright for these kinds of circumstances where a crawler is linking to non-existent web pages that are getting uncovered through Googlebot.3. URLs along with the noindex tag will certainly create a "crawled/not catalogued" item in Search Console which those will not possess a bad result on the remainder of the web site.Read the concern and answer on LinkedIn:.Why will Google.com index web pages when they can't even see the information?Included Graphic by Shutterstock/Krakenimages. com.