Публикация от account_disabled в Nov 27, 2023 6:56:23 GMT
ranking well. کیش April 20, 2012 at 10:04 pm i want to say some thing about “Robots.txt: Pages that are included in a Robots.txt file tell the search engines NOT to crawl these pages. However pages in a Robots.txt file can still accrue PageRank and can be indexed in search results, says Matt Cutts.” How Can A page being indexed without crawling? As pr my knowledge a page cant be indexed without crawling Andrew Shotland April 22, 2012 at 9:10 am There are several ways this can happen. For example, the URL was already indexed before yo Asia Mobile Number List
blocked it in /robots.txt. And in my experience, if there are en
ough links to a URL Google does not seem to respect the /robots.txt directive 100% of the time and often you get “thin” URL indexing. Weed Butter Guy November 5, 2012 at 8:23 am The prolbem with the /robots.txt is that they don’t have to follow it. It’s just a guideline. Like google please don’t go here. Since when did you not do something just because someone asked….. Wayne Cochran December 17, 2012 at 7:03 am One question that
I have is that I have a login page on my site that allows login thru Facebook, Twitter, etc. Due to two plugins that had different conflicting settings, and indexed, so all of links to the applications on social media sites were getting followed. In a case such as this, since I do not want “normal” visitors accessing this page, would this be an instance where it would be OK to noindex, nofollow. And how long does it usually take for Google to remove a page from indexing? Andrew Shotland December 17, 2012 at 9:26 am If these URLs are already in Google’s index, I would tag them with “noindex” first and wait for them to be purged. Once they are purged, you could either block them from being crawled via rel=”nofollow” tags on the links to them and/or via blocking them in /robots.txt. I would do both, just to be sure. In the trade, we call that “double-bagging”. Re how long it takes to.
blocked it in /robots.txt. And in my experience, if there are en
ough links to a URL Google does not seem to respect the /robots.txt directive 100% of the time and often you get “thin” URL indexing. Weed Butter Guy November 5, 2012 at 8:23 am The prolbem with the /robots.txt is that they don’t have to follow it. It’s just a guideline. Like google please don’t go here. Since when did you not do something just because someone asked….. Wayne Cochran December 17, 2012 at 7:03 am One question that
I have is that I have a login page on my site that allows login thru Facebook, Twitter, etc. Due to two plugins that had different conflicting settings, and indexed, so all of links to the applications on social media sites were getting followed. In a case such as this, since I do not want “normal” visitors accessing this page, would this be an instance where it would be OK to noindex, nofollow. And how long does it usually take for Google to remove a page from indexing? Andrew Shotland December 17, 2012 at 9:26 am If these URLs are already in Google’s index, I would tag them with “noindex” first and wait for them to be purged. Once they are purged, you could either block them from being crawled via rel=”nofollow” tags on the links to them and/or via blocking them in /robots.txt. I would do both, just to be sure. In the trade, we call that “double-bagging”. Re how long it takes to.