If a link to a website has a parameter in it and that parameter is blocked by robots .txt file, will that link lose its ranking value?

If a link to a website has a parameter in it and that parameter is blocked by robots .txt file, will that link lose its ranking value?

Webmaster Guru Asked on September 15, 2016 in Ranking Signal.
Add Comment
1 Answer(s)

JOHN MEULLER (Webmaster Analyst from Google):

Not necessarily. So what would probably happen here is this would be like a link to a robotted URL, just a general robotted URL. We don’t know what content is on that URL. But we do know there is some context for that URL based on the link, based on the anchor text, where that link is coming from– all of that. So from our point of view, what we will see is people are pointing at a URL on your website. It’s blocked by robots.txt. We don’t know what’s there. So what we’ll do is we’ll index that URL without the content and try to show that in the search results. So that’s something that sometimes makes sense. If there are pages that, for technical reasons, we can’t really crawl them because it causes issues on your server, then maybe a robotted URL is fine there. And that’s something we could show in the search results regardless. But on the other hand, if you’re using the robots. txt file to simplify duplicate content, then that’s probably not what you want to have happen. You don’t want to have these duplicates showing up in the search results. You want to have Google recognize these duplicates, fold them into one, and make that one URL a lot stronger. So if that’s the case, if you’re trying to solve a duplicate content problem here, I’d recommend looking at the other ways of handling duplicate content instead. So maybe you can set up a redirect for these parameters. If these are session IDs or something that you don’t really care about, maybe you can set up a redirect to your preferred version. You can almost always set a rel=canonical to your preferred version. Make sure that, of course, these URLs are crawlable, so that we can look at that URL and see, oh, it’s the duplicate of the other one that we already have. Therefore, all of these links that are going to this parameterized URL could be combined with that other URL that you have. And we can show that URL a little bit better in rankings. So those are the two aspects there. On the one hand, if you don’t want this URL crawled, maybe leaving it robotted is fine. On the other hand, if you’re doing this for duplicate content, I would not recommend using robots.txt. I’d recommend using one of the other methods. 

Webmaster Guru Answered on September 15, 2016.
Add Comment

Your Answer

By posting your answer, you agree to the privacy policy and terms of service.