I have a client that uses Demandware in order to sell on his own shop.And now, this causes duplicate content, the same content under different URLs, besides blocking crawling and indexation via robots.txt, what would you recommend?

I have a client that uses Demandware in order to sell on his own shop.And now, this causes duplicate content, the same content under different URLs, besides blocking crawling and indexation via robots.txt, what would you recommend?

Webmaster Guru Asked on November 29, 2016 in Duplicate Content.
Add Comment
1 Answer(s)

JOHN MEULLER (Webmaster Analyst from Google):

So I don’t really know what Demandware is. I am going to assume this is a kind of an affiliate setup maybe where someone else is selling the same content as well. Or maybe this is a kind of shop set up. I don’t know. But in general, when it comes to duplicate content where you have the same content under different URLs, our main recommendation is to try to solve that within your website directly, for example, by using a rel canonical on those pages so that you tell us which of these pages you think belong together, and then we can fold them together. Blocking, crawling by robots.txt is usually not recommended unless you really have the situation where crawling your content itself causes so many problems that you need to block that. But just for normal duplicate content, that’s not something that you would need to do. The thing that would happen with the robots.txt file if you use that to block crawling, is that we would not be able to recognize that this is duplicate content. And if anyone were to link to one of those URLs that you have blocked, which could also be within your website, then we wouldn’t know what to do with that. And we wouldn’t be able to forward this context of that link. Whereas, if you use a rel canonical on those pages, then we would know, oh, you mean this link that’s going to this page is actually meant to go to this page here. So we can forward that and make that final destination page a little bit stronger so that it shows a bit more visibly in search as well.

Webmaster Guru Answered on November 29, 2016.
Add Comment

Your Answer

By posting your answer, you agree to the privacy policy and terms of service.