If you’ve ever had any of your links shared on twitter, you have potentially just lost a load of link equity which was coming from embedded feeds.
Many sites use the Twitter API to display their user or brand feeds via a backend script (normally PHP or ASP), and as Twitter’s URL shortening service (t.co) has been passing link juice for years, it was easy to gain some (albiet shortlived) links by doing something to get a major twitter account to tweet about your site, or retweet you. This was possible because the twitter feeds display t.co links; t.co uses a 301 redirect, and (until recently) there were no robots directives which stopped search crawlers from passing on that link juice (note most of the links on twitter.com itself had the rel=”nofollow” attribute – more about that in a second)
Embedded tweets have always passed PageRank through a 301
These embedded tweet feeds are quite widespread – you can find them everywhere from corporate blogs to governmental sites. Also, as I wrote about last year, the main Twitter site was leaking PageRank through dofollow links for around six months.
No more following
A few days ago, on June the 13th 2013, Twitter changed the t.co/robots.txt file to block all crawlers.
Before – t.co/robots.txt mirrored the main twitter.com/robots.txt; allowing the t.co short URL’s to pass link juice:
Now – It’s a sad day as t.co brutally blocks out the world, like an angsty teenager who insists that no one could ever understand
How could they do this to me?
There’s no official word yet – it doesn’t really seem like anyone’s noticed. My theory is that they’re cracking down on blackhatters who were taking advantage of the above leak; but of course I would think that – I live and breathe SEO. They could have been trying to chill out the massive server demand that search robots make on them; or even trying to stop browser plugins which swap out twitter links with the real equivalents.
or maybe they realised that copying and pasting the robots.txt file from twitter.com to t.co was a fucking retarded thing to do. Who am I to say? My guess is as good as yours.
What can I do?
If you REALLY want to pass link juice from any random link you post on twitter, you could write a script which looks up the real URL before sending it to the browser. It’s probably not worth it though – the big gains in this bug were always getting links on sites you didn’t control by doing something to get the account owners to retweet you.