Home | Search Engine Optimization | Canonical Tag Delays with Googlebot in Web vs Mobile Index

Canonical Tag Delays with Googlebot in Web vs Mobile Index

John Doherty —  December 20, 2011
  • Buffer
  • Vote on Hacker News
  • Sharebar
  • Buffer
  • Vote on Hacker News

I have recently been noticing an intriguing trend when I publish blog posts. When my posts publish, they are automatically tweeted out by a plugin called WordTwit. My posts are always indexed within two to four minutes.

What I am noticing, though, is that my posts are increasingly being indexed with the “?utm_source=rss” parameter, like so:

Click to Enlarge

Within a matter of a few hours, usually 6-8, Google picks up the canonical version of the post and the old drops out of the index. What is intriguing, though, is that I see something different on a mobile device. See this screenshot from my phone:

No UTM source parameter!

And yes, the canonical tag is present:

Click to Enlarge

What’s Going On?

I dug into log files to see what is happening from the perspective of Googlebot/Googlbot-Mobile to see if it might provide me a bit of insight.

What I found is that Googlebot Mobile never hit my site. Instead, Googlebot hit my site twice within 10 seconds, though it hit different URLs.

Check it out:

Click to Enlarge

This was at 9:15:15am and 9:15:25am (central time, which is an hour behind my NYC time).

Weirdly, an iPhone device hit the RSS version at 10:02, which I can only presume was someone reading the post on their mobile device:

Not Googlebot though.

The above was not Googlebot, however, because it did not come from the 66.249.67 IP range.

Then again at 10:06, Googlebot hit the correct URL:

Click to Enlarge


What I cannot figure out is why the mobile version would show correctly, yet it takes about 6-8 hours for the canonical version to show up in the main index? The only thing I can think of is that the index that is displayed to mobile devices is in a different data center and thus the data takes a while to populate across all the servers. But 8 hours? That seems like a lot.

Now let’s see what happens with those post.

Anyone have other thoughts? Why would Google show the non-canonical version in the regular index for a few hours, yet it shows correctly on mobile?

John Doherty

Posts

I'm the new (as of October 2013) Online Marketing Manager of Hotpads.com, soon to be based in San Francisco. Previous to Hotpads I worked at Distilled for 2 years as an online marketing consultant. In my spare time I shoot lifestyle photography, explore new and interesting food in New York, ski, rock climb, and update my Twitter and Google+ accounts.

2 responses to Canonical Tag Delays with Googlebot in Web vs Mobile Index

  1. Have you tested this on other sites too? And instead of tweeting the post, try a Facebook share or a social bookmark site, just as a ping. And are you using any sitemap plugins that might ping google?

    • Hey Bob, I have not. I like your idea of publishing on other social networks to see what the difference might be. I will do this with future posts and report back!

      Thanks for the comment!