Home | Search Engine Optimization | Canonical Tag Delays with Googlebot in Web vs Mobile Index

Canonical Tag Delays with Googlebot in Web vs Mobile Index

John Doherty —  December 20, 2011
  • Buffer
  • Vote on Hacker News
  • Sharebar
  • Buffer
  • Vote on Hacker News

I have recently been noticing an intriguing trend when I publish blog posts. When my posts publish, they are automatically tweeted out by a plugin called WordTwit. My posts are always indexed within two to four minutes.

What I am noticing, though, is that my posts are increasingly being indexed with the “?utm_source=rss” parameter, like so:

Click to Enlarge

Within a matter of a few hours, usually 6-8, Google picks up the canonical version of the post and the old drops out of the index. What is intriguing, though, is that I see something different on a mobile device. See this screenshot from my phone:

No UTM source parameter!

And yes, the canonical tag is present:

Click to Enlarge

What’s Going On?

I dug into log files to see what is happening from the perspective of Googlebot/Googlbot-Mobile to see if it might provide me a bit of insight.

What I found is that Googlebot Mobile never hit my site. Instead, Googlebot hit my site twice within 10 seconds, though it hit different URLs.

Check it out:

Click to Enlarge

This was at 9:15:15am and 9:15:25am (central time, which is an hour behind my NYC time).

Weirdly, an iPhone device hit the RSS version at 10:02, which I can only presume was someone reading the post on their mobile device:

Not Googlebot though.

The above was not Googlebot, however, because it did not come from the 66.249.67 IP range.

Then again at 10:06, Googlebot hit the correct URL:

Click to Enlarge


What I cannot figure out is why the mobile version would show correctly, yet it takes about 6-8 hours for the canonical version to show up in the main index? The only thing I can think of is that the index that is displayed to mobile devices is in a different data center and thus the data takes a while to populate across all the servers. But 8 hours? That seems like a lot.

Now let’s see what happens with those post.

Anyone have other thoughts? Why would Google show the non-canonical version in the regular index for a few hours, yet it shows correctly on mobile?

John Doherty

Posts

I'm the Senior Marketing Manager of HotPads.com, based in San Francisco. Previous to Hotpads I worked at Distilled for 2 years as an online marketing consultant. In my spare time I shoot lifestyle photography, ski, rock climb, and update my Twitter and Google+ accounts.

2 responses to Canonical Tag Delays with Googlebot in Web vs Mobile Index

  1. Have you tested this on other sites too? And instead of tweeting the post, try a Facebook share or a social bookmark site, just as a ping. And are you using any sitemap plugins that might ping google?

    • Hey Bob, I have not. I like your idea of publishing on other social networks to see what the difference might be. I will do this with future posts and report back!

      Thanks for the comment!