Google clarifies how Google's crawlers handle cache control headers

Google has added a new section to its crawler and fetcher documentation for HTTP caching, which clarifies how Google’s crawlers handle cache control headers. With that, Gary Illyes from Google also wrote a blog post named HTTP caching asking site owners to allow Google to cache its pages.

What is new. The new documentation now says:

“Google’s crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header.”

Google does not support other HTTP caching directives and if both ETag and Last-Modified response header fields are present in the HTTP response, Google’s crawlers use the ETag value.

Caching decreased. Gary Illyes from Google said in the blog post that the “number of requests that can be returned from local caches has decreased.” He said, “10 years ago about 0.026% of the total fetches were cacheable, which is already not that impressive; today that number is 0.017%.”

“If you’re in the business of making your users happy and perhaps also want to potentially save a few bucks on your hosting bill, talk to your hosting or CMS provider, or your developers about how to enable HTTP caching for your site. If nothing else, your users will like you a bit more,” Gary Illyes added.

Why we care. Caching may help Google crawl your site more efficiently, which may make for a happier Googlebot. There is no mention of any SEO or ranking benefit to using caching, there is also no mention of crawl budget benefit.

The above is the detailed content of Google clarifies how Google's crawlers handle cache control headers. For more information, please follow other related articles on the PHP Chinese website!