Grow Your Business Online.
Submit a ticket My Tickets
Welcome
Login  Sign up

I'm getting a crawl error on Cloudflare cdn-cgi files.

If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow

Google from crawling that with a rule in robots.txt for:


Disallow: /cdn-cgi/



An additional option using WordPress:


Solution provided by Ben


You can make modifications in your /wp-includes/functions.php file. to create an additional Disallow rule.

A sample of what it would look like is below:

function do_robots() { 
header( 'Content-Type: text/plain; charset=utf-8' );

do_action( 'do_robotstxt' );

$output = "User-agent: *\n";
$public = get_option( 'blog_public' ); 
if ( '0' == $public ) { 
$output .= "Disallow: /\n"; 
} else { 
$site_url = parse_url( site_url() ); 
$path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : ''; 
$output .= "Disallow: $path/wp-admin/\n"; 
$output .= "Disallow: $path/wp-includes/\n"; 
$output .= "Disallow: $path/cdn-cgi/\n"; 
}



Note:

This won't impact the actual crawl for your site at all. The error for /cdn-cgi/ is also not going to impact

your rankings at all. Google is attempting to crawl a Cloudflare resource, one that doesn't need to be

crawled, and they possibly can't read the DOM or JavaScript in that resource.



Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.