Check Server HTTP Headers with CURL

As you may have noticed, I’ve changed my web site’s domain recently. Therefore, I had to redirect all requests to the new address. This has been done and it works as expected, but how about taking a closer look at the HTTP responses the web server returns to the client if an old URL is requested?

This is where CURL comes handy once again. CURL’s command line options include two very useful switches, -I and -L:

-I : when used, CURL prints only the server response’s HTTP headers, instead of the page data.
-L : if the initial web server response indicates that the requested page has moved to a new location (redirect), CURL’s default behaviour is not to request the page at that new location, but just print the HTTP error message. This switch instructs CURL to make another request asking for the page at the new location whenever the web server returns a 3xx HTTP code.

The combination of the above two switches results in having all the server responses’ headers printed to the terminal, until CURL receives a code other than 3xx.

So, here is a real-life example. I have made two major changes to my web site so far; one included a URL structure modification when I moved from a pure HTML web site to WordPress, while the second was a domain change from the free domain test.example.net to the current paid domain name. So, this makes at least two permanent redirections. I say “at least”, because other necessary redirections could take place as well, for example, redirections of the example.org version of the domain to the www.example.org version, or redirections required for permalinks.

So, here is CURL’s output when I requested an old page of my web site:

$ curl -I -L http://test.example.net/servers/vnc.html

HTTP/1.1 301 Moved Permanently
Date: Fri, 06 Oct 2006 01:49:06 GMT
Server: Apache/2.2.2
Location: http://www.test.example.net/servers/vnc.html
Connection: close
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 301 Moved Permanently
Date: Fri, 06 Oct 2006 01:49:06 GMT
Server: Apache/2.2.2
Location: http://www.g-loaded.eu/servers/vnc.html
Connection: close
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 301 Moved Permanently
Date: Fri, 06 Oct 2006 01:49:06 GMT
Server: Apache/2.2.2
Location: http://www.g-loaded.eu/2005/11/10/configure-vnc-server-in-fedora/
Connection: close
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 200 OK
Date: Fri, 06 Oct 2006 01:49:06 GMT
Server: Apache/2.2.2
X-Powered-By: PHP/5.1.4
X-Pingback: http://www.g-loaded.eu/xmlrpc.php
Status: 200 OK
Connection: close
Content-Type: text/html; charset=UTF-8

The above output shows that the client has to send 4 HTTP requests and receive 4 respective responses from the server in order to reach that old page’s final location:

The first server response informs the client that the page has been permanently moved to the www version of the old domain.
The second response indicates that the page has been permanently moved to the www version of the new domain using the old URL structure.
The third indicates a permanent move to the new URL structure.
The final response informs the client with a 200 OK code that it has reached the page’s final location.

In this example, CURL provides a clear view of what situation a browser, a search engine bot or any other HTTP client encounters when it tries to reach an old page.

I am not an expert, but I guess that too many redirects are not a good thing, not only by considering the small increase of the server load, but also that search engine bots might not like them very much. On the other hand, redirections are a necessary evil if you want links from other websites pointing to your old domain or your old URL structure to continue to be valid, which in fact adds “value” to your website, as search engine experts would say. So it’s a matter of choice.

Related Articles