How to Use curl to Get HTTP Headers

Configuring a webserver like nginx or Apache is pretty simple to get a webpage served. Things can get trickier when you’re making configuration changes that are not visible when you check the page in a browser. These are changes like enabling GZIP compression or HTTP/2.

The page will look exactly the same if they are enabled or not. Sure, you can use a third-party site to evaluate your website but that always feels like cheating.

This is where curl comes in. curl is a command line utility that transfers data using URLs. It supports, well, pretty much everything but the feature that we’re intereseted in here is that it will print the HTTP headers that are sent with a web page request.

The headers let the client and server pass additional information about HTTP sessions. This additional information will include information about if compression or HTTP/2 is supported. This means that you can discover if your new web server will serve pages via HTTP/2 or with GZIP by looking at the HTTP headers.

The first option to use with curl is the -I flag which only requests the headers:

$ curl -I http://bash-prompt.net
HTTP/1.1 301 Moved Permanently
Server: nginx/1.14.2
Date: Wed, 22 Apr 2020 10:49:32 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://bash-prompt.net/
Strict-Transport-Security: max-age=15768000

This immediatly tells us that this page is not availble at http://bash-prompt.net but is abailable at https://bash-prompt.net/ by a 301 redirect. This is how Let’s Encrypt auto-configure nginx or Apache to direct all traffic to the HTTPS version of a site.

Following the redirect and running curl against https://bash-prompt.net gives us:

$ curl -I https://bash-prompt.net
HTTP/2 200
server: nginx/1.14.2
date: Wed, 22 Apr 2020 10:54:04 GMT
content-type: text/html
content-length: 3687
last-modified: Sat, 18 Apr 2020 11:37:57 GMT
vary: Accept-Encoding
etag: "5e9ae695-e67"
strict-transport-security: max-age=15768000
accept-ranges: bytes

This lets us know that the site is being served with HTTP/2, the webserver and other information. In order to check if the site is being served with compression you need to ask the webserver if a particular compression type is supported. This is done with by using -H Accept-Encoding: <TYPE> option.

This will check for the three supported methods of compression gzip, br (Brotoli) or lzma:

E.g.:

$ curl -I -H 'Accept-Encoding: gzip' https://bash-prompt.net
HTTP/2 200 
server: nginx/1.14.2
date: Wed, 22 Apr 2020 11:22:50 GMT
content-type: text/html
last-modified: Sat, 18 Apr 2020 11:37:57 GMT
vary: Accept-Encoding
etag: W/"5e9ae695-e67"
strict-transport-security: max-age=15768000
content-encoding: gzip

The content-encoding: gzip reply indicates that GZIP is supported by the webserver.

Finally, if you pass the -v with outputs the entire connection which is really useful to see the TLS negiotation going on:

$ curl -Iv https://bash-prompt.net
*   Trying 138.68.52.22:443...
* Connected to bash-prompt.net (138.68.52.22) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=bash-prompt.net
*  start date: Apr 12 22:07:16 2020 GMT
*  expire date: Jul 11 22:07:16 2020 GMT
*  subjectAltName: host "bash-prompt.net" matched cert's "bash-prompt.net"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55dfb32878b0)
> HEAD / HTTP/2
> Host: bash-prompt.net
> user-agent: curl/7.69.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
HTTP/2 200 
< server: nginx/1.14.2
server: nginx/1.14.2
< date: Wed, 22 Apr 2020 11:24:38 GMT
date: Wed, 22 Apr 2020 11:24:38 GMT
< content-type: text/html
content-type: text/html
< content-length: 3687
content-length: 3687
< last-modified: Sat, 18 Apr 2020 11:37:57 GMT
last-modified: Sat, 18 Apr 2020 11:37:57 GMT
< vary: Accept-Encoding
vary: Accept-Encoding
< etag: "5e9ae695-e67"
etag: "5e9ae695-e67"
< strict-transport-security: max-age=15768000
strict-transport-security: max-age=15768000
< accept-ranges: bytes
accept-ranges: bytes

< 
* Connection #0 to host bash-prompt.net left intact