When installing a configuring and modifying a website it is frequently necessary to make changes to the way that the webserver is serving the site e.g. enabling mod_deflate or enabling HTTP2. However, these changes can be somewhat difficult to verify by viewing the site in a browser.
There are lots of websites that will do the checking for you such as GiftOfSpeed.com which will check if GZIP is enabled. However, this being Linux, if you’re on the command line and you have a problem, it’s an almost given certainty that some clever person met the same problem before you and there’s a tool that will do the job.
Using Telnet To Get Basic Information
I have written about using Telnet before to establish if a networked application is listening for network connections. Telnet will work for HTTP and HTTPS to establish if you are able to connect to the webserver.
HTTP is a text protocol which means that we can actually “talk” to the webserver and request an asset such as a page or an image. Along with the asset will come some header information which will give additional information such as a redirect.
The syntax for communication with a webserver is very simple. First, open the connection:
# telnet bash-prompt.net 80
telnet bash-prompt.net 80
Trying 78.31.109.134...
Connected to bash-prompt.net.
Escape character is '^]'.
Now we need to enter whether we want to GET or POST (ask for or send data to the server), the asset we want and the protocol. Requesting the default home page for Bash-Prompt.net gives us:
GET / HTTP/1.1
The / refers to the root directory of the site. When no specific file is listed the webserver will, according to its configuration, decide what the default file is going to be. As this could be index.html, index.htm or index.php it’s usually easier to leave it simply as / unless you know what file you want.
HTTP/1.1 is the protocol and version. All webserver will be able to serve HTTP1.1 so for testing simply leave it as that.
The next line is technically optional but required in practice because almost all webserver run more than one site on a single IP address. This is known as “Virtual Hosting”. If there is more than one site on an IP the web server will only be able to determine which site you want by the supplied host-header.
In the case of Apache2, the supplied host-header must match the ServerName or ServerAlias in the VirtualHost configuration for the site. The host-header is supplied as follows:
Host: bash-prompt.net
After you have entered the Host: line hit Enter twice. Once you hit Enter` on an empty line the webserver will process the request and send back your file as a text stream along with some header information. This is what the whole transaction looks like for connecting to this website:
# telnet bash-prompt.net 80
Trying 78.31.109.134...
Connected to bash-prompt.net.
Escape character is '^]'.
GET / HTTP/1.1
Host: bash-prompt.net
HTTP/1.1 301 Moved Permanently
Date: Fri, 09 Mar 2018 07:22:05 GMT
Server: Apache
Location: https://bash-prompt.net/
Content-Length: 232
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://bash-prompt.net/">here</a>.</p>
</body></html>
Connection closed by foreign host.
As you can see, my webserver is listening on port 80 for HTTP requests but will respond with a 301 redirection response which all modern browsers will automatically follow to the new location i.e. https://bash-prompt.net.
Using Telnet very quickly allows you to check that the webserver is:
- Listening for requests.
- Listening on the correct IP.
- Knows about the host-header you supplied.
All of which allows you to test that the webserver is working before it goes live on the internet or before DNS entries are live.
Using CURL For More Advanced Information; HTTP2 and GZIP/mod_deflate
Telnet is fine for investigating the more basic setting of a web server but we sometimes need to know more specific information particularly when enabling certain features.
CURL is primarily a tool for transferring data, such as download files and is used in a huge number of applications. CURL also possess the ability to print out header information when making a request to a website. These headers will tell us if certain features are enabled or not.
The -I flag instructs curl to display the default header information and not download the website file. Using the -I option for this Bash-Prompt.net gives the following output:
curl -I http://bash-prompt.net
HTTP/1.1 301 Moved Permanently
Date: Fri, 09 Mar 2018 07:40:10 GMT
Server: Apache
Location: https://bash-prompt.net/
Content-Type: text/html; charset=iso-8859-1
This tells us, just Telnet did, that there is a 301 redirection in place. Running the same command on the HTTPS URL gives the following:
# curl -I https://bash-prompt.net
HTTP/1.1 200 OK
Date: Fri, 09 Mar 2018 07:50:26 GMT
Server: Apache
Last-Modified: Thu, 08 Feb 2018 08:38:39 GMT
ETag: "19f7-564af56a262c0"
Accept-Ranges: bytes
Content-Length: 6647
Vary: Accept-Encoding
Content-Type: text/html
If you have successfully enabled HTTP2 then the output will change to:
# curl -I https://bash-prompt.net
HTTP/2 200
date: Fri, 09 Mar 2018 07:51:54 GMT
server: Apache
last-modified: Thu, 08 Feb 2018 08:38:39 GMT
etag: "19f7-564af56a262c0"
accept-ranges: bytes
content-length: 6647
vary: Accept-Encoding
content-type: text/html
The protocol line now reads HTTP/2 indicating that HTTPS connections will now get served via the HTTP2 protocol.
CURL will show if your site is using GZIP by instructing it to ask for a specific header response from the webserver. This is done with the -H flag followed by the header e.g. -H 'Accept-Encoding: gzip, deflate'. This gives us the command and its output:
# curl -I -H 'Accept-Encoding: gzip, deflate' https://bash-prompt.net
HTTP/2 200
date: Fri, 09 Mar 2018 07:56:22 GMT
server: Apache
last-modified: Thu, 08 Feb 2018 08:38:39 GMT
etag: "19f7-564af56a262c0-gzip"
accept-ranges: bytes
vary: Accept-Encoding
content-encoding: gzip
content-length: 2175
content-type: text/html
There are several allowed HTTP compression algorithms which can also be requested e.g. Googles Brotli can be tested by adding br to the list of algorithms e.g.:
curl -I -H 'Accept-Encoding: gzip, deflate, br' https://bash-prompt.net
Finally, if you would like to see the entire transaction between CURL and the webserver add -v. CURL will show everything it sends and every answer it receives including the HTTPS certificate details, protocols, compression etc. For Bash-Prompt.net this looks like:
curl -I -v -H 'Accept-Encoding: gzip,deflate' https://bash-prompt.net
* Rebuilt URL to: https://bash-prompt.net/
* Trying 78.31.109.134...
* TCP_NODELAY set
* Connected to bash-prompt.net (78.31.109.134) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=bash-prompt.net
* start date: Feb 13 03:27:13 2018 GMT
* expire date: May 14 03:27:13 2018 GMT
* subjectAltName: host "bash-prompt.net" matched cert's "bash-prompt.net"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56291a87eda0)
> HEAD / HTTP/1.1
> Host: bash-prompt.net
> User-Agent: curl/7.52.1
> Accept: */*
> Accept-Encoding: gzip,deflate
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
HTTP/2 200
< date: Fri, 09 Mar 2018 08:06:44 GMT
date: Fri, 09 Mar 2018 08:06:44 GMT
< server: Apache
server: Apache
< last-modified: Thu, 08 Feb 2018 08:38:39 GMT
last-modified: Thu, 08 Feb 2018 08:38:39 GMT
< etag: "19f7-564af56a262c0-gzip"
etag: "19f7-564af56a262c0-gzip"
< accept-ranges: bytes
accept-ranges: bytes
< vary: Accept-Encoding
vary: Accept-Encoding
< content-encoding: gzip
content-encoding: gzip
< content-length: 2175
content-length: 2175
< content-type: text/html
content-type: text/html
<
* Curl_http_done: called premature == 0
* Connection #0 to host bash-prompt.net left intact
These two tools will enable you to learn exactly how your webserver is serving pages without having to resort to external sites.