Overview of packet tracing with the ssldump utility

sol10209: Overview of packet tracing with the ssldump utility
The Secure Socket Layer (SSL) protocol is used to encrypt sensitive data for transmission on the Internet. If a BIG-IP LTM system is contributing to a technical issue, it may be helpful to decrypt the application data to better understand the issue. The ssldump utility is an SSL/TLS network protocol analyzer, which identifies TCP connections from a chosen packet trace or network interface and attempts to interpret them as SSL/TLS traffic. When the ssldump utility identifies SSL/TLS traffic, it decodes the records and displays them in text to standard output. If provided with the private key that was used to encrypt the connections, the ssldump utility may also be able to decrypt the connections and display the application data traffic.

You can use the ssldump utility to examine, decrypt, and decode SSL-encrypted packet streams managed by the BIG-IP system. The ssldump utility can act on packet streams real-time as they traverse the system, or on a packet capture file saved in the libpcap format, such as that produced by the tcpdump utility. Although it is possible for the ssldump utility to decode and display live traffic real-time as it traverses the BIG-IP system, it is rarely the most effective method to examine the voluminous and complex output of the ssldump utility. Capturing the target traffic to a file using the tcpdump utility, then decoding the file using the ssldump utility offers a better opportunity to examine the traffic in detail.

Capturing the target traffic

Examining the SSL handshake and other SSL record messages

Examining the decrypted application data

Capturing the target traffic

To begin, you can capture a packet trace that contains the SSL traffic you want to examine.

Note: For more information about capturing using the tcpdump utility on the BIG-IP system, refer to .

When you capture SSL conversations for ssldump examination, observe the following guidelines:

• If you use a browser to test, first close all existing browser windows and then use a newly-opened browser window to reproduce the issue to ensure a new session key is used. The ssldump utility cannot decrypt traffic for which the handshake including the key exchange was not seen.
• To write the captured packets to a file for examination with the ssldump utility, you must specify the -w option with the name of the file to which the captured data should be stored.
• Use the -i option to specify the interface or VLAN from which traffic is to be captured.
• Use the appropriate tcpdump filters to include only the traffic you want to examine.
• If you want to decrypt and examine the application data, you must capture the entire packet by specifying a value of 0 or the maximum size of the target packet to the -s option.
• Consider using the -v (verbose) option to increase the level of detail captured.

Note: For more information, refer to the Capturing Packet Data section in .

For example, if you want to save for examination client-side traffic to a specific SSL virtual server listening on the VLAN external, the following command includes the appropriate options and filters on the virtual server's IP address and port:

tcpdump -vvv -s 0 -nni external -w /var/tmp/www-ssl-client.cap host 10.1.1.100 and port 443

If you want to examine server-side traffic from one client to any pool member, use the -i option to specify the VLAN on which the servers reside, and filter on the client IP address, the server subnet, and the port on which the servers are listening. To do so, use the following command:

tcpdump -vvv -s 0 -nni internal -w /var/tmp/www-ssl-server.cap host 192.168.22.33 and net 10.1.1.0/24 and port 8080 

The traffic matching the specified filter is saved to the indicated capture file.

The options used are:

• -vvv  Maximum verbosity
• -s      Snaplength (0 captures full packets)
• -nn    Do not resolve host or service names
• -i       Interface - can be ifname or vlan name
• -w     Write output to file

  Note: For more information regarding tcpdump options, refer to the man page for tcpdump.

Examining the SSL handshake and other SSL record messages

SSL connections are established on top of an existing TCP connection using an SSL handshake that accomplishes the following:

• The client and server negotiate security capabilities, such as the public-key algorithm, the symmetric key algorithm, and compression algorithms.
• The server transmits its certificate to the client, allowing the client to validate the identity of the server.
• The client and server exchange session key information.
• The client may also send its certificate to the server, allowing the server to validate the identity of the client.

The handshake transactions consist of a number of SSL record messages. These messages can be examined by executing the ssldump utility using the -r option to specify the path and name of the tcpdump capture file to be examined. Other useful options include the following:

• -n    Do not resolve host names.
• -A    Print all fields (ssldump, by default, prints only the most interesting).
• -e    Print absolute timestamps.
• -d    Display application data, including traffic before session initiates.
• -M    Output a pre-master secret log file (v. 11.2.0 and later)

Note: Refer to the ssldump man page for more detailed information about ssldump options.

For example, the following command displays all of the SSL record messages found in the tcpdump capture file named www-ssl-client.cap:

ssldump -nr /var/tmp/www-ssl-client.cap

The SSL records printed by the ssldump utility appear similar to the following example:

New TCP connection #2: 172.16.31.22(32866) <-> 192.168.1.8(8389)
2 1  0.0002 (0.0002)  C>S  Handshake
      ClientHello
        Version 3.0
        resume [32]=
          a3 ca ad 46 95 5d 64 bb 33 ec b5 12 91 21 a3 50
          d2 c0 c5 f6 67 c3 cc 9e c0 4a 71 1b 92 dc 58 55
        cipher suites
        SSL_DHE_RSA_WITH_AES_256_CBC_SHA
        SSL_DHE_DSS_WITH_AES_256_CBC_SHA
        SSL_RSA_WITH_AES_256_CBC_SHA
        SSL_DHE_RSA_WITH_AES_128_CBC_SHA
        SSL_DHE_DSS_WITH_AES_128_CBC_SHA
        SSL_RSA_WITH_RC4_128_MD5
        SSL_RSA_WITH_RC4_128_SHA
        SSL_RSA_WITH_AES_128_CBC_SHA
        SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        Unknown value 0xfeff
        SSL_RSA_WITH_3DES_EDE_CBC_SHA
        SSL_DHE_RSA_WITH_DES_CBC_SHA
        SSL_DHE_DSS_WITH_DES_CBC_SHA
        Unknown value 0xfefe
        SSL_RSA_WITH_DES_CBC_SHA
        SSL_RSA_EXPORT1024_WITH_RC4_56_SHA
        SSL_RSA_EXPORT1024_WITH_DES_CBC_SHA
        SSL_RSA_EXPORT_WITH_RC4_40_MD5
        SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5
        compression methods
                  NULL
2 2  0.0277 (0.0274)  S>C  Handshake
      ServerHello
        Version 3.0
        session_id[32]=
          a3 ca ad 46 95 5d 64 bb 33 ec b5 12 91 21 a3 50
          d2 c0 c5 f6 67 c3 cc 9e c0 4a 71 1b 92 dc 58 55
        cipherSuite         SSL_RSA_WITH_3DES_EDE_CBC_SHA
        compressionMethod                   NULL
2 3  0.0277 (0.0000)  S>C  ChangeCipherSpec
2 4  0.0277 (0.0000)  S>C  Handshake
2 5  0.0282 (0.0005)  C>S  ChangeCipherSpec
2 6  0.0282 (0.0000)  C>S  Handshake
2 7  0.0282 (0.0000)  C>S  application_data
2 8  0.0289 (0.0006)  S>C  application_data
2 9  0.0289 (0.0000)  S>C  application_data
2 10 0.0292 (0.0003)  C>S  application_data
2 11 0.0296 (0.0003)  S>C  application_data
2 12 0.0296 (0.0000)  S>C  application_data
1 24 3.5016 (0.5372)  S>C  application_data
1 25 3.5016 (0.0000)  S>C  application_data
2 13 0.5424 (0.5128)  C>S  application_data
2 14 0.5429 (0.0005)  S>C  application_data
2 15 0.5429 (0.0000)  S>C  application_data
1 26 6.0378 (2.5362)  C>S  application_data
1 27 6.0411 (0.0033)  S>C  application_data
1 28 6.0411 (0.0000)  S>C  application_data
2 16 3.1243 (2.5814)  C>S  application_data
2 17 3.1455 (0.0212)  S>C  application_data
2 18 3.1455 (0.0000)  S>C  application_data
1 29 9.2325 (3.1914)  C>S  application_data
1 30 9.2359 (0.0033)  S>C  application_data
1 31 9.2359 (0.0000)  S>C  application_data
1 32 9.3185 (0.0826)  C>S  application_data
2 19 6.3589 (3.2133)  C>S  application_data
1 33 9.3276 (0.0090)  S>C  application_data
1 34 9.3276 (0.0000)  S>C  application_data
2 20 6.3632 (0.0043)  S>C  application_data
2 21 6.3632 (0.0000)  S>C  application_data
1 35 12.3565 (3.0289)  C>S  application_data
1 36 12.3682 (0.0116)  S>C  application_data
1 37 12.3682 (0.0000)  S>C  application_data

The first line defines a new TCP connection which appears similar to the following example:

New TCP connection #2: 172.16.31.22(32866) <-> 192.168.1.8(8080)

To differentiate records belonging to different connections, each connection is numbered. The example defines connection 2. The host that sends the first SYN is printed on the left and the host that responds is printed on the right. In most cases, the SSL client is printed on the left with the SSL server on the right. In this case we have a connection from 172.16.31.22 port 32866 to 192.168.1.8 port 8080.

Subsequent lines represent SSL records sent between the client and the server. The printout of each SSL record begins with a record line. It contains the connection number with which the record is associated, and the sequence number of the record itself, followed by two timestamps. The first timestamp is the time, in seconds, since the beginning of the connection. The second timestamp is the time, in seconds, since the previous record on the same connection. The next column indicates the origin of the message. Communications originating from the client are indicated in the next column by C>S (client to server), while messages originating from the server are marked with S>C (server to client). The last column indicates the type of SSL record message that was received, which can be either Handshake, IAlert, ChangeCipherSpec, or application_data. Finally, the ssldump utility may print record-specific data on the rest of the line. For Handshake records, the utility prints the handshake message. For application_data messages, the utility prints the decrypted application data.

By default, the ssldump utility decodes and displays useful details of some SSL record messages.

For example:

• ClientHello

  Details the version, offered cipher suites, and session id, if provided.

• ServerHello

  Details the version, session_id, chosen cipher suite, and compression method.

• Alert

  Details type and level, if provided.

The following example is the ssldump output for the second record on connection 2:

2 2  0.0277 (0.0274)  S>C  Handshake
      ServerHello
        Version 3.0
        session_id[32]=
          a3 ca ad 46 95 5d 64 bb 33 ec b5 12 91 21 a3 50
          d2 c0 c5 f6 67 c3 cc 9e c0 4a 71 1b 92 dc 58 55
        cipherSuite         SSL_RSA_WITH_3DES_EDE_CBC_SHA
        compressionMethod                   NULL

The record was sent by the server, and is a Handshake record that contains a Server Hello message.

Examining the decrypted application data

When troubleshooting SSL issues, it may be necessary to examine the decrypted application data. To do this, the ssldump utility must have access to either the (asymmetric) private key from the server you want to debug, or the (symmetric) pre-master secret keys. Using the pre-master secret keys allows you to examine the decrypted application data in Wireshark without having access to the (asymmetric) private key.

Warning: Decrypting the SSL application data may expose sensitive information such as credit card numbers and passwords.

Examining the decrypted application data using the (symmetric) pre-master secret keys

Beginning in BIG-IP 11.2.0, the ssldump -M option allows you to create a pre-master secret (PMS) key log file. You can load the PMS log file into Wireshark (1.6 and later) along with the capture file, and use it to decrypt the application data without having access to the server's private key. This option gives F5 Technical Support the ability to fully decrypt sessions in the targeted capture file without revealing sensitive information about the private key.

To run ssldump using the -M option to the create a pre-master secret key log file, perform the following procedure:

1. Log in to the BIG-IP command line.
2. Capture the packet trace containing the SSL traffic (see the Capturing the target traffic section).
   
3. To create a pre-master secret key log file, use the following ssldump syntax:

   ssldump -r /path/to/capture_file -k /path/to/private_key -M /path/to/pre-master-key_log_file

   For example, the following ssldump command reads the www-ssl-client1.cap capture file using the test.org key file to decrypt the session, and creates the PMS log file called client1.pms:

   ssldump -r /var/tmp/www-ssl-client1.cap -k /config/filestore/files_d/Common_d/certificate_key_d/\:Common\:test.org.key_1 -M /var/tmp/client1.pms

   You can now load the capture file and the PMS log file into Wireshark (1.6 and later).

Note: To load the pre-master secret (PMS) key log file in Wireshark, navigate to Edit > Preferences > Protocols > SSL, and enter the path and filename of the PMS key in the (Pre)-Master-Secret log filename field.

Note: For F5 Technical Support to examine the decrypted application data, you must submit the capture file and PMS key log file.

Examining the decrypted application data using the (asymmetric) private key

To decrypt display application data, the ssldump utility will need a copy of the private key from the server you want to debug. If you do not have the key, the application data cannot be decrypted, and you will only be able to examine and decode the SSL handshake packets.

Important: Not all ciphers provide the ability to decrypt SSL traffic using a utility such as ssldump. Depending on the cipher negotiated, the ssldump utility may not be able to derive enough information from the SSL handshake and the server’s private key to decrypt the application data. Examples of such SSL ciphers would be the Diffie-Hellman Ephemeral (DHE) cipher suites and export-grade RSA cipher suites. If it is necessary to decrypt application data for a virtual server, you can change the cipher suite used in your SSL profile to accept only those ciphers. To do so, make a note of the cipher string currently configured in the SSL profile, then temporarily modify the SSL profile to specify NONE:RC4+RSA as the custom cipher string. For specific configuration steps, refer to the examples in .

In the previous example, the ssldump command that is provided prints the application_data record type but does not display the application data itself. Since no key was provided, the application data has not been decrypted. To print the decrypted application data, use the -k option to specify the path and name of the file containing the server's private key.

For example:

ssldump -Aed -nr /var/tmp/www-ssl-client.cap -k /config/ssl/ssl.key/www-ssl.key

Note: In BIG-IP 11.x, the SSL profile keys are stored in the /config/filestore/files_d/_d/certificate_key_d/ directory.

Decoded application data records printed by ssldump appear similar to the following example:

  3 7  1247932735.6286 (0.0031)  C>SV3.1(436)  application_data
      ---------------------------------------------------------------
      GET / HTTP/1.1
      Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
      Accept-Language: en-us
      User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; .NET CLR 1.1.4322; msn OptimizedIE8;ENUS)
      Accept-Encoding: gzip, deflate
      Host: 172.24.72.169
      Connection: Keep-Alive

  3 8  1247932735.8015 (0.1728)  S>CV3.1(1540)  application_data
      ---------------------------------------------------------------
      HTTP/1.1 200 OK
      Date: Sat, 18 Jul 2009 15:25:37 GMT
      Server: Apache/2.2.3 (Debian) PHP/4.4.4-8+etch4 mod_ssl/2.2.3 OpenSSL/0.9.8g mod_perl/2.0.2 Perl/v5.8.8
      Last-Modified: Tue, 08 Jan 2008 22:59:06 GMT
      ETag: "3306ee-8be-ec750280"
      Accept-Ranges: bytes
      Content-Length: 2238
      Keep-Alive: timeout=15, max=100
      Connection: Keep-Alive
      Content-Type: text/html

Locating a BIG-IP virtual server's private key

The private key that the BIG-IP system uses for a virtual server is specified in the Client SSL profile applied to the virtual server. The key file locations are listed below:

• BIG-IP 9.x - 10.x:
  
  /config/ssl/ssl.key/
  
  For example:

  /config/ssl/ssl.key/

• BIG-IP 11. x:
  
  /config/filestore/files_d/_d/certificate_key_d/
  
  For example:

  /config/filestore/files_d/Common_d/certificate_key_d/:Common:

To determine the key used for a particular virtual server, examine the virtual server configuration to determine the name of the ssl profiles applied to the virtual server, then examine the SSL profile configuration to determine the name of the key file.

Resumed TLS handshake

One of the most common reasons an ssldump may not decrypt application data, is if the data is contained within a resumed TLS session. Public key operations are expensive in terms of processing power during the initial setup and key exchange. TLS specifications allow a secure shortcut by using the session_id to resume an SSL connection for which the key exchange was already performed.

In situations where the SSL communication is using a resumed session, ssldump will not be able decrypt the application data unless the capture file contains the initial handshake containing the asymmetric key exchange and session_id. The ssldump utility relies on the information that is exchanged during the initial session setup to decrypt the data.

The following packet capture output example shows a resumed connection:

A client requesting to resume an SSL session appears similar to the following example:

New TCP connection #2: 192.0.2.42(37811) <-> 198.51.100.47(443)
2 1 1296947622.6312 (0.0006) C>SV3.1(156) Handshake
        ClientHello
            Version 3.1
            random[32]=
                4d 4d d9 b3 c1 db aa 8e 2a fc 95 ce c0 66 dc 49
                a9 49 a5 bf bc c3 3d 67 e5 ba be 77 44 44 6f 8b
                resume [32]=
                c4 44 ea 86 e2 ba f5 40 4b 44 b4 c2 3a d8 b4 ad
                4c dc 13 0d 6c 48 f2 70 19 c3 05 f4 06 e5 ab a9
                cipher suites
                TLS_DHE_RSA_WITH_AES_128_CBC_SHA
                Unknown value 0x45
                TLS_DHE_RSA_WITH_AES_256_CBC_SHA
                Unknown value 0x88
                TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
                TLS_DHE_DSS_WITH_AES_128_CBC_SHA
                Unknown value 0x44
                TLS_DHE_DSS_WITH_AES_256_CBC_SHA
                Unknown value 0x87
                TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
                TLS_DHE_DSS_WITH_RC4_128_SHA
                Unknown value 0x90
                Unknown value 0x91
                Unknown value 0x8f
                Unknown value 0x8e
                TLS_RSA_WITH_AES_128_CBC_SHA
                Unknown value 0x41
                TLS_RSA_WITH_AES_256_CBC_SHA
                Unknown value 0x84
                TLS_RSA_WITH_3DES_EDE_CBC_SHA
                TLS_RSA_WITH_RC4_128_SHA
                TLS_RSA_WITH_RC4_128_MD5
                Unknown value 0x8c
                Unknown value 0x8d
                Unknown value 0x8b
                Unknown value 0x8a
                compression methods
                                  NULL

A server agreeing to resume the session returns the ServerHello with the same session_id as displayed below:

2 2 1296947622.6313 (0.0001) S>CV3.1(74) Handshake
        ServerHello
            Version 3.1
            random[32]=
                19 21 d7 55 c1 14 65 63 54 23 62 b7 c4 30 a2 f0
                b8 c4 20 06 86 ed 9c 1f 9e 46 0f 42 79 45 8a 29
            session_id[32]=
                c4 44 ea 86 e2 ba f5 40 4b 44 b4 c2 3a d8 b4 ad
                4c dc 13 0d 6c 48 f2 70 19 c3 05 f4 06 e5 ab a9
            cipherSuite TLS_RSA_WITH_RC4_128_SHA
            compressionMethod NULL

The previous example is a resumed session; the client sends a ClientHello that includes a resume[], the server responds with a ServerHello that contains the same session_id the client sent.

T