Troubleshoot BIG-IP Using Tcpdump (Advanced)

Tcpdump Advanced Topic

(1) Capturing extended TMM data with tcpdump

#tcpdump -s0 -i <interface>:<noise amplitude> -w <filename> [filter]

For example:

tcpdump -s0 -ni 0.0:nnn [filter]

The noise amplitude defines the level of TMM details included in the packet capture. The following noise levels may be captured:

• n: Low details
  • Ingress: A flag indicating whether TMM is sending or receiving the packet.
    • > A zero (0) indicates that TMM is sending the packet.
    • > A non-zero number indicates that TMM is receiving the packet.
  • Slot: The chassis slot number of the TMM that is handling the packet.
    • > TMM on slot 1 “f5ethtrailer.slot == 1”
  • TMM: The number of the TMM that is handling the packet.
    • > TMM number 0 “f5ethtrailer.tmm == 0”
  • VIP: The name of the virtual server that is handling the connection.
• nn: Low and medium details
  • Flow ID: A number identifying a flow within TMM. The same flow ID can be used for different flows in different TMMs. Also, the same flow ID can be re-used for a different flow within the same TMM at a different time.
  • Peer ID: A number identifying the peer flow within TMM. Note:
    • > The same peer ID can be used for different flows in different TMMs.
    • > Also, the same peer ID can be re-used for a different flow within the same TMM at a different time.
  • Reset Cause: In BIG-IP 11.2.0 and later, the reset cause (if available) is included for TCP reset packets.
    • > For more information, refer to K13223: Configuring the BIG-IP system to log TCP RST packets.
  • Connflow Flags: Diagnostic information used by F5 Technical Support.
  • Flow Type: Diagnostic information used by F5 Technical Support.
  • High Availability Unit: Diagnostic information used by F5 Technical Support.
  • Ingress Slot: Diagnostic information used by F5 Technical Support.
  • Ingress Port: Diagnostic information used by F5 Technical Support.
• nnn: Low, medium, and high details
  • Peer IP Protocol: The IP protocol of the peer flow.
    • > This field is not populated prior to BIG-IP 11.0.0.
  • Peer VLAN: The VLAN ID number that is associated with the peer flow.
  • Peer Remote Address: The IP address of the host on the far end of the peer flow.
  • Peer Local Address: The IP address used by TMM for the peer flow.
  • Peer Remote Port: The protocol port of the host on the far end of the peer flow.
  • Peer Local Port: The protocol port used by TMM for the peer flow.

For Example:

• [Client-host] 192.168.201.1:56869 -> 192.168.201.10:80 [F5-VS]
  • Flow ID: 0x000056000178b900
  • Peer ID: 0x0000000000000000
  • no peer connection information
• [F5-Self-IP] 192.168.202.101:40693-> 192.168.202.145:80 [Web-server1]
  • Flow ID: 0x000056000178ba00
  • Peer ID: 0x000056000178b900
  • Peer remote address: 192.168.201.1
  • Peer local address: 192.168.201.10
  • Peer remote port: 56869
  • Peer local port: 80
• [Web-server1] 192.168.202.145:80 -> 192.168.202.101:40693 [F5-Self-IP]
  • Flow ID: 0x000056000178ba00
  • Peer ID: 0x000056000178b900
  • Peer remote address: 192.168.201.1
  • Peer local address: 192.168.201.10
  • Peer remote port: 56869
  • Peer local port: 80
• [F5-VS] 192.168.201.10:80 -> 192.168.201.1:56869 [Client-host]
  • Flow ID: 0x000056000178b900
  • Peer ID: 0x000056000178ba00
  • Peer remote address: 192.168.202.145
  • Peer local address: 192.168.202.101
  • Peer remote port: 80
  • Peer local port: 40693

(2) Capturing traffic with TMM information for a specific traffic flow

Beginning in BIG-IP 11.2.0, you can use the ‘p’ interface modifier with the ‘n’ modifier to capture traffic with TMM information for a specific flow and its related peer flow.

The p modifier allows you to capture a specific traffic flow through the BIG-IP system from end to end, even when the configuration uses a SNAT or OneConnect.

For Example:

tcpdump -vnni 0.0:nnnp -s0 host 192.168.201.10 and tcp port 80 -w /var/tmp/test1.pcap

• [Client-host] 192.168.201.1:56869 -> 192.168.201.10:80 [F5-VS]
• [F5-Self-IP] 192.168.202.101:40693-> 192.168.202.145:80 [Web-server1]
• [Web-server1] 192.168.202.145:80 -> 192.168.202.101:40693 [F5-Self-IP]
• [F5-VS] 192.168.201.10:80 -> 192.168.201.1:56869 [Client-host]

Mitigating the side-effect of capturing using ‘p’ modifier 

Note: This modifier continues to produce flow information for the life of the connection. Subsequent tcpdump captures reveal flow information from previous tcpdump captures using the :p modifier if the connection is still active. To clear flow information from previous use, run the tcpdump command without the ‘p’ modifier and filter the capture to match none of the earlier captured flows.

1. If you have not already done so, terminate all tcpdump captures that use the p modifier.
2. To clear the p modifier flag, run a tcpdump capture without using the p modifier, and filter the capture to match none of the earlier captured flows.
   • For example, the following command searches for traffic to or from an IP address that does not exist in the network environment:
   • tcpdump -ni 0.0:nnn -s0 -w /var/tmp/capture2.pcap host 1.2.3.4 and port 1
3. Leave the command in step 2 running until all of the long-lived flows from the previously captured flows that were using the p modifier have had a chance to send a frame through the BIG-IP system, even though those long-lived flows have nothing to be captured on the command in step 2.
   • This is the only way to clear the p modifier flag for the traffic filter that matches the previous captured flows.
4. Press Ctrl+C to terminate the tcpdump capture.

(2.1) Capturing traffic with TMM information and ePVA diagnostics

Note: This modifier produces large amounts of data and can cause significant resource utilization. This additional resource demand may cause poor performance or a system failure if the BIG-IP system is at high resource utilization. Use this modifier only with very specific filters.

By default, the FastL4 profile used with a Performance Layer4 virtual server uses the PVA/ePVA to accelerate traffic processing through the BIG-IP.

• To determine whether your platform contains a PVA chip, use the “tmsh show /sys hardware | grep -i pva”
• For all packets to be captured by regular tcpdump, the PVA/ePVA must be disabled.

Beginning in BIG-IP 14.1.0, you can use the –f5 parameter to dump ePVA debug information in the tcpdump. The parameter can be specified as follows:

• –f5 epva:hwoff (enables debug information related to hardware offloading from the ePVA provider)
• –f5 epva:sc (enables debug information related to SYN cookies from the ePVA provider)
• –f5 epva:all (enables all debug information from the ePVA provider)

For Example

tcpdump –f5 epva:all -i 0.0 tcp port 80

tcpdump –f5 epva:all -vnni 0.0:nnnp -s0 host 192.168.201.10 and tcp port 80 -w /var/tmp/test1.pcap

(3) Display TMM information in Wireshark