Output of flowgrind

After starting a test, flowgrind first prints a preamble with the date of the test, the number of flows as well as the reporting interval length and the throughput measurement unit. Last but not least the preamble contains the version number of flowgrind which is important to reproduce and understand the results of the test should the capabilities of flowgrind change in future versions.

While the test runs the controller collects the interval reports from the daemons participating the test and prints the report lines. It is not visible in this example output, but flowgrind does not make any guarantees on the order the reports from different flows or endpoints are printed. Only the reports from the same flow and endpoint are shown ordered. The reason is that gathering and displaying the results is done asynchronously. This needs to be taken into consideration when parsing the output.

During the test, a header with the column names and units is displayed periodically. The columns are resized dynamically to make best use of the available space. This can be seen in the sample output, at the second time the header gets displayed the width of the cwnd column changes. If desired, groups of columns can be disabled from showing by a configuration option. The columns seen in the figure show all the metrics measured by flowgrind:

#: The endpoint, either S for source or R for destination.
ID: The flow identifier.
begin and end: The boundaries of the measuring interval in seconds. The time shown is the elapsed time since receiving the RPC message to start the test from the daemon’s point of view.
through: The transmitting goodput of the flow endpoint during this measurement interval, measured in Mb/s.
IAT and RTT: The 1-way and 2-way block delays respectively block IAT and block RTT. For both delays the minimum and maximum encoutered values in that interval are displayed in addition to the artihmetic mean. If no block acknowledgement arrived during that report interval, inf is displayed.
cwnd: Size of TCP congestion window in number of segements. All TCP specific metrics are obtained from the Linux kernel through the TCP_INFO socket option at the end of every reporting interval.
ssth: The slowstart threshold of the sender in number of segments.
uack, sack and fack: Statistics about unacknowledged, selectively acknowledged or forward acknowledged segments.
lost: Number of segments assumed lost during reporting interval.
reor: Segment reordering metric. The Linux kernel can detect and cope with reordering without loss of performance if the distance a segment gets displaced does not exceed the reordering metric.
rtt and rttvar: TCP round-trip time and its variance given in ms.
rto: The retransmission timeout given in ms.
castate: Internal state of congestion control state machine as implemented in the Linux kernel. Can be one of open, disorder, cwr, recovery or loss:
- Open is the normal state. It indicates that there are no issues with the connection.
- Disorder is similar to Open but is entered upon receiving duplicate ACKs or selective acknowledgements as special attention might be neded in the near future.
- CWR is entered when the size of the congestion window got lowered due to receiving an ICMP Source Quench message or a notification from Explicit Congestion Notification (ECN).
- Recovery indicates that the congestion window got lowered and a segment is fast-retransmitted.
- Loss is entered if the RTO expires. Again the size of the congestion window got lowered in this state.
mss and mtu: Maximum segment size and maximum transmission unit in bytes.
status: The state of the flow inside flowgrind for diagnostic purposes. It is a tuple of two values, the first for sending and the second for receiving. Ideally the states of both the source and destination endpoints of a flow should be symmetrical but since they are not synchronized they may not change at the same time. The possible values are:
- c Direction completed sending/receiving.
- d Waiting for initial delay.
- f Fault state.
- l Active state, nothing yet transmitted or received.
- n Normal activity, some data got transmitted or received.
- o Flow has zero duration in that direction, no data is going to be exchanged.

After the test finishes, a final report line header is displayed for each flow endpoint. Flowgrind tries to guess the used network type from the value of the MSS and MTU, in this case it correctly identified the used network as Ethernet. The final report line also contains the size of the send buffer and the size of the receive buffer, the advertised initial window. For comparison the size of the data blocks is displayed as well for each direction. Afterwards the initial delay and the flow duration are shown. Following these scheduling options some information about the used socket options may be shown, though no special options were configured for this simple test. Afterwards the average throughput is shown and the number of data blocks sent and received is printed.

Following the final report headers the flow metrics for the entire test duration are displayed. In the final report the columns containing information returned by the TCP_INFO socket option show the values measured just at the time the final report was taken, right before the socket gets destroyed.