12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793 |
- ============
- SNMP counter
- ============
- This document explains the meaning of SNMP counters.
- General IPv4 counters
- =====================
- All layer 4 packets and ICMP packets will change these counters, but
- these counters won't be changed by layer 2 packets (such as STP) or
- ARP packets.
- * IpInReceives
- Defined in `RFC1213 ipInReceives`_
- .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
- The number of packets received by the IP layer. It gets increasing at the
- beginning of ip_rcv function, always be updated together with
- IpExtInOctets. It will be increased even if the packet is dropped
- later (e.g. due to the IP header is invalid or the checksum is wrong
- and so on). It indicates the number of aggregated segments after
- GRO/LRO.
- * IpInDelivers
- Defined in `RFC1213 ipInDelivers`_
- .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
- The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
- ICMP and so on. If no one listens on a raw socket, only kernel
- supported protocols will be delivered, if someone listens on the raw
- socket, all valid IP packets will be delivered.
- * IpOutRequests
- Defined in `RFC1213 ipOutRequests`_
- .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
- The number of packets sent via IP layer, for both single cast and
- multicast packets, and would always be updated together with
- IpExtOutOctets.
- * IpExtInOctets and IpExtOutOctets
- They are Linux kernel extensions, no RFC definitions. Please note,
- RFC1213 indeed defines ifInOctets and ifOutOctets, but they
- are different things. The ifInOctets and ifOutOctets include the MAC
- layer header size but IpExtInOctets and IpExtOutOctets don't, they
- only include the IP layer header and the IP layer data.
- * IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
- They indicate the number of four kinds of ECN IP packets, please refer
- `Explicit Congestion Notification`_ for more details.
- .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
- These 4 counters calculate how many packets received per ECN
- status. They count the real frame number regardless the LRO/GRO. So
- for the same packet, you might find that IpInReceives count 1, but
- IpExtInNoECTPkts counts 2 or more.
- * IpInHdrErrors
- Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is
- dropped due to the IP header error. It might happen in both IP input
- and IP forward paths.
- .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
- * IpInAddrErrors
- Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two
- scenarios: (1) The IP address is invalid. (2) The destination IP
- address is not a local address and IP forwarding is not enabled
- .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
- * IpExtInNoRoutes
- This counter means the packet is dropped when the IP stack receives a
- packet and can't find a route for it from the route table. It might
- happen when IP forwarding is enabled and the destination IP address is
- not a local address and there is no route for the destination IP
- address.
- * IpInUnknownProtos
- Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the
- layer 4 protocol is unsupported by kernel. If an application is using
- raw socket, kernel will always deliver the packet to the raw socket
- and this counter won't be increased.
- .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
- * IpExtInTruncatedPkts
- For IPv4 packet, it means the actual data size is smaller than the
- "Total Length" field in the IPv4 header.
- * IpInDiscards
- Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped
- in the IP receiving path and due to kernel internal reasons (e.g. no
- enough memory).
- .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
- * IpOutDiscards
- Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is
- dropped in the IP sending path and due to kernel internal reasons.
- .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
- * IpOutNoRoutes
- Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is
- dropped in the IP sending path and no route is found for it.
- .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
- ICMP counters
- =============
- * IcmpInMsgs and IcmpOutMsgs
- Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
- .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
- .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
- As mentioned in the RFC1213, these two counters include errors, they
- would be increased even if the ICMP packet has an invalid type. The
- ICMP output path will check the header of a raw socket, so the
- IcmpOutMsgs would still be updated if the IP header is constructed by
- a userspace program.
- * ICMP named types
- | These counters include most of common ICMP types, they are:
- | IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
- | IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
- | IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
- | IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
- | IcmpInRedirects: `RFC1213 icmpInRedirects`_
- | IcmpInEchos: `RFC1213 icmpInEchos`_
- | IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
- | IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
- | IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
- | IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
- | IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
- | IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
- | IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
- | IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
- | IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
- | IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
- | IcmpOutEchos: `RFC1213 icmpOutEchos`_
- | IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
- | IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
- | IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
- | IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
- | IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
- .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
- .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
- .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
- .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
- .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
- .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
- .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
- .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
- .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
- .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
- .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
- .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
- .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
- .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
- .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
- .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
- .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
- .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
- .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
- .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
- .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
- .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
- Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
- Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
- straightforward. The 'In' counter means kernel receives such a packet
- and the 'Out' counter means kernel sends such a packet.
- * ICMP numeric types
- They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
- ICMP type number. These counters track all kinds of ICMP packets. The
- ICMP type number definition could be found in the `ICMP parameters`_
- document.
- .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
- For example, if the Linux kernel sends an ICMP Echo packet, the
- IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
- packet, IcmpMsgInType0 would increase 1.
- * IcmpInCsumErrors
- This counter indicates the checksum of the ICMP packet is
- wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
- before updating IcmpMsgInType[N]. If a packet has bad checksum, the
- IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
- * IcmpInErrors and IcmpOutErrors
- Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
- .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
- .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
- When an error occurs in the ICMP packet handler path, these two
- counters would be updated. The receiving packet path use IcmpInErrors
- and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
- is increased, IcmpInErrors would always be increased too.
- relationship of the ICMP counters
- ---------------------------------
- The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
- are updated at the same time. The sum of IcmpMsgInType[N] plus
- IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
- receives an ICMP packet, kernel follows below logic:
- 1. increase IcmpInMsgs
- 2. if has any error, update IcmpInErrors and finish the process
- 3. update IcmpMsgOutType[N]
- 4. handle the packet depending on the type, if has any error, update
- IcmpInErrors and finish the process
- So if all errors occur in step (2), IcmpInMsgs should be equal to the
- sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
- step (4), IcmpInMsgs should be equal to the sum of
- IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
- IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
- IcmpInErrors.
- General TCP counters
- ====================
- * TcpInSegs
- Defined in `RFC1213 tcpInSegs`_
- .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
- The number of packets received by the TCP layer. As mentioned in
- RFC1213, it includes the packets received in error, such as checksum
- error, invalid TCP header and so on. Only one error won't be included:
- if the layer 2 destination address is not the NIC's layer 2
- address. It might happen if the packet is a multicast or broadcast
- packet, or the NIC is in promiscuous mode. In these situations, the
- packets would be delivered to the TCP layer, but the TCP layer will discard
- these packets before increasing TcpInSegs. The TcpInSegs counter
- isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
- counter would only increase 1.
- * TcpOutSegs
- Defined in `RFC1213 tcpOutSegs`_
- .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
- The number of packets sent by the TCP layer. As mentioned in RFC1213,
- it excludes the retransmitted packets. But it includes the SYN, ACK
- and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
- GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
- increase 2.
- * TcpActiveOpens
- Defined in `RFC1213 tcpActiveOpens`_
- .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
- It means the TCP layer sends a SYN, and come into the SYN-SENT
- state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
- increase 1.
- * TcpPassiveOpens
- Defined in `RFC1213 tcpPassiveOpens`_
- .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
- It means the TCP layer receives a SYN, replies a SYN+ACK, come into
- the SYN-RCVD state.
- * TcpExtTCPRcvCoalesce
- When packets are received by the TCP layer and are not be read by the
- application, the TCP layer will try to merge them. This counter
- indicate how many packets are merged in such situation. If GRO is
- enabled, lots of packets would be merged by GRO, these packets
- wouldn't be counted to TcpExtTCPRcvCoalesce.
- * TcpExtTCPAutoCorking
- When sending packets, the TCP layer will try to merge small packets to
- a bigger one. This counter increase 1 for every packet merged in such
- situation. Please refer to the LWN article for more details:
- https://lwn.net/Articles/576263/
- * TcpExtTCPOrigDataSent
- This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
- explaination below::
- TCPOrigDataSent: number of outgoing packets with original data (excluding
- retransmission but including data-in-SYN). This counter is different from
- TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is
- more useful to track the TCP retransmission rate.
- * TCPSynRetrans
- This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
- explaination below::
- TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
- retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
- * TCPFastOpenActiveFail
- This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
- explaination below::
- TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because
- the remote does not accept it or the attempts timed out.
- .. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd
- * TcpExtListenOverflows and TcpExtListenDrops
- When kernel receives a SYN from a client, and if the TCP accept queue
- is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.
- At the same time kernel will also add 1 to TcpExtListenDrops. When a
- TCP socket is in LISTEN state, and kernel need to drop a packet,
- kernel would always add 1 to TcpExtListenDrops. So increase
- TcpExtListenOverflows would let TcpExtListenDrops increasing at the
- same time, but TcpExtListenDrops would also increase without
- TcpExtListenOverflows increasing, e.g. a memory allocation fail would
- also let TcpExtListenDrops increase.
- Note: The above explanation is based on kernel 4.10 or above version, on
- an old kernel, the TCP stack has different behavior when TCP accept
- queue is full. On the old kernel, TCP stack won't drop the SYN, it
- would complete the 3-way handshake. As the accept queue is full, TCP
- stack will keep the socket in the TCP half-open queue. As it is in the
- half open queue, TCP stack will send SYN+ACK on an exponential backoff
- timer, after client replies ACK, TCP stack checks whether the accept
- queue is still full, if it is not full, moves the socket to the accept
- queue, if it is full, keeps the socket in the half-open queue, at next
- time client replies ACK, this socket will get another chance to move
- to the accept queue.
- TCP Fast Open
- =============
- * TcpEstabResets
- Defined in `RFC1213 tcpEstabResets`_.
- .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
- * TcpAttemptFails
- Defined in `RFC1213 tcpAttemptFails`_.
- .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
- * TcpOutRsts
- Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates
- the 'segments sent containing the RST flag', but in linux kernel, this
- couner indicates the segments kerenl tried to send. The sending
- process might be failed due to some errors (e.g. memory alloc failed).
- .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
- * TcpExtTCPSpuriousRtxHostQueues
- When the TCP stack wants to retransmit a packet, and finds that packet
- is not lost in the network, but the packet is not sent yet, the TCP
- stack would give up the retransmission and update this counter. It
- might happen if a packet stays too long time in a qdisc or driver
- queue.
- * TcpEstabResets
- The socket receives a RST packet in Establish or CloseWait state.
- * TcpExtTCPKeepAlive
- This counter indicates many keepalive packets were sent. The keepalive
- won't be enabled by default. A userspace program could enable it by
- setting the SO_KEEPALIVE socket option.
- * TcpExtTCPSpuriousRTOs
- The spurious retransmission timeout detected by the `F-RTO`_
- algorithm.
- .. _F-RTO: https://tools.ietf.org/html/rfc5682
- TCP Fast Path
- =============
- When kernel receives a TCP packet, it has two paths to handler the
- packet, one is fast path, another is slow path. The comment in kernel
- code provides a good explanation of them, I pasted them below::
- It is split into a fast path and a slow path. The fast path is
- disabled when:
- - A zero window was announced from us
- - zero window probing
- is only handled properly on the slow path.
- - Out of order segments arrived.
- - Urgent data is expected.
- - There is no buffer space left
- - Unexpected TCP flags/window values/header lengths are received
- (detected by checking the TCP header against pred_flags)
- - Data is sent in both directions. The fast path only supports pure senders
- or pure receivers (this means either the sequence number or the ack
- value must stay constant)
- - Unexpected TCP option.
- Kernel will try to use fast path unless any of the above conditions
- are satisfied. If the packets are out of order, kernel will handle
- them in slow path, which means the performance might be not very
- good. Kernel would also come into slow path if the "Delayed ack" is
- used, because when using "Delayed ack", the data is sent in both
- directions. When the TCP window scale option is not used, kernel will
- try to enable fast path immediately when the connection comes into the
- established state, but if the TCP window scale option is used, kernel
- will disable the fast path at first, and try to enable it after kernel
- receives packets.
- * TcpExtTCPPureAcks and TcpExtTCPHPAcks
- If a packet set ACK flag and has no data, it is a pure ACK packet, if
- kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
- if kernel handles it in the slow path, TcpExtTCPPureAcks will
- increase 1.
- * TcpExtTCPHPHits
- If a TCP packet has data (which means it is not a pure ACK packet),
- and this packet is handled in the fast path, TcpExtTCPHPHits will
- increase 1.
- TCP abort
- =========
- * TcpExtTCPAbortOnData
- It means TCP layer has data in flight, but need to close the
- connection. So TCP layer sends a RST to the other side, indicate the
- connection is not closed very graceful. An easy way to increase this
- counter is using the SO_LINGER option. Please refer to the SO_LINGER
- section of the `socket man page`_:
- .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
- By default, when an application closes a connection, the close function
- will return immediately and kernel will try to send the in-flight data
- async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
- to a positive number, the close function won't return immediately, but
- wait for the in-flight data are acked by the other side, the max wait
- time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
- when the application closes a connection, kernel will send a RST
- immediately and increase the TcpExtTCPAbortOnData counter.
- * TcpExtTCPAbortOnClose
- This counter means the application has unread data in the TCP layer when
- the application wants to close the TCP connection. In such a situation,
- kernel will send a RST to the other side of the TCP connection.
- * TcpExtTCPAbortOnMemory
- When an application closes a TCP connection, kernel still need to track
- the connection, let it complete the TCP disconnect process. E.g. an
- app calls the close method of a socket, kernel sends fin to the other
- side of the connection, then the app has no relationship with the
- socket any more, but kernel need to keep the socket, this socket
- becomes an orphan socket, kernel waits for the reply of the other side,
- and would come to the TIME_WAIT state finally. When kernel has no
- enough memory to keep the orphan socket, kernel would send an RST to
- the other side, and delete the socket, in such situation, kernel will
- increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
- TcpExtTCPAbortOnMemory:
- 1. the memory used by the TCP protocol is higher than the third value of
- the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:
- .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
- 2. the orphan socket count is higher than net.ipv4.tcp_max_orphans
- * TcpExtTCPAbortOnTimeout
- This counter will increase when any of the TCP timers expire. In such
- situation, kernel won't send RST, just give up the connection.
- * TcpExtTCPAbortOnLinger
- When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
- for the fin packet from the other side, kernel could send a RST and
- delete the socket immediately. This is not the default behavior of
- Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
- you could let kernel follow this behavior.
- * TcpExtTCPAbortFailed
- The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
- satisfied. If an internal error occurs during this process,
- TcpExtTCPAbortFailed will be increased.
- .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
- TCP Hybrid Slow Start
- =====================
- The Hybrid Slow Start algorithm is an enhancement of the traditional
- TCP congestion window Slow Start algorithm. It uses two pieces of
- information to detect whether the max bandwidth of the TCP path is
- approached. The two pieces of information are ACK train length and
- increase in packet delay. For detail information, please refer the
- `Hybrid Slow Start paper`_. Either ACK train length or packet delay
- hits a specific threshold, the congestion control algorithm will come
- into the Congestion Avoidance state. Until v4.20, two congestion
- control algorithms are using Hybrid Slow Start, they are cubic (the
- default congestion control algorithm) and cdg. Four snmp counters
- relate with the Hybrid Slow Start algorithm.
- .. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf
- * TcpExtTCPHystartTrainDetect
- How many times the ACK train length threshold is detected
- * TcpExtTCPHystartTrainCwnd
- The sum of CWND detected by ACK train length. Dividing this value by
- TcpExtTCPHystartTrainDetect is the average CWND which detected by the
- ACK train length.
- * TcpExtTCPHystartDelayDetect
- How many times the packet delay threshold is detected.
- * TcpExtTCPHystartDelayCwnd
- The sum of CWND detected by packet delay. Dividing this value by
- TcpExtTCPHystartDelayDetect is the average CWND which detected by the
- packet delay.
- TCP retransmission and congestion control
- =========================================
- The TCP protocol has two retransmission mechanisms: SACK and fast
- recovery. They are exclusive with each other. When SACK is enabled,
- the kernel TCP stack would use SACK, or kernel would use fast
- recovery. The SACK is a TCP option, which is defined in `RFC2018`_,
- the fast recovery is defined in `RFC6582`_, which is also called
- 'Reno'.
- The TCP congestion control is a big and complex topic. To understand
- the related snmp counter, we need to know the states of the congestion
- control state machine. There are 5 states: Open, Disorder, CWR,
- Recovery and Loss. For details about these states, please refer page 5
- and page 6 of this document:
- https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf
- .. _RFC2018: https://tools.ietf.org/html/rfc2018
- .. _RFC6582: https://tools.ietf.org/html/rfc6582
- * TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery
- When the congestion control comes into Recovery state, if sack is
- used, TcpExtTCPSackRecovery increases 1, if sack is not used,
- TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP
- stack begins to retransmit the lost packets.
- * TcpExtTCPSACKReneging
- A packet was acknowledged by SACK, but the receiver has dropped this
- packet, so the sender needs to retransmit this packet. In this
- situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver
- could drop a packet which has been acknowledged by SACK, although it is
- unusual, it is allowed by the TCP protocol. The sender doesn't really
- know what happened on the receiver side. The sender just waits until
- the RTO expires for this packet, then the sender assumes this packet
- has been dropped by the receiver.
- * TcpExtTCPRenoReorder
- The reorder packet is detected by fast recovery. It would only be used
- if SACK is disabled. The fast recovery algorithm detects recorder by
- the duplicate ACK number. E.g., if retransmission is triggered, and
- the original retransmitted packet is not lost, it is just out of
- order, the receiver would acknowledge multiple times, one for the
- retransmitted packet, another for the arriving of the original out of
- order packet. Thus the sender would find more ACks than its
- expectation, and the sender knows out of order occurs.
- * TcpExtTCPTSReorder
- The reorder packet is detected when a hole is filled. E.g., assume the
- sender sends packet 1,2,3,4,5, and the receiving order is
- 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will
- fill the hole), two conditions will let TcpExtTCPTSReorder increase
- 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
- 3 is retransmitted but the timestamp of the packet 3's ACK is earlier
- than the retransmission timestamp.
- * TcpExtTCPSACKReorder
- The reorder packet detected by SACK. The SACK has two methods to
- detect reorder: (1) DSACK is received by the sender. It means the
- sender sends the same packet more than one times. And the only reason
- is the sender believes an out of order packet is lost so it sends the
- packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and
- the sender has received SACKs for packet 2 and 5, now the sender
- receives SACK for packet 4 and the sender doesn't retransmit the
- packet yet, the sender would know packet 4 is out of order. The TCP
- stack of kernel will increase TcpExtTCPSACKReorder for both of the
- above scenarios.
- * TcpExtTCPSlowStartRetrans
- The TCP stack wants to retransmit a packet and the congestion control
- state is 'Loss'.
- * TcpExtTCPFastRetrans
- The TCP stack wants to retransmit a packet and the congestion control
- state is not 'Loss'.
- * TcpExtTCPLostRetransmit
- A SACK points out that a retransmission packet is lost again.
- * TcpExtTCPRetransFail
- The TCP stack tries to deliver a retransmission packet to lower layers
- but the lower layers return an error.
- * TcpExtTCPSynRetrans
- The TCP stack retransmits a SYN packet.
- DSACK
- =====
- The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report
- duplicate packets to the sender. There are two kinds of
- duplications: (1) a packet which has been acknowledged is
- duplicate. (2) an out of order packet is duplicate. The TCP stack
- counts these two kinds of duplications on both receiver side and
- sender side.
- .. _RFC2883 : https://tools.ietf.org/html/rfc2883
- * TcpExtTCPDSACKOldSent
- The TCP stack receives a duplicate packet which has been acked, so it
- sends a DSACK to the sender.
- * TcpExtTCPDSACKOfoSent
- The TCP stack receives an out of order duplicate packet, so it sends a
- DSACK to the sender.
- * TcpExtTCPDSACKRecv
- The TCP stack receives a DSACK, which indicates an acknowledged
- duplicate packet is received.
- * TcpExtTCPDSACKOfoRecv
- The TCP stack receives a DSACK, which indicate an out of order
- duplicate packet is received.
- invalid SACK and DSACK
- ======================
- When a SACK (or DSACK) block is invalid, a corresponding counter would
- be updated. The validation method is base on the start/end sequence
- number of the SACK block. For more details, please refer the comment
- of the function tcp_is_sackblock_valid in the kernel source code. A
- SACK option could have up to 4 blocks, they are checked
- individually. E.g., if 3 blocks of a SACk is invalid, the
- corresponding counter would be updated 3 times. The comment of the
- `Add counters for discarded SACK blocks`_ patch has additional
- explaination:
- .. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32
- * TcpExtTCPSACKDiscard
- This counter indicates how many SACK blocks are invalid. If the invalid
- SACK block is caused by ACK recording, the TCP stack will only ignore
- it and won't update this counter.
- * TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo
- When a DSACK block is invalid, one of these two counters would be
- updated. Which counter will be updated depends on the undo_marker flag
- of the TCP socket. If the undo_marker is not set, the TCP stack isn't
- likely to re-transmit any packets, and we still receive an invalid
- DSACK block, the reason might be that the packet is duplicated in the
- middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo
- will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld
- will be updated. As implied in its name, it might be an old packet.
- SACK shift
- ==========
- The linux networking stack stores data in sk_buff struct (skb for
- short). If a SACK block acrosses multiple skb, the TCP stack will try
- to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
- 10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and
- 15 in skb2 would be moved to skb1. This operation is 'shift'. If a
- SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has
- seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be
- discard, this operation is 'merge'.
- * TcpExtTCPSackShifted
- A skb is shifted
- * TcpExtTCPSackMerged
- A skb is merged
- * TcpExtTCPSackShiftFallback
- A skb should be shifted or merged, but the TCP stack doesn't do it for
- some reasons.
- TCP out of order
- ================
- * TcpExtTCPOFOQueue
- The TCP layer receives an out of order packet and has enough memory
- to queue it.
- * TcpExtTCPOFODrop
- The TCP layer receives an out of order packet but doesn't have enough
- memory, so drops it. Such packets won't be counted into
- TcpExtTCPOFOQueue.
- * TcpExtTCPOFOMerge
- The received out of order packet has an overlay with the previous
- packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge
- packets will also be counted into TcpExtTCPOFOQueue.
- TCP PAWS
- ========
- PAWS (Protection Against Wrapped Sequence numbers) is an algorithm
- which is used to drop old packets. It depends on the TCP
- timestamps. For detail information, please refer the `timestamp wiki`_
- and the `RFC of PAWS`_.
- .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
- .. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps
- * TcpExtPAWSActive
- Packets are dropped by PAWS in Syn-Sent status.
- * TcpExtPAWSEstab
- Packets are dropped by PAWS in any status other than Syn-Sent.
- TCP ACK skip
- ============
- In some scenarios, kernel would avoid sending duplicate ACKs too
- frequently. Please find more details in the tcp_invalid_ratelimit
- section of the `sysctl document`_. When kernel decides to skip an ACK
- due to tcp_invalid_ratelimit, kernel would update one of below
- counters to indicate the ACK is skipped in which scenario. The ACK
- would only be skipped if the received packet is either a SYN packet or
- it has no data.
- .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
- * TcpExtTCPACKSkippedSynRecv
- The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
- TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
- waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
- in the Syn-Recv status. But in several scenarios, the TCP stack need
- to send an ACK. E.g., the TCP stack receives the same SYN packet
- repeately, the received packet does not pass the PAWS check, or the
- received packet sequence number is out of window. In these scenarios,
- the TCP stack needs to send ACK. If the ACk sending frequency is higher than
- tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
- increase TcpExtTCPACKSkippedSynRecv.
- * TcpExtTCPACKSkippedPAWS
- The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
- numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
- or Time-Wait statuses, the skipped ACK would be counted to
- TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or
- TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
- would be counted to TcpExtTCPACKSkippedPAWS.
- * TcpExtTCPACKSkippedSeq
- The sequence number is out of window and the timestamp passes the PAWS
- check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
- * TcpExtTCPACKSkippedFinWait2
- The ACK is skipped in Fin-Wait-2 status, the reason would be either
- PAWS check fails or the received sequence number is out of window.
- * TcpExtTCPACKSkippedTimeWait
- Tha ACK is skipped in Time-Wait status, the reason would be either
- PAWS check failed or the received sequence number is out of window.
- * TcpExtTCPACKSkippedChallenge
- The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
- 3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
- `RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these
- three scenarios, In some TCP status, the linux TCP stack would also
- send challenge ACKs if the ACK number is before the first
- unacknowledged number (more strict than `RFC 5961 section 5.2`_).
- .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
- .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
- .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
- TCP receive window
- ==================
- * TcpExtTCPWantZeroWindowAdv
- Depending on current memory usage, the TCP stack tries to set receive
- window to zero. But the receive window might still be a no-zero
- value. For example, if the previous window size is 10, and the TCP
- stack receives 3 bytes, the current window size would be 7 even if the
- window size calculated by the memory usage is zero.
- * TcpExtTCPToZeroWindowAdv
- The TCP receive window is set to zero from a no-zero value.
- * TcpExtTCPFromZeroWindowAdv
- The TCP receive window is set to no-zero value from zero.
- Delayed ACK
- ===========
- The TCP Delayed ACK is a technique which is used for reducing the
- packet count in the network. For more details, please refer the
- `Delayed ACK wiki`_
- .. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
- * TcpExtDelayedACKs
- A delayed ACK timer expires. The TCP stack will send a pure ACK packet
- and exit the delayed ACK mode.
- * TcpExtDelayedACKLocked
- A delayed ACK timer expires, but the TCP stack can't send an ACK
- immediately due to the socket is locked by a userspace program. The
- TCP stack will send a pure ACK later (after the userspace program
- unlock the socket). When the TCP stack sends the pure ACK later, the
- TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
- mode.
- * TcpExtDelayedACKLost
- It will be updated when the TCP stack receives a packet which has been
- ACKed. A Delayed ACK loss might cause this issue, but it would also be
- triggered by other reasons, such as a packet is duplicated in the
- network.
- Tail Loss Probe (TLP)
- =====================
- TLP is an algorithm which is used to detect TCP packet loss. For more
- details, please refer the `TLP paper`_.
- .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
- * TcpExtTCPLossProbes
- A TLP probe packet is sent.
- * TcpExtTCPLossProbeRecovery
- A packet loss is detected and recovered by TLP.
- TCP Fast Open description
- =========================
- TCP Fast Open is a technology which allows data transfer before the
- 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
- general description.
- .. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open
- * TcpExtTCPFastOpenActive
- When the TCP stack receives an ACK packet in the SYN-SENT status, and
- the ACK packet acknowledges the data in the SYN packet, the TCP stack
- understand the TFO cookie is accepted by the other side, then it
- updates this counter.
- * TcpExtTCPFastOpenActiveFail
- This counter indicates that the TCP stack initiated a TCP Fast Open,
- but it failed. This counter would be updated in three scenarios: (1)
- the other side doesn't acknowledge the data in the SYN packet. (2) The
- SYN packet which has the TFO cookie is timeout at least once. (3)
- after the 3-way handshake, the retransmission timeout happens
- net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
- fast open after the handshake.
- * TcpExtTCPFastOpenPassive
- This counter indicates how many times the TCP stack accepts the fast
- open request.
- * TcpExtTCPFastOpenPassiveFail
- This counter indicates how many times the TCP stack rejects the fast
- open request. It is caused by either the TFO cookie is invalid or the
- TCP stack finds an error during the socket creating process.
- * TcpExtTCPFastOpenListenOverflow
- When the pending fast open request number is larger than
- fastopenq->max_qlen, the TCP stack will reject the fast open request
- and update this counter. When this counter is updated, the TCP stack
- won't update TcpExtTCPFastOpenPassive or
- TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
- TCP_FASTOPEN socket operation and it could not be larger than
- net.core.somaxconn. For example:
- setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));
- * TcpExtTCPFastOpenCookieReqd
- This counter indicates how many times a client wants to request a TFO
- cookie.
- SYN cookies
- ===========
- SYN cookies are used to mitigate SYN flood, for details, please refer
- the `SYN cookies wiki`_.
- .. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies
- * TcpExtSyncookiesSent
- It indicates how many SYN cookies are sent.
- * TcpExtSyncookiesRecv
- How many reply packets of the SYN cookies the TCP stack receives.
- * TcpExtSyncookiesFailed
- The MSS decoded from the SYN cookie is invalid. When this counter is
- updated, the received packet won't be treated as a SYN cookie and the
- TcpExtSyncookiesRecv counter wont be updated.
- Challenge ACK
- =============
- For details of challenge ACK, please refer the explaination of
- TcpExtTCPACKSkippedChallenge.
- * TcpExtTCPChallengeACK
- The number of challenge acks sent.
- * TcpExtTCPSYNChallenge
- The number of challenge acks sent in response to SYN packets. After
- updates this counter, the TCP stack might send a challenge ACK and
- update the TcpExtTCPChallengeACK counter, or it might also skip to
- send the challenge and update the TcpExtTCPACKSkippedChallenge.
- prune
- =====
- When a socket is under memory pressure, the TCP stack will try to
- reclaim memory from the receiving queue and out of order queue. One of
- the reclaiming method is 'collapse', which means allocate a big sbk,
- copy the contiguous skbs to the single big skb, and free these
- contiguous skbs.
- * TcpExtPruneCalled
- The TCP stack tries to reclaim memory for a socket. After updates this
- counter, the TCP stack will try to collapse the out of order queue and
- the receiving queue. If the memory is still not enough, the TCP stack
- will try to discard packets from the out of order queue (and update the
- TcpExtOfoPruned counter)
- * TcpExtOfoPruned
- The TCP stack tries to discard packet on the out of order queue.
- * TcpExtRcvPruned
- After 'collapse' and discard packets from the out of order queue, if
- the actually used memory is still larger than the max allowed memory,
- this counter will be updated. It means the 'prune' fails.
- * TcpExtTCPRcvCollapsed
- This counter indicates how many skbs are freed during 'collapse'.
- examples
- ========
- ping test
- ---------
- Run the ping command against the public dns server 8.8.8.8::
- nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
- PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
- 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
- --- 8.8.8.8 ping statistics ---
- 1 packets transmitted, 1 received, 0% packet loss, time 0ms
- rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
- The nstayt result::
- nstatuser@nstat-a:~$ nstat
- #kernel
- IpInReceives 1 0.0
- IpInDelivers 1 0.0
- IpOutRequests 1 0.0
- IcmpInMsgs 1 0.0
- IcmpInEchoReps 1 0.0
- IcmpOutMsgs 1 0.0
- IcmpOutEchos 1 0.0
- IcmpMsgInType0 1 0.0
- IcmpMsgOutType8 1 0.0
- IpExtInOctets 84 0.0
- IpExtOutOctets 84 0.0
- IpExtInNoECTPkts 1 0.0
- The Linux server sent an ICMP Echo packet, so IpOutRequests,
- IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
- server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
- IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
- was passed to the ICMP layer via IP layer, so IpInDelivers was
- increased 1. The default ping data size is 48, so an ICMP Echo packet
- and its corresponding Echo Reply packet are constructed by:
- * 14 bytes MAC header
- * 20 bytes IP header
- * 16 bytes ICMP header
- * 48 bytes data (default value of the ping command)
- So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
- tcp 3-way handshake
- -------------------
- On server side, we run::
- nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
- Listening on [0.0.0.0] (family 0, port 9000)
- On client side, we run::
- nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
- Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
- The server listened on tcp 9000 port, the client connected to it, they
- completed the 3-way handshake.
- On server side, we can find below nstat output::
- nstatuser@nstat-b:~$ nstat | grep -i tcp
- TcpPassiveOpens 1 0.0
- TcpInSegs 2 0.0
- TcpOutSegs 1 0.0
- TcpExtTCPPureAcks 1 0.0
- On client side, we can find below nstat output::
- nstatuser@nstat-a:~$ nstat | grep -i tcp
- TcpActiveOpens 1 0.0
- TcpInSegs 1 0.0
- TcpOutSegs 2 0.0
- When the server received the first SYN, it replied a SYN+ACK, and came into
- SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
- SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
- packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
- of the 3-way handshake is a pure ACK without data, so
- TcpExtTCPPureAcks increased 1.
- When the client sent SYN, the client came into the SYN-SENT state, so
- TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
- ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
- 1, TcpOutSegs increased 2.
- TCP normal traffic
- ------------------
- Run nc on server::
- nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
- Listening on [0.0.0.0] (family 0, port 9000)
- Run nc on client::
- nstatuser@nstat-a:~$ nc -v nstat-b 9000
- Connection to nstat-b 9000 port [tcp/*] succeeded!
- Input a string in the nc client ('hello' in our example)::
- nstatuser@nstat-a:~$ nc -v nstat-b 9000
- Connection to nstat-b 9000 port [tcp/*] succeeded!
- hello
- The client side nstat output::
- nstatuser@nstat-a:~$ nstat
- #kernel
- IpInReceives 1 0.0
- IpInDelivers 1 0.0
- IpOutRequests 1 0.0
- TcpInSegs 1 0.0
- TcpOutSegs 1 0.0
- TcpExtTCPPureAcks 1 0.0
- TcpExtTCPOrigDataSent 1 0.0
- IpExtInOctets 52 0.0
- IpExtOutOctets 58 0.0
- IpExtInNoECTPkts 1 0.0
- The server side nstat output::
- nstatuser@nstat-b:~$ nstat
- #kernel
- IpInReceives 1 0.0
- IpInDelivers 1 0.0
- IpOutRequests 1 0.0
- TcpInSegs 1 0.0
- TcpOutSegs 1 0.0
- IpExtInOctets 58 0.0
- IpExtOutOctets 52 0.0
- IpExtInNoECTPkts 1 0.0
- Input a string in nc client side again ('world' in our exmaple)::
- nstatuser@nstat-a:~$ nc -v nstat-b 9000
- Connection to nstat-b 9000 port [tcp/*] succeeded!
- hello
- world
- Client side nstat output::
- nstatuser@nstat-a:~$ nstat
- #kernel
- IpInReceives 1 0.0
- IpInDelivers 1 0.0
- IpOutRequests 1 0.0
- TcpInSegs 1 0.0
- TcpOutSegs 1 0.0
- TcpExtTCPHPAcks 1 0.0
- TcpExtTCPOrigDataSent 1 0.0
- IpExtInOctets 52 0.0
- IpExtOutOctets 58 0.0
- IpExtInNoECTPkts 1 0.0
- Server side nstat output::
- nstatuser@nstat-b:~$ nstat
- #kernel
- IpInReceives 1 0.0
- IpInDelivers 1 0.0
- IpOutRequests 1 0.0
- TcpInSegs 1 0.0
- TcpOutSegs 1 0.0
- TcpExtTCPHPHits 1 0.0
- IpExtInOctets 58 0.0
- IpExtOutOctets 52 0.0
- IpExtInNoECTPkts 1 0.0
- Compare the first client-side nstat and the second client-side nstat,
- we could find one difference: the first one had a 'TcpExtTCPPureAcks',
- but the second one had a 'TcpExtTCPHPAcks'. The first server-side
- nstat and the second server-side nstat had a difference too: the
- second server-side nstat had a TcpExtTCPHPHits, but the first
- server-side nstat didn't have it. The network traffic patterns were
- exactly the same: the client sent a packet to the server, the server
- replied an ACK. But kernel handled them in different ways. When the
- TCP window scale option is not used, kernel will try to enable fast
- path immediately when the connection comes into the established state,
- but if the TCP window scale option is used, kernel will disable the
- fast path at first, and try to enable it after kerenl receives
- packets. We could use the 'ss' command to verify whether the window
- scale option is used. e.g. run below command on either server or
- client::
- nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
- Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
- tcp 0 0 192.168.122.250:40654 192.168.122.251:9000
- ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
- The 'wscale:7,7' means both server and client set the window scale
- option to 7. Now we could explain the nstat output in our test:
- In the first nstat output of client side, the client sent a packet, server
- reply an ACK, when kernel handled this ACK, the fast path was not
- enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
- In the second nstat output of client side, the client sent a packet again,
- and received another ACK from the server, in this time, the fast path is
- enabled, and the ACK was qualified for fast path, so it was handled by
- the fast path, so this ACK was counted into TcpExtTCPHPAcks.
- In the first nstat output of server side, fast path was not enabled,
- so there was no 'TcpExtTCPHPHits'.
- In the second nstat output of server side, the fast path was enabled,
- and the packet received from client qualified for fast path, so it
- was counted into 'TcpExtTCPHPHits'.
- TcpExtTCPAbortOnClose
- ---------------------
- On the server side, we run below python script::
- import socket
- import time
- port = 9000
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- s.bind(('0.0.0.0', port))
- s.listen(1)
- sock, addr = s.accept()
- while True:
- time.sleep(9999999)
- This python script listen on 9000 port, but doesn't read anything from
- the connection.
- On the client side, we send the string "hello" by nc::
- nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
- Then, we come back to the server side, the server has received the "hello"
- packet, and the TCP layer has acked this packet, but the application didn't
- read it yet. We type Ctrl-C to terminate the server script. Then we
- could find TcpExtTCPAbortOnClose increased 1 on the server side::
- nstatuser@nstat-b:~$ nstat | grep -i abort
- TcpExtTCPAbortOnClose 1 0.0
- If we run tcpdump on the server side, we could find the server sent a
- RST after we type Ctrl-C.
- TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
- ---------------------------------------------------
- Below is an example which let the orphan socket count be higher than
- net.ipv4.tcp_max_orphans.
- Change tcp_max_orphans to a smaller value on client::
- sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
- Client code (create 64 connection to server)::
- nstatuser@nstat-a:~$ cat client_orphan.py
- import socket
- import time
- server = 'nstat-b' # server address
- port = 9000
- count = 64
- connection_list = []
- for i in range(64):
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- s.connect((server, port))
- connection_list.append(s)
- print("connection_count: %d" % len(connection_list))
- while True:
- time.sleep(99999)
- Server code (accept 64 connection from client)::
- nstatuser@nstat-b:~$ cat server_orphan.py
- import socket
- import time
- port = 9000
- count = 64
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- s.bind(('0.0.0.0', port))
- s.listen(count)
- connection_list = []
- while True:
- sock, addr = s.accept()
- connection_list.append((sock, addr))
- print("connection_count: %d" % len(connection_list))
- Run the python scripts on server and client.
- On server::
- python3 server_orphan.py
- On client::
- python3 client_orphan.py
- Run iptables on server::
- sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
- Type Ctrl-C on client, stop client_orphan.py.
- Check TcpExtTCPAbortOnMemory on client::
- nstatuser@nstat-a:~$ nstat | grep -i abort
- TcpExtTCPAbortOnMemory 54 0.0
- Check orphane socket count on client::
- nstatuser@nstat-a:~$ ss -s
- Total: 131 (kernel 0)
- TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
- Transport Total IP IPv6
- * 0 - -
- RAW 1 0 1
- UDP 1 1 0
- TCP 14 13 1
- INET 16 14 2
- FRAG 0 0 0
- The explanation of the test: after run server_orphan.py and
- client_orphan.py, we set up 64 connections between server and
- client. Run the iptables command, the server will drop all packets from
- the client, type Ctrl-C on client_orphan.py, the system of the client
- would try to close these connections, and before they are closed
- gracefully, these connections became orphan sockets. As the iptables
- of the server blocked packets from the client, the server won't receive fin
- from the client, so all connection on clients would be stuck on FIN_WAIT_1
- stage, so they will keep as orphan sockets until timeout. We have echo
- 10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
- only keep 10 orphan sockets, for all other orphan sockets, the client
- system sent RST for them and delete them. We have 64 connections, so
- the 'ss -s' command shows the system has 10 orphan sockets, and the
- value of TcpExtTCPAbortOnMemory was 54.
- An additional explanation about orphan socket count: You could find the
- exactly orphan socket count by the 'ss -s' command, but when kernel
- decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
- doesn't always check the exactly orphan socket count. For increasing
- performance, kernel checks an approximate count firstly, if the
- approximate count is more than tcp_max_orphans, kernel checks the
- exact count again. So if the approximate count is less than
- tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
- would find TcpExtTCPAbortOnMemory is not increased at all. If
- tcp_max_orphans is large enough, it won't occur, but if you decrease
- tcp_max_orphans to a small value like our test, you might find this
- issue. So in our test, the client set up 64 connections although the
- tcp_max_orphans is 10. If the client only set up 11 connections, we
- can't find the change of TcpExtTCPAbortOnMemory.
- Continue the previous test, we wait for several minutes. Because of the
- iptables on the server blocked the traffic, the server wouldn't receive
- fin, and all the client's orphan sockets would timeout on the
- FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
- 10 timeout on the client::
- nstatuser@nstat-a:~$ nstat | grep -i abort
- TcpExtTCPAbortOnTimeout 10 0.0
- TcpExtTCPAbortOnLinger
- ----------------------
- The server side code::
- nstatuser@nstat-b:~$ cat server_linger.py
- import socket
- import time
- port = 9000
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- s.bind(('0.0.0.0', port))
- s.listen(1)
- sock, addr = s.accept()
- while True:
- time.sleep(9999999)
- The client side code::
- nstatuser@nstat-a:~$ cat client_linger.py
- import socket
- import struct
- server = 'nstat-b' # server address
- port = 9000
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
- s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
- s.connect((server, port))
- s.close()
- Run server_linger.py on server::
- nstatuser@nstat-b:~$ python3 server_linger.py
- Run client_linger.py on client::
- nstatuser@nstat-a:~$ python3 client_linger.py
- After run client_linger.py, check the output of nstat::
- nstatuser@nstat-a:~$ nstat | grep -i abort
- TcpExtTCPAbortOnLinger 1 0.0
- TcpExtTCPRcvCoalesce
- --------------------
- On the server, we run a program which listen on TCP port 9000, but
- doesn't read any data::
- import socket
- import time
- port = 9000
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- s.bind(('0.0.0.0', port))
- s.listen(1)
- sock, addr = s.accept()
- while True:
- time.sleep(9999999)
- Save the above code as server_coalesce.py, and run::
- python3 server_coalesce.py
- On the client, save below code as client_coalesce.py::
- import socket
- server = 'nstat-b'
- port = 9000
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- s.connect((server, port))
- Run::
- nstatuser@nstat-a:~$ python3 -i client_coalesce.py
- We use '-i' to come into the interactive mode, then a packet::
- >>> s.send(b'foo')
- 3
- Send a packet again::
- >>> s.send(b'bar')
- 3
- On the server, run nstat::
- ubuntu@nstat-b:~$ nstat
- #kernel
- IpInReceives 2 0.0
- IpInDelivers 2 0.0
- IpOutRequests 2 0.0
- TcpInSegs 2 0.0
- TcpOutSegs 2 0.0
- TcpExtTCPRcvCoalesce 1 0.0
- IpExtInOctets 110 0.0
- IpExtOutOctets 104 0.0
- IpExtInNoECTPkts 2 0.0
- The client sent two packets, server didn't read any data. When
- the second packet arrived at server, the first packet was still in
- the receiving queue. So the TCP layer merged the two packets, and we
- could find the TcpExtTCPRcvCoalesce increased 1.
- TcpExtListenOverflows and TcpExtListenDrops
- -------------------------------------------
- On server, run the nc command, listen on port 9000::
- nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
- Listening on [0.0.0.0] (family 0, port 9000)
- On client, run 3 nc commands in different terminals::
- nstatuser@nstat-a:~$ nc -v nstat-b 9000
- Connection to nstat-b 9000 port [tcp/*] succeeded!
- The nc command only accepts 1 connection, and the accept queue length
- is 1. On current linux implementation, set queue length to n means the
- actual queue length is n+1. Now we create 3 connections, 1 is accepted
- by nc, 2 in accepted queue, so the accept queue is full.
- Before running the 4th nc, we clean the nstat history on the server::
- nstatuser@nstat-b:~$ nstat -n
- Run the 4th nc on the client::
- nstatuser@nstat-a:~$ nc -v nstat-b 9000
- If the nc server is running on kernel 4.10 or higher version, you
- won't see the "Connection to ... succeeded!" string, because kernel
- will drop the SYN if the accept queue is full. If the nc client is running
- on an old kernel, you would see that the connection is succeeded,
- because kernel would complete the 3 way handshake and keep the socket
- on half open queue. I did the test on kernel 4.15. Below is the nstat
- on the server::
- nstatuser@nstat-b:~$ nstat
- #kernel
- IpInReceives 4 0.0
- IpInDelivers 4 0.0
- TcpInSegs 4 0.0
- TcpExtListenOverflows 4 0.0
- TcpExtListenDrops 4 0.0
- IpExtInOctets 240 0.0
- IpExtInNoECTPkts 4 0.0
- Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time
- between the 4th nc and the nstat was longer, the value of
- TcpExtListenOverflows and TcpExtListenDrops would be larger, because
- the SYN of the 4th nc was dropped, the client was retrying.
- IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes
- -------------------------------------------------
- server A IP address: 192.168.122.250
- server B IP address: 192.168.122.251
- Prepare on server A, add a route to server B::
- $ sudo ip route add 8.8.8.8/32 via 192.168.122.251
- Prepare on server B, disable send_redirects for all interfaces::
- $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
- $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
- $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
- $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
- We want to let sever A send a packet to 8.8.8.8, and route the packet
- to server B. When server B receives such packet, it might send a ICMP
- Redirect message to server A, set send_redirects to 0 will disable
- this behavior.
- First, generate InAddrErrors. On server B, we disable IP forwarding::
- $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
- On server A, we send packets to 8.8.8.8::
- $ nc -v 8.8.8.8 53
- On server B, we check the output of nstat::
- $ nstat
- #kernel
- IpInReceives 3 0.0
- IpInAddrErrors 3 0.0
- IpExtInOctets 180 0.0
- IpExtInNoECTPkts 3 0.0
- As we have let server A route 8.8.8.8 to server B, and we disabled IP
- forwarding on server B, Server A sent packets to server B, then server B
- dropped packets and increased IpInAddrErrors. As the nc command would
- re-send the SYN packet if it didn't receive a SYN+ACK, we could find
- multiple IpInAddrErrors.
- Second, generate IpExtInNoRoutes. On server B, we enable IP
- forwarding::
- $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
- Check the route table of server B and remove the default route::
- $ ip route show
- default via 192.168.122.1 dev ens3 proto static
- 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251
- $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static
- On server A, we contact 8.8.8.8 again::
- $ nc -v 8.8.8.8 53
- nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable
- On server B, run nstat::
- $ nstat
- #kernel
- IpInReceives 1 0.0
- IpOutRequests 1 0.0
- IcmpOutMsgs 1 0.0
- IcmpOutDestUnreachs 1 0.0
- IcmpMsgOutType3 1 0.0
- IpExtInNoRoutes 1 0.0
- IpExtInOctets 60 0.0
- IpExtOutOctets 88 0.0
- IpExtInNoECTPkts 1 0.0
- We enabled IP forwarding on server B, when server B received a packet
- which destination IP address is 8.8.8.8, server B will try to forward
- this packet. We have deleted the default route, there was no route for
- 8.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP
- Destination Unreachable" message to server A.
- Third, generate IpOutNoRoutes. Run ping command on server B::
- $ ping -c 1 8.8.8.8
- connect: Network is unreachable
- Run nstat on server B::
- $ nstat
- #kernel
- IpOutNoRoutes 1 0.0
- We have deleted the default route on server B. Server B couldn't find
- a route for the 8.8.8.8 IP address, so server B increased
- IpOutNoRoutes.
- TcpExtTCPACKSkippedSynRecv
- --------------------------
- In this test, we send 3 same SYN packets from client to server. The
- first SYN will let server create a socket, set it to Syn-Recv status,
- and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
- again, and record the reply time (the duplicate ACK reply time). The
- third SYN will let server check the previous duplicate ACK reply time,
- and decide to skip the duplicate ACK, then increase the
- TcpExtTCPACKSkippedSynRecv counter.
- Run tcpdump to capture a SYN packet::
- nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
- tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
- Open another terminal, run nc command::
- nstatuser@nstat-a:~$ nc nstat-b 9000
- As the nstat-b didn't listen on port 9000, it should reply a RST, and
- the nc command exited immediately. It was enough for the tcpdump
- command to capture a SYN packet. A linux server might use hardware
- offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
- might be not correct. We call tcprewrite to fix it::
- nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
- On nstat-b, we run nc to listen on port 9000::
- nstatuser@nstat-b:~$ nc -lkv 9000
- Listening on [0.0.0.0] (family 0, port 9000)
- On nstat-a, we blocked the packet from port 9000, or nstat-a would send
- RST to nstat-b::
- nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
- Send 3 SYN repeatly to nstat-b::
- nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
- Check snmp cunter on nstat-b::
- nstatuser@nstat-b:~$ nstat | grep -i skip
- TcpExtTCPACKSkippedSynRecv 1 0.0
- As we expected, TcpExtTCPACKSkippedSynRecv is 1.
- TcpExtTCPACKSkippedPAWS
- -----------------------
- To trigger PAWS, we could send an old SYN.
- On nstat-b, let nc listen on port 9000::
- nstatuser@nstat-b:~$ nc -lkv 9000
- Listening on [0.0.0.0] (family 0, port 9000)
- On nstat-a, run tcpdump to capture a SYN::
- nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
- tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
- On nstat-a, run nc as a client to connect nstat-b::
- nstatuser@nstat-a:~$ nc -v nstat-b 9000
- Connection to nstat-b 9000 port [tcp/*] succeeded!
- Now the tcpdump has captured the SYN and exit. We should fix the
- checksum::
- nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
- Send the SYN packet twice::
- nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
- On nstat-b, check the snmp counter::
- nstatuser@nstat-b:~$ nstat | grep -i skip
- TcpExtTCPACKSkippedPAWS 1 0.0
- We sent two SYN via tcpreplay, both of them would let PAWS check
- failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
- for the second SYN, and updated TcpExtTCPACKSkippedPAWS.
- TcpExtTCPACKSkippedSeq
- ----------------------
- To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid
- timestamp (to pass PAWS check) but the sequence number is out of
- window. The linux TCP stack would avoid to skip if the packet has
- data, so we need a pure ACK packet. To generate such a packet, we
- could create two sockets: one on port 9000, another on port 9001. Then
- we capture an ACK on port 9001, change the source/destination port
- numbers to match the port 9000 socket. Then we could trigger
- TcpExtTCPACKSkippedSeq via this packet.
- On nstat-b, open two terminals, run two nc commands to listen on both
- port 9000 and port 9001::
- nstatuser@nstat-b:~$ nc -lkv 9000
- Listening on [0.0.0.0] (family 0, port 9000)
- nstatuser@nstat-b:~$ nc -lkv 9001
- Listening on [0.0.0.0] (family 0, port 9001)
- On nstat-a, run two nc clients::
- nstatuser@nstat-a:~$ nc -v nstat-b 9000
- Connection to nstat-b 9000 port [tcp/*] succeeded!
- nstatuser@nstat-a:~$ nc -v nstat-b 9001
- Connection to nstat-b 9001 port [tcp/*] succeeded!
- On nstat-a, run tcpdump to capture an ACK::
- nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
- tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
- On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
- string 'foo' in our example::
- nstatuser@nstat-b:~$ nc -lkv 9001
- Listening on [0.0.0.0] (family 0, port 9001)
- Connection from nstat-a 42132 received!
- foo
- On nstat-a, the tcpdump should have caputred the ACK. We should check
- the source port numbers of the two nc clients::
- nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
- State Recv-Q Send-Q Local Address:Port Peer Address:Port
- ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000
- ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001
- Run tcprewrite, change port 9001 to port 9000, chagne port 42132 to
- port 50208::
- nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum
- Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
- nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
- Check TcpExtTCPACKSkippedSeq on nstat-b::
- nstatuser@nstat-b:~$ nstat | grep -i skip
- TcpExtTCPACKSkippedSeq 1 0.0
|