snmp_counter.rst 65 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793
  1. ============
  2. SNMP counter
  3. ============
  4. This document explains the meaning of SNMP counters.
  5. General IPv4 counters
  6. =====================
  7. All layer 4 packets and ICMP packets will change these counters, but
  8. these counters won't be changed by layer 2 packets (such as STP) or
  9. ARP packets.
  10. * IpInReceives
  11. Defined in `RFC1213 ipInReceives`_
  12. .. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26
  13. The number of packets received by the IP layer. It gets increasing at the
  14. beginning of ip_rcv function, always be updated together with
  15. IpExtInOctets. It will be increased even if the packet is dropped
  16. later (e.g. due to the IP header is invalid or the checksum is wrong
  17. and so on). It indicates the number of aggregated segments after
  18. GRO/LRO.
  19. * IpInDelivers
  20. Defined in `RFC1213 ipInDelivers`_
  21. .. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28
  22. The number of packets delivers to the upper layer protocols. E.g. TCP, UDP,
  23. ICMP and so on. If no one listens on a raw socket, only kernel
  24. supported protocols will be delivered, if someone listens on the raw
  25. socket, all valid IP packets will be delivered.
  26. * IpOutRequests
  27. Defined in `RFC1213 ipOutRequests`_
  28. .. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28
  29. The number of packets sent via IP layer, for both single cast and
  30. multicast packets, and would always be updated together with
  31. IpExtOutOctets.
  32. * IpExtInOctets and IpExtOutOctets
  33. They are Linux kernel extensions, no RFC definitions. Please note,
  34. RFC1213 indeed defines ifInOctets and ifOutOctets, but they
  35. are different things. The ifInOctets and ifOutOctets include the MAC
  36. layer header size but IpExtInOctets and IpExtOutOctets don't, they
  37. only include the IP layer header and the IP layer data.
  38. * IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts
  39. They indicate the number of four kinds of ECN IP packets, please refer
  40. `Explicit Congestion Notification`_ for more details.
  41. .. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6
  42. These 4 counters calculate how many packets received per ECN
  43. status. They count the real frame number regardless the LRO/GRO. So
  44. for the same packet, you might find that IpInReceives count 1, but
  45. IpExtInNoECTPkts counts 2 or more.
  46. * IpInHdrErrors
  47. Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is
  48. dropped due to the IP header error. It might happen in both IP input
  49. and IP forward paths.
  50. .. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27
  51. * IpInAddrErrors
  52. Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two
  53. scenarios: (1) The IP address is invalid. (2) The destination IP
  54. address is not a local address and IP forwarding is not enabled
  55. .. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27
  56. * IpExtInNoRoutes
  57. This counter means the packet is dropped when the IP stack receives a
  58. packet and can't find a route for it from the route table. It might
  59. happen when IP forwarding is enabled and the destination IP address is
  60. not a local address and there is no route for the destination IP
  61. address.
  62. * IpInUnknownProtos
  63. Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the
  64. layer 4 protocol is unsupported by kernel. If an application is using
  65. raw socket, kernel will always deliver the packet to the raw socket
  66. and this counter won't be increased.
  67. .. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27
  68. * IpExtInTruncatedPkts
  69. For IPv4 packet, it means the actual data size is smaller than the
  70. "Total Length" field in the IPv4 header.
  71. * IpInDiscards
  72. Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped
  73. in the IP receiving path and due to kernel internal reasons (e.g. no
  74. enough memory).
  75. .. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28
  76. * IpOutDiscards
  77. Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is
  78. dropped in the IP sending path and due to kernel internal reasons.
  79. .. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28
  80. * IpOutNoRoutes
  81. Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is
  82. dropped in the IP sending path and no route is found for it.
  83. .. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29
  84. ICMP counters
  85. =============
  86. * IcmpInMsgs and IcmpOutMsgs
  87. Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_
  88. .. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41
  89. .. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43
  90. As mentioned in the RFC1213, these two counters include errors, they
  91. would be increased even if the ICMP packet has an invalid type. The
  92. ICMP output path will check the header of a raw socket, so the
  93. IcmpOutMsgs would still be updated if the IP header is constructed by
  94. a userspace program.
  95. * ICMP named types
  96. | These counters include most of common ICMP types, they are:
  97. | IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_
  98. | IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_
  99. | IcmpInParmProbs: `RFC1213 icmpInParmProbs`_
  100. | IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_
  101. | IcmpInRedirects: `RFC1213 icmpInRedirects`_
  102. | IcmpInEchos: `RFC1213 icmpInEchos`_
  103. | IcmpInEchoReps: `RFC1213 icmpInEchoReps`_
  104. | IcmpInTimestamps: `RFC1213 icmpInTimestamps`_
  105. | IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_
  106. | IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_
  107. | IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_
  108. | IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_
  109. | IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_
  110. | IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_
  111. | IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_
  112. | IcmpOutRedirects: `RFC1213 icmpOutRedirects`_
  113. | IcmpOutEchos: `RFC1213 icmpOutEchos`_
  114. | IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_
  115. | IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_
  116. | IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_
  117. | IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_
  118. | IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_
  119. .. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41
  120. .. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41
  121. .. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42
  122. .. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42
  123. .. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42
  124. .. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42
  125. .. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42
  126. .. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42
  127. .. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43
  128. .. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43
  129. .. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43
  130. .. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44
  131. .. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44
  132. .. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44
  133. .. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44
  134. .. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44
  135. .. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45
  136. .. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45
  137. .. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45
  138. .. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45
  139. .. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45
  140. .. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46
  141. Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP
  142. Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are
  143. straightforward. The 'In' counter means kernel receives such a packet
  144. and the 'Out' counter means kernel sends such a packet.
  145. * ICMP numeric types
  146. They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the
  147. ICMP type number. These counters track all kinds of ICMP packets. The
  148. ICMP type number definition could be found in the `ICMP parameters`_
  149. document.
  150. .. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml
  151. For example, if the Linux kernel sends an ICMP Echo packet, the
  152. IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply
  153. packet, IcmpMsgInType0 would increase 1.
  154. * IcmpInCsumErrors
  155. This counter indicates the checksum of the ICMP packet is
  156. wrong. Kernel verifies the checksum after updating the IcmpInMsgs and
  157. before updating IcmpMsgInType[N]. If a packet has bad checksum, the
  158. IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated.
  159. * IcmpInErrors and IcmpOutErrors
  160. Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_
  161. .. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41
  162. .. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43
  163. When an error occurs in the ICMP packet handler path, these two
  164. counters would be updated. The receiving packet path use IcmpInErrors
  165. and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors
  166. is increased, IcmpInErrors would always be increased too.
  167. relationship of the ICMP counters
  168. ---------------------------------
  169. The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they
  170. are updated at the same time. The sum of IcmpMsgInType[N] plus
  171. IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel
  172. receives an ICMP packet, kernel follows below logic:
  173. 1. increase IcmpInMsgs
  174. 2. if has any error, update IcmpInErrors and finish the process
  175. 3. update IcmpMsgOutType[N]
  176. 4. handle the packet depending on the type, if has any error, update
  177. IcmpInErrors and finish the process
  178. So if all errors occur in step (2), IcmpInMsgs should be equal to the
  179. sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in
  180. step (4), IcmpInMsgs should be equal to the sum of
  181. IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4),
  182. IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus
  183. IcmpInErrors.
  184. General TCP counters
  185. ====================
  186. * TcpInSegs
  187. Defined in `RFC1213 tcpInSegs`_
  188. .. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48
  189. The number of packets received by the TCP layer. As mentioned in
  190. RFC1213, it includes the packets received in error, such as checksum
  191. error, invalid TCP header and so on. Only one error won't be included:
  192. if the layer 2 destination address is not the NIC's layer 2
  193. address. It might happen if the packet is a multicast or broadcast
  194. packet, or the NIC is in promiscuous mode. In these situations, the
  195. packets would be delivered to the TCP layer, but the TCP layer will discard
  196. these packets before increasing TcpInSegs. The TcpInSegs counter
  197. isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs
  198. counter would only increase 1.
  199. * TcpOutSegs
  200. Defined in `RFC1213 tcpOutSegs`_
  201. .. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48
  202. The number of packets sent by the TCP layer. As mentioned in RFC1213,
  203. it excludes the retransmitted packets. But it includes the SYN, ACK
  204. and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of
  205. GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will
  206. increase 2.
  207. * TcpActiveOpens
  208. Defined in `RFC1213 tcpActiveOpens`_
  209. .. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47
  210. It means the TCP layer sends a SYN, and come into the SYN-SENT
  211. state. Every time TcpActiveOpens increases 1, TcpOutSegs should always
  212. increase 1.
  213. * TcpPassiveOpens
  214. Defined in `RFC1213 tcpPassiveOpens`_
  215. .. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47
  216. It means the TCP layer receives a SYN, replies a SYN+ACK, come into
  217. the SYN-RCVD state.
  218. * TcpExtTCPRcvCoalesce
  219. When packets are received by the TCP layer and are not be read by the
  220. application, the TCP layer will try to merge them. This counter
  221. indicate how many packets are merged in such situation. If GRO is
  222. enabled, lots of packets would be merged by GRO, these packets
  223. wouldn't be counted to TcpExtTCPRcvCoalesce.
  224. * TcpExtTCPAutoCorking
  225. When sending packets, the TCP layer will try to merge small packets to
  226. a bigger one. This counter increase 1 for every packet merged in such
  227. situation. Please refer to the LWN article for more details:
  228. https://lwn.net/Articles/576263/
  229. * TcpExtTCPOrigDataSent
  230. This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
  231. explaination below::
  232. TCPOrigDataSent: number of outgoing packets with original data (excluding
  233. retransmission but including data-in-SYN). This counter is different from
  234. TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is
  235. more useful to track the TCP retransmission rate.
  236. * TCPSynRetrans
  237. This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
  238. explaination below::
  239. TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down
  240. retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
  241. * TCPFastOpenActiveFail
  242. This counter is explained by `kernel commit f19c29e3e391`_, I pasted the
  243. explaination below::
  244. TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because
  245. the remote does not accept it or the attempts timed out.
  246. .. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd
  247. * TcpExtListenOverflows and TcpExtListenDrops
  248. When kernel receives a SYN from a client, and if the TCP accept queue
  249. is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows.
  250. At the same time kernel will also add 1 to TcpExtListenDrops. When a
  251. TCP socket is in LISTEN state, and kernel need to drop a packet,
  252. kernel would always add 1 to TcpExtListenDrops. So increase
  253. TcpExtListenOverflows would let TcpExtListenDrops increasing at the
  254. same time, but TcpExtListenDrops would also increase without
  255. TcpExtListenOverflows increasing, e.g. a memory allocation fail would
  256. also let TcpExtListenDrops increase.
  257. Note: The above explanation is based on kernel 4.10 or above version, on
  258. an old kernel, the TCP stack has different behavior when TCP accept
  259. queue is full. On the old kernel, TCP stack won't drop the SYN, it
  260. would complete the 3-way handshake. As the accept queue is full, TCP
  261. stack will keep the socket in the TCP half-open queue. As it is in the
  262. half open queue, TCP stack will send SYN+ACK on an exponential backoff
  263. timer, after client replies ACK, TCP stack checks whether the accept
  264. queue is still full, if it is not full, moves the socket to the accept
  265. queue, if it is full, keeps the socket in the half-open queue, at next
  266. time client replies ACK, this socket will get another chance to move
  267. to the accept queue.
  268. TCP Fast Open
  269. =============
  270. * TcpEstabResets
  271. Defined in `RFC1213 tcpEstabResets`_.
  272. .. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48
  273. * TcpAttemptFails
  274. Defined in `RFC1213 tcpAttemptFails`_.
  275. .. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48
  276. * TcpOutRsts
  277. Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates
  278. the 'segments sent containing the RST flag', but in linux kernel, this
  279. couner indicates the segments kerenl tried to send. The sending
  280. process might be failed due to some errors (e.g. memory alloc failed).
  281. .. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52
  282. * TcpExtTCPSpuriousRtxHostQueues
  283. When the TCP stack wants to retransmit a packet, and finds that packet
  284. is not lost in the network, but the packet is not sent yet, the TCP
  285. stack would give up the retransmission and update this counter. It
  286. might happen if a packet stays too long time in a qdisc or driver
  287. queue.
  288. * TcpEstabResets
  289. The socket receives a RST packet in Establish or CloseWait state.
  290. * TcpExtTCPKeepAlive
  291. This counter indicates many keepalive packets were sent. The keepalive
  292. won't be enabled by default. A userspace program could enable it by
  293. setting the SO_KEEPALIVE socket option.
  294. * TcpExtTCPSpuriousRTOs
  295. The spurious retransmission timeout detected by the `F-RTO`_
  296. algorithm.
  297. .. _F-RTO: https://tools.ietf.org/html/rfc5682
  298. TCP Fast Path
  299. =============
  300. When kernel receives a TCP packet, it has two paths to handler the
  301. packet, one is fast path, another is slow path. The comment in kernel
  302. code provides a good explanation of them, I pasted them below::
  303. It is split into a fast path and a slow path. The fast path is
  304. disabled when:
  305. - A zero window was announced from us
  306. - zero window probing
  307. is only handled properly on the slow path.
  308. - Out of order segments arrived.
  309. - Urgent data is expected.
  310. - There is no buffer space left
  311. - Unexpected TCP flags/window values/header lengths are received
  312. (detected by checking the TCP header against pred_flags)
  313. - Data is sent in both directions. The fast path only supports pure senders
  314. or pure receivers (this means either the sequence number or the ack
  315. value must stay constant)
  316. - Unexpected TCP option.
  317. Kernel will try to use fast path unless any of the above conditions
  318. are satisfied. If the packets are out of order, kernel will handle
  319. them in slow path, which means the performance might be not very
  320. good. Kernel would also come into slow path if the "Delayed ack" is
  321. used, because when using "Delayed ack", the data is sent in both
  322. directions. When the TCP window scale option is not used, kernel will
  323. try to enable fast path immediately when the connection comes into the
  324. established state, but if the TCP window scale option is used, kernel
  325. will disable the fast path at first, and try to enable it after kernel
  326. receives packets.
  327. * TcpExtTCPPureAcks and TcpExtTCPHPAcks
  328. If a packet set ACK flag and has no data, it is a pure ACK packet, if
  329. kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1,
  330. if kernel handles it in the slow path, TcpExtTCPPureAcks will
  331. increase 1.
  332. * TcpExtTCPHPHits
  333. If a TCP packet has data (which means it is not a pure ACK packet),
  334. and this packet is handled in the fast path, TcpExtTCPHPHits will
  335. increase 1.
  336. TCP abort
  337. =========
  338. * TcpExtTCPAbortOnData
  339. It means TCP layer has data in flight, but need to close the
  340. connection. So TCP layer sends a RST to the other side, indicate the
  341. connection is not closed very graceful. An easy way to increase this
  342. counter is using the SO_LINGER option. Please refer to the SO_LINGER
  343. section of the `socket man page`_:
  344. .. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html
  345. By default, when an application closes a connection, the close function
  346. will return immediately and kernel will try to send the in-flight data
  347. async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger
  348. to a positive number, the close function won't return immediately, but
  349. wait for the in-flight data are acked by the other side, the max wait
  350. time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0,
  351. when the application closes a connection, kernel will send a RST
  352. immediately and increase the TcpExtTCPAbortOnData counter.
  353. * TcpExtTCPAbortOnClose
  354. This counter means the application has unread data in the TCP layer when
  355. the application wants to close the TCP connection. In such a situation,
  356. kernel will send a RST to the other side of the TCP connection.
  357. * TcpExtTCPAbortOnMemory
  358. When an application closes a TCP connection, kernel still need to track
  359. the connection, let it complete the TCP disconnect process. E.g. an
  360. app calls the close method of a socket, kernel sends fin to the other
  361. side of the connection, then the app has no relationship with the
  362. socket any more, but kernel need to keep the socket, this socket
  363. becomes an orphan socket, kernel waits for the reply of the other side,
  364. and would come to the TIME_WAIT state finally. When kernel has no
  365. enough memory to keep the orphan socket, kernel would send an RST to
  366. the other side, and delete the socket, in such situation, kernel will
  367. increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger
  368. TcpExtTCPAbortOnMemory:
  369. 1. the memory used by the TCP protocol is higher than the third value of
  370. the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_:
  371. .. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html
  372. 2. the orphan socket count is higher than net.ipv4.tcp_max_orphans
  373. * TcpExtTCPAbortOnTimeout
  374. This counter will increase when any of the TCP timers expire. In such
  375. situation, kernel won't send RST, just give up the connection.
  376. * TcpExtTCPAbortOnLinger
  377. When a TCP connection comes into FIN_WAIT_2 state, instead of waiting
  378. for the fin packet from the other side, kernel could send a RST and
  379. delete the socket immediately. This is not the default behavior of
  380. Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option,
  381. you could let kernel follow this behavior.
  382. * TcpExtTCPAbortFailed
  383. The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is
  384. satisfied. If an internal error occurs during this process,
  385. TcpExtTCPAbortFailed will be increased.
  386. .. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50
  387. TCP Hybrid Slow Start
  388. =====================
  389. The Hybrid Slow Start algorithm is an enhancement of the traditional
  390. TCP congestion window Slow Start algorithm. It uses two pieces of
  391. information to detect whether the max bandwidth of the TCP path is
  392. approached. The two pieces of information are ACK train length and
  393. increase in packet delay. For detail information, please refer the
  394. `Hybrid Slow Start paper`_. Either ACK train length or packet delay
  395. hits a specific threshold, the congestion control algorithm will come
  396. into the Congestion Avoidance state. Until v4.20, two congestion
  397. control algorithms are using Hybrid Slow Start, they are cubic (the
  398. default congestion control algorithm) and cdg. Four snmp counters
  399. relate with the Hybrid Slow Start algorithm.
  400. .. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf
  401. * TcpExtTCPHystartTrainDetect
  402. How many times the ACK train length threshold is detected
  403. * TcpExtTCPHystartTrainCwnd
  404. The sum of CWND detected by ACK train length. Dividing this value by
  405. TcpExtTCPHystartTrainDetect is the average CWND which detected by the
  406. ACK train length.
  407. * TcpExtTCPHystartDelayDetect
  408. How many times the packet delay threshold is detected.
  409. * TcpExtTCPHystartDelayCwnd
  410. The sum of CWND detected by packet delay. Dividing this value by
  411. TcpExtTCPHystartDelayDetect is the average CWND which detected by the
  412. packet delay.
  413. TCP retransmission and congestion control
  414. =========================================
  415. The TCP protocol has two retransmission mechanisms: SACK and fast
  416. recovery. They are exclusive with each other. When SACK is enabled,
  417. the kernel TCP stack would use SACK, or kernel would use fast
  418. recovery. The SACK is a TCP option, which is defined in `RFC2018`_,
  419. the fast recovery is defined in `RFC6582`_, which is also called
  420. 'Reno'.
  421. The TCP congestion control is a big and complex topic. To understand
  422. the related snmp counter, we need to know the states of the congestion
  423. control state machine. There are 5 states: Open, Disorder, CWR,
  424. Recovery and Loss. For details about these states, please refer page 5
  425. and page 6 of this document:
  426. https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf
  427. .. _RFC2018: https://tools.ietf.org/html/rfc2018
  428. .. _RFC6582: https://tools.ietf.org/html/rfc6582
  429. * TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery
  430. When the congestion control comes into Recovery state, if sack is
  431. used, TcpExtTCPSackRecovery increases 1, if sack is not used,
  432. TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP
  433. stack begins to retransmit the lost packets.
  434. * TcpExtTCPSACKReneging
  435. A packet was acknowledged by SACK, but the receiver has dropped this
  436. packet, so the sender needs to retransmit this packet. In this
  437. situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver
  438. could drop a packet which has been acknowledged by SACK, although it is
  439. unusual, it is allowed by the TCP protocol. The sender doesn't really
  440. know what happened on the receiver side. The sender just waits until
  441. the RTO expires for this packet, then the sender assumes this packet
  442. has been dropped by the receiver.
  443. * TcpExtTCPRenoReorder
  444. The reorder packet is detected by fast recovery. It would only be used
  445. if SACK is disabled. The fast recovery algorithm detects recorder by
  446. the duplicate ACK number. E.g., if retransmission is triggered, and
  447. the original retransmitted packet is not lost, it is just out of
  448. order, the receiver would acknowledge multiple times, one for the
  449. retransmitted packet, another for the arriving of the original out of
  450. order packet. Thus the sender would find more ACks than its
  451. expectation, and the sender knows out of order occurs.
  452. * TcpExtTCPTSReorder
  453. The reorder packet is detected when a hole is filled. E.g., assume the
  454. sender sends packet 1,2,3,4,5, and the receiving order is
  455. 1,2,4,5,3. When the sender receives the ACK of packet 3 (which will
  456. fill the hole), two conditions will let TcpExtTCPTSReorder increase
  457. 1: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet
  458. 3 is retransmitted but the timestamp of the packet 3's ACK is earlier
  459. than the retransmission timestamp.
  460. * TcpExtTCPSACKReorder
  461. The reorder packet detected by SACK. The SACK has two methods to
  462. detect reorder: (1) DSACK is received by the sender. It means the
  463. sender sends the same packet more than one times. And the only reason
  464. is the sender believes an out of order packet is lost so it sends the
  465. packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and
  466. the sender has received SACKs for packet 2 and 5, now the sender
  467. receives SACK for packet 4 and the sender doesn't retransmit the
  468. packet yet, the sender would know packet 4 is out of order. The TCP
  469. stack of kernel will increase TcpExtTCPSACKReorder for both of the
  470. above scenarios.
  471. * TcpExtTCPSlowStartRetrans
  472. The TCP stack wants to retransmit a packet and the congestion control
  473. state is 'Loss'.
  474. * TcpExtTCPFastRetrans
  475. The TCP stack wants to retransmit a packet and the congestion control
  476. state is not 'Loss'.
  477. * TcpExtTCPLostRetransmit
  478. A SACK points out that a retransmission packet is lost again.
  479. * TcpExtTCPRetransFail
  480. The TCP stack tries to deliver a retransmission packet to lower layers
  481. but the lower layers return an error.
  482. * TcpExtTCPSynRetrans
  483. The TCP stack retransmits a SYN packet.
  484. DSACK
  485. =====
  486. The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report
  487. duplicate packets to the sender. There are two kinds of
  488. duplications: (1) a packet which has been acknowledged is
  489. duplicate. (2) an out of order packet is duplicate. The TCP stack
  490. counts these two kinds of duplications on both receiver side and
  491. sender side.
  492. .. _RFC2883 : https://tools.ietf.org/html/rfc2883
  493. * TcpExtTCPDSACKOldSent
  494. The TCP stack receives a duplicate packet which has been acked, so it
  495. sends a DSACK to the sender.
  496. * TcpExtTCPDSACKOfoSent
  497. The TCP stack receives an out of order duplicate packet, so it sends a
  498. DSACK to the sender.
  499. * TcpExtTCPDSACKRecv
  500. The TCP stack receives a DSACK, which indicates an acknowledged
  501. duplicate packet is received.
  502. * TcpExtTCPDSACKOfoRecv
  503. The TCP stack receives a DSACK, which indicate an out of order
  504. duplicate packet is received.
  505. invalid SACK and DSACK
  506. ======================
  507. When a SACK (or DSACK) block is invalid, a corresponding counter would
  508. be updated. The validation method is base on the start/end sequence
  509. number of the SACK block. For more details, please refer the comment
  510. of the function tcp_is_sackblock_valid in the kernel source code. A
  511. SACK option could have up to 4 blocks, they are checked
  512. individually. E.g., if 3 blocks of a SACk is invalid, the
  513. corresponding counter would be updated 3 times. The comment of the
  514. `Add counters for discarded SACK blocks`_ patch has additional
  515. explaination:
  516. .. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32
  517. * TcpExtTCPSACKDiscard
  518. This counter indicates how many SACK blocks are invalid. If the invalid
  519. SACK block is caused by ACK recording, the TCP stack will only ignore
  520. it and won't update this counter.
  521. * TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo
  522. When a DSACK block is invalid, one of these two counters would be
  523. updated. Which counter will be updated depends on the undo_marker flag
  524. of the TCP socket. If the undo_marker is not set, the TCP stack isn't
  525. likely to re-transmit any packets, and we still receive an invalid
  526. DSACK block, the reason might be that the packet is duplicated in the
  527. middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo
  528. will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld
  529. will be updated. As implied in its name, it might be an old packet.
  530. SACK shift
  531. ==========
  532. The linux networking stack stores data in sk_buff struct (skb for
  533. short). If a SACK block acrosses multiple skb, the TCP stack will try
  534. to re-arrange data in these skb. E.g. if a SACK block acknowledges seq
  535. 10 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and
  536. 15 in skb2 would be moved to skb1. This operation is 'shift'. If a
  537. SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has
  538. seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be
  539. discard, this operation is 'merge'.
  540. * TcpExtTCPSackShifted
  541. A skb is shifted
  542. * TcpExtTCPSackMerged
  543. A skb is merged
  544. * TcpExtTCPSackShiftFallback
  545. A skb should be shifted or merged, but the TCP stack doesn't do it for
  546. some reasons.
  547. TCP out of order
  548. ================
  549. * TcpExtTCPOFOQueue
  550. The TCP layer receives an out of order packet and has enough memory
  551. to queue it.
  552. * TcpExtTCPOFODrop
  553. The TCP layer receives an out of order packet but doesn't have enough
  554. memory, so drops it. Such packets won't be counted into
  555. TcpExtTCPOFOQueue.
  556. * TcpExtTCPOFOMerge
  557. The received out of order packet has an overlay with the previous
  558. packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge
  559. packets will also be counted into TcpExtTCPOFOQueue.
  560. TCP PAWS
  561. ========
  562. PAWS (Protection Against Wrapped Sequence numbers) is an algorithm
  563. which is used to drop old packets. It depends on the TCP
  564. timestamps. For detail information, please refer the `timestamp wiki`_
  565. and the `RFC of PAWS`_.
  566. .. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17
  567. .. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps
  568. * TcpExtPAWSActive
  569. Packets are dropped by PAWS in Syn-Sent status.
  570. * TcpExtPAWSEstab
  571. Packets are dropped by PAWS in any status other than Syn-Sent.
  572. TCP ACK skip
  573. ============
  574. In some scenarios, kernel would avoid sending duplicate ACKs too
  575. frequently. Please find more details in the tcp_invalid_ratelimit
  576. section of the `sysctl document`_. When kernel decides to skip an ACK
  577. due to tcp_invalid_ratelimit, kernel would update one of below
  578. counters to indicate the ACK is skipped in which scenario. The ACK
  579. would only be skipped if the received packet is either a SYN packet or
  580. it has no data.
  581. .. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst
  582. * TcpExtTCPACKSkippedSynRecv
  583. The ACK is skipped in Syn-Recv status. The Syn-Recv status means the
  584. TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is
  585. waiting for an ACK. Generally, the TCP stack doesn't need to send ACK
  586. in the Syn-Recv status. But in several scenarios, the TCP stack need
  587. to send an ACK. E.g., the TCP stack receives the same SYN packet
  588. repeately, the received packet does not pass the PAWS check, or the
  589. received packet sequence number is out of window. In these scenarios,
  590. the TCP stack needs to send ACK. If the ACk sending frequency is higher than
  591. tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and
  592. increase TcpExtTCPACKSkippedSynRecv.
  593. * TcpExtTCPACKSkippedPAWS
  594. The ACK is skipped due to PAWS (Protect Against Wrapped Sequence
  595. numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2
  596. or Time-Wait statuses, the skipped ACK would be counted to
  597. TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or
  598. TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK
  599. would be counted to TcpExtTCPACKSkippedPAWS.
  600. * TcpExtTCPACKSkippedSeq
  601. The sequence number is out of window and the timestamp passes the PAWS
  602. check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait.
  603. * TcpExtTCPACKSkippedFinWait2
  604. The ACK is skipped in Fin-Wait-2 status, the reason would be either
  605. PAWS check fails or the received sequence number is out of window.
  606. * TcpExtTCPACKSkippedTimeWait
  607. Tha ACK is skipped in Time-Wait status, the reason would be either
  608. PAWS check failed or the received sequence number is out of window.
  609. * TcpExtTCPACKSkippedChallenge
  610. The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines
  611. 3 kind of challenge ACK, please refer `RFC 5961 section 3.2`_,
  612. `RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these
  613. three scenarios, In some TCP status, the linux TCP stack would also
  614. send challenge ACKs if the ACK number is before the first
  615. unacknowledged number (more strict than `RFC 5961 section 5.2`_).
  616. .. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7
  617. .. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9
  618. .. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11
  619. TCP receive window
  620. ==================
  621. * TcpExtTCPWantZeroWindowAdv
  622. Depending on current memory usage, the TCP stack tries to set receive
  623. window to zero. But the receive window might still be a no-zero
  624. value. For example, if the previous window size is 10, and the TCP
  625. stack receives 3 bytes, the current window size would be 7 even if the
  626. window size calculated by the memory usage is zero.
  627. * TcpExtTCPToZeroWindowAdv
  628. The TCP receive window is set to zero from a no-zero value.
  629. * TcpExtTCPFromZeroWindowAdv
  630. The TCP receive window is set to no-zero value from zero.
  631. Delayed ACK
  632. ===========
  633. The TCP Delayed ACK is a technique which is used for reducing the
  634. packet count in the network. For more details, please refer the
  635. `Delayed ACK wiki`_
  636. .. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment
  637. * TcpExtDelayedACKs
  638. A delayed ACK timer expires. The TCP stack will send a pure ACK packet
  639. and exit the delayed ACK mode.
  640. * TcpExtDelayedACKLocked
  641. A delayed ACK timer expires, but the TCP stack can't send an ACK
  642. immediately due to the socket is locked by a userspace program. The
  643. TCP stack will send a pure ACK later (after the userspace program
  644. unlock the socket). When the TCP stack sends the pure ACK later, the
  645. TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK
  646. mode.
  647. * TcpExtDelayedACKLost
  648. It will be updated when the TCP stack receives a packet which has been
  649. ACKed. A Delayed ACK loss might cause this issue, but it would also be
  650. triggered by other reasons, such as a packet is duplicated in the
  651. network.
  652. Tail Loss Probe (TLP)
  653. =====================
  654. TLP is an algorithm which is used to detect TCP packet loss. For more
  655. details, please refer the `TLP paper`_.
  656. .. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01
  657. * TcpExtTCPLossProbes
  658. A TLP probe packet is sent.
  659. * TcpExtTCPLossProbeRecovery
  660. A packet loss is detected and recovered by TLP.
  661. TCP Fast Open description
  662. =========================
  663. TCP Fast Open is a technology which allows data transfer before the
  664. 3-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a
  665. general description.
  666. .. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open
  667. * TcpExtTCPFastOpenActive
  668. When the TCP stack receives an ACK packet in the SYN-SENT status, and
  669. the ACK packet acknowledges the data in the SYN packet, the TCP stack
  670. understand the TFO cookie is accepted by the other side, then it
  671. updates this counter.
  672. * TcpExtTCPFastOpenActiveFail
  673. This counter indicates that the TCP stack initiated a TCP Fast Open,
  674. but it failed. This counter would be updated in three scenarios: (1)
  675. the other side doesn't acknowledge the data in the SYN packet. (2) The
  676. SYN packet which has the TFO cookie is timeout at least once. (3)
  677. after the 3-way handshake, the retransmission timeout happens
  678. net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole
  679. fast open after the handshake.
  680. * TcpExtTCPFastOpenPassive
  681. This counter indicates how many times the TCP stack accepts the fast
  682. open request.
  683. * TcpExtTCPFastOpenPassiveFail
  684. This counter indicates how many times the TCP stack rejects the fast
  685. open request. It is caused by either the TFO cookie is invalid or the
  686. TCP stack finds an error during the socket creating process.
  687. * TcpExtTCPFastOpenListenOverflow
  688. When the pending fast open request number is larger than
  689. fastopenq->max_qlen, the TCP stack will reject the fast open request
  690. and update this counter. When this counter is updated, the TCP stack
  691. won't update TcpExtTCPFastOpenPassive or
  692. TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the
  693. TCP_FASTOPEN socket operation and it could not be larger than
  694. net.core.somaxconn. For example:
  695. setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));
  696. * TcpExtTCPFastOpenCookieReqd
  697. This counter indicates how many times a client wants to request a TFO
  698. cookie.
  699. SYN cookies
  700. ===========
  701. SYN cookies are used to mitigate SYN flood, for details, please refer
  702. the `SYN cookies wiki`_.
  703. .. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies
  704. * TcpExtSyncookiesSent
  705. It indicates how many SYN cookies are sent.
  706. * TcpExtSyncookiesRecv
  707. How many reply packets of the SYN cookies the TCP stack receives.
  708. * TcpExtSyncookiesFailed
  709. The MSS decoded from the SYN cookie is invalid. When this counter is
  710. updated, the received packet won't be treated as a SYN cookie and the
  711. TcpExtSyncookiesRecv counter wont be updated.
  712. Challenge ACK
  713. =============
  714. For details of challenge ACK, please refer the explaination of
  715. TcpExtTCPACKSkippedChallenge.
  716. * TcpExtTCPChallengeACK
  717. The number of challenge acks sent.
  718. * TcpExtTCPSYNChallenge
  719. The number of challenge acks sent in response to SYN packets. After
  720. updates this counter, the TCP stack might send a challenge ACK and
  721. update the TcpExtTCPChallengeACK counter, or it might also skip to
  722. send the challenge and update the TcpExtTCPACKSkippedChallenge.
  723. prune
  724. =====
  725. When a socket is under memory pressure, the TCP stack will try to
  726. reclaim memory from the receiving queue and out of order queue. One of
  727. the reclaiming method is 'collapse', which means allocate a big sbk,
  728. copy the contiguous skbs to the single big skb, and free these
  729. contiguous skbs.
  730. * TcpExtPruneCalled
  731. The TCP stack tries to reclaim memory for a socket. After updates this
  732. counter, the TCP stack will try to collapse the out of order queue and
  733. the receiving queue. If the memory is still not enough, the TCP stack
  734. will try to discard packets from the out of order queue (and update the
  735. TcpExtOfoPruned counter)
  736. * TcpExtOfoPruned
  737. The TCP stack tries to discard packet on the out of order queue.
  738. * TcpExtRcvPruned
  739. After 'collapse' and discard packets from the out of order queue, if
  740. the actually used memory is still larger than the max allowed memory,
  741. this counter will be updated. It means the 'prune' fails.
  742. * TcpExtTCPRcvCollapsed
  743. This counter indicates how many skbs are freed during 'collapse'.
  744. examples
  745. ========
  746. ping test
  747. ---------
  748. Run the ping command against the public dns server 8.8.8.8::
  749. nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1
  750. PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
  751. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms
  752. --- 8.8.8.8 ping statistics ---
  753. 1 packets transmitted, 1 received, 0% packet loss, time 0ms
  754. rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms
  755. The nstayt result::
  756. nstatuser@nstat-a:~$ nstat
  757. #kernel
  758. IpInReceives 1 0.0
  759. IpInDelivers 1 0.0
  760. IpOutRequests 1 0.0
  761. IcmpInMsgs 1 0.0
  762. IcmpInEchoReps 1 0.0
  763. IcmpOutMsgs 1 0.0
  764. IcmpOutEchos 1 0.0
  765. IcmpMsgInType0 1 0.0
  766. IcmpMsgOutType8 1 0.0
  767. IpExtInOctets 84 0.0
  768. IpExtOutOctets 84 0.0
  769. IpExtInNoECTPkts 1 0.0
  770. The Linux server sent an ICMP Echo packet, so IpOutRequests,
  771. IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The
  772. server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs,
  773. IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply
  774. was passed to the ICMP layer via IP layer, so IpInDelivers was
  775. increased 1. The default ping data size is 48, so an ICMP Echo packet
  776. and its corresponding Echo Reply packet are constructed by:
  777. * 14 bytes MAC header
  778. * 20 bytes IP header
  779. * 16 bytes ICMP header
  780. * 48 bytes data (default value of the ping command)
  781. So the IpExtInOctets and IpExtOutOctets are 20+16+48=84.
  782. tcp 3-way handshake
  783. -------------------
  784. On server side, we run::
  785. nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000
  786. Listening on [0.0.0.0] (family 0, port 9000)
  787. On client side, we run::
  788. nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000
  789. Connection to 192.168.122.251 9000 port [tcp/*] succeeded!
  790. The server listened on tcp 9000 port, the client connected to it, they
  791. completed the 3-way handshake.
  792. On server side, we can find below nstat output::
  793. nstatuser@nstat-b:~$ nstat | grep -i tcp
  794. TcpPassiveOpens 1 0.0
  795. TcpInSegs 2 0.0
  796. TcpOutSegs 1 0.0
  797. TcpExtTCPPureAcks 1 0.0
  798. On client side, we can find below nstat output::
  799. nstatuser@nstat-a:~$ nstat | grep -i tcp
  800. TcpActiveOpens 1 0.0
  801. TcpInSegs 1 0.0
  802. TcpOutSegs 2 0.0
  803. When the server received the first SYN, it replied a SYN+ACK, and came into
  804. SYN-RCVD state, so TcpPassiveOpens increased 1. The server received
  805. SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2
  806. packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK
  807. of the 3-way handshake is a pure ACK without data, so
  808. TcpExtTCPPureAcks increased 1.
  809. When the client sent SYN, the client came into the SYN-SENT state, so
  810. TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent
  811. ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased
  812. 1, TcpOutSegs increased 2.
  813. TCP normal traffic
  814. ------------------
  815. Run nc on server::
  816. nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
  817. Listening on [0.0.0.0] (family 0, port 9000)
  818. Run nc on client::
  819. nstatuser@nstat-a:~$ nc -v nstat-b 9000
  820. Connection to nstat-b 9000 port [tcp/*] succeeded!
  821. Input a string in the nc client ('hello' in our example)::
  822. nstatuser@nstat-a:~$ nc -v nstat-b 9000
  823. Connection to nstat-b 9000 port [tcp/*] succeeded!
  824. hello
  825. The client side nstat output::
  826. nstatuser@nstat-a:~$ nstat
  827. #kernel
  828. IpInReceives 1 0.0
  829. IpInDelivers 1 0.0
  830. IpOutRequests 1 0.0
  831. TcpInSegs 1 0.0
  832. TcpOutSegs 1 0.0
  833. TcpExtTCPPureAcks 1 0.0
  834. TcpExtTCPOrigDataSent 1 0.0
  835. IpExtInOctets 52 0.0
  836. IpExtOutOctets 58 0.0
  837. IpExtInNoECTPkts 1 0.0
  838. The server side nstat output::
  839. nstatuser@nstat-b:~$ nstat
  840. #kernel
  841. IpInReceives 1 0.0
  842. IpInDelivers 1 0.0
  843. IpOutRequests 1 0.0
  844. TcpInSegs 1 0.0
  845. TcpOutSegs 1 0.0
  846. IpExtInOctets 58 0.0
  847. IpExtOutOctets 52 0.0
  848. IpExtInNoECTPkts 1 0.0
  849. Input a string in nc client side again ('world' in our exmaple)::
  850. nstatuser@nstat-a:~$ nc -v nstat-b 9000
  851. Connection to nstat-b 9000 port [tcp/*] succeeded!
  852. hello
  853. world
  854. Client side nstat output::
  855. nstatuser@nstat-a:~$ nstat
  856. #kernel
  857. IpInReceives 1 0.0
  858. IpInDelivers 1 0.0
  859. IpOutRequests 1 0.0
  860. TcpInSegs 1 0.0
  861. TcpOutSegs 1 0.0
  862. TcpExtTCPHPAcks 1 0.0
  863. TcpExtTCPOrigDataSent 1 0.0
  864. IpExtInOctets 52 0.0
  865. IpExtOutOctets 58 0.0
  866. IpExtInNoECTPkts 1 0.0
  867. Server side nstat output::
  868. nstatuser@nstat-b:~$ nstat
  869. #kernel
  870. IpInReceives 1 0.0
  871. IpInDelivers 1 0.0
  872. IpOutRequests 1 0.0
  873. TcpInSegs 1 0.0
  874. TcpOutSegs 1 0.0
  875. TcpExtTCPHPHits 1 0.0
  876. IpExtInOctets 58 0.0
  877. IpExtOutOctets 52 0.0
  878. IpExtInNoECTPkts 1 0.0
  879. Compare the first client-side nstat and the second client-side nstat,
  880. we could find one difference: the first one had a 'TcpExtTCPPureAcks',
  881. but the second one had a 'TcpExtTCPHPAcks'. The first server-side
  882. nstat and the second server-side nstat had a difference too: the
  883. second server-side nstat had a TcpExtTCPHPHits, but the first
  884. server-side nstat didn't have it. The network traffic patterns were
  885. exactly the same: the client sent a packet to the server, the server
  886. replied an ACK. But kernel handled them in different ways. When the
  887. TCP window scale option is not used, kernel will try to enable fast
  888. path immediately when the connection comes into the established state,
  889. but if the TCP window scale option is used, kernel will disable the
  890. fast path at first, and try to enable it after kerenl receives
  891. packets. We could use the 'ss' command to verify whether the window
  892. scale option is used. e.g. run below command on either server or
  893. client::
  894. nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 )
  895. Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
  896. tcp 0 0 192.168.122.250:40654 192.168.122.251:9000
  897. ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98
  898. The 'wscale:7,7' means both server and client set the window scale
  899. option to 7. Now we could explain the nstat output in our test:
  900. In the first nstat output of client side, the client sent a packet, server
  901. reply an ACK, when kernel handled this ACK, the fast path was not
  902. enabled, so the ACK was counted into 'TcpExtTCPPureAcks'.
  903. In the second nstat output of client side, the client sent a packet again,
  904. and received another ACK from the server, in this time, the fast path is
  905. enabled, and the ACK was qualified for fast path, so it was handled by
  906. the fast path, so this ACK was counted into TcpExtTCPHPAcks.
  907. In the first nstat output of server side, fast path was not enabled,
  908. so there was no 'TcpExtTCPHPHits'.
  909. In the second nstat output of server side, the fast path was enabled,
  910. and the packet received from client qualified for fast path, so it
  911. was counted into 'TcpExtTCPHPHits'.
  912. TcpExtTCPAbortOnClose
  913. ---------------------
  914. On the server side, we run below python script::
  915. import socket
  916. import time
  917. port = 9000
  918. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  919. s.bind(('0.0.0.0', port))
  920. s.listen(1)
  921. sock, addr = s.accept()
  922. while True:
  923. time.sleep(9999999)
  924. This python script listen on 9000 port, but doesn't read anything from
  925. the connection.
  926. On the client side, we send the string "hello" by nc::
  927. nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000
  928. Then, we come back to the server side, the server has received the "hello"
  929. packet, and the TCP layer has acked this packet, but the application didn't
  930. read it yet. We type Ctrl-C to terminate the server script. Then we
  931. could find TcpExtTCPAbortOnClose increased 1 on the server side::
  932. nstatuser@nstat-b:~$ nstat | grep -i abort
  933. TcpExtTCPAbortOnClose 1 0.0
  934. If we run tcpdump on the server side, we could find the server sent a
  935. RST after we type Ctrl-C.
  936. TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout
  937. ---------------------------------------------------
  938. Below is an example which let the orphan socket count be higher than
  939. net.ipv4.tcp_max_orphans.
  940. Change tcp_max_orphans to a smaller value on client::
  941. sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans"
  942. Client code (create 64 connection to server)::
  943. nstatuser@nstat-a:~$ cat client_orphan.py
  944. import socket
  945. import time
  946. server = 'nstat-b' # server address
  947. port = 9000
  948. count = 64
  949. connection_list = []
  950. for i in range(64):
  951. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  952. s.connect((server, port))
  953. connection_list.append(s)
  954. print("connection_count: %d" % len(connection_list))
  955. while True:
  956. time.sleep(99999)
  957. Server code (accept 64 connection from client)::
  958. nstatuser@nstat-b:~$ cat server_orphan.py
  959. import socket
  960. import time
  961. port = 9000
  962. count = 64
  963. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  964. s.bind(('0.0.0.0', port))
  965. s.listen(count)
  966. connection_list = []
  967. while True:
  968. sock, addr = s.accept()
  969. connection_list.append((sock, addr))
  970. print("connection_count: %d" % len(connection_list))
  971. Run the python scripts on server and client.
  972. On server::
  973. python3 server_orphan.py
  974. On client::
  975. python3 client_orphan.py
  976. Run iptables on server::
  977. sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP
  978. Type Ctrl-C on client, stop client_orphan.py.
  979. Check TcpExtTCPAbortOnMemory on client::
  980. nstatuser@nstat-a:~$ nstat | grep -i abort
  981. TcpExtTCPAbortOnMemory 54 0.0
  982. Check orphane socket count on client::
  983. nstatuser@nstat-a:~$ ss -s
  984. Total: 131 (kernel 0)
  985. TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0
  986. Transport Total IP IPv6
  987. * 0 - -
  988. RAW 1 0 1
  989. UDP 1 1 0
  990. TCP 14 13 1
  991. INET 16 14 2
  992. FRAG 0 0 0
  993. The explanation of the test: after run server_orphan.py and
  994. client_orphan.py, we set up 64 connections between server and
  995. client. Run the iptables command, the server will drop all packets from
  996. the client, type Ctrl-C on client_orphan.py, the system of the client
  997. would try to close these connections, and before they are closed
  998. gracefully, these connections became orphan sockets. As the iptables
  999. of the server blocked packets from the client, the server won't receive fin
  1000. from the client, so all connection on clients would be stuck on FIN_WAIT_1
  1001. stage, so they will keep as orphan sockets until timeout. We have echo
  1002. 10 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would
  1003. only keep 10 orphan sockets, for all other orphan sockets, the client
  1004. system sent RST for them and delete them. We have 64 connections, so
  1005. the 'ss -s' command shows the system has 10 orphan sockets, and the
  1006. value of TcpExtTCPAbortOnMemory was 54.
  1007. An additional explanation about orphan socket count: You could find the
  1008. exactly orphan socket count by the 'ss -s' command, but when kernel
  1009. decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel
  1010. doesn't always check the exactly orphan socket count. For increasing
  1011. performance, kernel checks an approximate count firstly, if the
  1012. approximate count is more than tcp_max_orphans, kernel checks the
  1013. exact count again. So if the approximate count is less than
  1014. tcp_max_orphans, but exactly count is more than tcp_max_orphans, you
  1015. would find TcpExtTCPAbortOnMemory is not increased at all. If
  1016. tcp_max_orphans is large enough, it won't occur, but if you decrease
  1017. tcp_max_orphans to a small value like our test, you might find this
  1018. issue. So in our test, the client set up 64 connections although the
  1019. tcp_max_orphans is 10. If the client only set up 11 connections, we
  1020. can't find the change of TcpExtTCPAbortOnMemory.
  1021. Continue the previous test, we wait for several minutes. Because of the
  1022. iptables on the server blocked the traffic, the server wouldn't receive
  1023. fin, and all the client's orphan sockets would timeout on the
  1024. FIN_WAIT_1 state finally. So we wait for a few minutes, we could find
  1025. 10 timeout on the client::
  1026. nstatuser@nstat-a:~$ nstat | grep -i abort
  1027. TcpExtTCPAbortOnTimeout 10 0.0
  1028. TcpExtTCPAbortOnLinger
  1029. ----------------------
  1030. The server side code::
  1031. nstatuser@nstat-b:~$ cat server_linger.py
  1032. import socket
  1033. import time
  1034. port = 9000
  1035. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  1036. s.bind(('0.0.0.0', port))
  1037. s.listen(1)
  1038. sock, addr = s.accept()
  1039. while True:
  1040. time.sleep(9999999)
  1041. The client side code::
  1042. nstatuser@nstat-a:~$ cat client_linger.py
  1043. import socket
  1044. import struct
  1045. server = 'nstat-b' # server address
  1046. port = 9000
  1047. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  1048. s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
  1049. s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1))
  1050. s.connect((server, port))
  1051. s.close()
  1052. Run server_linger.py on server::
  1053. nstatuser@nstat-b:~$ python3 server_linger.py
  1054. Run client_linger.py on client::
  1055. nstatuser@nstat-a:~$ python3 client_linger.py
  1056. After run client_linger.py, check the output of nstat::
  1057. nstatuser@nstat-a:~$ nstat | grep -i abort
  1058. TcpExtTCPAbortOnLinger 1 0.0
  1059. TcpExtTCPRcvCoalesce
  1060. --------------------
  1061. On the server, we run a program which listen on TCP port 9000, but
  1062. doesn't read any data::
  1063. import socket
  1064. import time
  1065. port = 9000
  1066. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  1067. s.bind(('0.0.0.0', port))
  1068. s.listen(1)
  1069. sock, addr = s.accept()
  1070. while True:
  1071. time.sleep(9999999)
  1072. Save the above code as server_coalesce.py, and run::
  1073. python3 server_coalesce.py
  1074. On the client, save below code as client_coalesce.py::
  1075. import socket
  1076. server = 'nstat-b'
  1077. port = 9000
  1078. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  1079. s.connect((server, port))
  1080. Run::
  1081. nstatuser@nstat-a:~$ python3 -i client_coalesce.py
  1082. We use '-i' to come into the interactive mode, then a packet::
  1083. >>> s.send(b'foo')
  1084. 3
  1085. Send a packet again::
  1086. >>> s.send(b'bar')
  1087. 3
  1088. On the server, run nstat::
  1089. ubuntu@nstat-b:~$ nstat
  1090. #kernel
  1091. IpInReceives 2 0.0
  1092. IpInDelivers 2 0.0
  1093. IpOutRequests 2 0.0
  1094. TcpInSegs 2 0.0
  1095. TcpOutSegs 2 0.0
  1096. TcpExtTCPRcvCoalesce 1 0.0
  1097. IpExtInOctets 110 0.0
  1098. IpExtOutOctets 104 0.0
  1099. IpExtInNoECTPkts 2 0.0
  1100. The client sent two packets, server didn't read any data. When
  1101. the second packet arrived at server, the first packet was still in
  1102. the receiving queue. So the TCP layer merged the two packets, and we
  1103. could find the TcpExtTCPRcvCoalesce increased 1.
  1104. TcpExtListenOverflows and TcpExtListenDrops
  1105. -------------------------------------------
  1106. On server, run the nc command, listen on port 9000::
  1107. nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000
  1108. Listening on [0.0.0.0] (family 0, port 9000)
  1109. On client, run 3 nc commands in different terminals::
  1110. nstatuser@nstat-a:~$ nc -v nstat-b 9000
  1111. Connection to nstat-b 9000 port [tcp/*] succeeded!
  1112. The nc command only accepts 1 connection, and the accept queue length
  1113. is 1. On current linux implementation, set queue length to n means the
  1114. actual queue length is n+1. Now we create 3 connections, 1 is accepted
  1115. by nc, 2 in accepted queue, so the accept queue is full.
  1116. Before running the 4th nc, we clean the nstat history on the server::
  1117. nstatuser@nstat-b:~$ nstat -n
  1118. Run the 4th nc on the client::
  1119. nstatuser@nstat-a:~$ nc -v nstat-b 9000
  1120. If the nc server is running on kernel 4.10 or higher version, you
  1121. won't see the "Connection to ... succeeded!" string, because kernel
  1122. will drop the SYN if the accept queue is full. If the nc client is running
  1123. on an old kernel, you would see that the connection is succeeded,
  1124. because kernel would complete the 3 way handshake and keep the socket
  1125. on half open queue. I did the test on kernel 4.15. Below is the nstat
  1126. on the server::
  1127. nstatuser@nstat-b:~$ nstat
  1128. #kernel
  1129. IpInReceives 4 0.0
  1130. IpInDelivers 4 0.0
  1131. TcpInSegs 4 0.0
  1132. TcpExtListenOverflows 4 0.0
  1133. TcpExtListenDrops 4 0.0
  1134. IpExtInOctets 240 0.0
  1135. IpExtInNoECTPkts 4 0.0
  1136. Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time
  1137. between the 4th nc and the nstat was longer, the value of
  1138. TcpExtListenOverflows and TcpExtListenDrops would be larger, because
  1139. the SYN of the 4th nc was dropped, the client was retrying.
  1140. IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes
  1141. -------------------------------------------------
  1142. server A IP address: 192.168.122.250
  1143. server B IP address: 192.168.122.251
  1144. Prepare on server A, add a route to server B::
  1145. $ sudo ip route add 8.8.8.8/32 via 192.168.122.251
  1146. Prepare on server B, disable send_redirects for all interfaces::
  1147. $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0
  1148. $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0
  1149. $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0
  1150. $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0
  1151. We want to let sever A send a packet to 8.8.8.8, and route the packet
  1152. to server B. When server B receives such packet, it might send a ICMP
  1153. Redirect message to server A, set send_redirects to 0 will disable
  1154. this behavior.
  1155. First, generate InAddrErrors. On server B, we disable IP forwarding::
  1156. $ sudo sysctl -w net.ipv4.conf.all.forwarding=0
  1157. On server A, we send packets to 8.8.8.8::
  1158. $ nc -v 8.8.8.8 53
  1159. On server B, we check the output of nstat::
  1160. $ nstat
  1161. #kernel
  1162. IpInReceives 3 0.0
  1163. IpInAddrErrors 3 0.0
  1164. IpExtInOctets 180 0.0
  1165. IpExtInNoECTPkts 3 0.0
  1166. As we have let server A route 8.8.8.8 to server B, and we disabled IP
  1167. forwarding on server B, Server A sent packets to server B, then server B
  1168. dropped packets and increased IpInAddrErrors. As the nc command would
  1169. re-send the SYN packet if it didn't receive a SYN+ACK, we could find
  1170. multiple IpInAddrErrors.
  1171. Second, generate IpExtInNoRoutes. On server B, we enable IP
  1172. forwarding::
  1173. $ sudo sysctl -w net.ipv4.conf.all.forwarding=1
  1174. Check the route table of server B and remove the default route::
  1175. $ ip route show
  1176. default via 192.168.122.1 dev ens3 proto static
  1177. 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251
  1178. $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static
  1179. On server A, we contact 8.8.8.8 again::
  1180. $ nc -v 8.8.8.8 53
  1181. nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable
  1182. On server B, run nstat::
  1183. $ nstat
  1184. #kernel
  1185. IpInReceives 1 0.0
  1186. IpOutRequests 1 0.0
  1187. IcmpOutMsgs 1 0.0
  1188. IcmpOutDestUnreachs 1 0.0
  1189. IcmpMsgOutType3 1 0.0
  1190. IpExtInNoRoutes 1 0.0
  1191. IpExtInOctets 60 0.0
  1192. IpExtOutOctets 88 0.0
  1193. IpExtInNoECTPkts 1 0.0
  1194. We enabled IP forwarding on server B, when server B received a packet
  1195. which destination IP address is 8.8.8.8, server B will try to forward
  1196. this packet. We have deleted the default route, there was no route for
  1197. 8.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP
  1198. Destination Unreachable" message to server A.
  1199. Third, generate IpOutNoRoutes. Run ping command on server B::
  1200. $ ping -c 1 8.8.8.8
  1201. connect: Network is unreachable
  1202. Run nstat on server B::
  1203. $ nstat
  1204. #kernel
  1205. IpOutNoRoutes 1 0.0
  1206. We have deleted the default route on server B. Server B couldn't find
  1207. a route for the 8.8.8.8 IP address, so server B increased
  1208. IpOutNoRoutes.
  1209. TcpExtTCPACKSkippedSynRecv
  1210. --------------------------
  1211. In this test, we send 3 same SYN packets from client to server. The
  1212. first SYN will let server create a socket, set it to Syn-Recv status,
  1213. and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK
  1214. again, and record the reply time (the duplicate ACK reply time). The
  1215. third SYN will let server check the previous duplicate ACK reply time,
  1216. and decide to skip the duplicate ACK, then increase the
  1217. TcpExtTCPACKSkippedSynRecv counter.
  1218. Run tcpdump to capture a SYN packet::
  1219. nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000
  1220. tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
  1221. Open another terminal, run nc command::
  1222. nstatuser@nstat-a:~$ nc nstat-b 9000
  1223. As the nstat-b didn't listen on port 9000, it should reply a RST, and
  1224. the nc command exited immediately. It was enough for the tcpdump
  1225. command to capture a SYN packet. A linux server might use hardware
  1226. offload for the TCP checksum, so the checksum in the /tmp/syn.pcap
  1227. might be not correct. We call tcprewrite to fix it::
  1228. nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum
  1229. On nstat-b, we run nc to listen on port 9000::
  1230. nstatuser@nstat-b:~$ nc -lkv 9000
  1231. Listening on [0.0.0.0] (family 0, port 9000)
  1232. On nstat-a, we blocked the packet from port 9000, or nstat-a would send
  1233. RST to nstat-b::
  1234. nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP
  1235. Send 3 SYN repeatly to nstat-b::
  1236. nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done
  1237. Check snmp cunter on nstat-b::
  1238. nstatuser@nstat-b:~$ nstat | grep -i skip
  1239. TcpExtTCPACKSkippedSynRecv 1 0.0
  1240. As we expected, TcpExtTCPACKSkippedSynRecv is 1.
  1241. TcpExtTCPACKSkippedPAWS
  1242. -----------------------
  1243. To trigger PAWS, we could send an old SYN.
  1244. On nstat-b, let nc listen on port 9000::
  1245. nstatuser@nstat-b:~$ nc -lkv 9000
  1246. Listening on [0.0.0.0] (family 0, port 9000)
  1247. On nstat-a, run tcpdump to capture a SYN::
  1248. nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000
  1249. tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
  1250. On nstat-a, run nc as a client to connect nstat-b::
  1251. nstatuser@nstat-a:~$ nc -v nstat-b 9000
  1252. Connection to nstat-b 9000 port [tcp/*] succeeded!
  1253. Now the tcpdump has captured the SYN and exit. We should fix the
  1254. checksum::
  1255. nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum
  1256. Send the SYN packet twice::
  1257. nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done
  1258. On nstat-b, check the snmp counter::
  1259. nstatuser@nstat-b:~$ nstat | grep -i skip
  1260. TcpExtTCPACKSkippedPAWS 1 0.0
  1261. We sent two SYN via tcpreplay, both of them would let PAWS check
  1262. failed, the nstat-b replied an ACK for the first SYN, skipped the ACK
  1263. for the second SYN, and updated TcpExtTCPACKSkippedPAWS.
  1264. TcpExtTCPACKSkippedSeq
  1265. ----------------------
  1266. To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid
  1267. timestamp (to pass PAWS check) but the sequence number is out of
  1268. window. The linux TCP stack would avoid to skip if the packet has
  1269. data, so we need a pure ACK packet. To generate such a packet, we
  1270. could create two sockets: one on port 9000, another on port 9001. Then
  1271. we capture an ACK on port 9001, change the source/destination port
  1272. numbers to match the port 9000 socket. Then we could trigger
  1273. TcpExtTCPACKSkippedSeq via this packet.
  1274. On nstat-b, open two terminals, run two nc commands to listen on both
  1275. port 9000 and port 9001::
  1276. nstatuser@nstat-b:~$ nc -lkv 9000
  1277. Listening on [0.0.0.0] (family 0, port 9000)
  1278. nstatuser@nstat-b:~$ nc -lkv 9001
  1279. Listening on [0.0.0.0] (family 0, port 9001)
  1280. On nstat-a, run two nc clients::
  1281. nstatuser@nstat-a:~$ nc -v nstat-b 9000
  1282. Connection to nstat-b 9000 port [tcp/*] succeeded!
  1283. nstatuser@nstat-a:~$ nc -v nstat-b 9001
  1284. Connection to nstat-b 9001 port [tcp/*] succeeded!
  1285. On nstat-a, run tcpdump to capture an ACK::
  1286. nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001
  1287. tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
  1288. On nstat-b, send a packet via the port 9001 socket. E.g. we sent a
  1289. string 'foo' in our example::
  1290. nstatuser@nstat-b:~$ nc -lkv 9001
  1291. Listening on [0.0.0.0] (family 0, port 9001)
  1292. Connection from nstat-a 42132 received!
  1293. foo
  1294. On nstat-a, the tcpdump should have caputred the ACK. We should check
  1295. the source port numbers of the two nc clients::
  1296. nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee
  1297. State Recv-Q Send-Q Local Address:Port Peer Address:Port
  1298. ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000
  1299. ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001
  1300. Run tcprewrite, change port 9001 to port 9000, chagne port 42132 to
  1301. port 50208::
  1302. nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum
  1303. Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b::
  1304. nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done
  1305. Check TcpExtTCPACKSkippedSeq on nstat-b::
  1306. nstatuser@nstat-b:~$ nstat | grep -i skip
  1307. TcpExtTCPACKSkippedSeq 1 0.0