eql.txt 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528
  1. EQL Driver: Serial IP Load Balancing HOWTO
  2. Simon "Guru Aleph-Null" Janes, simon@ncm.com
  3. v1.1, February 27, 1995
  4. This is the manual for the EQL device driver. EQL is a software device
  5. that lets you load-balance IP serial links (SLIP or uncompressed PPP)
  6. to increase your bandwidth. It will not reduce your latency (i.e. ping
  7. times) except in the case where you already have lots of traffic on
  8. your link, in which it will help them out. This driver has been tested
  9. with the 1.1.75 kernel, and is known to have patched cleanly with
  10. 1.1.86. Some testing with 1.1.92 has been done with the v1.1 patch
  11. which was only created to patch cleanly in the very latest kernel
  12. source trees. (Yes, it worked fine.)
  13. 1. Introduction
  14. Which is worse? A huge fee for a 56K leased line or two phone lines?
  15. It's probably the former. If you find yourself craving more bandwidth,
  16. and have a ISP that is flexible, it is now possible to bind modems
  17. together to work as one point-to-point link to increase your
  18. bandwidth. All without having to have a special black box on either
  19. side.
  20. The eql driver has only been tested with the Livingston PortMaster-2e
  21. terminal server. I do not know if other terminal servers support load-
  22. balancing, but I do know that the PortMaster does it, and does it
  23. almost as well as the eql driver seems to do it (-- Unfortunately, in
  24. my testing so far, the Livingston PortMaster 2e's load-balancing is a
  25. good 1 to 2 KB/s slower than the test machine working with a 28.8 Kbps
  26. and 14.4 Kbps connection. However, I am not sure that it really is
  27. the PortMaster, or if it's Linux's TCP drivers. I'm told that Linux's
  28. TCP implementation is pretty fast though.--)
  29. I suggest to ISPs out there that it would probably be fair to charge
  30. a load-balancing client 75% of the cost of the second line and 50% of
  31. the cost of the third line etc...
  32. Hey, we can all dream you know...
  33. 2. Kernel Configuration
  34. Here I describe the general steps of getting a kernel up and working
  35. with the eql driver. From patching, building, to installing.
  36. 2.1. Patching The Kernel
  37. If you do not have or cannot get a copy of the kernel with the eql
  38. driver folded into it, get your copy of the driver from
  39. ftp://slaughter.ncm.com/pub/Linux/LOAD_BALANCING/eql-1.1.tar.gz.
  40. Unpack this archive someplace obvious like /usr/local/src/. It will
  41. create the following files:
  42. ______________________________________________________________________
  43. -rw-r--r-- guru/ncm 198 Jan 19 18:53 1995 eql-1.1/NO-WARRANTY
  44. -rw-r--r-- guru/ncm 30620 Feb 27 21:40 1995 eql-1.1/eql-1.1.patch
  45. -rwxr-xr-x guru/ncm 16111 Jan 12 22:29 1995 eql-1.1/eql_enslave
  46. -rw-r--r-- guru/ncm 2195 Jan 10 21:48 1995 eql-1.1/eql_enslave.c
  47. ______________________________________________________________________
  48. Unpack a recent kernel (something after 1.1.92) someplace convenient
  49. like say /usr/src/linux-1.1.92.eql. Use symbolic links to point
  50. /usr/src/linux to this development directory.
  51. Apply the patch by running the commands:
  52. ______________________________________________________________________
  53. cd /usr/src
  54. patch </usr/local/src/eql-1.1/eql-1.1.patch
  55. ______________________________________________________________________
  56. 2.2. Building The Kernel
  57. After patching the kernel, run make config and configure the kernel
  58. for your hardware.
  59. After configuration, make and install according to your habit.
  60. 3. Network Configuration
  61. So far, I have only used the eql device with the DSLIP SLIP connection
  62. manager by Matt Dillon (-- "The man who sold his soul to code so much
  63. so quickly."--) . How you configure it for other "connection"
  64. managers is up to you. Most other connection managers that I've seen
  65. don't do a very good job when it comes to handling more than one
  66. connection.
  67. 3.1. /etc/rc.d/rc.inet1
  68. In rc.inet1, ifconfig the eql device to the IP address you usually use
  69. for your machine, and the MTU you prefer for your SLIP lines. One
  70. could argue that MTU should be roughly half the usual size for two
  71. modems, one-third for three, one-fourth for four, etc... But going
  72. too far below 296 is probably overkill. Here is an example ifconfig
  73. command that sets up the eql device:
  74. ______________________________________________________________________
  75. ifconfig eql 198.67.33.239 mtu 1006
  76. ______________________________________________________________________
  77. Once the eql device is up and running, add a static default route to
  78. it in the routing table using the cool new route syntax that makes
  79. life so much easier:
  80. ______________________________________________________________________
  81. route add default eql
  82. ______________________________________________________________________
  83. 3.2. Enslaving Devices By Hand
  84. Enslaving devices by hand requires two utility programs: eql_enslave
  85. and eql_emancipate (-- eql_emancipate hasn't been written because when
  86. an enslaved device "dies", it is automatically taken out of the queue.
  87. I haven't found a good reason to write it yet... other than for
  88. completeness, but that isn't a good motivator is it?--)
  89. The syntax for enslaving a device is "eql_enslave <master-name>
  90. <slave-name> <estimated-bps>". Here are some example enslavings:
  91. ______________________________________________________________________
  92. eql_enslave eql sl0 28800
  93. eql_enslave eql ppp0 14400
  94. eql_enslave eql sl1 57600
  95. ______________________________________________________________________
  96. When you want to free a device from its life of slavery, you can
  97. either down the device with ifconfig (eql will automatically bury the
  98. dead slave and remove it from its queue) or use eql_emancipate to free
  99. it. (-- Or just ifconfig it down, and the eql driver will take it out
  100. for you.--)
  101. ______________________________________________________________________
  102. eql_emancipate eql sl0
  103. eql_emancipate eql ppp0
  104. eql_emancipate eql sl1
  105. ______________________________________________________________________
  106. 3.3. DSLIP Configuration for the eql Device
  107. The general idea is to bring up and keep up as many SLIP connections
  108. as you need, automatically.
  109. 3.3.1. /etc/slip/runslip.conf
  110. Here is an example runslip.conf:
  111. ______________________________________________________________________
  112. name sl-line-1
  113. enabled
  114. baud 38400
  115. mtu 576
  116. ducmd -e /etc/slip/dialout/cua2-288.xp -t 9
  117. command eql_enslave eql $interface 28800
  118. address 198.67.33.239
  119. line /dev/cua2
  120. name sl-line-2
  121. enabled
  122. baud 38400
  123. mtu 576
  124. ducmd -e /etc/slip/dialout/cua3-288.xp -t 9
  125. command eql_enslave eql $interface 28800
  126. address 198.67.33.239
  127. line /dev/cua3
  128. ______________________________________________________________________
  129. 3.4. Using PPP and the eql Device
  130. I have not yet done any load-balancing testing for PPP devices, mainly
  131. because I don't have a PPP-connection manager like SLIP has with
  132. DSLIP. I did find a good tip from LinuxNET:Billy for PPP performance:
  133. make sure you have asyncmap set to something so that control
  134. characters are not escaped.
  135. I tried to fix up a PPP script/system for redialing lost PPP
  136. connections for use with the eql driver the weekend of Feb 25-26 '95
  137. (Hereafter known as the 8-hour PPP Hate Festival). Perhaps later this
  138. year.
  139. 4. About the Slave Scheduler Algorithm
  140. The slave scheduler probably could be replaced with a dozen other
  141. things and push traffic much faster. The formula in the current set
  142. up of the driver was tuned to handle slaves with wildly different
  143. bits-per-second "priorities".
  144. All testing I have done was with two 28.8 V.FC modems, one connecting
  145. at 28800 bps or slower, and the other connecting at 14400 bps all the
  146. time.
  147. One version of the scheduler was able to push 5.3 K/s through the
  148. 28800 and 14400 connections, but when the priorities on the links were
  149. very wide apart (57600 vs. 14400) the "faster" modem received all
  150. traffic and the "slower" modem starved.
  151. 5. Testers' Reports
  152. Some people have experimented with the eql device with newer
  153. kernels (than 1.1.75). I have since updated the driver to patch
  154. cleanly in newer kernels because of the removal of the old "slave-
  155. balancing" driver config option.
  156. o icee from LinuxNET patched 1.1.86 without any rejects and was able
  157. to boot the kernel and enslave a couple of ISDN PPP links.
  158. 5.1. Randolph Bentson's Test Report
  159. From bentson@grieg.seaslug.org Wed Feb 8 19:08:09 1995
  160. Date: Tue, 7 Feb 95 22:57 PST
  161. From: Randolph Bentson <bentson@grieg.seaslug.org>
  162. To: guru@ncm.com
  163. Subject: EQL driver tests
  164. I have been checking out your eql driver. (Nice work, that!)
  165. Although you may already done this performance testing, here
  166. are some data I've discovered.
  167. Randolph Bentson
  168. bentson@grieg.seaslug.org
  169. ---------------------------------------------------------
  170. A pseudo-device driver, EQL, written by Simon Janes, can be used
  171. to bundle multiple SLIP connections into what appears to be a
  172. single connection. This allows one to improve dial-up network
  173. connectivity gradually, without having to buy expensive DSU/CSU
  174. hardware and services.
  175. I have done some testing of this software, with two goals in
  176. mind: first, to ensure it actually works as described and
  177. second, as a method of exercising my device driver.
  178. The following performance measurements were derived from a set
  179. of SLIP connections run between two Linux systems (1.1.84) using
  180. a 486DX2/66 with a Cyclom-8Ys and a 486SLC/40 with a Cyclom-16Y.
  181. (Ports 0,1,2,3 were used. A later configuration will distribute
  182. port selection across the different Cirrus chips on the boards.)
  183. Once a link was established, I timed a binary ftp transfer of
  184. 289284 bytes of data. If there were no overhead (packet headers,
  185. inter-character and inter-packet delays, etc.) the transfers
  186. would take the following times:
  187. bits/sec seconds
  188. 345600 8.3
  189. 234600 12.3
  190. 172800 16.7
  191. 153600 18.8
  192. 76800 37.6
  193. 57600 50.2
  194. 38400 75.3
  195. 28800 100.4
  196. 19200 150.6
  197. 9600 301.3
  198. A single line running at the lower speeds and with large packets
  199. comes to within 2% of this. Performance is limited for the higher
  200. speeds (as predicted by the Cirrus databook) to an aggregate of
  201. about 160 kbits/sec. The next round of testing will distribute
  202. the load across two or more Cirrus chips.
  203. The good news is that one gets nearly the full advantage of the
  204. second, third, and fourth line's bandwidth. (The bad news is
  205. that the connection establishment seemed fragile for the higher
  206. speeds. Once established, the connection seemed robust enough.)
  207. #lines speed mtu seconds theory actual %of
  208. kbit/sec duration speed speed max
  209. 3 115200 900 _ 345600
  210. 3 115200 400 18.1 345600 159825 46
  211. 2 115200 900 _ 230400
  212. 2 115200 600 18.1 230400 159825 69
  213. 2 115200 400 19.3 230400 149888 65
  214. 4 57600 900 _ 234600
  215. 4 57600 600 _ 234600
  216. 4 57600 400 _ 234600
  217. 3 57600 600 20.9 172800 138413 80
  218. 3 57600 900 21.2 172800 136455 78
  219. 3 115200 600 21.7 345600 133311 38
  220. 3 57600 400 22.5 172800 128571 74
  221. 4 38400 900 25.2 153600 114795 74
  222. 4 38400 600 26.4 153600 109577 71
  223. 4 38400 400 27.3 153600 105965 68
  224. 2 57600 900 29.1 115200 99410.3 86
  225. 1 115200 900 30.7 115200 94229.3 81
  226. 2 57600 600 30.2 115200 95789.4 83
  227. 3 38400 900 30.3 115200 95473.3 82
  228. 3 38400 600 31.2 115200 92719.2 80
  229. 1 115200 600 31.3 115200 92423 80
  230. 2 57600 400 32.3 115200 89561.6 77
  231. 1 115200 400 32.8 115200 88196.3 76
  232. 3 38400 400 33.5 115200 86353.4 74
  233. 2 38400 900 43.7 76800 66197.7 86
  234. 2 38400 600 44 76800 65746.4 85
  235. 2 38400 400 47.2 76800 61289 79
  236. 4 19200 900 50.8 76800 56945.7 74
  237. 4 19200 400 53.2 76800 54376.7 70
  238. 4 19200 600 53.7 76800 53870.4 70
  239. 1 57600 900 54.6 57600 52982.4 91
  240. 1 57600 600 56.2 57600 51474 89
  241. 3 19200 900 60.5 57600 47815.5 83
  242. 1 57600 400 60.2 57600 48053.8 83
  243. 3 19200 600 62 57600 46658.7 81
  244. 3 19200 400 64.7 57600 44711.6 77
  245. 1 38400 900 79.4 38400 36433.8 94
  246. 1 38400 600 82.4 38400 35107.3 91
  247. 2 19200 900 84.4 38400 34275.4 89
  248. 1 38400 400 86.8 38400 33327.6 86
  249. 2 19200 600 87.6 38400 33023.3 85
  250. 2 19200 400 91.2 38400 31719.7 82
  251. 4 9600 900 94.7 38400 30547.4 79
  252. 4 9600 400 106 38400 27290.9 71
  253. 4 9600 600 110 38400 26298.5 68
  254. 3 9600 900 118 28800 24515.6 85
  255. 3 9600 600 120 28800 24107 83
  256. 3 9600 400 131 28800 22082.7 76
  257. 1 19200 900 155 19200 18663.5 97
  258. 1 19200 600 161 19200 17968 93
  259. 1 19200 400 170 19200 17016.7 88
  260. 2 9600 600 176 19200 16436.6 85
  261. 2 9600 900 180 19200 16071.3 83
  262. 2 9600 400 181 19200 15982.5 83
  263. 1 9600 900 305 9600 9484.72 98
  264. 1 9600 600 314 9600 9212.87 95
  265. 1 9600 400 332 9600 8713.37 90
  266. 5.2. Anthony Healy's Report
  267. Date: Mon, 13 Feb 1995 16:17:29 +1100 (EST)
  268. From: Antony Healey <ahealey@st.nepean.uws.edu.au>
  269. To: Simon Janes <guru@ncm.com>
  270. Subject: Re: Load Balancing
  271. Hi Simon,
  272. I've installed your patch and it works great. I have trialed
  273. it over twin SL/IP lines, just over null modems, but I was
  274. able to data at over 48Kb/s [ISDN link -Simon]. I managed a
  275. transfer of up to 7.5 Kbyte/s on one go, but averaged around
  276. 6.4 Kbyte/s, which I think is pretty cool. :)