vrf.rst 16 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451
  1. .. SPDX-License-Identifier: GPL-2.0
  2. ====================================
  3. Virtual Routing and Forwarding (VRF)
  4. ====================================
  5. The VRF Device
  6. ==============
  7. The VRF device combined with ip rules provides the ability to create virtual
  8. routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
  9. Linux network stack. One use case is the multi-tenancy problem where each
  10. tenant has their own unique routing tables and in the very least need
  11. different default gateways.
  12. Processes can be "VRF aware" by binding a socket to the VRF device. Packets
  13. through the socket then use the routing table associated with the VRF
  14. device. An important feature of the VRF device implementation is that it
  15. impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
  16. (ie., they do not need to be run in each VRF). The design also allows
  17. the use of higher priority ip rules (Policy Based Routing, PBR) to take
  18. precedence over the VRF device rules directing specific traffic as desired.
  19. In addition, VRF devices allow VRFs to be nested within namespaces. For
  20. example network namespaces provide separation of network interfaces at the
  21. device layer, VLANs on the interfaces within a namespace provide L2 separation
  22. and then VRF devices provide L3 separation.
  23. Design
  24. ------
  25. A VRF device is created with an associated route table. Network interfaces
  26. are then enslaved to a VRF device::
  27. +-----------------------------+
  28. | vrf-blue | ===> route table 10
  29. +-----------------------------+
  30. | | |
  31. +------+ +------+ +-------------+
  32. | eth1 | | eth2 | ... | bond1 |
  33. +------+ +------+ +-------------+
  34. | |
  35. +------+ +------+
  36. | eth8 | | eth9 |
  37. +------+ +------+
  38. Packets received on an enslaved device and are switched to the VRF device
  39. in the IPv4 and IPv6 processing stacks giving the impression that packets
  40. flow through the VRF device. Similarly on egress routing rules are used to
  41. send packets to the VRF device driver before getting sent out the actual
  42. interface. This allows tcpdump on a VRF device to capture all packets into
  43. and out of the VRF as a whole\ [1]_. Similarly, netfilter\ [2]_ and tc rules
  44. can be applied using the VRF device to specify rules that apply to the VRF
  45. domain as a whole.
  46. .. [1] Packets in the forwarded state do not flow through the device, so those
  47. packets are not seen by tcpdump. Will revisit this limitation in a
  48. future release.
  49. .. [2] Iptables on ingress supports PREROUTING with skb->dev set to the real
  50. ingress device and both INPUT and PREROUTING rules with skb->dev set to
  51. the VRF device. For egress POSTROUTING and OUTPUT rules can be written
  52. using either the VRF device or real egress device.
  53. Setup
  54. -----
  55. 1. VRF device is created with an association to a FIB table.
  56. e.g,::
  57. ip link add vrf-blue type vrf table 10
  58. ip link set dev vrf-blue up
  59. 2. An l3mdev FIB rule directs lookups to the table associated with the device.
  60. A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
  61. l3mdev rule for IPv4 and IPv6 when the first device is created with a
  62. default preference of 1000. Users may delete the rule if desired and add
  63. with a different priority or install per-VRF rules.
  64. Prior to the v4.8 kernel iif and oif rules are needed for each VRF device::
  65. ip ru add oif vrf-blue table 10
  66. ip ru add iif vrf-blue table 10
  67. 3. Set the default route for the table (and hence default route for the VRF)::
  68. ip route add table 10 unreachable default metric 4278198272
  69. This high metric value ensures that the default unreachable route can
  70. be overridden by a routing protocol suite. FRRouting interprets
  71. kernel metrics as a combined admin distance (upper byte) and priority
  72. (lower 3 bytes). Thus the above metric translates to [255/8192].
  73. 4. Enslave L3 interfaces to a VRF device::
  74. ip link set dev eth1 master vrf-blue
  75. Local and connected routes for enslaved devices are automatically moved to
  76. the table associated with VRF device. Any additional routes depending on
  77. the enslaved device are dropped and will need to be reinserted to the VRF
  78. FIB table following the enslavement.
  79. The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
  80. addresses as VRF enslavement changes::
  81. sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
  82. 5. Additional VRF routes are added to associated table::
  83. ip route add table 10 ...
  84. Applications
  85. ------------
  86. Applications that are to work within a VRF need to bind their socket to the
  87. VRF device::
  88. setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
  89. or to specify the output device using cmsg and IP_PKTINFO.
  90. By default the scope of the port bindings for unbound sockets is
  91. limited to the default VRF. That is, it will not be matched by packets
  92. arriving on interfaces enslaved to an l3mdev and processes may bind to
  93. the same port if they bind to an l3mdev.
  94. TCP & UDP services running in the default VRF context (ie., not bound
  95. to any VRF device) can work across all VRF domains by enabling the
  96. tcp_l3mdev_accept and udp_l3mdev_accept sysctl options::
  97. sysctl -w net.ipv4.tcp_l3mdev_accept=1
  98. sysctl -w net.ipv4.udp_l3mdev_accept=1
  99. These options are disabled by default so that a socket in a VRF is only
  100. selected for packets in that VRF. There is a similar option for RAW
  101. sockets, which is enabled by default for reasons of backwards compatibility.
  102. This is so as to specify the output device with cmsg and IP_PKTINFO, but
  103. using a socket not bound to the corresponding VRF. This allows e.g. older ping
  104. implementations to be run with specifying the device but without executing it
  105. in the VRF. This option can be disabled so that packets received in a VRF
  106. context are only handled by a raw socket bound to the VRF, and packets in the
  107. default VRF are only handled by a socket not bound to any VRF::
  108. sysctl -w net.ipv4.raw_l3mdev_accept=0
  109. netfilter rules on the VRF device can be used to limit access to services
  110. running in the default VRF context as well.
  111. --------------------------------------------------------------------------------
  112. Using iproute2 for VRFs
  113. =======================
  114. iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
  115. section lists both commands where appropriate -- with the vrf keyword and the
  116. older form without it.
  117. 1. Create a VRF
  118. To instantiate a VRF device and associate it with a table::
  119. $ ip link add dev NAME type vrf table ID
  120. As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
  121. covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
  122. device create.
  123. 2. List VRFs
  124. To list VRFs that have been created::
  125. $ ip [-d] link show type vrf
  126. NOTE: The -d option is needed to show the table id
  127. For example::
  128. $ ip -d link show type vrf
  129. 11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  130. link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
  131. vrf table 1 addrgenmode eui64
  132. 12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  133. link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
  134. vrf table 10 addrgenmode eui64
  135. 13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  136. link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
  137. vrf table 66 addrgenmode eui64
  138. 14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
  139. link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
  140. vrf table 81 addrgenmode eui64
  141. Or in brief output::
  142. $ ip -br link show type vrf
  143. mgmt UP 72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
  144. red UP b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
  145. blue UP 36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
  146. green UP e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
  147. 3. Assign a Network Interface to a VRF
  148. Network interfaces are assigned to a VRF by enslaving the netdevice to a
  149. VRF device::
  150. $ ip link set dev NAME master NAME
  151. On enslavement connected and local routes are automatically moved to the
  152. table associated with the VRF device.
  153. For example::
  154. $ ip link set dev eth0 master mgmt
  155. 4. Show Devices Assigned to a VRF
  156. To show devices that have been assigned to a specific VRF add the master
  157. option to the ip command::
  158. $ ip link show vrf NAME
  159. $ ip link show master NAME
  160. For example::
  161. $ ip link show vrf red
  162. 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
  163. link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
  164. 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
  165. link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
  166. 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
  167. link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
  168. Or using the brief output::
  169. $ ip -br link show vrf red
  170. eth1 UP 02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
  171. eth2 UP 02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
  172. eth5 DOWN 02:00:00:00:02:06 <BROADCAST,MULTICAST>
  173. 5. Show Neighbor Entries for a VRF
  174. To list neighbor entries associated with devices enslaved to a VRF device
  175. add the master option to the ip command::
  176. $ ip [-6] neigh show vrf NAME
  177. $ ip [-6] neigh show master NAME
  178. For example::
  179. $ ip neigh show vrf red
  180. 10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
  181. 10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
  182. $ ip -6 neigh show vrf red
  183. 2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
  184. 6. Show Addresses for a VRF
  185. To show addresses for interfaces associated with a VRF add the master
  186. option to the ip command::
  187. $ ip addr show vrf NAME
  188. $ ip addr show master NAME
  189. For example::
  190. $ ip addr show vrf red
  191. 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
  192. link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
  193. inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
  194. valid_lft forever preferred_lft forever
  195. inet6 2002:1::2/120 scope global
  196. valid_lft forever preferred_lft forever
  197. inet6 fe80::ff:fe00:202/64 scope link
  198. valid_lft forever preferred_lft forever
  199. 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
  200. link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
  201. inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
  202. valid_lft forever preferred_lft forever
  203. inet6 2002:2::2/120 scope global
  204. valid_lft forever preferred_lft forever
  205. inet6 fe80::ff:fe00:203/64 scope link
  206. valid_lft forever preferred_lft forever
  207. 7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
  208. link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
  209. Or in brief format::
  210. $ ip -br addr show vrf red
  211. eth1 UP 10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
  212. eth2 UP 10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
  213. eth5 DOWN
  214. 7. Show Routes for a VRF
  215. To show routes for a VRF use the ip command to display the table associated
  216. with the VRF device::
  217. $ ip [-6] route show vrf NAME
  218. $ ip [-6] route show table ID
  219. For example::
  220. $ ip route show vrf red
  221. unreachable default metric 4278198272
  222. broadcast 10.2.1.0 dev eth1 proto kernel scope link src 10.2.1.2
  223. 10.2.1.0/24 dev eth1 proto kernel scope link src 10.2.1.2
  224. local 10.2.1.2 dev eth1 proto kernel scope host src 10.2.1.2
  225. broadcast 10.2.1.255 dev eth1 proto kernel scope link src 10.2.1.2
  226. broadcast 10.2.2.0 dev eth2 proto kernel scope link src 10.2.2.2
  227. 10.2.2.0/24 dev eth2 proto kernel scope link src 10.2.2.2
  228. local 10.2.2.2 dev eth2 proto kernel scope host src 10.2.2.2
  229. broadcast 10.2.2.255 dev eth2 proto kernel scope link src 10.2.2.2
  230. $ ip -6 route show vrf red
  231. local 2002:1:: dev lo proto none metric 0 pref medium
  232. local 2002:1::2 dev lo proto none metric 0 pref medium
  233. 2002:1::/120 dev eth1 proto kernel metric 256 pref medium
  234. local 2002:2:: dev lo proto none metric 0 pref medium
  235. local 2002:2::2 dev lo proto none metric 0 pref medium
  236. 2002:2::/120 dev eth2 proto kernel metric 256 pref medium
  237. local fe80:: dev lo proto none metric 0 pref medium
  238. local fe80:: dev lo proto none metric 0 pref medium
  239. local fe80::ff:fe00:202 dev lo proto none metric 0 pref medium
  240. local fe80::ff:fe00:203 dev lo proto none metric 0 pref medium
  241. fe80::/64 dev eth1 proto kernel metric 256 pref medium
  242. fe80::/64 dev eth2 proto kernel metric 256 pref medium
  243. ff00::/8 dev red metric 256 pref medium
  244. ff00::/8 dev eth1 metric 256 pref medium
  245. ff00::/8 dev eth2 metric 256 pref medium
  246. unreachable default dev lo metric 4278198272 error -101 pref medium
  247. 8. Route Lookup for a VRF
  248. A test route lookup can be done for a VRF::
  249. $ ip [-6] route get vrf NAME ADDRESS
  250. $ ip [-6] route get oif NAME ADDRESS
  251. For example::
  252. $ ip route get 10.2.1.40 vrf red
  253. 10.2.1.40 dev eth1 table red src 10.2.1.2
  254. cache
  255. $ ip -6 route get 2002:1::32 vrf red
  256. 2002:1::32 from :: dev eth1 table red proto kernel src 2002:1::2 metric 256 pref medium
  257. 9. Removing Network Interface from a VRF
  258. Network interfaces are removed from a VRF by breaking the enslavement to
  259. the VRF device::
  260. $ ip link set dev NAME nomaster
  261. Connected routes are moved back to the default table and local entries are
  262. moved to the local table.
  263. For example::
  264. $ ip link set dev eth0 nomaster
  265. --------------------------------------------------------------------------------
  266. Commands used in this example::
  267. cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
  268. 1 mgmt
  269. 10 red
  270. 66 blue
  271. 81 green
  272. EOF
  273. function vrf_create
  274. {
  275. VRF=$1
  276. TBID=$2
  277. # create VRF device
  278. ip link add ${VRF} type vrf table ${TBID}
  279. if [ "${VRF}" != "mgmt" ]; then
  280. ip route add table ${TBID} unreachable default metric 4278198272
  281. fi
  282. ip link set dev ${VRF} up
  283. }
  284. vrf_create mgmt 1
  285. ip link set dev eth0 master mgmt
  286. vrf_create red 10
  287. ip link set dev eth1 master red
  288. ip link set dev eth2 master red
  289. ip link set dev eth5 master red
  290. vrf_create blue 66
  291. ip link set dev eth3 master blue
  292. vrf_create green 81
  293. ip link set dev eth4 master green
  294. Interface addresses from /etc/network/interfaces:
  295. auto eth0
  296. iface eth0 inet static
  297. address 10.0.0.2
  298. netmask 255.255.255.0
  299. gateway 10.0.0.254
  300. iface eth0 inet6 static
  301. address 2000:1::2
  302. netmask 120
  303. auto eth1
  304. iface eth1 inet static
  305. address 10.2.1.2
  306. netmask 255.255.255.0
  307. iface eth1 inet6 static
  308. address 2002:1::2
  309. netmask 120
  310. auto eth2
  311. iface eth2 inet static
  312. address 10.2.2.2
  313. netmask 255.255.255.0
  314. iface eth2 inet6 static
  315. address 2002:2::2
  316. netmask 120
  317. auto eth3
  318. iface eth3 inet static
  319. address 10.2.3.2
  320. netmask 255.255.255.0
  321. iface eth3 inet6 static
  322. address 2002:3::2
  323. netmask 120
  324. auto eth4
  325. iface eth4 inet static
  326. address 10.2.4.2
  327. netmask 255.255.255.0
  328. iface eth4 inet6 static
  329. address 2002:4::2
  330. netmask 120