As part of a task this week, I needed to build an iptables-based firewall for a training network that implemented 1-to-1 NAT to a set of servers in a DMZ. This is a fairly standard configuration, but with one small wrinkle: we only had a small range of “public” IP addresses (198.51.100.112/29), and we were using that range both for our NATed servers and for the link between the firewall and the “ISP”. Specifically, our IP scheme looks like this:
- ISP router – 198.51.100.118
- Firewall – 198.51.100.113 (external) / 192.168.111.113 (internal)
- DMZ servers – 192.168.111.114-117 (internal) / 198.51.100.114-117 (provided via NAT)
Setting up the NAT itself is easy, with two rules per server:
iptables -t nat -A PREROUTING -d 198.51.100.114 -j DNAT --to-destination 192.168.111.114
iptables -t nat -A POSTROUTING -s 192.168.111.114 -j SNAT --to-source 198.51.100.114
It’s so easy that you just slap the rules down to try it out, and guess what? The DMZ servers cannot ping the ISP router. As I always tell my students, we teach you to sniff network traffic for a reason, because there’s no better way to know whether and/or what your machine is actually putting on the wire. In this case, sniffing on the firewall shows the problem immediately. The ping packet does get sent from the firewall to the ISP router with the correct address translation, but when the router goes to reply, it tries to ARP for the MAC address of the DMZ server.
With a moment’s though, of course it does. The ISP router is configured with interface IP address 198.51.100.118/29, so if it sends a packet to 198.51.100.114 it thinks that host is on its network. This is a known problem, with a known solution: proxy ARP. What needs to happen is that the firewall needs to respond to ARP requests for the DMZ host with its own MAC address.
So I knew the solution, but had never set it up on a Linux-based firewall before, so off to the Internet. There are a lot of good sources for configuring almost anything you want on a Linux machine, and proxy ARP is no exception. Unfortunately there are also a lot of bad sources, but if you’re in the field long enough, you get a pretty good feel up front about this. But there is a third class of sites, which I like to think of as the junk DNA of the Internet: sites that were once good sources, but due to the passage of time, are no longer good.
Because of this junk DNA, finding good info on proxy ARP turned out to be a big problem, due to a combination of several factors:
- The methods for implementing proxy ARP have changed between the 2.2, 2.4, and 2.6 releases of the Linux kernel, and pages offering instruction are not always well labelled as to what version is applicable.
- Regarding the commands you execute to implement proxy ARP, the older versions (setting the proxy_arp flag in the kernel and using the arp command to create the proxy) are much more intuitive than the newer version (which uses the route table).
- The older commands actually appear to work on more modern kernels, but they do not actually enable proxy ARP.
For the record, the official solution to enable proxy ARP in the Linux 2.6 kernel is to create a host route for the external NAT address, so to make my configuration work, I needed to run the following command:
ip route add 198.51.100.114/32 dev lo
In my case, I know I’m going to NAT the address away, so I never actually expect to route any packets with this destination address; I bury the route to the loopback adapter to make this a de facto null route.
But here’s my question: why is one configuring proxy ARP in the routing table? I believe the answer is that a common use case for proxy ARP is when a router is used to segment hosts that are, at least from an IP addressing perspective, genuinely in the same network. In this case, you would have to configure the router with real routes to the hosts in these segments, and by doing this the required proxy ARP is “done for you”.
But here’s my question: why is one configuring proxy ARP in the routing table? I’m not specifically pointing fingers at Linux kernel developers, but more at our modern “Apple syndrome”. The genius of Apple is that they have made several technologies so easy to use that even my 9 year-old can figure them out; through using auto-discovery and configuration defaults, you take the device out of the box and it just works. For one use case of proxy ARP, the Linux kernel has made it “just work”, at the expense of making it highly unintuitive for other use cases.
For some reason that I am unable to fathom, our technological world has made ease of use come at the expense of maintainability…the tools available to diagnose and solve problems either do not exist, or are so expensive and/or unintuitive to use. If something goes wrong, get a new one. Want to tinker and get more capability? Sorry, can’t do that.
There are always exceptions to the rule: the “makers” who would rather build their own than use someone else’s. But we are creating a bifurcation in society between those who can do and those who can use. This split is probably good for my business, but I worry it is unhealthy.