My friend Sean Phipps and I solved an interesting network problem recently, and I believe the solution is noteworthy simply because of the number of things we tried that didn’t work.
Let me set up the scenario, with names and addresses changed to protect the innocent, and some irrelevancies removed. We have two networks: Network A has external address 192.0.2.1 and protects internal range 10.10.10.0/24. Network B has external address 198.51.100.100 in front of internal range 10.50.100.0/24. The gateways of both networks are Cisco ASAs, and it is possible to route through a tunnel between the ASAs to communicate between the two internal networks.
Remote access (RAS) to the internal networks is possible only through the Network A gateway; the design relies on the tunnel between the ASAs to provide RAS to Network B. When a client connects to the ASA, it receives an IP address in the 10.100.0.0/24 range; the ASA provides stateful access to these IPs, so that they can initiate connects into Network A and Network B, but assets in those networks cannot connect to the VPN clients. Finally, split tunneling is enable on the VPN clients; the Anyconnect client only forces traffic to the 10.0.0.0/8 range through the tunnel.
No criticism of the design: it’s really solid and well-administered. Until the tunnel between the ASAs crashes while the administrator is out of town. So here’s the problem we ware faced with: is it possible to provide RAS clients access to a web server 10.50.100.200 on Network B without the ASA tunnel being up? Our rules of the road: we cannot change any infrastructure configuration, but we have administrative access to machines within both Network A and Network B, and hosts within Network B do have RAS to Network A through the VPN gateway.
The obvious solution is to set up a machine (let’s call him Faux Gateway, and assume he has IP address 10.50.100.100) in Network B that connects to Network A through the VPN, configure that machine to route packets between the two networks. Since I’m bothering to write, the obvious solution obviously does not work; when Faux Gateway connects to the VPN, its tunnel IP address is 10.100.0.1 (it’s really randomly assigned in this range, but let’s just fix it here) and the ASA prevents RAS clients from talking to each other. This also prevents us from setting up an SSH or other redirector on the Faux Gateway.
This problem is relatively easily soluble using an SSH server in Network A…let’s call him SSH Server, and assume he has IP address 10.10.10.10. We still need our Faux Gateway to connect to the Network A RAS, and from that server we issue the following command
ssh firstname.lastname@example.org -R10.10.10.10:80:10.50.100.200:80
For those of you who don’t speak SSH, this syntax sets up a socket listener on 10.10.10.10 on port 80. Any machine that connects to this listener will have their traffic forwarded through the SSH tunnel to the Faux Gateway, which will create a connection to 10.50.100.200 on port 80 and forward the traffic received from the tunnel through. Neat, huh?!?
Except it doesn’t work. Look at Faux Gateway’s routing table. Since it has a physical interface on Network B with IP address 10.50.100.100, it has a directly connected route to the 10.50.100.0/24 network. But when the machine connects to the VPN Gateway, the Anyconnect client installs a new route to this network through the VPN! This is exactly what Anyconnect should do…after all when the tunnel is up, this is the correct configuration. The problem is that Anyconnect will not let you affect this route in any way, shape or form. You cannot change the route. You cannot replace the route. You cannot put in an alternative route with a lower metric. You cannot put in a host route to the 10.50.100.200 web server. As Jesse Ventura might say, Anyconnect is dug in like an Alabama tick, and it is not going to let go of its routes. It ain’t got time to bleed.
We struggled with this problem for a long time. Could we convert to IPv6 using socat? Anyconnect had IPv6 locked down too. Could we write our own network stack? Could we dork the Anyconnect binaries? Theoretically, but that really is changing the rules. For the right stakes that game might be worth winning, but that wasn’t our problem. We wanted an easy way to do it.
One possibility that we think could have worked if we had spent the time and effort, would be to equip our Faux Gateway with a second IP address, say 172.16.1.1, on its physical NIC attached to Network B. We could then set up another SSH server in Network B with a similar configuration (physical NIC on network B, with two IP addresses, say 10.50.100.101 and 172.16.1.2), then create a tunnel from the Faux Gateway to this SSH server via:
ssh email@example.com -L8888:10.50.100.200:80
This should work because 172.16.1.2 should use the default route, and since we are doing split tunneling, the default route still goes through the physical interface instead of the VPN tunnel.
We would also need to tear down the original tunnel to the SSH server on Network A and replace it with:
ssh firstname.lastname@example.org -R10.10.10.10:80:172.16.1.2:8888
The net effect of this would be that when connecting to the Network A SSH server and having the traffic tunneled to the Faux Gateway, instead of trying to go directly to the web server and getting traffic dumped into the VPN tunnel, we instead push the connection through the tunnel to the Network B SSH server, which can safely push the connection to the web server.
If you understand that description, you know why we didn’t want to do it this way. If you didn’t understand that description, you also know why we didn’t want to do it this way. It cannot really be described as easy.
But the real solution actually is easy. The problem is the routing table: if we try to send a packet to the 10.50.100.0/24 network that hits the routing table, it is going through the tunnel, period. So we need to not send a packet with anything in that network as the destination IP address…the light bulb goes on: NAT! We can make this work with two simple iptables rules. First, before we hit the routing table, we need to change our destination IP address to an address that will use the default route instead of the VPN tunnel:
iptables -t nat -I PREROUTING -d 10.50.100.200 -p tcp --dport 80 -j DNAT --to-destination 172.16.1.2
This instructs any packet being processed by the Faux Gateway that has destination IP address 10.50.100.200 and destination port 80 to have its destination IP address rewritten to 172.16.1.2 before a routing decision is made. With this destination IP address, the routing decision is made for the physical NIC…now we have to translate it back before putting it on the wire:
iptables -t nat -I OUTPUT -d 172.16.1.2 -p tcp --dport 80 -j DNAT --to-destination 10.50.100.200
This converts the address back after the routing decision is made, and the packet gets to the intended destination! What about the return packets? Well, they have no problem, because we want them to return through the tunnel. The Faux Gateway gets the return packet with a destination IP address of (say) 10.100.0.18, which it gleefully routes through the tunnel.
So with an SSH server in Network A and our Faux Gateway in Network B, it takes three simple commands to re-establish remote access to the web server.