[SOLVED] iptables mangle + specific route works for icmp but not for service

Neck · 03-26-2010, 06:22 PM

Hello LQ people, been getting white hair with this, hope someone can help me.

So my setup is: I have 2 servers, they're linked with a vpn using 10.1.0.x ips. Both are directly on the net with a public IP. So both have 2 interfaces eth0 and tun0.

I set up one of the server to act as router with nat, request sent to it are redirected through vpn to the other server. Until there all work great.

Now the problem is, the other server will send answers to the net (through eth0) since it's the default route. I got it working fully by replacing the default gateway with the VPN one (i.e. 10.1.0.1), however this is annoying since it prevents directly connecting to the server from the net.
So I decided to set up policy routing with packet marking (http://www.debian-administration.org...Policy_routing).

Now this is where it starts to be fun. Iptables has a rule to catch all packets with source "10.1.0.6" and mark them. Then there is a "fwmark" entry in ip rules to catch it and redirect it to a specific table.

table is as follow:

Code:

10.1.0.1 dev tun0  scope link 
default via 10.1.0.1 dev tun0

That is for the situation. Now the problem:

Code:

# done from the target server
ping -I 10.1.0.6 linuxquestion.org

Works fine. But if I do:

Code:

# done from a random computer external to network
telnet NATSERVERIP PORT

it fails. And if I use tcpdump I can see the packets properly arriving through tun0, but sent back through eth0.
tun0: (dont mind timestamp not from same batch)

Code:

20:47:57.108498 IP 10.1.0.6 > X.X.X.X: ICMP echo request, id 58387, seq 4, length 64
20:47:57.136775 IP X.X.X.X > 10.1.0.6: ICMP echo reply, id 58387, seq 4, length 64
20:49:03.164488 IP X.X.X.X.34657 > 10.1.0.6.9000: S 674528219:674528219(0) win 5840 <mss 1368,sackOK,timestamp 6146680 0,nop,wscale 6>

eth0:

Code:

21:55:58.921525 IP 10.1.0.6.9000 > X.X.X.X.34414: S 3516043788:3516043788(0) ack 2055095368 win 5792 <mss 1460,sackOK,timestamp 30904896 7127002,nop,wscale 7>

in summary why:

Code:

Mar 26 20:47:57 server kernel: iptables-testIN= OUT=eth0 SRC=10.1.0.6 DST=X.X.X.X LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=58387 SEQ=4

is routed properly. While

Code:

Mar 26 20:49:13 server kernel: iptables-testIN= OUT=eth0 SRC=10.1.0.6 DST=X.X.X.X LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=9000 DPT=34657 WINDOW=5792 RES=0x00 ACK SYN URGP=0

is routed through eth0, knowing both are marked.

Thanks anyone who can give me a clue !

PS: both servers are running under debian 5 lenny.

mpier · 03-28-2010, 07:25 AM

Hi,
could you shortly explain what you want to achieve? Maybe simple NAT will be adequate? What about routing tables, rules, ...? I think, you should be so clearly as possible to get any help.

Neck · 03-29-2010, 02:36 AM

Hello,

Thanks for the reply, I managed to get this to work just a few hours ago. What i was trying to achieve was:

Code:

    -----------[ net ]---------
    |                         |
    |eth0                     |eth0
[ServerB]                 [ServerA]
       \_______vpn_________/
     tun0                 tun0

Both server A and B are connected to the net through eth0 and have a public IP; both are connected with a VPN and have private 10.1.0.x IP.

Now I wanted ServerA to forward some request to ServerB. So i set up a NAT table on it. This work, but ServerB was sending replies through eth0 rather than tun0.
i.e. path was
net -> ServerA -> vpn -> ServerB -> net
rather than
net -> ServerA -> vpn -> ServerB -> vpn -> ServerA -> net

Now the solution... I feel like I've been trying to kill a fly with an atomic bomb. So, removed iptables rules, etc; all I needed was a special routing table and a rule for it to be used by all packet originated from 10.1.0.6. In the end set up look like:

Code:

# ip route show table main
192.168.0.0/22 dev eth0  proto kernel  scope link  src 192.168.0.125
default via 192.168.0.1 dev eth0

# ip route show table private
10.1.0.5 dev tun0  proto kernel  scope link  src 10.1.0.6
default via 10.1.0.5 dev tun0

# ip rule
0:	from all lookup local
100:	from 10.1.0.6 lookup private
32766:	from all lookup main
32767:	from all lookup default

Also it appear like I tried to use the wrong gateway, because i tried something similar earlier, but the VPN uses 10.1.0.5 as gateway rather than 10.1.0.1 (not too sure why but it works so eh

).

Thanks a lot for anyone who spent some time reading and thinking.

zhjim · 03-29-2010, 02:46 AM

Did you just try a simple route? I mean without marking and creating an extra table?
Something like

Code:

ip route add 10.1.0.6/32 dev tun0

Maybe this would be the right one

Code:

ip route add 10.1.0.6/32 via 10.1.0.5 dev tun0

Just a normal routing rule..
At least that's what its for...

Maybe there would arise a problem with resolving name and the wrong route depending on this. But dunno how or if you use hostnames or IP addresses. But a selfmade /etc/hosts will solve this too.

Neck · 03-29-2010, 04:44 AM

I did, but it wouldn't work, because the ServerA only does DNAT meaning the source address can be any IP like 1.2.3.4.
What you propose would work perfectly to make the VPN available on the ServerB (i.e. ping 10.1.0.5 would work) and actually i think its done by default when starting the VPN client. However, a packet with the target "1.2.3.4" wouldn't follow this route and therefor would be sent out on eth0. I think the only way to give a route by matching the source IP rather than the destination is a rule, I might be wrong though, still new to this

zhjim · 03-29-2010, 07:07 AM

As long as it's working its alright

I'm still a bit confused which server has which IP on the VPN site. I starting to grasp it though...

And as far as my understanding goes and interpreting your rules you do nothing else then

Code:

ip route add 10.1.0.5 via 10.1.0.6 dev tun0

and also defining a default route for everything else coming from the 10.1.0.6 ip.

So for my understanding the kernel wants to send out the stuff over tun0 cause it came in from tun0. If you would not have the default route within your private rule it would send this stuff out over the normal default route aka inet.

Gotta see if there is a kernel var that can be set for this behaviour. Just like /proc/sys/net/ipv4/ip_forward

Finally got my head around what custom rules are good for

Neck · 03-29-2010, 07:59 AM

Quote:

Originally Posted by zhjim

As long as it's working its alright

I'm still a bit confused which server has which IP on the VPN site. I starting to grasp it though...

ServerA has 10.1.0.1 with a point to point on 10.1.0.2
ServerB has 10.1.0.6 with a point to point on 10.1.0.5
I don't really know whats these point to point ips, and didn't have time to look it up yet.

Quote:

And as far as my understanding goes and interpreting your rules you do nothing else then

Code:

ip route add 10.1.0.5 via 10.1.0.6 dev tun0

and also defining a default route for everything else coming from the 10.1.0.6 ip.

So for my understanding the kernel wants to send out the stuff over tun0 cause it came in from tun0. If you would not have the default route within your private rule it would send this stuff out over the normal default route aka inet.

10.1.0.5 and 10.1.0.6 are the same server. And by default the process running on it does not send thing back to tun0 even though it came from it. Hence why my specific rule to force it.
In english it would be something along: "If you send any packet with the source ip 10.1.0.6 then use this specific route table, else just use default".

Quote:

Finally got my head around what custom rules are good for

Yay