Complicated Routing problem

[Andy Holyer]

Short form of the question:

If I can ping/traceroute C from B, and B from A, then shouldn't it
logically follow that I must be able to ping/traceroute C from A? I
can't!

Long form:
---
here's a fairly large preamble to all of this, so please bear with me.

We are a wireless ISP, providing Broadband to people in rural areas
further away from their nearest exchange than the 3.5km allowed by wired
DSL.

One large chunk of our network was bought wholesale from another ISP as
they were about to go bust. It consists of a network of cut-down Red Hat
boxes acting as routers (the OS is stored on a Flash card on the
motherboard. The cutting down of Linux was fairly drastic, so many
useful diagnostic tools are missin - for example there's no ssh.

The whole network is hidden behind a NAT at the entry box of the
network. Each village has its own "Head Box" and there is a cipe pipe to
it. Subscribers have their own boxes which associate with the head box.

We wish to extend our network by adding one of our own router boxes - we
run the Commercial Star-OS Operating system on our boxes. These are
administered using ssh.

We have connected the new box to an existing head boox using an ethernet
crossover cable, assigning the box an external IP address, and adding
routing statements to allow the box to be accessed by ssh. This works
very efficiantly.

This time we're trying to add a box to a client box, not a head box. The
connection is like this:

========================
| Network Gateway |
========================
| |
Cipe Tunnel
| |
========================
| Head Box |
========================
| |
IPSec
| |
========================
| Customer Box |
========================
| |
Ethernet
| |
========================
| StarOs Box |
========================
(Most of this came with the old network, so we're stuck with it).

I have assigned the static address of the two ethernet ports to be
212.24.92.249 and 212.24.92.250 (The StarOS box is 250).
I've added routing commands to the Gateway to route
212.24.92.248/30 to the Head Box over cipe, and on the head box to route
it to the Customer box.

Complicated, but should work, right?

It doesn't. If I log onto the Head box, I can ping 212.24.92.250, ans
traceroute gose straight to it; If i go back one step backwards I can't
ping and if I try and traceroute it routes as far as the headbox and
then returns "* * *" until it reaches 30 jumps.

Oh, and one other thing? The Customer box is getting perfect delivery of
Internet. You can connect a PC to it and surf the net perfectly.

What's going wrong? I've tried routing over the unencrypted IP
connections over which the tunnels are built, and the same thing happens.

Most sincere thanks for any hints or pointers as to what to look at.
I've posted below the routing tables of the Head Box; with the way
they've implemented IPSec it makes fairly gruesome reading, so only look
if you've a strong stomach.

Thanks in Advance, Andy Holyer

-------------- Cut Here --------------------------------
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
10.0.7.12 10.0.7.20 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.24 10.0.7.20 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.10 10.0.7.15 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.21 10.0.7.15 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.6 10.0.7.4 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.7 10.0.7.13 255.255.255.255 UGH 0 0 0
wlan0
192.168.105.7 * 255.255.255.255 UH 0 0 0
cipcb0
10.0.7.17 10.0.7.15 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.2 10.0.7.15 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.18 10.0.7.20 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.3 10.0.7.15 255.255.255.255 UGH 0 0 0
wlan0
10.0.7.19 10.0.7.20 255.255.255.255 UGH 0 0 0
wlan0
212.24.92.236 10.0.7.13 255.255.255.252 UG 0 0 0
ipsec0
212.24.92.248 10.0.7.15 255.255.255.252 UG 0 0 0
wlan0
212.24.92.240 10.0.7.14 255.255.255.252 UG 0 0 0
ipsec0
212.24.92.244 10.0.7.9 255.255.255.252 UG 0 0 0
ipsec0
212.24.92.220 * 255.255.255.252 U 0 0 0
eth1
212.24.92.248 10.0.7.15 255.255.255.248 UG 0 0 0
wlan0
10.8.6.0 10.0.7.6 255.255.255.0 UG 0 0 0
ipsec0
10.8.22.0 10.0.7.22 255.255.255.0 UG 0 0 0
ipsec0
10.8.7.0 10.0.7.7 255.255.255.0 UG 0 0 0
ipsec0
10.8.23.0 10.0.7.23 255.255.255.0 UG 0 0 0
ipsec0
10.8.4.0 10.0.7.4 255.255.255.0 UG 0 0 0
ipsec0
10.8.20.0 10.0.7.20 255.255.255.0 UG 0 0 0
ipsec0
192.168.21.0 10.0.7.15 255.255.255.0 UG 0 0 0
wlan0
10.8.5.0 10.0.7.5 255.255.255.0 UG 0 0 0
ipsec0
10.0.7.0 * 255.255.255.0 U 0 0 0
wlan0
10.0.7.0 * 255.255.255.0 U 0 0 0
ipsec0
192.168.20.0 10.0.7.15 255.255.255.0 UG 0 0 0
wlan0
10.8.21.0 10.0.7.21 255.255.255.0 UG 0 0 0
ipsec0
10.8.2.0 10.0.7.2 255.255.255.0 UG 0 0 0
ipsec0
10.8.18.0 10.0.7.18 255.255.255.0 UG 0 0 0
ipsec0
192.168.19.0 * 255.255.255.0 U 0 0 0
eth1
192.168.2.0 192.168.11.1 255.255.255.0 UG 0 0 0
wlan1
10.8.3.0 10.0.7.3 255.255.255.0 UG 0 0 0
ipsec0
192.168.18.0 * 255.255.255.0 U 0 0 0
eth0
10.8.19.0 10.0.7.19 255.255.255.0 UG 0 0 0
ipsec0
10.8.16.0 10.0.7.16 255.255.255.0 UG 0 0 0
ipsec0
10.8.17.0 10.0.7.17 255.255.255.0 UG 0 0 0
ipsec0
10.8.14.0 10.0.7.14 255.255.255.0 UG 0 0 0
ipsec0
10.8.15.0 10.0.7.15 255.255.255.0 UG 0 0 0
ipsec0
10.8.12.0 10.0.7.12 255.255.255.0 UG 0 0 0
ipsec0
192.168.12.0 10.0.7.15 255.255.255.0 UG 0 0 0
wlan0
10.8.13.0 10.0.7.13 255.255.255.0 UG 0 0 0
ipsec0
10.8.10.0 10.0.7.10 255.255.255.0 UG 0 0 0
ipsec0
192.168.11.0 * 255.255.255.0 U 0 0 0
wlan1
10.8.11.0 10.0.7.11 255.255.255.0 UG 0 0 0
ipsec0
10.8.8.0 10.0.7.8 255.255.255.0 UG 0 0 0
ipsec0
10.8.24.0 10.0.7.24 255.255.255.0 UG 0 0 0
ipsec0
10.8.9.0 10.0.7.9 255.255.255.0 UG 0 0 0
ipsec0
10.8.25.0 10.0.7.25 255.255.255.0 UG 0 0 0
ipsec0
default 192.168.105.7 0.0.0.0 UG 0 0 0
cipcb0
#

--
Andy Holyer, Systems Administrator
Hedgehog Broadband Ltd, 11 Marlborough Place, Brighton UK BN1 1UB
[url]http://www.hhbb.co.uk/[/url]

[Dave Hinz]

On Thu, 02 Sep 2004 13:13:11 +0100, Andy Holyer <andyh@hhbb.co.uk> wrote:

> Short form of the question:
>
> If I can ping/traceroute C from B, and B from A, then shouldn't it
> logically follow that I must be able to ping/traceroute C from A? I
> can't!

Sounds like B isn't routing, or isn't routing properly.

> We are a wireless ISP, providing Broadband to people in rural areas
> further away from their nearest exchange than the 3.5km allowed by wired
> DSL.

I'm doing something similar; neighbor with a T1, several of us using
802.11b to bounce signal around to get connectivity.

> The cutting down of Linux was fairly drastic, so many
> useful diagnostic tools are missin - for example there's no ssh.

Ouch. But telnet is on? Ouch.

> It doesn't. If I log onto the Head box, I can ping 212.24.92.250, ans
> traceroute gose straight to it; If i go back one step backwards I can't
> ping and if I try and traceroute it routes as far as the headbox and
> then returns "* * *" until it reaches 30 jumps.
>
> Oh, and one other thing? The Customer box is getting perfect delivery of
> Internet. You can connect a PC to it and surf the net perfectly.

So you _are_ routing then. Is ICMP turned off, so you're just seeing
what looks like a problem and actually isn't?

Dave Hinz

[Moe Trin]

In article <andyh-51A396.13131102092004@mercury.nildram.net>, Andy Holyer wrote:

>Short form of the question:
>
>If I can ping/traceroute C from B, and B from A, then shouldn't it
>logically follow that I must be able to ping/traceroute C from A?

No. Does C know to use B to reach A? Does A know to use B to reach C?

>One large chunk of our network was bought wholesale from another ISP as
>they were about to go bust. It consists of a network of cut-down Red Hat
>boxes acting as routers (the OS is stored on a Flash card on the
>motherboard. The cutting down of Linux was fairly drastic, so many
>useful diagnostic tools are missin - for example there's no ssh.

None the less, that sounds like a good point to be looking at. Does
the RH box have 'tcpdump'? If not, grab a laptop and a hub and some
Ethernet cables, and hook that into the Ethernet connection that
plugs into the headbox. What do you see?

>I've posted below the routing tables of the Head Box; with the way
>they've implemented IPSec it makes fairly gruesome reading, so only look
>if you've a strong stomach.

And without knowing what is what and where, it also causes my eyes to
water. Look at the routing tables on all hosts involved. Do they know
how to pass packets for destination A and C? Do A and C know how to
reach each other. This may mean using explicit routes, rather than
relying on defaults.

Old guy

[Andy Holyer]

In article <slrncjfj3b.sa9.ibuprofin@atlantis.phx.az.us>,
[email]ibuprofin@painkiller.example.tld[/email] (Moe Trin) wrote:

> In article <andyh-51A396.13131102092004@mercury.nildram.net>, Andy Holyer
> wrote:

> >Short form of the question:
> >
> >If I can ping/traceroute C from B, and B from A, then shouldn't it
> >logically follow that I must be able to ping/traceroute C from A?

>
> No. Does C know to use B to reach A? Does A know to use B to reach C?
>

Yes, and Yes.

> >One large chunk of our network was bought wholesale from another ISP as
> >they were about to go bust. It consists of a network of cut-down Red Hat
> >boxes acting as routers (the OS is stored on a Flash card on the
> >motherboard. The cutting down of Linux was fairly drastic, so many
> >useful diagnostic tools are missin - for example there's no ssh.

>
> None the less, that sounds like a good point to be looking at. Does
> the RH box have 'tcpdump'? If not, grab a laptop and a hub and some
> Ethernet cables, and hook that into the Ethernet connection that
> plugs into the headbox. What do you see?
>

No tcpdump. Could you expand on what you were saying about using a
laptop? Just to make things really tricky, the headbox is mounted to the
top of a gable about 15 meteres above the ground.

> >I've posted below the routing tables of the Head Box; with the way
> >they've implemented IPSec it makes fairly gruesome reading, so only look
> >if you've a strong stomach.

>
> And without knowing what is what and where, it also causes my eyes to
> water. Look at the routing tables on all hosts involved. Do they know
> how to pass packets for destination A and C? Do A and C know how to
> reach each other. This may mean using explicit routes, rather than
> relying on defaults.

As far as I can see, all is fine. That's what makes it all so
frustrating.

--
Andy Holyer, Systems Administrator
Hedgehog Broadband Ltd, 11 Marlborough Place, Brighton UK BN1 1UB
[url]http://www.hhbb.co.uk/[/url]

[Andy Holyer]

In article <2posbmFmhccvU2@uni-berlin.de>,
Dave Hinz <DaveHinz@spamcop.net> wrote:

> On Thu, 02 Sep 2004 13:13:11 +0100, Andy Holyer <andyh@hhbb.co.uk> wrote:

>

> > The cutting down of Linux was fairly drastic, so many
> > useful diagnostic tools are missin - for example there's no ssh.

>
> Ouch. But telnet is on? Ouch.
>

My thoughts entirely. I lost a FreeBSD Box to some script kiddies using
a telnetd exploit and it was only the fact that they did it on the night
of 10 September 2001 that saved my hid - the customer had something else
to worry about the next morning, if you get my drift.

Haven't touched telnet since then.

--
Andy Holyer, Systems Administrator
Hedgehog Broadband Ltd, 11 Marlborough Place, Brighton UK BN1 1UB
[url]http://www.hhbb.co.uk/[/url]

[Moe Trin]

Mail-Copies-To: poster

Sorry - can't do that anymore.

In article <andyh-624C45.10051503092004@mercury.nildram.net>, Andy Holyer wrote:

>In article <slrncjfj3b.sa9.ibuprofin@atlantis.phx.az.us>,
> [email]ibuprofin@painkiller.example.tld[/email] (Moe Trin) wrote:

>> No. Does C know to use B to reach A? Does A know to use B to reach C?
>>

>Yes, and Yes.

Hate to tell you how often that one is the problem. "I can talk fine to box
X". Of course box X is turning about in circles trying to figue out where
the heck this unknown host is, so that it can reply, or go after the
tormentor with a broadsword. The other problem is where some router has
an entry that sends the packets to the Falkland Islands or South Georgia,
and the penguin running the router down there had to put in a firewall rule
to stop being bothered by this stuff.

>No tcpdump. Could you expand on what you were saying about using a
>laptop?

Laptop, network sniffer - just about anything that lets you see the
packets that are passing by. It's called 'divide and conquer". Can you
see packets from A going to C? What about C going to A. Are ther also
some ICMP Type 3 (or 5) packets trying to scream "NNNnnnoooooooo!!!"

>Just to make things really tricky, the headbox is mounted to the
>top of a gable about 15 meteres above the ground.

Oh, joy!!! Well, I suppose it keeps the local children from stealing
the hardware. ;-)

>As far as I can see, all is fine. That's what makes it all so
>frustrating.

Pushing a bit - could there be a firewall someplace that's dropping
stuff? Another idea - traceroute from A to C and vice versa, or
is there someone dropping ICMP packets? If so, if you know the IP
addresses of the intermediate hosts, you could even try telnetting
to them. You're not looking to get in, just seeing if the connection
gets refused or rejected (which means you got there, but the server
doesn't want to speak to you).

Old guy

[Andy Holyer]

In article <slrncjhvu5.267.ibuprofin@atlantis.phx.az.us>,
[email]ibuprofin@painkiller.example.tld[/email] (Moe Trin) wrote:

> Pushing a bit - could there be a firewall someplace that's dropping
> stuff? Another idea - traceroute from A to C and vice versa, or
> is there someone dropping ICMP packets? If so, if you know the IP
> addresses of the intermediate hosts, you could even try telnetting
> to them. You're not looking to get in, just seeing if the connection
> gets refused or rejected (which means you got there, but the server
> doesn't want to speak to you).

There's something subtly wrong with the setup of the secure tunnels,
both IPSEC and CIPE, in that they don't return pings from a traceroute -
you just get *.*.* from that box.

I can traceroute from A to B over the unencrypted hops ant then it goes
wrong when we get there. Here's to output:

(On B):
# traceroute 192.168.12.2
traceroute to 192.168.12.2 (192.168.12.2), 30 hops max, 40 byte packets
1 192.168.12.2 (192.168.12.2) 8.883 ms 7.729 ms 5.14 ms
#

Not surprising, as 192.168.12.1/24 is one of B's WLAN IP addresses

Now from A:

# traceroute 192.168.12.2
traceroute to 192.168.12.2 (192.168.12.2), 30 hops max, 40 byte packets
1 192.168.2.2 (192.168.2.2) 4.146 ms 3.358 ms 4.157 ms
2 192.168.4.2 (192.168.4.2) 4.359 ms 5.624 ms 4.611 ms
3 192.168.6.2 (192.168.6.2) 7.86 ms 8.081 ms 7.792 ms
4 192.168.9.2 (192.168.9.2) 13.121 ms 8.946 ms 7.934 ms
5 192.168.11.2 (192.168.11.2) 11.699 ms 11.248 ms 11.121 ms
6 * * *
7 * * *
....and so it continues until traceroute runs out of road.

My suspicion is that for some reason it's trying to use one of its IPSEC
tunnels, but in that case, why does it work when telnetted in? It's
suypicous that B's default route is back over the cipe tunnel to A, but
that's true of all the other boxes as well, each of them has a cipe
tunnel back to the mail access box.

Curiouser and curiouser

--
Andy Holyer, Systems Administrator
Hedgehog Broadband Ltd, 11 Marlborough Place, Brighton UK BN1 1UB
[url]http://www.hhbb.co.uk/[/url]

[Moe Trin]

In article <andyh-163E13.09333706092004@mercury.nildram.net>, Andy Holyer
wrote:

>There's something subtly wrong with the setup of the secure tunnels,
>both IPSEC and CIPE, in that they don't return pings from a traceroute -
>you just get *.*.* from that box.

Unix traceroute uses UDP pasckets (by default starting at 33434 and
incrementing per hop) starting with a TTL of 1, and incrementing that.
It depends on receiving an ICMP Time Exceeded error (Type 11 code 0)
from intermediate hops, and a Port Unreachable (Type 3 code 3) when
it reaches it's destination. (It's the b0rken microsoft 'tracert'
that uses pings for this, although the Unix version can be made to
do so with the -I option.) Could your tunnel be blocking or
dropping the ICMP stuff? As suggested, try telnetting to each box
and looking at the response (hopefully, a ACK RST packet, rather
than an ICMP error). Don't forget that telnet accepts a port number
arguement when connecting.

># traceroute 192.168.12.2
>traceroute to 192.168.12.2 (192.168.12.2), 30 hops max, 40 byte packets
> 1 192.168.12.2 (192.168.12.2) 8.883 ms 7.729 ms 5.14 ms
>#

Well, you got there.

> 4 192.168.9.2 (192.168.9.2) 13.121 ms 8.946 ms 7.934 ms
> 5 192.168.11.2 (192.168.11.2) 11.699 ms 11.248 ms 11.121 ms
> 6 * * *
> 7 * * *
>...and so it continues until traceroute runs out of road.

What are the missing steps between 192.168.11.2 and 192.168.12.2?
Can you reach those steps by any other means? Can any of them run
tcpdump to see what packets are passing by?

>My suspicion is that for some reason it's trying to use one of its IPSEC
>tunnels, but in that case, why does it work when telnetted in?

Different protocol? Telnet solely depends on TCP, while traceroute
must have ICMP, and probably needs UDP in *nix. Take that back,
telnet _MAY_ need a UDP connection from each end to their name servers,
but the symptoms would be noticable if that weren't working.

>Curiouser and curiouser

Just another happy day in networking land ;-)

Old guy

[Andy Holyer]

[email]ibuprofin@painkiller.example.tld[/email] (Moe Trin) wrote in message news:

> ># traceroute 192.168.12.2
> >traceroute to 192.168.12.2 (192.168.12.2), 30 hops max, 40 byte packets
> > 1 192.168.12.2 (192.168.12.2) 8.883 ms 7.729 ms 5.14 ms
> >#

>
> Well, you got there.
>

> > 4 192.168.9.2 (192.168.9.2) 13.121 ms 8.946 ms 7.934 ms
> > 5 192.168.11.2 (192.168.11.2) 11.699 ms 11.248 ms 11.121 ms
> > 6 * * *
> > 7 * * *
> >...and so it continues until traceroute runs out of road.

>
> What are the missing steps between 192.168.11.2 and 192.168.12.2?
> Can you reach those steps by any other means? Can any of them run
> tcpdump to see what packets are passing by?
>

There are no missing steps. wlan0 of the box is configured to
192.168.11.2/24, and wlan1 is 192.168.12.1/24. Here's (part of) the
ifconfig to prove it:

wlan0 Link encap:Ethernet HWaddr 00:02:6F:04:54:AC
inet addr:10.0.7.1 Bcast:10.0.7.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2464996 errors:0 dropped:0 overruns:0 frame:0
TX packets:2908055 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:643652625 (613.8 MiB) TX bytes:1520352586 (1449.9
MiB)
Interrupt:10 Base address:0x100

wlan0:0 Link encap:Ethernet HWaddr 00:02:6F:04:54:AC
inet addr:192.168.12.1 Bcast:192.168.12.255
Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:10 Base address:0x100

wlan1 Link encap:Ethernet HWaddr 00:02:6F:05:4A:92
inet addr:192.168.11.2 Bcast:192.168.11.255
Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2582605 errors:0 dropped:0 overruns:0 frame:0
TX packets:2172520 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:1745937590 (1665.0 MiB) TX bytes:809435215 (771.9
MiB)
Interrupt:10 Base address:0x140