Strange results from a tcpdump, can anyone help?

Strange results from a tcpdump, can anyone help?

NewsGroups | Search | Tools
 comp.dcom.lans.ethernet  Post an article  get this group's latest topics as an RSS feed add this group's latest topics to your My MSN content add this group's latest topics to your My Yahoo content  add this group's latest topics to your Google content  YahooMyWeb Yahoo!  Google Google  Windows Live Favorites Windows Live  del.icio.us del.icio.us  digg digg  Add to Netscape Netscape
Subject Author Date
Strange results from a tcpdump, can anyone help? maethlin 03-29-2006
Posted by maethlin on March 29, 2006, 2:51 pm
If you were  Registered and logged in, you could reply and use other advanced thread options


Hello there, I was refered here by a helpful soul who claims that
members of this group may have a deeper understanding of network issues
that could help me figure out this problem. I'm reposting the
pertinent details below:
================================================

I'm a bit of a newb to the world of networking, so please bear with me.

I work in an environment with many separate vlans spanning several
switches (say about a dozen). Today we had an incident where suddenly
traffic was going ballistic on most ports in the network. Doing a
tcpdump on a particular host on this network, you could actually see
unicast traffic that was neither destined to or coming from the host.
Or, to put it another way, it almost looked like the host was on a hub,
where you could see packets travelling between other hosts on the
network to other destinations.

We shut off some ports where some new windows servers were brought up
today. As soon as those ports were taken offline, then tcpdumps on the
other hosts went to normal (i.e. the only traffic you could see were
broadcasts, or unicasts to and from that host).

Can anyone think of a likely explanation for this?

Please let me know if I'm not making sense!

Thanks in advance,

=====================================================

An additional wrinkle I've noticed while studying the tcpdump:

All the traffic I'm seeing that is not supposed to be there (i.e. http
traffic from various other switches/hosts on the vlan) tends to be
packets from the same vlan (vlan 82) destined to other hosts outside
this vlan. In other words, the packets have src ips originating from
within the vlan and dst ips are all external, and the src ips are from
hosts that are not confined to a particular switch (at a brief glance,
I'm seeing src packets coming from switch08, switch05, switch06, as
well as other hosts on switch01 - where the tcpdump was taken).

If it was simply a bad switch with a bad port that had lost it's mac
tables and was now broadcasting everywhere in the vlan, I would expect
to see packets in the tcpdump with all the src ips from a single
switch, and dst ips both internal and external to the vlan.

That doesn't seem to be the case here.


Posted by Albert Manfredi on March 29, 2006, 5:01 pm
If you were  Registered and logged in, you could reply and use other advanced thread options


> Hello there, I was refered here by a helpful soul who claims that
> members of this group may have a deeper understanding of network
> issues
> that could help me figure out this problem. I'm reposting the
> pertinent details below:
> ================================================
>
> I'm a bit of a newb to the world of networking, so please bear with
> me.
>
> I work in an environment with many separate vlans spanning several
> switches (say about a dozen). Today we had an incident where suddenly
> traffic was going ballistic on most ports in the network. Doing a
> tcpdump on a particular host on this network, you could actually see
> unicast traffic that was neither destined to or coming from the host.
> Or, to put it another way, it almost looked like the host was on a
> hub,
> where you could see packets travelling between other hosts on the
> network to other destinations.
>
> We shut off some ports where some new windows servers were brought up
> today. As soon as those ports were taken offline, then tcpdumps on
> the
> other hosts went to normal (i.e. the only traffic you could see were
> broadcasts, or unicasts to and from that host).
>
> Can anyone think of a likely explanation for this?
>
> Please let me know if I'm not making sense!
>
> Thanks in advance,
>
> =====================================================
>
> An additional wrinkle I've noticed while studying the tcpdump:
>
> All the traffic I'm seeing that is not supposed to be there (i.e. http
> traffic from various other switches/hosts on the vlan) tends to be
> packets from the same vlan (vlan 82) destined to other hosts outside
> this vlan. In other words, the packets have src ips originating from
> within the vlan and dst ips are all external, and the src ips are from
> hosts that are not confined to a particular switch (at a brief glance,
> I'm seeing src packets coming from switch08, switch05, switch06, as
> well as other hosts on switch01 - where the tcpdump was taken).
>
> If it was simply a bad switch with a bad port that had lost it's mac
> tables and was now broadcasting everywhere in the vlan, I would expect
> to see packets in the tcpdump with all the src ips from a single
> switch, and dst ips both internal and external to the vlan.
>
> That doesn't seem to be the case here.

Traffic from the VLAN destined for hosts outside the VLAN is traffic
that has to go to a router. Can you see whether the MAC DA on these
frames is the MAC address of the default router? If it is, then the
question would be why traffic destined to a particular host, the router,
is showing up at other ports. As if that router MAC address is not being
learned, so all frames to the router are being flooded to all active
ports in the catenet.

Is there anything odd about the network? Are these all one-way UDP
datagrams, for example?

Bert


Posted by maethlin on March 29, 2006, 5:30 pm
If you were  Registered and logged in, you could reply and use other advanced thread options


Will get you some answers to your questions Bert, thank you for the
insight! I have additional info though, through some troubleshooting -
we've narrowed it down to certain ports causing the problem when they
are brought up with portfast turned off. As long as portfast is on,
bringing the ports up causes no problem (as they don't participate in
the STP).

Also sniffing traffic, you can see topology changes happening when
those ports are brought up. Oddly though, the ports are just connected
to plain old w2k3 servers.


Posted by maethlin on March 29, 2006, 9:00 pm
If you were  Registered and logged in, you could reply and use other advanced thread options


So Bert, to answer your question, I'm looking at some of the packets
(that were supposed to go outside the vlan) and the MAC DA on them
seems to be the core switch interface (all the access switches I am
discussing are plugged into a core switch). So yes, it would seem that
traffic destined to the core is somehow showing up at other ports - in
this case almost every port on that vlan.

Nothing odd about the network that I can tell. They are not one-way
UDP datagrams - this is regular http and tcp traffic.


Posted by anoop on March 29, 2006, 11:47 pm
If you were  Registered and logged in, you could reply and use other advanced thread options



maethlin wrote:
> So Bert, to answer your question, I'm looking at some of the packets
> (that were supposed to go outside the vlan) and the MAC DA on them
> seems to be the core switch interface (all the access switches I am
> discussing are plugged into a core switch). So yes, it would seem that
> traffic destined to the core is somehow showing up at other ports - in
> this case almost every port on that vlan.

At the time when the problem happens, look at the switch's forwarding
table and see if the MAC address of the router has been learned there.

Anoop


Similar ThreadsPosted
Flow control frames: will I see them with tcpdump? January 30, 2006, 10:05 am
watching outgoing ping packets via tcpdump/ethereal January 25, 2006, 10:14 am
Strange ethrenet frame October 4, 2006, 10:54 am
strange ethernet electric problem June 22, 2007, 5:39 am
Strange switch behaviour in VLAN network July 6, 2005, 9:53 am

other useful resources:
The Federal Communications Commission (FCC)
Telecommunications Industry Association
Electronic and Software Security Products and Services
International Telecommunication Union

Custom CGI Perl and PHP programming by 1-Script.com

Contact Us | Privacy Policy
The site map in XML format XML site map