This document discusses the challenges network analysts face in investigating issues at 10GbE network speeds. It outlines how traditional tools like Wireshark struggle with the volume of data at high speeds. The document then introduces EndaceVision and Endace Packets as a solution, which leverages purpose-built hardware for 100% accurate network recording at 10GbE speeds and faster protocol decoding. It argues these tools allow analysts to more effectively search, filter, and drill down into precise packets of interest when troubleshooting complex network issues.
To assure you that there’s no waving of hands, here’s a 60-second view of the traffic we’re using for the demo. This is a live capture in our demo lab of replayed previously captured data, so, while it’s got spikes in the small scale, you won’t see the daily variations you would on your own network. You can see from the timestamps on the screen that this data was taken yesterday, and that there are sustained traffic spikes above 6Gbps.
Here’s what it looks like when viewing 10 minutes.
Here’s 1 hour
And just for fun, here’s 2 days.The trend you’re seeing is that, as we zoom out, the line flattens. There just aren’t enough pixels to show all of the spikes, so we’re having to do the same thing as all of the other tools out there: average the samples. But as network analysts, we like those spikes. We know that there are bursts and microbursts. So here’s what we at Endace have done.
I turned on the Bursts display, which tracks the maximum bandwidth value for each display point, with a sample size of 1ms.
Now I’m going to zoom back to 1 hour.
Here’s 10 minutes.
And back down to 1 minute.
Now I’m going to pivot my view to Traffic Breakdown over Time, with a breakdown on Applications. This lets me see what’s going on at Layer 7. For this capture, we’ve got mostly RTP traffic out there – that’s VoIP. I can also see that someone is using a lot of iTunes, plus some http video, Amazon.com, etc. While RTP is probably business traffic, iTunes and Amazon almost definitely aren’t.
I’m going to filter in on iTunes to see who’s using it.
Here’s the filter applied. I’m going to pivot my view to see who’s sending this traffic.
I’m still looking at the same data, but I pivoted to show the top talkers, with the iTunes filter. I also zoomed my timescale out to show the last 24 hours.There’s 1 internal host who’s really pulling the majority of traffic. You can see it on the left. The vast majority of its traffic is colored blue, to indicate that it’s receiving that data. Similarly, the 2 primary external servers are colored green.Just a reminder, this is our demo lab, doing a live capture of traffic that’s being replayed. I doubt iTunes in reality is capable of feeding 10T to a single client in 24 hours.
I can also change the filter to remove the iTunes and see who else the node is talking to.
Next question is what else this node is doing. It looks like, apart from iTunes, there’s not much else – some Facebook, a little generic http.
We can uncover some of that generic http with the Conversations view.
What I really want is to know who this node is. Yes, I probably have other tools to find out – dynamic DNS might tell me. But I’m a packet geek, and I trust what the packets tell me. So I’m going to download the mdns packets and see the advertisements from the node itself.I’m going to download the packets to the probe. It keeps them off my laptop to keep it from potentially going into PCI scope. It also means I don’t tie up my laptop in the download if it’s a large file, and my teammates can also access it if they need to.
Here’s where we look at the download – there’s a progress indicator, time remaining, etc, but the cool part is that I don’t have to wait for the download to complete, so I clicked on Packets.Notice that you can also download these directly as either PCAP or ERF. ERF is the Endace format for packet capture.
Endace Packets looks like Wireshark, because it’s the tool that our customers said they use most often.There’s a lot of mdns out there, let me filter it down to something tighter.
I filtered for DNS responses of type PTR, mapping the IP or IPv6 address to the hostname, then used the “contains” filter to narrow down the search to mdns .local names. I’ll dig a little more into the first packet now.
And there’s the culprit. The device identifies itself as “Neil L’s iPhone”. Now I can go have a chat with Neil about proper use of the local resources.
Just to be thorough, I also did a local download – on the probe – for some of that unidentified http traffic. Here I’ve got a filter applied to focus only on the packets which have a http request URI, which will tell me what domains the iPhone is connecting to.
Since EndacePackets does name resolution, it also does name de-resolution, which is useful for cases like this, where everything is going either to the Amazon cloud or to cloudfront. Just hover over a name and the address will pop up.
And there’s the real URI for the request. You can also see the User-Agent. CFNetwork is a sockets API in IOS, so it looks like this is an app, probably pulling down an advertisement.
I’ll scroll to the right and you can see the list of relative URIs – this BYOD stuff gets pretty chatty, but that’s a different vendor.