[prev in list] [next in list] [prev in thread] [next in thread]
List: nmap-dev
Subject: Live Capture Performance to Rival Wireshark
From: Matthew Davis <matthewndavis () gmail ! com>
Date: 2022-08-20 22:46:59
Message-ID: CAAHmOOM27OyOvGQbtf4cP9yKnTzDA7h_UZORe0ZdGun_DnmKAw () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hello.
The project I work on is a Windows-only, 64-bit-only application that
collects and processes application layer messages across various
system-specific protocols. Our legacy application used the old Windows raw
socket mechanism (before migrating to WinPcap 4.1.3) to read the packets
off the wire. Our revamped application is using the OEM version of Npcap
with MUCH better success.
However...
A recent survey of our log files from the field indicates that we are
missing packets. Specifically, converting our log files to .pcapng and
opening them in Wireshark, we see about 1% of the packets showing the [TCP
Previous segment not captured] message. Due to the nature of this data,
this 1% loss is unacceptable to our users. As expected, this loss gets
more dramatic with additional network traffic. Testing in the lab shows
that Wireshark v3.6.7 captures the packets from a stress test with no
apparent packet loss, so I know the problem is on our end.
I think there are some application-level improvements we can make, but I'm
interested in knowing if we are configuring and executing our packet
capture in the most efficient way possible.
Here is a brief summary of how we are doing things currently.
- We call pcap_open() with a snap length of 65536, promiscuous mode
enabled, and a read timeout of 500ms.
- We call pcap_setbuff() to increase the size of the kernel buffer to 16MB.
- We then call pcap_next_ex() inside a for loop to get the next capture.
- Upon successful return, we allocate a byte array using pkt_header.caplen,
copy the pkt_data into the byte array, and add the byte array to a
pre-allocated list.
- We execute this for loop until the pre-allocated list is filled (to avoid
reallocation) or a predetermined timeout is exceeded on the application
side.
- When either of these conditions is satisfied, we hand the pre-allocated
list off to another thread, allocate a new list, and do the loop again.
Here are my questions.
1. Is pcap_next_ex() the most efficient way of transferring captures to
the application? It looks like pcap_loop() or pcap_dispatch() might allow
multiple captures to be returned via a single callback. Is that correct?
And if so, would that be the recommended way to get the captures in a high
data rate environment?
2. Our understanding is that the kernel buffer *IS* the ring buffer that
must be read from at least as fast as the data is received in order to
minimize/eliminate the occurrence of dropped packets. We understand the
size of the buffer won't prevent dropped packets if the application can't
keep up (it merely delays the moment when that occurs). But a bigger ring
buffer can accommodate data spikes, allowing the application to catch up
during data lulls. To this end, how big can we make the kernel buffer via
pcap_setbuff()? Is there a practical or rule-of-thumb limit? Is the ring
buffer associated with each handle??? If we collect simultaneously with
Npcap on multiple NICs, does the size of each ring buffer need to be
limited in any way? Should we use pcap_set_buffer_size() instead of
pcap_setbuff()? Can pcap_set_buffer_size() be called after pcap_open()
like pcap_setbuff() can? Or do we need to use the create-set-activate
pattern?
3. We are confused about the difference between the kernel buffer and the
user buffer. How does the user buffer work with pcap_next_ex()? Since
pcap_next_ex() only returns a single packet at a time, does the user buffer
even matter? Perhaps it comes into play with pcap_loop() or
pcap_dispatch() being able to return more data in the callback?
4. How does pcap_mintocopy() work with pcap_next_ex()? Again, since
pcap_next_ex() only returns a single packet at a time, does this even
apply? Perhaps with pcap_loop() or pcap_dispatch()?
5. Finally, can stats be enabled for the same handle that is capturing the
data? If we want to monitor, for example, the number of dropped packets
seen during a capture, how do we do that with the pcap_t returned by
pcap_open()? If we need two pcap_t handles, one for capture and one for
stats, does that imply a single ring buffer under the hood for a given NIC?
As stated earlier, we are using Windows 64-bit only with Npcap OEM 1.70.
I apologize for the length, but any help/guidance is GREATLY appreciated.
And hopefully this will help others out as well.
Matt Davis
[Attachment #5 (text/html)]
<div dir="ltr">Hello.<div><br></div><div>The project I work on is a Windows-only, \
64-bit-only application that collects and processes application layer messages across \
various system-specific protocols. Our legacy application used the old Windows raw \
socket mechanism (before migrating to WinPcap 4.1.3) to read the packets off the \
wire. Our revamped application is using the OEM version of Npcap with MUCH better \
success.</div><div><br></div><div>However...</div><div><br></div><div>A recent survey \
of our log files from the field indicates that we are missing packets. \
Specifically, converting our log files to .pcapng and opening them in Wireshark, we \
see about 1% of the packets showing the [TCP Previous segment not captured] message. \
Due to the nature of this data, this 1% loss is unacceptable to our users. As \
expected, this loss gets more dramatic with additional network traffic. Testing in \
the lab shows that Wireshark v3.6.7 captures the packets from a stress test with no \
apparent packet loss, so I know the problem is on our \
end.<br></div><div><br></div><div>I think there are some application-level \
improvements we can make, but I'm interested in knowing if we are configuring and \
executing our packet capture in the most efficient way \
possible.</div><div><br></div><div>Here is a brief summary of how we are doing things \
currently.</div><div><br></div><div>- We call pcap_open() with a snap length of \
65536, promiscuous mode enabled, and a read timeout of 500ms.</div><div>- We call \
pcap_setbuff() to increase the size of the kernel buffer to 16MB.</div><div>- We then \
call pcap_next_ex() inside a for loop to get the next capture.</div><div>- Upon \
successful return, we allocate a byte array using pkt_header.caplen, copy the \
pkt_data into the byte array, and add the byte array to a pre-allocated \
list.</div><div>- We execute this for loop until the pre-allocated list is filled (to \
avoid reallocation) or a predetermined timeout is exceeded on the application \
side.</div><div>- When either of these conditions is satisfied, we hand the \
pre-allocated list off to another thread, allocate a new list, and do the loop \
again.</div><div><br></div><div>Here are my questions.</div><div><br></div><div>1. \
Is pcap_next_ex() the most efficient way of transferring captures to the application? \
It looks like pcap_loop() or pcap_dispatch() might allow multiple captures to be \
returned via a single callback. Is that correct? And if so, would that be the \
recommended way to get the captures in a high data rate \
environment?</div><div><br></div><div>2. Our understanding is that the kernel \
buffer *IS* the ring buffer that must be read from at least as fast as the data is \
received in order to minimize/eliminate the occurrence of dropped packets. We \
understand the size of the buffer won't prevent dropped packets if the \
application can't keep up (it merely delays the moment when that occurs). But a \
bigger ring buffer can accommodate data spikes, allowing the application to catch up \
during data lulls. To this end, how big can we make the kernel buffer via \
pcap_setbuff()? Is there a practical or rule-of-thumb limit? Is the ring buffer \
associated with each handle??? If we collect simultaneously with Npcap on multiple \
NICs, does the size of each ring buffer need to be limited in any way? Should we \
use pcap_set_buffer_size() instead of pcap_setbuff()? Can pcap_set_buffer_size() be \
called after pcap_open() like pcap_setbuff() can? Or do we need to use the \
create-set-activate pattern?</div><div><br></div><div>3. We are confused about the \
difference between the kernel buffer and the user buffer. How does the user buffer \
work with pcap_next_ex()? Since pcap_next_ex() only returns a single packet at a \
time, does the user buffer even matter? Perhaps it comes into play with pcap_loop() \
or pcap_dispatch() being able to return more data in the \
callback?</div><div><br></div><div>4. How does pcap_mintocopy() work with \
pcap_next_ex()? Again, since pcap_next_ex() only returns a single packet at a time, \
does this even apply? Perhaps with pcap_loop() or \
pcap_dispatch()?</div><div><br></div><div>5. Finally, can stats be enabled for the \
same handle that is capturing the data? If we want to monitor, for example, the \
number of dropped packets seen during a capture, how do we do that with the pcap_t \
returned by pcap_open()? If we need two pcap_t handles, one for capture and one for \
stats, does that imply a single ring buffer under the hood for a given \
NIC?</div><div><br></div><div>As stated earlier, we are using Windows 64-bit only \
with Npcap OEM 1.70.</div><div><br></div><div>I apologize for the length, but any \
help/guidance is GREATLY appreciated. And hopefully this will help others out as \
well.</div><div><br></div><div>Matt Davis</div></div>
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at https://seclists.org/nmap-dev/
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic