I'm writing a network analysis program, and just today I've found out that it's too slow and missing packets.

Ideally, I want the thread that reads from the network card to have exclusive use of one of the CPU cores.

I'm looking at the following two functions:
sched_setaffinity
pthread_setaffinity_np

And I note the following in the Linux manual:
"For example, by dedicating one CPU to a particular process (i.e., setting the affinity mask of that process to specify a single CPU, and setting the affinity mask of all other processes to exclude that CPU), it is possible to ensure maximum execution speed for that process"

I can find lots and lots of code examples to set the affinity for one thread or for one process -- however I cannot find one code example showing me how to exclude all other threads.

There's absolutely no point in me restricting my thread to the 1st CPU core if other threads can run on that core too -- in fact that will make my program _slower_ because my thread can't be scheduled on the other cores. Every code sample I can find online seems to just restrict the thread in question without placing any restriction on other threads.

Anyone know how this is supposed to work?

On 03/07/2023 12:42, Frederick Virchanza Gotham wrote:
>
> I'm writing a network analysis program, and just today I've found out that it's too slow and missing packets.
>
> Ideally, I want the thread that reads from the network card to have exclusive use of one of the CPU cores.
>
> I'm looking at the following two functions:
> sched_setaffinity
> pthread_setaffinity_np
>
> And I note the following in the Linux manual:
> "For example, by dedicating one CPU to a particular process (i.e., setting the affinity mask of that process to specify a single CPU, and setting the affinity mask of all other processes to exclude that CPU), it is possible to ensure maximum execution speed for that process"
>
> I can find lots and lots of code examples to set the affinity for one thread or for one process -- however I cannot find one code example showing me how to exclude all other threads.
>
> There's absolutely no point in me restricting my thread to the 1st CPU core if other threads can run on that core too -- in fact that will make my program _slower_ because my thread can't be scheduled on the other cores. Every code sample I can find online seems to just restrict the thread in question without placing any restriction on other threads.
>
> Anyone know how this is supposed to work?

"cpu isolation" is, I believe, the term you want to look for on Google.
I don't know if this can only be set from boot parameters or configured
at run-time. You could also search for real-time linux, as that is
where these features are often used.

You will probably also want to disable some or all power-down modes and
clock speed switching, as these can give significant variations in
latencies. And you may need to pin the network card interrupts to a
processor core too.

Frederick Virchanza Gotham <cauldwell.thomas@gmail.com> writes:
>
>I'm writing a network analysis program, and just today I've found out that =
>it's too slow and missing packets.
>
>Ideally, I want the thread that reads from the network card to have exclusi=
>ve use of one of the CPU cores.
>
>I'm looking at the following two functions:
> sched_setaffinity
> pthread_setaffinity_np

This is A C++ group. A simple google search should be more productive,
and the answer will be dependent upon which operating system you are
targetting.

On 7/3/2023 3:42 AM, Frederick Virchanza Gotham wrote:
>
> I'm writing a network analysis program, and just today I've found out that it's too slow and missing packets.
>
> Ideally, I want the thread that reads from the network card to have exclusive use of one of the CPU cores.
>
> I'm looking at the following two functions:
> sched_setaffinity
> pthread_setaffinity_np
>
> And I note the following in the Linux manual:
> "For example, by dedicating one CPU to a particular process (i.e., setting the affinity mask of that process to specify a single CPU, and setting the affinity mask of all other processes to exclude that CPU), it is possible to ensure maximum execution speed for that process"
>
> I can find lots and lots of code examples to set the affinity for one thread or for one process -- however I cannot find one code example showing me how to exclude all other threads.

Not sure you can do that. The OS can do whatever it wants. It can pin
your thread to a CPU, but that does not mean it cannot use said CPU for
other programs threads...

> There's absolutely no point in me restricting my thread to the 1st CPU core if other threads can run on that core too -- in fact that will make my program _slower_ because my thread can't be scheduled on the other cores. Every code sample I can find online seems to just restrict the thread in question without placing any restriction on other threads.
>
> Anyone know how this is supposed to work?

"Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> writes:
>On 7/3/2023 3:42 AM, Frederick Virchanza Gotham wrote:
>>
>> I'm writing a network analysis program, and just today I've found out that it's too slow and missing packets.
>>
>> Ideally, I want the thread that reads from the network card to have exclusive use of one of the CPU cores.
>>
>> I'm looking at the following two functions:
>> sched_setaffinity
>> pthread_setaffinity_np
>>
>> And I note the following in the Linux manual:
>> "For example, by dedicating one CPU to a particular process (i.e., setting the affinity mask of that process to specify a single CPU, and setting the affinity mask of all other processes to exclude that CPU), it is possible to ensure maximum execution speed for that process"
>>
>> I can find lots and lots of code examples to set the affinity for one thread or for one process -- however I cannot find one code example showing me how to exclude all other threads.
>
>Not sure you can do that. The OS can do whatever it wants. It can pin
>your thread to a CPU, but that does not mean it cannot use said CPU for
>other programs threads...

Linux provides mechanisms that allow dedication of compute resources
(e.g. hardware thread/core) to a specific set of processes. So do
most other legacy Unix operating systems.

The mechanisms vary based on the host operating system. cgroups is
the current favored mechanism on linux.

Am 03.07.2023 um 12:42 schrieb Frederick Virchanza Gotham:
>
> I'm writing a network analysis program, and just today I've found out that it's too slow and missing packets.
>
> Ideally, I want the thread that reads from the network card to have exclusive use of one of the CPU cores.
>
> I'm looking at the following two functions:
> sched_setaffinity
> pthread_setaffinity_np

I think the packets are likely to be processed at the core where the
interrupt of the NIC hits if the core isn't currently running another
thread. The interrupt runs at an aribtrary core, so I think it's un-
likely that it will hit the core you reserve for the procesing.
And at all I think the system has enough buffering capabilities for
the packets which aren't delivered to your thread yet. So there must
sth. different that you're missing packets.
If you like you could try io_uring, which has the shortest path from
kernel to userspace if the frequency of packets is high. If you have
one packet in the receive queue the processing is as usual with a
kernel notification. If you have further packets in the queue and the
kernel notices that theres an unprocessed packed in the ring it pushes
further packets into the ring without any notifications form kernel-
to user-space.

>
> And I note the following in the Linux manual:
> "For example, by dedicating one CPU to a particular process (i.e., setting the affinity mask of that process to specify a single CPU, and setting the affinity mask of all other processes to exclude that CPU), it is possible to ensure maximum execution speed for that process"
>
> I can find lots and lots of code examples to set the affinity for one thread or for one process -- however I cannot find one code example showing me how to exclude all other threads.
>
> There's absolutely no point in me restricting my thread to the 1st CPU core if other threads can run on that core too -- in fact that will make my program _slower_ because my thread can't be scheduled on the other cores. Every code sample I can find online seems to just restrict the thread in question without placing any restriction on other threads.
>
> Anyone know how this is supposed to work?

Bonita Montero <Bonita.Montero@gmail.com> writes:
>Am 03.07.2023 um 12:42 schrieb Frederick Virchanza Gotham:
>>
>> I'm writing a network analysis program, and just today I've found out that it's too slow and missing packets.
>>
>> Ideally, I want the thread that reads from the network card to have exclusive use of one of the CPU cores.
>>
>> I'm looking at the following two functions:
>> sched_setaffinity
>> pthread_setaffinity_np
>
>I think the packets are likely to be processed at the core where the
>interrupt of the NIC hits if the core isn't currently running another
>thread. The interrupt runs at an aribtrary core,

This is not accurate. RSS is used to direct NIC interrupts to specific
cores, often depending on the contents of header fields (e.g. TCP packets
destined for port 80 may be sent to a different core than UDP packets or
RDMA packets). Rules are programmed into the NIC to designate the target
core in the packet parsing logic.

Most modern NICs support some form of RSS (receive side scaling) and the
parsing rules can be arbitrarily complex in advanced networking chips
such as the OcteonTx2 processors (parsing MPLS, various tunnelling
protocols, IPsec, higher-level protocols like TCP et cetera et alia).

Am 04.07.2023 um 15:53 schrieb Scott Lurndal:

> This is not accurate. RSS is used to direct NIC interrupts to
> specific cores, often depending on the contents of header fields ...

And which NIC supports that ?

> Most modern NICs support some form of RSS ...

I don't believe that and Linux doesn't use task-offloading.

Bonita Montero <Bonita.Montero@gmail.com> writes:
>Am 04.07.2023 um 15:53 schrieb Scott Lurndal:
>
>> This is not accurate. RSS is used to direct NIC interrupts to
>> specific cores, often depending on the contents of header fields ...
>
>And which NIC supports that ?

Start with the ubiquitous Intel E-1000 and pretty much
every other Intel pro NIC.

https://www.intel.com/content/www/us/en/support/articles/000006703/ethernet-products.html

>
>> Most modern NICs support some form of RSS ...
>

Most modern server NICs support some form of RSS. Cheap-ass consumer
gear (realtek, etc) - you get what you pay for.

On Tuesday, July 4, 2023 at 1:50:34 PM UTC+1, Bonita Montero wrote:
>
> So there must sth. different that you're missing packets.

I figured out what's going on.

In my code, when an Ethernet frame is read from the network card, I check the checksum of the IP header, and also the checksum of the TCP segment. If either is wrong, the entire Ethernet frame is discarded.

In my code, I calculate what the checksum should be for the IP header, and I saw that in cases where the frame is discarded, the incoming packet had the 'total packet length' in the IP header as 44, but in my code I was calculating the checksum of 46 bytes. I didn't know where the other 2 bytes were coming from, whether they were 'options' in the TCP header or whatever, but then in Wireshark I saw that the packet had a mysterious "00 00" at the end. I selected those two bytes and Wireshark told me "Ethernet padding bytes".

Turns out that if an Ethernet frame is less than 60 bytes, then it's padded with zeroes to bring it to 60 bytes.

Ethernet header is 14 bytes, and the IP packet was 44 bytes, bringing the incoming frame to 58 bytes. So the operating system / network card tagged two zero bytes on the end.

I was debugging it for approx 8 hours before I figured out what was wrong. Now I'm lost losing any packets.

Still though I just set the 'priority' of my network reading thread to maximum.

On Tuesday, July 4, 2023 at 6:25:16 PM UTC+1, Frederick Virchanza Gotham wrote:

> I was debugging it for approx 8 hours before I figured out what was wrong.. Now I'm lost losing any packets.

Typo:

Now I'm ***not*** losing any packets.

On 7/3/2023 2:18 PM, Scott Lurndal wrote:
> "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> writes:
>> On 7/3/2023 3:42 AM, Frederick Virchanza Gotham wrote:
>>>
>>> I'm writing a network analysis program, and just today I've found out that it's too slow and missing packets.
>>>
>>> Ideally, I want the thread that reads from the network card to have exclusive use of one of the CPU cores.
>>>
>>> I'm looking at the following two functions:
>>> sched_setaffinity
>>> pthread_setaffinity_np
>>>
>>> And I note the following in the Linux manual:
>>> "For example, by dedicating one CPU to a particular process (i.e., setting the affinity mask of that process to specify a single CPU, and setting the affinity mask of all other processes to exclude that CPU), it is possible to ensure maximum execution speed for that process"
>>>
>>> I can find lots and lots of code examples to set the affinity for one thread or for one process -- however I cannot find one code example showing me how to exclude all other threads.
>>
>> Not sure you can do that. The OS can do whatever it wants. It can pin
>> your thread to a CPU, but that does not mean it cannot use said CPU for
>> other programs threads...
>
> Linux provides mechanisms that allow dedication of compute resources
> (e.g. hardware thread/core) to a specific set of processes. So do
> most other legacy Unix operating systems.
>
> The mechanisms vary based on the host operating system. cgroups is
> the current favored mechanism on linux.

Thanks Scott. Actually, I remember using affinity masks to try to
emulate per-cpu data in user mode as an experiment.

On 7/4/2023 11:30 AM, Chris M. Thomasson wrote:
> On 7/3/2023 2:18 PM, Scott Lurndal wrote:
>> "Chris M. Thomasson" <chris.m.thomasson.1@gmail.com> writes:
>>> On 7/3/2023 3:42 AM, Frederick Virchanza Gotham wrote:
>>>>
>>>> I'm writing a network analysis program, and just today I've found
>>>> out that it's too slow and missing packets.
>>>>
>>>> Ideally, I want the thread that reads from the network card to have
>>>> exclusive use of one of the CPU cores.
>>>>
>>>> I'm looking at the following two functions:
>>>>           sched_setaffinity
>>>>           pthread_setaffinity_np
>>>>
>>>> And I note the following in the Linux manual:
>>>>           "For example, by dedicating one CPU to a particular
>>>> process (i.e., setting the affinity mask of that process to specify
>>>> a single CPU, and setting the affinity mask of all other processes
>>>> to exclude that CPU), it is possible to ensure maximum execution
>>>> speed for that process"
>>>>
>>>> I can find lots and lots of code examples to set the affinity for
>>>> one thread or for one process -- however I cannot find one code
>>>> example showing me how to exclude all other threads.
>>>
>>> Not sure you can do that. The OS can do whatever it wants. It can pin
>>> your thread to a CPU, but that does not mean it cannot use said CPU for
>>> other programs threads...
>>
>> Linux provides mechanisms that allow dedication of compute resources
>> (e.g. hardware thread/core) to a specific set of processes.   So do
>> most other legacy Unix operating systems.
>>
>> The mechanisms vary based on the host operating system.  cgroups is
>> the current favored mechanism on linux.
>
> Thanks Scott. Actually, I remember using affinity masks to try to
> emulate per-cpu data in user mode as an experiment.

Iirc, I think I used the ALOM to check a cpu temperature when I pinned a
hot thread with a high priority to a CPU.

Am 04.07.2023 um 19:08 schrieb Scott Lurndal:
> Bonita Montero <Bonita.Montero@gmail.com> writes:
>> Am 04.07.2023 um 15:53 schrieb Scott Lurndal:
>>
>>> This is not accurate. RSS is used to direct NIC interrupts to
>>> specific cores, often depending on the contents of header fields ...
>>
>> And which NIC supports that ?
>
> Start with the ubiquitous Intel E-1000 and pretty much
> every other Intel pro NIC.

The NICs have Task offloading for IP- and TCP Checksum calculation.
Bu they for sure don't manage streams ! And interrupt routing isn't
done by the NIC.

> Most modern server NICs support some form of RSS.
> Cheap-ass consumer gear (realtek, etc) - you get what you pay for.

Linux doesn't use task offloading according to this:

https://en.wikipedia.org/wiki/TCP_offload_engine

Am 04.07.2023 um 19:25 schrieb Frederick Virchanza Gotham:
> On Tuesday, July 4, 2023 at 1:50:34 PM UTC+1, Bonita Montero wrote:
>>
>> So there must sth. different that you're missing packets.
>
>
> I figured out what's going on.
>
> In my code, when an Ethernet frame is read from the network card, I check the checksum of the IP header, and also the checksum of the TCP segment. If either is wrong, the entire Ethernet frame is discarded.
>
> In my code, I calculate what the checksum should be for the IP header, and I saw that in cases where the frame is discarded, the incoming packet had the 'total packet length' in the IP header as 44, but in my code I was calculating the checksum of 46 bytes. I didn't know where the other 2 bytes were coming from, whether they were 'options' in the TCP header or whatever, but then in Wireshark I saw that the packet had a mysterious "00 00" at the end. I selected those two bytes and Wireshark told me "Ethernet padding bytes".
>
> Turns out that if an Ethernet frame is less than 60 bytes, then it's padded with zeroes to bring it to 60 bytes.
>
> Ethernet header is 14 bytes, and the IP packet was 44 bytes, bringing the incoming frame to 58 bytes. So the operating system / network card tagged two zero bytes on the end.
>
> I was debugging it for approx 8 hours before I figured out what was wrong. Now I'm lost losing any packets.
>
> Still though I just set the 'priority' of my network reading thread to maximum.

Why do you do the checksum-stuff yourself ?

On Wednesday, July 5, 2023 at 4:28:22 AM UTC+1, Bonita Montero wrote:
>
> Why do you do the checksum-stuff yourself ?

My program opens a raw socket and does port scanning. I last released a version about 14 years ago: ( http://www.virjacode.com/projects/dynamo/ ), and I hope to release a new version with way more features this year.

On 04/07/2023 18:25, Frederick Virchanza Gotham wrote:
> On Tuesday, July 4, 2023 at 1:50:34 PM UTC+1, Bonita Montero wrote:
>>
>> So there must sth. different that you're missing packets.
>
>
> I figured out what's going on.
>
> In my code, when an Ethernet frame is read from the network card, I check the checksum of the IP header, and also the checksum of the TCP segment. If either is wrong, the entire Ethernet frame is discarded.
>
> In my code, I calculate what the checksum should be for the IP header, and I saw that in cases where the frame is discarded, the incoming packet had the 'total packet length' in the IP header as 44, but in my code I was calculating the checksum of 46 bytes. I didn't know where the other 2 bytes were coming from, whether they were 'options' in the TCP header or whatever, but then in Wireshark I saw that the packet had a mysterious "00 00" at the end. I selected those two bytes and Wireshark told me "Ethernet padding bytes".
>
> Turns out that if an Ethernet frame is less than 60 bytes, then it's padded with zeroes to bring it to 60 bytes.
>
> Ethernet header is 14 bytes, and the IP packet was 44 bytes, bringing the incoming frame to 58 bytes. So the operating system / network card tagged two zero bytes on the end.
>
> I was debugging it for approx 8 hours before I figured out what was wrong. Now I'm lost losing any packets.
>
> Still though I just set the 'priority' of my network reading thread to maximum.

Thanks for letting us know. You've also just shown me how out of date I
am in HW knowledge - last time I looked a device only had one interrupt
line :(

So in fact you never were losing packets at the HW level, you were
throwing them away because you thought they were bad!

Andy

Am 05.07.2023 um 10:54 schrieb Frederick Virchanza Gotham:

> My program opens a raw socket and does port scanning. I last released a version about 14 years ago: ( http://www.virjacode.com/projects/dynamo/ ), and I hope to release a new version with way more features this year.

I don't see much sense in that. If you scan a external
port range you need your own IP as the sender's address
to get replies. And if you do it that way you could use
normal sockets.

On Wednesday, July 5, 2023 at 12:43:19 PM UTC+1, Bonita Montero wrote:
> Am 05.07.2023 um 10:54 schrieb Frederick Virchanza Gotham:
>
> > My program opens a raw socket and does port scanning. I last released a version about 14 years ago: ( http://www.virjacode.com/projects/dynamo/ ), and I hope to release a new version with way more features this year.
> I don't see much sense in that. If you scan a external
> port range you need your own IP as the sender's address
> to get replies. And if you do it that way you could use
> normal sockets.

I suppose I could use normal sockets but I don't want the overhead of the operating system managing tens of thousands (yes, tens of thousands) of sockets and managing the 3-way handshakes for all those sockets. It's a lot more efficient to use a raw socket to send a byte-by-byte hand-crafted TCP SYN packet to each host, from port numbers 1 through 49151.

Bonita Montero <Bonita.Montero@gmail.com> writes:
>Am 04.07.2023 um 19:08 schrieb Scott Lurndal:
>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>> Am 04.07.2023 um 15:53 schrieb Scott Lurndal:
>>>
>>>> This is not accurate. RSS is used to direct NIC interrupts to
>>>> specific cores, often depending on the contents of header fields ...
>>>
>>> And which NIC supports that ?
>>
>> Start with the ubiquitous Intel E-1000 and pretty much
>> every other Intel pro NIC.
>
>The NICs have Task offloading for IP- and TCP Checksum calculation.
>Bu they for sure don't manage streams ! And interrupt routing isn't
>done by the NIC.

Your ignorance is showing. My company makes NICs. Very powerful
NICs. Which manage streams (the programmable TCAMs allow software
to select the packet fields that get CAMd when selecting the target CPU
for the interrupt).

Most use PCI/PCIe MSI-X interrupts, which have a 32-bit payload field
allowing (depending on the host interrupt controller) up to
2^32 different interrupt vectors. Our system support 21 bit
payloads, supporting 2^21 physical interrupts _per_ virtual device
using the ARM GIC-700 interrupt controller.

Using SR-IOV allows direct shared, but mediated, access to the NIC
from guest operating systems.

>
>> Most modern server NICs support some form of RSS.
>> Cheap-ass consumer gear (realtek, etc) - you get what you pay for.
>
>Linux doesn't use task offloading according to this:
>
>https://en.wikipedia.org/wiki/TCP_offload_engine

That's also incorrect. Take a look at DPDK or ODP.

Vir Campestris <vir.campestris@invalid.invalid> writes:
>On 04/07/2023 18:25, Frederick Virchanza Gotham wrote:
>> On Tuesday, July 4, 2023 at 1:50:34 PM UTC+1, Bonita Montero wrote:
>>>
>>> So there must sth. different that you're missing packets.
>>
>>
>> I figured out what's going on.
>>
>> In my code, when an Ethernet frame is read from the network card, I check the checksum of the IP header, and also the checksum of the TCP segment. If either is wrong, the entire Ethernet frame is discarded.
>>
>> In my code, I calculate what the checksum should be for the IP header, and I saw that in cases where the frame is discarded, the incoming packet had the 'total packet length' in the IP header as 44, but in my code I was calculating the checksum of 46 bytes. I didn't know where the other 2 bytes were coming from, whether they were 'options' in the TCP header or whatever, but then in Wireshark I saw that the packet had a mysterious "00 00" at the end. I selected those two bytes and Wireshark told me "Ethernet padding bytes".
>>
>> Turns out that if an Ethernet frame is less than 60 bytes, then it's padded with zeroes to bring it to 60 bytes.
>>
>> Ethernet header is 14 bytes, and the IP packet was 44 bytes, bringing the incoming frame to 58 bytes. So the operating system / network card tagged two zero bytes on the end.
>>
>> I was debugging it for approx 8 hours before I figured out what was wrong. Now I'm lost losing any packets.
>>
>> Still though I just set the 'priority' of my network reading thread to maximum.
>
>Thanks for letting us know. You've also just shown me how out of date I
>am in HW knowledge - last time I looked a device only had one interrupt
>line :(

Most modern high throughput devices use message signaled interrupts (MSI or MSI-X); MSI-X
has a 32-bit payload field, allowing up to four billion distinct interrupt
"lines". With interrupt virtualization, each guest operating system has
its own 32-bit interrupt namespace, allowing an infinite number of interrupt
"lines". The days of sharing level sensitive interrupts are long gone,
except for supporting legacy hardware.

>
>So in fact you never were losing packets at the HW level, you were
>throwing them away because you thought they were bad!

Or RED kicked in.

https://en.wikipedia.org/wiki/Random_early_detection

Am 05.07.2023 um 15:28 schrieb Frederick Virchanza Gotham:

> I suppose I could use normal sockets but I don't want the overhead of the operating system managing tens of thousands (yes, tens of thousands) of sockets and managing the 3-way handshakes for all those sockets. ...

These information is hashed and the time for the lookup is O(1).

On Wednesday, July 5, 2023 at 4:05:24 PM UTC+1, Bonita Montero wrote:
> Am 05.07.2023 um 15:28 schrieb Frederick Virchanza Gotham:
>
> > I suppose I could use normal sockets but I don't want the overhead of the operating system managing tens of thousands (yes, tens of thousands) of sockets and managing the 3-way handshakes for all those sockets. ...
>
> These information is hashed and the time for the lookup is O(1).

Yeah but we're talking 40,000 file descriptors for one process.

Am 05.07.2023 um 15:49 schrieb Scott Lurndal:

> Your ignorance is showing. My company makes NICs. Very powerful
> NICs. Which manage streams (the programmable TCAMs allow software
> to select the packet fields that get CAMd when selecting the target
> CPU for the interrupt).

Show me the documentation.

Am 06.07.2023 um 00:04 schrieb Frederick Virchanza Gotham:
> On Wednesday, July 5, 2023 at 4:05:24 PM UTC+1, Bonita Montero wrote:
>> Am 05.07.2023 um 15:28 schrieb Frederick Virchanza Gotham:
>>
>>> I suppose I could use normal sockets but I don't want the overhead of the operating system managing tens of thousands (yes, tens of thousands) of sockets and managing the 3-way handshakes for all those sockets. ...
>>
>> These information is hashed and the time for the lookup is O(1).
>
>
> Yeah but we're talking 40,000 file descriptors for one process.

Even with a billion file descriptors the access time would
be the same, i.e. O(1).