![]() ![]() 500/5000 with two separate companies – Rapidlink and Advocate (11 consecutive years). He graduated from Georgia Institute of Technology in 1987 and holds a Bachelor’s degree in Electrical Engineering with Honors. After working with IT leaders at over 600 enterprises to assess over $50 Billion in spend, Advocate knows how to uncover more savings to invest, enable more business outcomes and create more influence overall to do what matters more. The firm would soon emerge as a leader in TEM … and then network optimization … and ultimately become a premier TBM consulting and services company. In 2001, Tim Wise and Scott Fogle founded Advocate to help enterprises create a clear strategy for structuring their growing internet-connected systems. In 2000, Scott started Dantis, a web hosting company, and was Chief Operating Officer. He met Tim Wise at RapidLink Telecommunications in 1998, as they began their entrepreneurial pursuits. Scott gained his insider knowledge of products, sales and professional services at leading companies like NCR, AT&T and Scientific-Atlanta. Scott co-leads Accenture’s Technology Value capability group in North America and continues to co-lead Advocate. In two previous posts we've discussed how to receive 1M UDP packets per second and how to reduce the round trip time.Scott Fogle is a Managing Director in Accenture’s Technology Strategy & Advisory group and Co-founder of Advocate, an Information Technology Financial Management (ITFM) consultancy acquired by Accenture in June 2022. We did the experiments on Linux and the performance was very good considering it's a general purpose operating system. Unfortunately the speed of vanilla Linux kernel networking is not sufficient for more specialized workloads. For example, here at CloudFlare, we are constantly dealing with large packet floods. This is not enough in our environment, especially since the network cards are capable of handling a much higher throughput. Modern 10Gbps NIC's can usually process at least 10M pps. It's apparent that the only way to squeeze more packets from our hardware is by working around the Linux kernel networking stack. This is called a "kernel bypass" and in this article we'll dig into various ways of achieving it. Let's prepare a small experiment to convince you that working around Linux is indeed necessary. ![]() Let's see how many packets can be handled by the kernel under perfect conditions. Passing packets to userspace is costly, so instead let's try to drop them as soon as they leave the network driver code. To my knowledge the fastest way to drop packets in Linux, without hacking the kernel sources, is by placing a DROP rule in the PREROUTING iptables chain: $ sudo iptables -t raw -I PREROUTING -p udp -dport 4321 -dst 192.168.254.1 -j DROPĮthtool statistics above show that the network card receives a line rate of 12M packets per second. By manipulating an indirection table on a NIC with ethtool -X, we direct all the packets to RX queue #0. As we can see the kernel is able to process 1.4M pps on that queue with a single CPU. Processing 1.4M pps on a single core is certainly a very good result, but unfortunately the stack doesn't scale. Let's see the numbers when we direct packets to four RX queues: $ sudo ethtool -X eth2 weight 1 1 1 1 When the packets hit many cores the numbers drop sharply. Even optimistically assuming the performance won't drop further when adding more cores, we would still need more than 20 CPU's to handle packets at line rate. The performance limitations of the Linux kernel network are nothing new. Over the years there had been many attempts to address them. ![]() The most common techniques involve creating specialized API's to aid with receiving packets from the hardware at very high speed. Unfortunately these techniques are in total flux and a single widely adopted approach hasn't emerged yet. Here is a list of the most widely known kernel bypass techniques. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |