Re: 1gbit LAN/NIC performance, queue speed bug?

> On Tuesday 16 November 2010, Robert Lewandowski wrote: >> Hello, >> >> PROBLEM: transfer speed is ONLY HALF if queue is defined in pf.conf >> although queue is 950Mbit (1000Mbit-5%) >> pf disabled: 768 Mbits/sec >> pf enabled, queue 950Mbit: 337 Mbits/sec >> >> ANALYSIS: >> >> - OpenBSD 4.8 default intallation. >> - Test made between OpenBSD 4.8 and Debian Linux. >> (between two Debian systems speed is more than 900Mbit/s) >> >> ********************************************************* >> LAN interface: Intel PRO/1000 PT Desktop Adapter (PCIe, model: >> EXPI9300PTBLK) >> DMESG: em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev >> 0x06: apic 1 int 16 (irq 5), address 00:1b:21:05:1f:39 >> ********************************************************* >> Default settings of TCP window size: >> net.inet.tcp.recvspace=16384 >> net.inet.tcp.sendspace=16384 >> ********************************************************* >> >> 1a) pf disabled >> >> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6 >> ------------------------------------------------------------ >> Client connecting to 10.0.0.6, TCP port 5001 >> TCP window size: 16.0 KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.0.0.8 port 27600 connected with 10.0.0.6 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0- 1.0 sec 54.7 MBytes 459 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 1.0- 2.0 sec 54.7 MBytes 458 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 2.0- 3.0 sec 54.7 MBytes 459 Mbits/sec >> >> 1b) pf enabled, no queue >> >> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6 >> ------------------------------------------------------------ >> Client connecting to 10.0.0.6, TCP port 5001 >> TCP window size: 16.0 KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.0.0.8 port 46912 connected with 10.0.0.6 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0- 1.0 sec 53.9 MBytes 452 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 1.0- 2.0 sec 52.6 MBytes 441 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 2.0- 3.0 sec 54.1 MBytes 454 Mbits/sec >> >> 1c) pf enabled, added queue to default pf.conf: >> >> altq on em0 cbq bandwidth 1Gb queue { q_lan } >> queue q_lan bandwidth 950Mb cbq (default) >> >> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6 >> ------------------------------------------------------------ >> Client connecting to 10.0.0.6, TCP port 5001 >> TCP window size: 16.0 KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.0.0.8 port 38266 connected with 10.0.0.6 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0- 1.0 sec 33.9 MBytes 284 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 1.0- 2.0 sec 35.0 MBytes 294 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 2.0- 3.0 sec 35.8 MBytes 300 Mbits/sec >> >> >> ********************************************************* >> TCP window size changed to 131072. >> net.inet.tcp.recvspace: 16384 -> 131072 >> net.inet.tcp.sendspace: 16384 -> 131072 >> ********************************************************* >> >> 1a) pf disabled >> >> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6 >> ------------------------------------------------------------ >> Client connecting to 10.0.0.6, TCP port 5001 >> TCP window size: 128 KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.0.0.8 port 32680 connected with 10.0.0.6 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0- 1.0 sec 91.5 MBytes 768 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 1.0- 2.0 sec 92.1 MBytes 773 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 2.0- 3.0 sec 91.2 MBytes 765 Mbits/sec >> >> 1b) pf enabled, no queue >> >> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6 >> ------------------------------------------------------------ >> Client connecting to 10.0.0.6, TCP port 5001 >> TCP window size: 128 KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.0.0.8 port 41092 connected with 10.0.0.6 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0- 1.0 sec 80.5 MBytes 675 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 1.0- 2.0 sec 80.1 MBytes 672 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 2.0- 3.0 sec 80.2 MBytes 673 Mbits/sec >> >> 1c) pf enabled, added queue to default pf.conf: >> >> altq on em0 cbq bandwidth 1Gb queue { q_lan } >> queue q_lan bandwidth 950Mb cbq (default) >> >> root@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6 >> ------------------------------------------------------------ >> Client connecting to 10.0.0.6, TCP port 5001 >> TCP window size: 128 KByte (default) >> ------------------------------------------------------------ >> [ 3] local 10.0.0.8 port 12499 connected with 10.0.0.6 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0- 1.0 sec 40.1 MBytes 337 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 1.0- 2.0 sec 40.1 MBytes 336 Mbits/sec >> [ ID] Interval Transfer Bandwidth >> [ 3] 2.0- 3.0 sec 40.0 MBytes 335 Mbits/sec >> >> ********************************************************* >> >> any ideas, suggestions about this situation? > > The default length for a queue is 50 packets - this only allows you to queue > around 75,000 bytes and the burstiness of TCP slow-start is likely to well > exceed this in your configuration (due to the BDP). I'd suggest increasing > the queue length - also run 'pfctl -vvs queue' or 'systat queue' and see > what's happening with regards to packets drops.