./Shade.sh

It seems its hard to get this nice piece of Hardware to utilize its two 10gbit SFP+ Links.
The CPU, a AMD V1500B, isn’t the fastest in comparison to todays embedded Processors on the Market.

After a lot reading through the OPNsense Forums and several FreeBSD Blogs / Bug Tracker, i maybe found the ideal Tuning Parameters to squeeze the most out of this tiny fanless Box.

It looks like, the most important things here are:

RSS with 3 Bits (ReceiveSideScaling)
machdep.hyperthreading_intr_allowed

Alongside other important Tuning Parameter, this did the most positive Impact and distributed finaly Interrupt Workload across all Cores (4 physical and 4 HTT Cores). Before only 4 Cores are utilized (you can check that with “top -P”), now all 8 Cores are part of Interrupt processing. This gave me additional 2Gbit/s so i can now squeeze around of max 7,5gbit/s with NAT / Routing out of this Box. Let me clarify here, i don’t use Zenarmor or Suricata on this box! With those IDS/IPS enabled it should be clear that throughput would be much lower, because this box is simply not well enough equipped.

The OPNSense DEC Hardware have Default Configs from Deciso, i’m not going to discuss this Parameter here. If you reinstalled your Box, you should import these Default Configs before adding my Parameter.

Here are all of my additional tunings:

hw.ibrs_disable=1
kern.ipc.maxsockbuf=16777216
machdep.hyperthreading_intr_allowed=1
net.inet.rss.bits=3
net.inet.rss.enabled=1
net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.soreceive_stream=1
net.isr.bindthreads=1
net.isr.defaultqlimit=2048
net.isr.dispatch=deferred
net.isr.maxthreads=-1

And here is the result with a internet IPerf3 Test

[SUM] 0.00-30.00 sec 25.0 GBytes 7.16 Gbits/sec 15958 sender
[SUM] 0.00-30.00 sec 24.9 GBytes 7.13 Gbits/sec receiver

CPU 0: 12.9% user,  0.0% nice, 17.3% system, 67.5% interrupt,  2.4% idle
CPU 1:  0.0% user,  0.0% nice, 56.1% system, 43.9% interrupt,  0.0% idle
CPU 2:  0.0% user,  0.0% nice, 78.8% system, 21.2% interrupt,  0.0% idle
CPU 3:  0.4% user,  0.0% nice, 70.2% system, 29.0% interrupt,  0.4% idle
CPU 4: 25.5% user,  0.0% nice, 16.9% system, 53.3% interrupt,  4.3% idle
CPU 5:  5.9% user,  0.0% nice,  3.9% system, 87.8% interrupt,  2.4% idle
CPU 6:  9.0% user,  0.0% nice, 36.1% system, 54.1% interrupt,  0.8% idle
CPU 7: 11.0% user,  0.0% nice, 12.5% system, 73.7% interrupt,  2.7% idle