performance tuning nginx...proxy 2 24 2 intel(r) xeon(r) platinum ... linux sysctl settings 26 •...

Performance Tuning NGINX

Name: Amir Rawdat

Currently: Technical Marketing Engineer at NGINX inc.

Previously: - Customer Applications Engineer at Nokia inc.

Multi-Process Architecture with QPI Bus

Web Server Topology

wrk nginx

Reverse Proxy Topology

�6

wrk nginx nginx

Technical Specifications

# Sockets # Cores per Socket

# Threads per Core

Model Name

RAM OS NIC

Client 2 22 2 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

128 GB Ubuntu Xenial

40GbE QSFP+

Web Server & Reverse Proxy

2 24 2 Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz

192 GB Ubuntu Xenial

40GbE QSFP+

Multi-Processor Architecture

Duplicate NGINX Configurations

#1

�9

Multi-Processor Architecture

NGINX Configuration (Instance 1)

�11

user root; worker_processes 48 ; worker_cpu_affinity auto

000000000000000000000000111111111111111111111111000000000000000000000000111111111111111111111111; worker_rlimit_nofile 1024000; error_log  /home/ubuntu/access.error error;

….. …….

NGINX Configuration (Instance 2)

�12

user root; worker_processes 48 ; worker_cpu_affinity auto

111111111111111111111111000000000000000000000000111111111111111111111111000000000000000000000000; worker_rlimit_nofile 1024000; error_log  /home/ubuntu/access.error error;

……. …….

Deploying NGINX Instances

�13

$ nginx –c /path/to/configuration/instance-1 $ nginx –c /path/to/configuration/instance-2 $ ps aux | grep nginx

nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx_0.conf nginx: worker process nginx: worker process

nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx_1.conf nginx: worker process nginx: worker process

$ pkill nginx

Additional NGINX Configuration Directives

#2

�14

Web Server (Instance 1)

events {    worker_connections 1000000; } http {      access_log off;      keepalive_timeout  315;      keepalive_requests 10000000;            sendfile       on;      tcp_nopush     on;      tcp_nodelay    on;

server {        listen 10.10.16.10:443 backlog=250000 reuseport;        root /usr/share/nginx/bin;  } }

Web Server (Instance 2)

�16

events {    worker_connections 1000000; } http {      access_log off;      keepalive_timeout  315;      keepalive_requests 10000000;            sendfile       on;      tcp_nopush     on;      tcp_nodelay    on;

server {        listen 10.10.11.23:443 backlog=250000 reuseport;        root /usr/share/nginx/bin;  } }

Reverse Proxy (Instance 1)

�17

server {   listen 10.10.10.18:443 ssl backlog=102400 reuseport;        ssl_certificate /etc/ssl/certs/nginx.pem;        ssl_certificate_key /etc/ssl/private/nginx.key;        ssl_session_cache off;        ssl_session_tickets off;       location / {         proxy_http_version 1.1;         proxy_set_header Connection "";         proxy_pass http://webserver_0;     }  }    upstream webserver_0 {        server 10.10.10.11:80;        keepalive 200;  } }

Reverse Proxy (Instance 2)

�18

server {   listen 10.10.15.9:443 ssl backlog=102400 reuseport;        ssl_certificate /etc/ssl/certs/nginx.pem;        ssl_certificate_key /etc/ssl/private/nginx.key;        ssl_session_cache off;        ssl_session_tickets off;       location / {         proxy_http_version 1.1;         proxy_set_header Connection "";         proxy_pass http://webserver_1;     }  }    upstream webserver_1 {      server 10.10.15.12:80;     keepalive 200;  } }

Performance Test Results

�19

Web Server (Instance 1)

events {    worker_connections 1000000; } http {      access_log off;      keepalive_timeout  315;      keepalive_requests 10000000;            sendfile       on;      tcp_nopush     on;      tcp_nodelay    on;

server {        listen 10.10.16.10:443 backlog=250000 reuseport;        root /usr/share/nginx/bin;  } }

Web Server (Instance 2)

�21

events {    worker_connections 1000000; } http {      access_log off;      keepalive_timeout  315;      keepalive_requests 10000000;            sendfile       on;      tcp_nopush     on;      tcp_nodelay    on;

server {        listen 10.10.11.23:443 backlog=250000 reuseport;        root /usr/share/nginx/bin2;  } }

Performance Test Results

�22

Performance Test Results

�23

Performance Tip: Linux Sysctl Settings

#3

�24

Linux Sysctl Settings

�25

• Increase memory thresholds to prevent packet dropping ◦ sysctl -w net.ipv4.tcp_rmem=”4096 87380 4194304” ◦ sysctl -w net.ipv4.tcp_wmem=”4096 65536 4194304”

• Increase the size of the processor queues ◦ sysctl -w net.core.net_dev_max_backlog=250000

• Setting the maximum TCP buffer sizes ◦ sysctl -w net.core.rmem_max=4194304 ◦ sysctl -w net.core.wmem_max=4194304

Linux Sysctl Settings

�26

• Disable TCP timestamps ◦ sysctl -w net.ipv4.tcp_timestamps=0

• Defines the local port range that is used by TCP and UDP to choose the local port ◦ sysctl -w net.ipv4.ip_local_port_range = 32768 60999

• Enable reuse of TIME-WAIT sockets for new connections when it is safe from protocol viewpoint. ◦ sysctl -w net.ipv4.tcp_tw_reuse = 1

Linux Sysctl Settings

�27

Performance Tip: Enabling RSS and TPS

4

�28

Enabling RSS and TPS

�29

• /etc/init.d/irq_balance stop • git clone https://github.com/ANLAB-KAIST/mlnx-en.git • cd /mlnx-en/ofed-scripts • ./set_irq_affinity_bynode.sh <numa node id> <interface-name> • set_irq_affinity -x local <interface-name>

Enabling RSS and TPS

�30

Enabling RSS and TPS

�31

Performance Test Results

�32

Performance Numbers with NGINX and Intel QuickAssist

�33

Performance with QuickAssist

Performance with QuickAssist

�35

Summary

�36

• Deploy two nginx instances • Using additional nginx configuration directives • Linux sysctl parameter tuning • Setting IRQ affinity using RSS and TPS • Consult appendix for additional information and

performance tips

Contact information here

Thank you

amir.rawdat@nginx.com

Appendix7

Client Traffic Script

taskset -c 0-21,44-65 wrk -t 44 -c 1000 -d 180s -H 'Connection: Close' https://10.10.16.10:443/$1 >> output.txt &

taskset -c 22-43,66-87 wrk -t 44 -c 1000 -d 180s -H 'Connection: Close' https://10.10.11.23:443/$1 >> output1.txt &

## $1 is the requested static file size

Architectural Specifications

Architectural Specifications

�41

Technical Specifications

# Sockets

# Cores per Socket

# Threads per Core

Model Name

RAM OS NIC

nbdw32 2 22 2 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

128 GB Ubuntu Xenial

40GbE QSFP+

Web Server

2 24 2 Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz

192 GB Ubuntu Xenial

40GbE QSFP+

Technical Specifications

# Sockets # Cores per Socket

# Threads per Core

Model Name

RAM OS NIC

Client 2 22 2 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

128 GB Ubuntu Xenial

40GbE QSFP+

Reverse Proxy

2 24 2 Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz

192 GB Ubuntu Xenial

40GbE QSFP+

Web Server 2 22 2 Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

128 GB Ubuntu Xenial

40GbE QSFP+

Reverse proxy (Instance 1)

�44

user root; worker_processes 48 ; worker_cpu_affinity auto

000000000000000000000000111111111111111111111111000000000000000000000000111111111111111111111111; worker_rlimit_nofile 1024000; error_log  /home/ubuntu/access.error crit; events {    worker_connections 1000000; }

http {      access_log off;      keepalive_timeout  315;      keepalive_requests 10000000;            sendfile       on;      tcp_nopush     on;      tcp_nodelay    on;

Reverse proxy (Instance 1)

�45

server {   listen 10.10.10.18:443 ssl backlog=102400 reuseport;        ssl_certificate /etc/ssl/certs/nginx.pem;        ssl_certificate_key /etc/ssl/private/nginx.key;        ssl_session_cache off;        ssl_session_tickets off;       location / {         proxy_http_version 1.1;         proxy_set_header Connection "";         proxy_pass http://webserver_0;     }  }    upstream webserver_0 {        server 10.10.10.11:80;        keepalive 200;  } }

Reverse proxy (Instance 2)

�46

user root; worker_processes 48 ; worker_cpu_affinity auto

111111111111111111111111000000000000000000000000111111111111111111111111000000000000000000000000; worker_rlimit_nofile 1024000; error_log  /home/ubuntu/access.error crit; events {    worker_connections 1000000; }

http {      access_log off;      keepalive_timeout  315;      keepalive_requests 10000000;            sendfile       on;      tcp_nopush     on;      tcp_nodelay    on;

Reverse proxy (Instance 2)

�47

server {   listen 10.10.15.9:443 ssl backlog=102400 reuseport;        ssl_certificate /etc/ssl/certs/nginx.pem;        ssl_certificate_key /etc/ssl/private/nginx.key;        ssl_session_cache off;        ssl_session_tickets off;       location / {         proxy_http_version 1.1;         proxy_set_header Connection "";         proxy_pass http://webserver_1;     }  }    upstream webserver_1 {      server 10.10.15.12:80;     keepalive 200;  } }