Categories
Linux Network

Test Unix socket

This is a small test/benchmark to se if the use of a unix domain socket in Nginx would affect performance in any significant way for an api

The api is hosted by .NET Core and containerized. Test is done by using apache benchmark. doing 100 concurrent request 50 times. The request that is done has a payload of 1,7 KB.

ab -n 5000 -c 100 "http://localhost/v3/api/"

Result

This is the average request per second after 10 tests against each configuration.

Requests per second
HTTP100
HTTP (Keepalive enabled)118
Unix Socket98
Unix Socket (Keepalive enabled)120

Keepalive

Nginx option keepalive has an impact on both http and unix sockets. It looks to be far more important than the actual connection type.

upstream backend {
        keepalive 100;
        server 10.265.44.3:8080;
}

Categories
Linux Network

Start NRPE after openvpn tunnel is connected

NRPE will not start if its server address is a openvpn ip and the tunnel is not yet established when NRPE tries to start. to solve this. create a new systemd file

systemctl -all | grep ovpn
cp /usr/lib/systemd/system/nrpe.service /etc/systemd/system/nrpe.service

Add your tun device to Requires and After. Note you need the systemD name of your tune device. in this case. OpenVPN was configured to use tun ovpn-gwSamuel check systemctl for its correct name. The result should look something like this. Notice the esacped dash sign in the name.

[Unit]
Description=Nagios Remote Program Executor
Documentation=http://www.nagios.org/documentation
Conflicts=nrpe.socket
Requires=network.target sys-devices-virtual-net-ovpn\x2dgwSamuel.device
After=network-online.target sys-devices-virtual-net-ovpn\x2dgwSamuel.device

[Install]
WantedBy=multi-user.target

[Service]
Type=forking
User=nrpe
Group=nrpe
EnvironmentFile=/etc/sysconfig/nrpe
ExecStart=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d $NRPE_SSL_OPT
Categories
Linux Network

Extending snmpd with sensor data from ipmitool

Create script

Create a script that returns the data you want to monitor through snmp. the output should be cleaned from whitespaces.

My examples monitor sensor data from ipmi by creating cron script that outputs ipmitool sensor data to  a temp file every two minutes. And then parsing it with snmp_ipmi from github. Cron script below.

# .--------------------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
 */2 * * * * root ipmitool sensor list > /tmp/ipmi_sensor_list

Add script to snmpd

Edit /etc/snmp/snmpd.conf and add your scripts

rouser OP5View

extend "System Fan 1" /usr/lib64/nagios/plugins/custom/ipmi/snmp_ipmi -f /tmp/ipmi_sensor_list -u "System Fan 1"
extend "System Fan 3" /usr/lib64/nagios/plugins/custom/ipmi/snmp_ipmi -f /tmp/ipmi_sensor_list -u "System Fan 3"
extend "System Fan 4" /usr/lib64/nagios/plugins/custom/ipmi/snmp_ipmi -f /tmp/ipmi_sensor_list -u "System Fan 4"

Test

restart snmpd and run the following command to see your snmp data. example output below.

# snmpwalk -u OP5View -A password -a SHA -X encryptionkey -x AES -l authPriv -v3 localhost .1.3.6.1.4.1.8072.1.3
NET-SNMP-EXTEND-MIB::nsExtendNumEntries.0 = INTEGER: 4
NET-SNMP-EXTEND-MIB::nsExtendCommand."test" = STRING: /bin/echo
NET-SNMP-EXTEND-MIB::nsExtendCommand."System Fan 1" = STRING: /usr/lib64/nagios/plugins/custom/ipmi/snmp_ipmi
NET-SNMP-EXTEND-MIB::nsExtendCommand."System Fan 3" = STRING: /usr/lib64/nagios/plugins/custom/ipmi/snmp_ipmi
NET-SNMP-EXTEND-MIB::nsExtendCommand."System Fan 4" = STRING: /usr/lib64/nagios/plugins/custom/ipmi/snmp_ipmi
NET-SNMP-EXTEND-MIB::nsExtendArgs."test" = STRING: hello
NET-SNMP-EXTEND-MIB::nsExtendArgs."System Fan 1" = STRING: -f /tmp/ipmi_sensor_list -u \"System Fan 1\"
NET-SNMP-EXTEND-MIB::nsExtendArgs."System Fan 3" = STRING: -f /tmp/ipmi_sensor_list -u \"System Fan 3\"
NET-SNMP-EXTEND-MIB::nsExtendArgs."System Fan 4" = STRING: -f /tmp/ipmi_sensor_list -u \"System Fan 4\"
NET-SNMP-EXTEND-MIB::nsExtendInput."test" = STRING:
NET-SNMP-EXTEND-MIB::nsExtendInput."System Fan 1" = STRING:
NET-SNMP-EXTEND-MIB::nsExtendInput."System Fan 3" = STRING:
NET-SNMP-EXTEND-MIB::nsExtendInput."System Fan 4" = STRING:
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."test" = INTEGER: 5
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."System Fan 1" = INTEGER: 5
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."System Fan 3" = INTEGER: 5
NET-SNMP-EXTEND-MIB::nsExtendCacheTime."System Fan 4" = INTEGER: 5
NET-SNMP-EXTEND-MIB::nsExtendExecType."test" = INTEGER: exec(1)
NET-SNMP-EXTEND-MIB::nsExtendExecType."System Fan 1" = INTEGER: exec(1)
NET-SNMP-EXTEND-MIB::nsExtendExecType."System Fan 3" = INTEGER: exec(1)
NET-SNMP-EXTEND-MIB::nsExtendExecType."System Fan 4" = INTEGER: exec(1)
NET-SNMP-EXTEND-MIB::nsExtendRunType."test" = INTEGER: run-on-read(1)
NET-SNMP-EXTEND-MIB::nsExtendRunType."System Fan 1" = INTEGER: run-on-read(1)
NET-SNMP-EXTEND-MIB::nsExtendRunType."System Fan 3" = INTEGER: run-on-read(1)
NET-SNMP-EXTEND-MIB::nsExtendRunType."System Fan 4" = INTEGER: run-on-read(1)
NET-SNMP-EXTEND-MIB::nsExtendStorage."test" = INTEGER: permanent(4)
NET-SNMP-EXTEND-MIB::nsExtendStorage."System Fan 1" = INTEGER: permanent(4)
NET-SNMP-EXTEND-MIB::nsExtendStorage."System Fan 3" = INTEGER: permanent(4)
NET-SNMP-EXTEND-MIB::nsExtendStorage."System Fan 4" = INTEGER: permanent(4)
NET-SNMP-EXTEND-MIB::nsExtendStatus."test" = INTEGER: active(1)
NET-SNMP-EXTEND-MIB::nsExtendStatus."System Fan 1" = INTEGER: active(1)
NET-SNMP-EXTEND-MIB::nsExtendStatus."System Fan 3" = INTEGER: active(1)
NET-SNMP-EXTEND-MIB::nsExtendStatus."System Fan 4" = INTEGER: active(1)
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."test" = STRING: hello
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."System Fan 1" = STRING: 1274.000
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."System Fan 3" = STRING: 1372.000
NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."System Fan 4" = STRING: 1372.000
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."test" = STRING: hello
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."System Fan 1" = STRING: 1274.000
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."System Fan 3" = STRING: 1372.000
NET-SNMP-EXTEND-MIB::nsExtendOutputFull."System Fan 4" = STRING: 1372.000
NET-SNMP-EXTEND-MIB::nsExtendOutNumLines."test" = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendOutNumLines."System Fan 1" = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendOutNumLines."System Fan 3" = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendOutNumLines."System Fan 4" = INTEGER: 1
NET-SNMP-EXTEND-MIB::nsExtendResult."test" = INTEGER: 0
NET-SNMP-EXTEND-MIB::nsExtendResult."System Fan 1" = INTEGER: 0
NET-SNMP-EXTEND-MIB::nsExtendResult."System Fan 3" = INTEGER: 0
NET-SNMP-EXTEND-MIB::nsExtendResult."System Fan 4" = INTEGER: 0
NET-SNMP-EXTEND-MIB::nsExtendOutLine."test".1 = STRING: hello
NET-SNMP-EXTEND-MIB::nsExtendOutLine."System Fan 1".1 = STRING: 1274.000
NET-SNMP-EXTEND-MIB::nsExtendOutLine."System Fan 3".1 = STRING: 1372.000
NET-SNMP-EXTEND-MIB::nsExtendOutLine."System Fan 4".1 = STRING: 1372.000

Get OID

Get you oid so you can search from another computer without MIB

snmptranslate -On 'NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."System Fan 1"'

Nagios / OP5 Test

Setup a snmp test in Nagios/OP5 using check_snmp. because the output from the script is strings check_snmp has to do string conversion to integer, which it can if you configure it correctly. The example below does one check_snmp request it returns the value for all 3 fans we configured above. if any of the 3 Fans RPM value is outside of 1300-1500 rpm a warning is generated, if the rpm value is outside of 1000-2000 a critical is generated.

_USER1_/check_snmp -H "192.168.80.11" -o "".1.3.6.1.4.1.8072.1.3.2.3.1.1.12.83.121.115.116.101.109.32.70.97.110.32.51,.1.3.6.1.4.1.8072.1.3.2.3.1.1.12.83.121.115.116.101.109.32.70.97.110.32.49,.1.3.6.1.4.1.8072.1.3.2.3.1.1.12.83.121.115.116.101.109.32.70.97.110.32.52"" -P 3 -U "OP5View " -a sha -A "password " -x aes -X "encryptionkey " -L authPriv -w 1300:1500,1300:1500,1300:1500 -c 1000:2000,1000:2000,1000:2000 --label=FAN1,FAN3,FAN4
Result code: OK
SNMP OK - FAN1 1453 FAN3 1453 FAN4 1453 | FAN1=1453 FAN3=1453 FAN4=1453

How it looks in OP5

Categories
Network

Enable IPMI from Linux

Install ipmitool
yum install ipmitool -y
Show Firmware
ipmitool mc info
Show Network
ipmitool lan print 1
Configure first network card (usally default) with static iP-number
ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr 192.168.61.119
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.61.1
Configure second network with DHCP
ipmitool lan set 2 ipsrc dhcp
Set SNMP community
ipmitool lan set 1 snmp dalesjo
List users
ipmitool user list 1
Change password on second user in list
ipmitool user set password 2
List Sensors
ipmitool sdr list
Restart IPMI
ipmitool mc reset
Sources

http://www.openfusion.net/linux/ipmi_on_centos

https://discuss.pivotal.io/hc/en-us/articles/206396927-How-to-work-on-IPMI-and-IPMITOOL

Categories
Network

Run letsencrypt as a normal user

Install and change permission to work with your normal user account.

yum install yum-utils -y
yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
yum install certbot -y

useradd certmaster
chown certmaster: /etc/letsencrypt/ -R
chown certmaster: /var/log/letsencrypt/ -R
chown certmaster: /var/lib/letsencrypt/ -R

You might want to allow your certmaster account to reload nginx or another service then a new certificate exists. To allow certmaster to reload nginx configuration, create sudoers file /etc/sudoers.d/certmaster with this content.

certmaster ALL=NOPASSWD: /usr/bin/systemctl reload nginx.service
certmaster ALL=NOPASSWD: /usr/sbin/nginx -t -c /etc/nginx/nginx.conf

Example of cron script to run.

# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
# Updates all certificates on saturday morning, checks nginx config and reloads
15 2 * * * certmaster chronic /usr/bin/certbot --no-self-upgrade renew && sudo /usr/sbin/nginx -t -c /etc/nginx/nginx.conf && sudo /usr/bin/systemctl reload nginx.service
Categories
Network

Encrypt private key OpenVPN

You can password protect your key file so that ed an extra password is needed to connect with openvpn. It will also work with embedded certificates in .ovpn

OpenVPN asking for a password

Categories
Network

Round-Robin webservers with letsencrypt

Nginx on Edge servers

Configure your edgeservers to redirect all http challanges to your main webserver.

Categories
Network

Embedding certificates in .ovpn

You can simplify OpenVPN distribution by only use one file for both config and certificates. A normal .ovpn file with separate .key and .crt files looks like this.

client dev tun 
dev-node Dalesjo VPN 
proto tcp 
remote my-server 443 
resolv-retry infinite 
nobind 
persist-key 
persist-tun 
remote-cert-tls server 
verb 3 
comp-lzo yes

ca DALESJO-OpenVPN.crt
cert DALESJO-SADAL.crt
key DALESJO-SADAL.key
tls-auth DALESJO-Server.tls 1
Categories
Network

Varnish reload

A reload of the varnish server does not destroy your cache but reloads your vcl.  Check your vcl file for errors before trying a reload.

varnishd -C -f /etc/varnish/default.vcl && systemctl reload varnish.service && systemctl status varnish.service varnishncsa.service
Categories
Linux Network

Add a second network card

To add a second ip-number to an network interface create a new ifcfg file ex /etc/sysconfig/network-scripts/ifcfg-enp1s0f0:0

TYPE="Ethernet"
BOOTPROTO="static"
DEVICE="enp1s0f0:1"
ONBOOT="no"
ZONE=public
IPADDR=80.67.x.x
PREFIX=31
GATEWAY=80.67.x.x

The default configuration for that interface might have parameters such as UUID, NAME those are not needed. change ONBOOT to yes after testing the configuration. To test run.

systemctl restart network.service