nginx dump
nginx 最近总是意外宕机,重启了5次左右,没超过一天 又宕机了 WTF。。。
Inspect
排查下进程
systemctl status nginx
宕机了,看看日志
cat /var/log/nginx/error.log
yaml
/02/20 20:06:43 [info] 1815946#1815946: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:82
2024/02/20 20:06:44 [notice] 1815950#1815950: signal process started
2024/02/20 20:06:47 [notice] 1815952#1815952: signal process started
Out of memory!
2024/02/20 20:06:47 [alert] 1765184#1765184: perl_parse() failed: 1
2024/02/20 20:06:48 [info] 1815953#1815953: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:82
2024/02/20 20:06:49 [notice] 1815956#1815956: signal process started
2024/02/20 20:06:51 [notice] 1815960#1815960: signal process started
2024/02/20 20:06:51 [error] 1815960#1815960: open() "/run/nginx.pid" failed (2: No such file or directory)
2024/02/20 20:06:52 [info] 1815964#1815964: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:82
2024/02/20 20:06:53 [notice] 1815967#1815967: signal process started
2024/02/20 20:06:58 [notice] 1815969#1815969: signal process started
2024/02/20 20:07:01 [notice] 1815971#1815971: signal process started
2024/02/20 20:07:01 [alert] 1815971#1815971: kill(1815962, 1) failed (3: No such process)
2024/02/20 20:07:01 [emerg] 1815972#1815972: bind() to 0.0.0.0:443 failed (98: Address already in use)
2024/02/20 20:07:01 [emerg] 1815972#1815972: bind() to 0.0.0.0:80 failed (98: Address already in use)
再看看别的日志,发现有恶意攻击
yaml
139.59.65.144 - - [20/Feb/2024:03:41:15 +0000] "GET / HTTP/1.1" 502 166 "-" "-"
139.59.65.144 - - [20/Feb/2024:03:41:15 +0000] "GET / HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Linux; Android 6.0; HTC One M9 Build/MRA97086) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.7570.98 Mobile Safari/537.3"
139.59.65.144 - - [20/Feb/2024:03:41:15 +0000] "GET /.vscode/sftp.json HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:16 +0000] "GET /about HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:16 +0000] "GET /debug/default/view?panel=config HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:16 +0000] "GET /v2/_catalog HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:17 +0000] "GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:17 +0000] "GET /server-status HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:18 +0000] "GET /login.action HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:18 +0000] "GET /_all_dbs HTTP/1.1" 502 166 "-" "Mozilla/5.0 (l9scan/2.0.0323e25383e22333e25343; +https://leakix.net)"
139.59.65.144 - - [20/Feb/2024:03:41:19 +0000] "GET /.DS_Store HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:19 +0000] "GET /.env HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:20 +0000] "GET /.git/config HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:20 +0000] "GET /s/0323e25383e22333e25343/_/;/META-INF/maven/com.atlassian.jira/jira-webapp-dist/pom.properties HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:20 +0000] "GET /config.json HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:21 +0000] "GET /telescope/requests HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
139.59.65.144 - - [20/Feb/2024:03:41:21 +0000] "GET /?rest_route=/wp/v2/users/ HTTP/1.1" 502 166 "-" "Go-http-client/1.1"
91.92.246.202 - - [20/Feb/2024:03:50:16 +0000] "GET /.env HTTP/1.1" 502 166 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0"
91.92.246.202 - - [20/Feb/2024:03:50:18 +0000] "GET /.env HTTP/1.1" 502 166 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0"
好家伙,这是准备偷我密码呢??? 这家伙恶意访问不存在文件,导致内存耗尽了
Solution
fail2ban
之前对 ssh
爆破的兄弟 已经加了 fail2ban
,nginx 再送你一套吧
yaml
[nginx-botsearch]
enabled = true
port = http,https
logpath = %(nginx_error_log)s
maxretry = 2
nginx 并发连接
yaml
worker_connections 768; #
Solution
整了半天都不是跟本原因 WTH
最后还是搜了一下,找到了根源
sudo apt purge libnginx-mod-http-perl
sudo systemctl restart nginx