server 受 ssh暴力登入攻擊的分析與防禦(2)--fail2ban 的使用

Last updated on 

關於這次攻擊事件的分析與防禦方法，最後是採用fail2ban 來做防禦，而非自己用 iptables+ipset 手刻規則，因為fail2ban可以整合各式防火牆、支援封鎖時間指數遞增、支援多種log格式分析(sshd, nginx,...)。封鎖時間會隨著錯誤次數成指數增長，所以使用越久效果會越好，同時對於不小心輸入錯誤的使用者一開始封鎖時間很短，足夠讓他們想起正確密碼又不至於等太久。至於 iptables+ipset 的進階用法將留到第三篇討論。

安裝 fail2ban

參考 How to Install and Configure Fail2ban on Ubuntu 20.04 | Linuxize 及 Fail2ban SSH config on Ubuntu 18.04 and RHEL/CentOS 8 | GoLinuxCloud

sudo apt update
sudo apt install fail2ban

設定 fail2ban 開機自動啟動服務：

sudo systemctl enable fail2ban

立刻啟動服務：

sudo systemctl start fail2ban

確認執行狀態：

sudo systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
     Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-02-01 17:18:10 CST; 2h 45min ago
       Docs: man:fail2ban(1)
    Process: 1561584 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
   Main PID: 1561585 (f2b/server)
      Tasks: 5 (limit: 77197)
     Memory: 20.2M
     CGroup: /system.slice/fail2ban.service
             └─1561585 /usr/bin/python3 /usr/bin/fail2ban-server -xf start

Feb 01 17:18:10 server systemd[1]: Starting Fail2Ban Service...
Feb 01 17:18:10 server systemd[1]: Started Fail2Ban Service.
Feb 01 17:18:10 server fail2ban-server[1561585]: Server ready

停止服務

sudo systemctl stop fail2ban

設置 ssh jail

在 /etc/fail2ban/jail.d/ 底下新增 00-sshd.conf 

sudo touch /etc/fail2ban/jail.d/00-sshd.conf

使用vi編輯內容為

sudo vi /etc/fail2ban/jail.d/00-sshd.conf

目前設定一天最多錯3次，第4次錯就從封鎖1分鐘開始，1分鐘後再錯，就倍增封鎖時間到2分鐘，2分鐘後再錯就封鎖4分鐘，再錯就8分鐘、16分鐘、32分鐘、64分鐘....以此類推。

[DEFAULT]
ignoreip = 127.0.0.1/8 ::1
bantime  = 1m
findtime = 1d
maxretry = 3
bantime.increment = true
bantime.maxtime = 2592000
bantime.rndtime = 86400
banaction = iptables-ipset-proto6-allports

[sshd]
enabled  = true
mode   = aggressive

關於 iptables-ipset-proto6 的用法，參考 [1]
關於 findtime, bantime, maxretry 的說明，參考 [2]
關於 bantime.increment, bantime.maxtime, bantime.rndtime 的說明，參考 [3]
關於 /var/log/auth.log 中出現 Feb 1 21:48:59 server sshd[1859236]: Connection closed by 177.92.4.106 port 34850 [preauth] 問題，參考[4]
關於 banaction ，參考[5]

每次編輯完後都要重新啟動 fail2ban 服務使修改結果生效：

sudo systemctl restart fail2ban

確認 ssh jail 狀態：

sudo fail2ban-client status
Status
|- Number of jail:      1
`- Jail list:   sshd
sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 2
|  |- Total failed:     111
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 19
   |- Total banned:     22
   `- Banned IP list:   92.63.199.9 60.12.136.62 157.230.63.81 222.187.239.31 221.181.185.148 221.181.185.223 185.234.217.184 221.181.185.135 221.181.185.141 49.233.53.111 221.181.185.19 37.187.113.47 111.231.77.115 222.187.222.55 221.181.185.198 221.181.185.29 221.181.185.143 202.83.170.254 221.181.185.140

檢查 fail2ban action

 sudo fail2ban-client get sshd actions
The jail sshd has the following actions:
iptables-multiport

注意：如果在上方的 jail conf 中沒有指定 action 欄位，預設使用 iptables-multiport 作為 ssh jail action，進一步查看內容：

sudo fail2ban-client get sshd action iptables-multiport actionstart
<iptables> -N f2b-sshd
<iptables> -A f2b-sshd -j RETURN
<iptables> -I INPUT -p tcp -m multiport --dports ssh -j f2b-sshd

這些 action定義在 /etc/fail2ban/action.d/iptables-multiport.conf 中

 vi /etc/fail2ban/action.d/iptables-multiport.conf

另外，應該可以從 iptables 看到所有規則：

sudo iptables --list-rules f2b-sshd
-N f2b-sshd
-A f2b-sshd -s 157.230.63.81/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 60.12.136.62/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -s 92.63.199.9/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN

若如上方 ssh jail conf 中指定 action = iptables-ipset-proto6-allports ，指令 sudo fail2ban-client get sshd actions 結果就會變成：

The jail sshd has the following actions:
iptables-ipset-proto6-allports

此時查看 action 就要改為 iptables-ipset-proto6-allports ：

sudo fail2ban-client get sshd action iptables-ipset-proto6-allports actionstart
ipset create <ipmset> hash:ip timeout 600<familyopt>
<iptables> -I INPUT -m set --match-set <ipmset> src -j <blocktype>

一樣可以在 /etc/fail2ban/action.d/iptables-ipset-proto6-allports.conf 找到

由於這個設定使用到 ipset，輸入

sudo ipset list f2b-sshd

可以查看 ip list

Name: f2b-sshd
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536 timeout 600
Size in memory: 29768
References: 1
Number of entries: 372
Members:
118.189.74.228 timeout 603128
45.249.247.162 timeout 603129
120.237.118.139 timeout 2780
118.122.38.61 timeout 603128
165.227.193.157 timeout 603128
106.13.197.35 timeout 603128
1.204.175.87 timeout 603127
......

輸入

sudo iptables --list-rules INPUT

可以看到只有一條規則跟 f2b-sshd 有關：

-A INPUT -m set --match-set f2b-sshd src -j REJECT --reject-with icmp-port-unreachable

其他設定

查看 jail2ban 記錄檔:

sudo watch tail /var/log/fail2ban.log -n 20

Get database name and location:

fail2ban-client get dbfile
Current database file is:
`- /var/lib/fail2ban/fail2ban.sqlite3

Ban IP Address or Network

fail2ban-client set sshd banip 192.168.10.0/24

Unban IP address or network

fail2ban-client set sshd unbanip 192.168.10.0/24

Fail2ban ignoreip

fail2ban-client get sshd ignoreip

add ignoreip

fail2ban-client set sshd addignoreip 192.168.11.12 192.168.10.0/24

Show ban ip with time

https://github.com/fail2ban/fail2ban/pull/2315#issuecomment-451779004

sudo fail2ban-client get sshd banip --with-time

Show ban ip ordered by bancount

https://serverfault.com/a/1026819/451844

get location of sqlite3:

which sqlite3

results: (注意 anaconda 首字母大小寫，否則會找不到路徑)

/share/apps/anaconda3/bin/sqlite3

get fail2ban.sqlite3 's schema:

sudo /share/apps/anaconda3/bin/sqlite3  /var/lib/fail2ban/fail2ban.sqlite3 '.schema'

results:

...
CREATE TABLE bips(ip TEXT NOT NULL, jail TEXT NOT NULL, timeofban INTEGER NOT NULL, bantime INTEGER NOT NULL, bancount INTEGER NOT NULL default 1, data JSON, PRIMARY KEY(ip, jail), FOREIGN KEY(jail) REFERENCES jails(name) );
...

use sudo with sqlite3:

sudo /share/apps/anaconda3/bin/sqlite3 /var/lib/fail2ban/fail2ban.sqlite3 "select jail,ip,bantime,bancount as count from bips group by ip order by count desc limit 100"

results:

sshd|68.183.25.127|1011363.16231006|15
sshd|201.48.192.60|1045428.02111748|15
...
sshd|221.181.185.223|1050783.05290286|15
sshd|141.98.80.60|1019079.54239064|15
sshd|177.70.121.39|524866.51508015|14
sshd|195.54.160.250|546595.121685704|14
sshd|45.93.201.193|547957.916561537|14
sshd|161.97.118.168|286359.853684653|13
sshd|194.165.16.42|256013.337475776|13
sshd|221.181.185.237|18386.8351457551|9
sshd|191.17.203.233|66873.605922362|4
sshd|191.23.56.122|82387.9567954801|3
sshd|209.141.45.21|39071.3000370249|3
sshd|103.247.196.56|62706.1043116257|2
...
sshd|222.92.183.227|44759.517825925|2
sshd|24.214.72.152|60|1
sshd|185.220.102.249|60|1
sshd|91.192.103.16|60|1

Start, Stop, Reload Jail

fail2ban-client stop sshd
fail2ban-client start sshd
fail2ban-client reload sshd

取得目前監控之 sshd log file 位置：

sudo fail2ban-client get sshd logpath
Current monitored log file(s):
`- /var/log/auth.log

設定欲監控之 sshd log file 位置：

fail2ban-client set sshd addlogpath /var/log/secure

問題排解

當 server 改變 timezone 時，由於 /var/log/auth.log 上的時間不會跟著改，但 fail2ban 卻已經使用新時間，因此會導致 fail2ban 完全失效，雖然有發現惡意攻擊ip，卻不斷顯示 expired bantime 而 ignore。

2021-02-22 14:20:21,808 fail2ban.filter         [1068]: INFO    [sshd] Found 113.160.132.243 - 2021-02-22 06:20:21
2021-02-22 14:20:22,262 fail2ban.actions        [1068]: INFO    [sshd] Ignore 113.160.132.243, expired bantime
2021-02-22 14:20:28,291 fail2ban.filter         [1068]: INFO    [sshd] Found 113.160.132.243 - 2021-02-22 06:20:28
2021-02-22 14:20:28,386 fail2ban.filter         [1068]: INFO    [sshd] Found 113.160.132.243 - 2021-02-22 06:20:28
2021-02-22 14:20:29,071 fail2ban.actions        [1068]: INFO    [sshd] Ignore 113.160.132.243, expired bantime
2021-02-22 14:20:34,882 fail2ban.filter         [1068]: INFO    [sshd] Found 113.160.132.243 - 2021-02-22 06:20:34
2021-02-22 14:20:34,980 fail2ban.filter         [1068]: INFO    [sshd] Found 113.160.132.243 - 2021-02-22 06:20:34
2021-02-22 14:20:41,465 fail2ban.filter         [1068]: INFO    [sshd] Found 113.160.132.243 - 2021-02-22 06:20:41
2021-02-22 14:20:41,562 fail2ban.filter         [1068]: INFO    [sshd] Found 113.160.132.243 - 2021-02-22 06:20:41
2021-02-22 14:20:41,687 fail2ban.actions        [1068]: INFO    [sshd] Ignore 113.160.132.243, expired bantime

因此，根據 https://github.com/fail2ban/fail2ban/issues/1986#issuecomment-606030023 的說法，只需要重新啟動 syslog service 即可：

A reboot is not really needed here, mostly it is enough to restart syslog service (e. g. systemctl restart rsyslog) or journal service (e. g. systemctl restart systemd-journald) etc.

[1] How can I add a action to fail2ban - iptables-ipset-proto | Page 2 | Plesk Forum

Using:

[own-badbot]
enabled = true
action = iptables-ipset-proto4[name=own-badbot, port="http,https,7080,7081"]
iptables-ipset-proto6-allports[name=own-badbot, port="http,https,7080,7081"]
iptables-ipset-proto6[name=own-badbot, port="http,https,7080,7081"]
logpath = /var/www/vhosts/system/*/logs/*access*log
/var/log/httpd/*access_log
filter = apache-badbots
findtime = 43200
bantime = 604800
maxretry = 3​

without ".conf" for the action - that works for me (Plesk 17.8.11)

Implement fail2ban with custom apache filter, ipset, and a sample based verification approach - Tech Knowledge Base - jaytaala.com Confluence

[DEFAULT]
ignoreip = 127.0.0.1/8 10.0.0.0/24
 
[apache-custom]
enabled  = true
logpath  = /var/log/apache*/access.log
action   = iptables-ipset-proto6[name=apache-custom, port="http,https", protocol=tcp, bantime=0]
findtime = 86400
bantime  = -1
maxretry = 1
 
[sshd]
enabled  = true
action   = iptables-ipset-proto6[name=ssh, port=ssh, protocol=tcp, bantime=0]
findtime = 3600
bantime  = -1
maxretry = 3

[2] Fail2ban SSH config on Ubuntu 18.04 and RHEL/CentOS 8 | GoLinuxCloud

  • findtime: The amount of time a user has when attempting to log in
  • bantime: The total number of seconds a host will be blocked from accessing the server if they are found to be in violation of the rules
  • port: If you are using a different port for SSHD. Since I am using default port 22, I have not used this.
  • ignoreip: The additional networks that you don't want to be blocked
  • maxretry: The number of failures that need to occur before Fail2ban takes action. The IP will be blocked for the number of minutes included in the bantime option.

時間格式可以使用 TIME ABBREVIATION FORMAT：

jail.conf(5) — fail2ban — Debian testing — Debian Manpages

TIME ABBREVIATION FORMAT

The time entries in fail2ban configuration (like findtime or bantime) can be provided as integer in seconds or as string using special abbreviation format (e. g. 600 is the same as 10m).


Abbreviation tokens:

years?, yea?, yy?
months?, mon?
weeks?, wee?, ww?
days?, da, dd?
hours?, hou?, hh?
minutes?, min?, mm?
seconds?, sec?, ss?

The question mark (?) means the optional character, so day as well as days can be used.


You can combine multiple tokens in format (separated with space resp. without separator), e. g.: 1y 6mo or 1d12h30m.
Note that tokens m as well as mm means minutes, for month use abbreviation mo or mon.

The time format can be tested using fail2ban-client:

fail2ban-client --str2sec 1d12h

[3] https://github.com/fail2ban/fail2ban/issues/2046#issuecomment-364874702

Please note that because the botnets relative easy could determine your ban-settings, this affects failure counters (meant maxretry) also (if bantime.increment allowed):

  • before first ban the IP should do 5 maxretry failed attempts inside findtime, to be banned;
  • after unban, it is known as "bad", so for next ban 3 attempts are enough;
  • next time 2 attempts only;
  • each next time will be banned immediately (already by 1 attempt)


The "bad" indicator is reset, if IP will be removed from database (manual unban or after triple time of the last ban, if no failures anymore), example:

  • ip was banned for 1 day. It will be removed at the earliest in 3 days;
  • ip was banned May 01 00:00:00 for 12 hours. It will be removed after May 02 12:00:00.

https://github.com/fail2ban/fail2ban/issues/1791#issuecomment-303343075

The only way I see is to use my ban-time-incr version (#1460) and to specify initial short bantime/maxretry values. The bantime will grows for such "bad" IPs with each attempt (ban).
And the short initial bantime avoids banning for long time by some typos of "good" users.

bantime = 1m
bantime.increment = true
bantime.factor = 2
bantime.maxtime = 10w

maxretry = 3
findtime = 1d

Incremental banning with Fail2Ban - Visei Internet

Have a look at the following section in jail.conf:

[DEFAULT]
# "bantime.increment" allows to use database for searching of previously banned ip's to increase a
# default ban time using special formula, default it is banTime * 1, 2, 4, 8, 16, 32...
#bantime.increment = true

# "bantime.rndtime" is the max number of seconds using for mixing with random time
# to prevent "clever" botnets calculate exact time IP can be unbanned again:
#bantime.rndtime =

# "bantime.maxtime" is the max number of seconds using the ban time can reach (don't grows further)
#bantime.maxtime =

# "bantime.factor" is a coefficient to calculate exponent growing of the formula or common multiplier,
# default value of factor is 1 and with default value of formula, the ban time
# grows by 1, 2, 4, 8, 16 ...
#bantime.factor = 1

# "bantime.formula" used by default to calculate next value of ban time, default value bellow,
# the same ban time growing will be reached by multipliers 1, 2, 4, 8, 16, 32...
#bantime.formula = ban.Time * (1<<(ban.Count if ban.Count<20 else 20)) * banFactor
#
# more aggressive example of formula has the same values only for factor "2.0 / 2.885385" :
#bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)

# "bantime.multipliers" used to calculate next value of ban time instead of formula, coresponding
# previously ban count and given "bantime.factor" (for multipliers default is 1);
# following example grows ban time by 1, 2, 4, 8, 16 ... and if last ban count greater as multipliers count,
# always used last multiplier (64 in example), for factor '1' and original ban time 600 - 10.6 hours
#bantime.multipliers = 1 2 4 8 16 32 64
# following example can be used for small initial ban time (bantime=60) - it grows more aggressive at begin,
# for bantime=60 the multipliers are minutes and equal: 1 min, 5 min, 30 min, 1 hour, 5 hour, 12 hour, 1 day, 2 day
#bantime.multipliers = 1 5 30 60 300 720 1440 2880

https://github.com/fail2ban/fail2ban/issues/2652#issuecomment-596107934

it is almost impossible to reach this exponential increased time by default settings, because by default bantime (1h) even with factor 1 already after 13th ban it will grow to almost 1 year, after 15th it is 4 years, after 18th - 30 years, after 19th - 60 years, and 20th ban will cause 1h * 2^20^, which corresponds to about 120 years. Considering all bans together the intruder is banned for almost 240 years!
Such long bans (as well as persistent bans) are ugly for several reasons, therefore it is even recommended to set bantime.maxtime to several weeks or 1 month. Longer bans would simply grow iptables / ipsets, so unnecessarily burden the network subsystem and not practicable at all.
If a (known as bad) intruder makes 1 attempt after 1 month ban, it will be banned immediately (again for 1 month), so why one should use longer banning? Anyway I use it several years and see extremely rare some attempts after 2-3 months banning (either they got "fixed", change IP/provider, or simply removed my protected hosts from its attack targets list).

[4] 關於 /var/log/auth.log 中出現 Feb 1 21:48:59 server sshd[1859236]: Connection closed by 177.92.4.106 port 34850 [preauth] 問題，解決方法如下：

regex - Modify fail2ban failregex to match failed public key authentications via ssh - Server Fault

[sshd]
mode   = aggressive

This mode now covers failed public keys.

[5] https://github.com/fail2ban/fail2ban/issues/2813#issuecomment-677654604

Maybe it is affected, because you set action not correctly - don't use action parameter directly (use banaction instead of):

- action = iptables-ipset-proto6-allports
+ banaction = iptables-ipset-proto6-allports

.....

However if you are affected by both bugs (I don't think so), as an interim solution unless you'd get upgraded version, you can set bantime in jail.local like here:

banaction = iptables-ipset-proto6[bantime=%(bantime)s]
banaction_allports = iptables-ipset-proto6-allports[bantime=%(bantime)s]

系列文章

server 受 ssh暴力登入攻擊的分析與防禦(1)--iptables 的使用

server 受 ssh暴力登入攻擊的分析與防禦(2)--fail2ban 的使用

server 受 ssh暴力登入攻擊的分析與防禦(3)--iptables + ipset 進階用法

server 受 ssh暴力登入攻擊的分析與防禦(4)--使用 captcha 及 2FA