What is using so much disk space on Unicorn?

Disk Space Usage 	65.13 GB 	118.05 GB 	55.18%

$ du -a /var | sort -n -r | head -n 10

56211764 /var
49952624 /var/lib
49377268 /var/lib/docker
36697616 /var/lib/docker/overlay2
12531672 /var/lib/docker/volumes
11563304 /var/lib/docker/volumes/767aeaa9df443368f491fa738193afbad38534b7079cdb92998089b84315a607
11563300 /var/lib/docker/volumes/767aeaa9df443368f491fa738193afbad38534b7079cdb92998089b84315a607/_data
4217284 /var/discourse
4214348 /var/discourse/shared
3832044 /var/discourse/shared/standalone
`

/var/lib/docker/overlay2 has a lot of new stuff starting Jul 7, previous entries were Jun 18.

I am not a docker but i did see this:

which kinda sounds the same.

In the last 9 days our server has gone from 55% full to 97% full. What should we do?

/dev/mapper/debian9--vg-debian9 119G 108G 4.3G 97% /

I think a lot of it is unused docker stuff (/var/lib/docker has 47G of files according to du); docker doesn’t really do manual garbage collection, so stuff has to be pruned: https://docs.docker.com/config/pruning/

2 Likes

Thanks, I pruned images and volumes not associated with any running on the system and regained about 30 gigs. Thanks!

/dev/mapper/debian9--vg-debian9  119G   85G   28G  76% /

We just ran out of space, which is why Discourse was down for about an hour until a few minutes ago.

Let’s figure out what’s taking up all this space so we don’t have the same problem tomorrow!

/var/log/mail.log is 17GB… and our hosting provider said people are sending spam through Unicorn…

Not a good combination!

Sounds like we need to lock down port 25…

noisebridge@ssdnodes-05208:~$ sudo find / -size +500M
/var/discourse/shared/standalone/backups/default/noisebridge-2019-07-05-033231-v20190603134013.tar.gz
/var/discourse/shared/standalone/backups/default/noisebridge-2019-07-12-033449-v20190603134013.tar.gz
/var/discourse/shared/standalone/backups/default/noisebridge-2019-06-21-033255-v20190603134013.tar.gz
/var/discourse/shared/standalone/backups/default/noisebridge-2019-07-19-033523-v20190603134013.tar.gz
/var/discourse/shared/standalone/backups/default/noisebridge-2019-06-28-033331-v20190603134013.tar.gz
/var/www/html/logs/discuss.access.log
/var/log/mail.info.1
/var/log/mail.info
/var/log/syslog
/var/log/syslog.1
/var/log/mail.log.1
/var/log/mail.log
/home/rando/storage/atmos-rockpro64.img.gz

And just how big are these 500mb+ files? This big:

-rw-r----- 1 root  adm       17G Jul 21 06:27 /var/log/mail.info.1
-rw-r----- 1 root  adm       17G Jul 21 06:27 /var/log/mail.log.1
-rw-r----- 1 root  adm       17G Jul 25 20:49 /var/log/mail.info
-rw-r----- 1 root  adm       17G Jul 25 20:49 /var/log/mail.log
-rw-r--r-- 1 rando rando    7.5G Feb 26 18:46 /home/rando/storage/atmos-rockpro64.img.gz
-rw-r----- 1 root  adm      4.6G Jul 25 06:29 /var/log/syslog.1
-rw-r----- 1 root  adm      2.7G Jul 25 20:49 /var/log/syslog
-rw-r--r-- 1 nginx root     1.1G Jul 25 20:49 /var/www/html/logs/discuss.access.log
-rw-r--r-- 1 rando www-data 622M Jul 18 22:36 /var/discourse/shared/standalone/backups/default/noisebridge-2019-07-19-033523-v20190603134013.tar.gz
-rw-r--r-- 1 rando www-data 612M Jul 11 22:35 /var/discourse/shared/standalone/backups/default/noisebridge-2019-07-12-033449-v20190603134013.tar.gz
-rw-r--r-- 1 rando www-data 575M Jul  4 22:32 /var/discourse/shared/standalone/backups/default/noisebridge-2019-07-05-033231-v20190603134013.tar.gz
-rw-r--r-- 1 rando www-data 569M Jun 27 22:33 /var/discourse/shared/standalone/backups/default/noisebridge-2019-06-28-033331-v20190603134013.tar.gz
-rw-r--r-- 1 rando www-data 564M Jun 20 22:33 /var/discourse/shared/standalone/backups/default/noisebridge-2019-06-21-033255-v20190603134013.tar.gz

Do we need to keep the older Discourse backups?

We can solve 2 problems at once by getting rid of the open relay.

Fixing…

$ sudo postsuper -d ALL
postsuper: Deleted: 84288 messages

Just deleted the 84,288 additional spam emails that Postfix was about to send out from Unicorn :face_with_raised_eyebrow:

I’ve stopped Postfix, thus stopping the spam.

I don’t yet see how the spammers were connecting to our Postfix server because it was listening on 127.0.0.1:25, not 0.0.0.0:25. I portscanned from the outside and indeed port 25 is closed.

EDIT: Answer:

noisebridge@ssdnodes-05208:~$ sudo docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS                    PORTS                              NAMES
2ea465c374e6        local_discourse/mail-receiver   "/sbin/boot"             5 months ago        Up 3 days                 0.0.0.0:2533->25/tcp               mail-receiver

That is, the Discourse mail-receiver was accepting connections from the outside world and forwarding them to local port 25.

But now Postfix is stopped until we (I?) further understand whether :2533 needs to be open to the world at all, and if so, how we can lock it down so that spammers can’t use it for Bad :tm: .

3 Likes

Thank you for looking into this and taking swift action @elimisteve as we literally need it working properly. :heavy_heart_exclamation:

1 Like