Citrix NetScalerVPX out of disk space

My NetScaler VPX ran out of disk space. I did several tricks like adding a second disk and mounting ít into the update directors, to be able to upgrade my NetScaler.

Today I found time to investigate this issue. I had no success flooowing CTX133588. I so I started looking for the largest files and directories in /var.

root@82e6de130138# du -a /var | sort -n -r | head -n 10
13033540        /var
11023956        /var/ns_system_backup
466274  /var/log
461740  /var/log/db
459822  /var/wi
434188  /var/netscaler
305070  /var/log/db/default
248492  /var/nslog
183824  /var/ns_system_backup/backup_7b829416-9965-475b-8dfb-d57d18ce9ba0.tgz
183824  /var/ns_system_backup/backup_1676b077-6132-47f5-88a5-aab4832e5c93.tgz
root@82e6de130138# df
Filesystem  1K-blocks    Used    Avail Capacity  Mounted on
/dev/md0       358830  344602     7052    98%    /
devfs               1       1        0   100%    /dev
procfs              4       4        0   100%    /proc
/dev/ad0s1a   1482638  941530   422498    69%    /flash
/dev/ad0s1e  14348494 2009796 11190820    15%    /var
root@82e6de130138#

What a big surprise! 11 GB in /var/ns_system_backup? How comes? I looked into it.

root@82e6de130138# ls -l
total 11019220
drwxr-xr-x  2 root  wheel       3584 Mar  8 17:13 WebServer
-rw-r--r--  1 root  wheel  188077595 Mar  5 17:14 backup_083f6140-a660-41d5-83f2-389403f29334.tgz
-rw-r--r--  1 root  wheel  187946493 Feb 20 17:13 backup_0bd31ba5-fa66-4e40-a912-d6c1a9ffdbdc.tgz
-rw-r--r--  1 root  wheel  187814966 Feb 14 05:13 backup_11653391-2dff-4c22-ba22-3cd9212cd7c2.tgz
-rw-r--r--  1 root  wheel  187751996 Feb  8 05:13 backup_15904e38-2e8f-49f1-ad8c-e153d9daf980.tgz
-rw-r--r--  1 root  wheel  188106376 Mar  8 17:15 backup_1676b077-6132-47f5-88a5-aab4832e5c93.tgz
-rw-r--r--  1 root  wheel  187932332 Feb 19 05:13 backup_189cc344-e721-40ae-81af-55935a0ee7ac.tgz
-rw-r--r--  1 root  wheel  187961509 Feb 22 05:13 backup_195f2b0a-8b21-4be9-b277-50220247b553.tgz
-rw-r--r--  1 root  wheel  188000564 Feb 26 05:14 backup_26ac5a11-0c8c-4034-b79b-64a94fee1ac1.tgz
-rw-r--r--  1 root  wheel  188098718 Mar  7 05:14 backup_3379fb1a-068e-4bc5-acfc-091108cece18.tgz
-rw-r--r--  1 root  wheel  187761902 Feb  9 05:13 backup_34fd7671-fad1-43b4-b662-0e3b7f221475.tgz

a total of 60 files, round about 183 MB each. In addition there was a subdirectory with the name of my admin partition, containing 60 more files, 7 kb each.

All these files had been relatively new. They had been created at 5:13 AM and PM, so my first guess was a cron job.

These tgz files are a full copy of /flash/nsconfig and selected subdirectories of /var. I could not find a cron job doing this backup, and I could not find any documentation about a /var/ns_system_backup folder.

I kept playing round, but did not find. I posted this question on dicussions.citrix.com and Carl Stalhood asked do you have MAS in place? Yes I do!

It’s MAS to be blamed for!

So I opened MAS, SystemSystem Administration and clicked on Instance Backup Settings. MAS does backup on managed devices twice a day by default. I changed to one backup in 10 days. It should keep just 1 backup, but it failed to delete. I also upgraded MAS and hope problem is solved by now …

Versions involved:

MAS: 11.1 52.26
NetScaler: 11.1 Build 52.13 and 51.26

Leave a Reply

Your email address will not be published. Required fields are marked *