I recently attempted to extend a filesystem on my Ubuntu system and ran into an interesting problem I figured would be worth sharing.
I was following the steps on my creating and extending logical volumes post and made it to the last step where you need to resize the filesystem using resize2fs. Initially when I ran the command I received:
$ sudo resize2fs -P /dev/archive_vg/archive_lv resize2fs 1.42 (29-Nov-2011) Please run 'e2fsck -f /dev/archive_vg/archive_lv' first.
OK, let me run that… over an hour later it finished. Great, let’s resize the filesystem!
$ sudo resize2fs /dev/archive_vg/archive_lv resize2fs 1.42 (29-Nov-2011) Resizing the filesystem on /dev/archive_vg/archive_lv to 1572860928 (4k) blocks. Killed
Excuse me? At first, I thought I had accidentally killed the command since I had been multi-tasking. In a panic, because I thought I may have permanently messed up the filesystem, I ran e2fsck again and waited another hour. The good news is the filesystem was fine. So I tried resize2fs again and received the same response. So what was going on?
What I should have done when I first hit the problem was check the logs as doing so would have given me a clue.
$ tail -f /var/log/syslog Dec 29 17:17:01 ubuntu CRON[2197]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Dec 29 17:34:15 ubuntu kernel: [20926.072059] select 1 (init), adj 0, size 247, to kill Dec 29 17:34:15 ubuntu kernel: [20926.072106] select 974 (liagent), adj 0, size 595, to kill Dec 29 17:34:15 ubuntu kernel: [20926.072112] select 2199 (resize2fs), adj 0, size 105738, to kill Dec 29 17:34:15 ubuntu kernel: [20926.072114] send sigkill to 2199 (resize2fs), adj 0, size 105738 Dec 29 17:36:30 ubuntu kernel: [21061.237413] kjournald starting. Commit interval 5 seconds Dec 29 17:36:30 ubuntu kernel: [21061.237803] EXT3-fs (dm-1): warning: mounting fs with errors, running e2fsck is recommended Dec 29 17:36:30 ubuntu kernel: [21061.239114] EXT3-fs (dm-1): using internal journal Dec 29 17:36:30 ubuntu kernel: [21061.239125] EXT3-fs (dm-1): mounted filesystem with ordered data mode
Well look at that, the kernel is killing processes including resize2fs. Why is it doing that? Well, it turns out my Ubuntu system only had 512 MB of RAM and I guess when you attempt to extend a 2TB filesystem into a 6TB filesystem you need a little more memory 🙂 I shut down the system, increased the memory to 4GB, fired it back up and tried again. This time I watched top as it ran. It managed to consume 330 MB of memory, but this time completed successfully.
In the end this turned out to be a corner case, but I hope it is useful for someone as I was unable to find anyone who had talked about it on Google.
© 2015, Steve Flanders. All rights reserved.
