Hey I've been looking at the agfl issues I mentioned a few weeks ago (got distracted by meltdown/spectre), and back then I was told I needed to run xfs_repair 4.7+ against all our xfs volumes that were formated pre-96f859d52bcb and are now running post-96f859d52bcb kernels. I've reviewed xfs_repair, but It's not immediately obvious why xfs_repair 4.7+ was required. Is there a specific commit in xfs_repair that is necessary? Also what's the best way to run programattically xfs_repair against boot volumes? I checked dracut but didn't find any xfs_repair module. what goes into the initramfs depends on your distro I suppose Also a side note... formatting a disk with a centos 7 3.10 kernel and then upgrading to a 4.14 kernel still shows the kernel as a v4 filesystem. ([ 194.700597] XFS (vdb2): Mounting V4 Filesystem) What is required to upgrade a v4 filesystem to v5? on my centos box, xfs_repair is there in initramfs? yes how is it supposed to be run. well i guess i expected it to be a dracut module. we really need to make this better but the xfs bits of the fs module are castrated. yeah... if you can boot to single user and have the root fs mounted ro you can repair it that'd be fine for one server. we have a farm of roughly 200 that probably need this done to. ok, so another option, please test it carefully first modify the fsck.xfs script to detect the "-f" option if it's present, call xfs_repair on the device then set fsck.mode=force on the kernel commandline lol centos 7 fsck.xfs is castrated as well. as for upgrading from v4 to v5, you can't fsck.xfs is like this upstream it's intentional probably because xfs_repair likely requires interaction. no because initscripts think they need to call fsck if the system wasn't cleanly shut down, and this is not the case for journaling filesystems sandeen: is it possible to format a volume as v5 from the default centos 3.10 kernel? there is no fsck work in general after an xfs crash chiluk, it's not the kernel that formats it, it's xfsprogs. really.. huh. and yes, you can format an xfs volume with recent centos7 xfsprogs I would've expected the kernel to take care of the actual heavy lifting of the format. it's simply bits to a disk, that's userspace task. yep... I think ext4 does some parts in the kernel so it can do lazy inode creation, and return from the mkfs sooner. alright so back to the problem at hand. what makes v4.7 of xfsprogs special in order to solve our agfl issues.. so, i'll start by saying that this problem was introduced when you went off the reservation and ran upstream on a distro box ;) and it'll get re-introduced if you go back to the distro kernel chiluk, yes, mkfs.ext4 plays silly games like that ;) yeah ... the realities of cloud life dictated the new kernel for overlayfs bits and yes I agree we went off the res a bit. although even though we're one of the first... many will eventually be hitting this. i.e. when people start doing centos 7 -> 8 upgrades well, they won't be supported by rhel or centos :) no, we'll handle the problem gracefully by then --> psychicist__ (~psychicis@ip127-8-212-87.adsl2.static.versatel.nl) has joined #xfs the issue is a disk structure that was coming out to different sizes on different architectures Yeah ... the decision to go elrepo kernel-ml was before my time. so djwong packed it to make it consistent, but then that made it different sizes across different kernels on teh same architecture :( yep.. I'm aware. so crossing that boundary of when the packing was added causes issues. on rhel7 we simply don't pack it so it doesn't change yep.. I'm tempted to remove the change in our upstream kernel.. or possibly push the fix into the elrepo kernels actually that's a great idea. I may do just that. or review the series i sent for 4.17 to autofix all that malarky? :) -*- sandeen blinks ... is there really no v4.6.0 tag in xfsprogs? djwong: linkage? "[RFC 0/5] xfs-4.17: fix v5 AGFL wrapping" if elrepo carries patches that might not be a bad idea. yeah they do carry the patch.. I'll pursue that option. that's because it was pushed down through the linux-stable process afaik. patching it out of rhel7 was a semi-unfortunate hack, but since we shipped 7.0GA w/o packing, it was ... most expedient. ubuntu v4.4 kernels have it as well. although I'm not sure if unstall medua started with the patch or acquired it later <-- rwareing_ (~rwareing@199.201.64.130) has quit (Quit: My MacBook Pro has gone to sleep. ZZZzzz…) -*- sandeen shrugs you build your own distro, you get to keep all the pieces! :) fun fun.. so sandeen djwong... do either of you know why running xfsprogs v4.7+ was the demarkation tag for resolving this? assuming I don't pursue one of the other crazy solutions. I did a quick spot check of commits and didn't see anything that rang a bell - is this the thing where we print that version when we hit the corruption? I don't remember. it was 4.5 not 4.7 the commit went into the 4.5 kernel.. (we have no 4.4 tag either?!) ok, so, that's when we simply packed the structure in xfs_repair as well yeah which is why I was confused when dchinner told me to use xfsprogs v4.7+ " you need to run xfs_repair from xfsprogs >= 4.7.0 to fix it up" that's back from way back. dchinner doesn't know what he's talking about, he's a wanna-be hanger-on ;) lol.. ok so now the question becomes ... does the centos v4.5 xfsprogs have the correct bits? -*- chiluk goes to download the srpm thanks for the help again guys... dchinner... if you figure out why you mentioned v4.7+ ... I'd love to hear it. no, if we packed it in our repair, it would not handle the unpacked disk format the kernel writes -*- sandeen wonders if it's ok w/ dchinner the tag-master if I go back and add those missing tags sometimes distros skip xfsprogs releases, so maybe rh never built a 4.5/4.6 package? speaking of which debian's xfsprogs package is old (4.9.0) now that I can fix. how important is it that the kernel match the version of xfsprogs? I'm almost wondering if xfsprogs shouldn't be included in linux-tools directly.. chiluk, it's generally supposed to be compatible except at very clear feature/compatibility breakpoints, and then it's still safe, just not functional :) i.e. old xfsprogs won't touch a v5 filesystem first patch in centos xfsprogs v4.5 package "xfsprogs-4.5.0-revert-AGFL-pack.patch" at least they are consistent. <-- psychicist__ (~psychicis@ip127-8-212-87.adsl2.static.versatel.nl) has quit (Quit: leaving) * Mon Jun 06 2016 Eric Sandeen 4.5.0-2 - Revert AGFL header packing (#1336920) yeah those guys are pretty smart ;) --> rwareing_ (~rwareing@199.201.64.130) has joined #xfs sandeen: we never released a xfsprogs 4.4.0 or 4.6.0 oh! hah ok I forgot about back in the day :) -*- sandeen didn't actually go look chiluk: I said "need xfsprogs >= 4.7.0" because I misread the git describe output :) `git describe ` indexes it's output from the head tag in teh tree at the time it was committed `git describe --contains ` indexes it's output from the next tag added to the tree /after/ the commit was made $ git describe --contains 9fccb9f6deaa v4.5.0-rc1~47 means it was in the 4.5.0-rc1 release, but I had a brain-fart and though it was committed after 4.5.0.... Ok thanks dchinner That's what I was hoping because I can't remember what I typed a second after typing .... what was I going to say? :P My faith in you has been reevaluated...;)