Update September 24, 2018: This issue is resolved on macOS Mojave.
Update March 30, 2018: This issue persists on macOS 10.13.4 (17E199)
This week we reported to Apple a serious flaw in macOS that can lead to data loss when using an APFS-formatted disk image. Until Apple issues a macOS update that resolves this problem, we're dropping support for APFS-formatted disk images.
Note: What I describe below applies to APFS sparse disk images only — ordinary APFS volumes (e.g. your SSD startup disk) are not affected by this problem. While the underlying problem here is very serious, this is not likely to be a widespread problem, and will be most applicable to a small subset of backups. Disk images are not used for most backup task activity, they are generally only applicable when making backups to network volumes. If you make backups to network volumes, read on to learn more.
Disk images are handy devices. They're files, but they act like a hard drive – you mount a disk image by double-clicking the file, then it behaves like it's another hard drive attached to your Mac. macOS has been using disk images for decades, and we find them particularly useful when making backups to network volumes. By formatting the disk image volume using an Apple-native format, we can do things like back up system files.
Naturally, when Apple introduced APFS in macOS High Sierra, we sought to offer support for using APFS on destination disk images when doing so would match the format of the source volume. As far as creating and mounting disk images is concerned, APFS and HFS+ are easily interchangeable, so adding support for APFS was very straightforward. Unnoticed by us, Apple, and thousands of developers, however, is a very subtle behavioral difference that is specific to APFS on a sparse disk image.
Earlier this week I noticed that an APFS-formatted sparsebundle disk image volume showed ample free space, despite that the underlying disk was completely full. Curious, I copied a video file to the disk image volume to see what would happen. The whole file copied without error! I opened the file, verified that the video played back start to finish, checksummed the file – as far as I could tell, the file was intact and whole on the disk image. When I unmounted and remounted the disk image, however, the video was corrupted. If you've ever lost data, you know the kick-in-the-gut feeling that would have ensued. Thankfully, I was just running some tests and the file that disappeared was just test data. Taking a closer look, I discovered two bugs in macOS's "diskimages-helper" service that lead to this result.
An APFS volume's free space doesn't reflect a smaller amount of free space on the underlying disk
In the past with HFS+ formatted disk images, the disk image volume would automatically adjust its... Read More