The details on how to clone bootcamp on Mac OS X

Before cloning Bootcamp, it is good to have an understanding of where we started, how we got here, and how it all works.

GPT

When Apple transitioned to Intel, Open Firmware (the code that was responsible for starting up your machine on a PPC Mac) was not replaced with BIOS, but with EFI (Extensible Firmware Interface). This technology, developed by Intel, allows Apple to keep the full features that we have come to expect with Open Firmware, but to adopt existing technology instead of re-writing Open Fireware for the new Intel Macs. Along with EFI came GPT, or GUID Partition tables, which tells how data is organized on the disk.

The important point on all of this is that Windows XP Pro does not recognize EFI, nor GPT. Bootcamp gets around this by requiring a firmware update that contains a BIOS compatibility module to EFI. When Windows boots up, it may not recognize EFI, but it will recognize the BIOS compatibilty mode and boot accordingly.

Windows boot process

When booting from the Bootcamp partition, the boot loader code of the Master Boot Record (MBR) on the selected disk will be read. Specifically the first 446 bytes of the disk. The MBR contains both the boot loader code and the partition table, but since we will be using diskutil to muck with the partition table, we won't need to deal with it directly; however, we do need to grab the boot loader code from the MBR using dd:

Now that we have the boot loader code, you'd think that we could just use Disk Utility to clone the NTFS partition, and we'd be done. However, Disk Utility doesn't understand NTFS completely, and we want to be able to do some fancy stuff like only backup unused blocks and compress the data. To do this, we need ntfsprogs. Located at http://www.linux-ntfs.org/, it compiles nicely on Mac OS X if you have the developer tools installed. However, the way it compiles and installs is that the binary tools require a library at /usr/local/lib, or something like that. A better solution is compile the library code statically inside the binary tools. That way, you can send the binary command line tool with ARD (or include the binary tool within a bundle of an application), and not worry about having to install the library in the correct location. This comes at a price, however. The binaries will all contain the same library code, and will bloat them by about 400k. Bah! 400K? You are too tiny and I laugh at you!

Running on Mac OS X, ntfsclone generates an error when it completes due to the fsync command not recognizing an option only avaiable on Linux, but you can either safely ignore these errors, or comment out all the "fsync_clone(fd_out);" lines by putting a "//" in front of them (the errors cause an exit code of not zero, which is a bummer, and good reason to comment out those lines).

To compile, do:

and within the ntfsprogs subdirectory, you'll find ntfsclone and ntfsresize. These are the two main tools we'll be using.

Assuming that you have Windows installed via BootCamp, you can now clone it to file using ntfsclone. However, we need to talk a bit about filesystems, partitions, and resizing.

If you have a NTFS filesystem that is 5 GB, and you try and restore it onto a partition that is less than 5 GB, it obviously will fail. If you restore an NTFS filesystem that was 5 GB onto a partition that is greater than 5 GB, you'll have unused space on the partition that will not be seen by Windows, and not used.

The real stickler comes when you create a 30 GB partition, install Windows and a couple of applications, and then clone it. The actual used space will only be about 5 GB, and the compressed image will be less than that. However, since the filesystem is 30GB, you HAVE to restore it onto a partition that is 30 GB or larger . This can be a real pain. However, there is a solution: ntfsresize.

NTFSRESIZE

ntfsresize will both tell you the minimum size that an NTFS partition can be shrunk down to, will shrink it for you, and will also expand it out to take up the entire partition. To find out how much you can shrink a partition, run it on the BootCamp partition. You'll need to unmount the partition first, either by dragging the volume to the trash, or using the diskutil unmount command:

Then run ntfsresize on it:

sh-2.05b$ sudo ./ntfsresize -i /dev/disk0s3
ntfsresize v1.13.0 (libntfs 9:0:0)
Device name : /dev/disk0s3
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 10737566208 bytes (10738 MB)
Current device size: 10737569792 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 7453 MB (69.4%)
Collecting resizing constraints ...
You might resize at 7452676096 bytes or 7453 MB (freeing 3285 MB).
Please make a test run using both the -n and -s options before real resizing!


So this partition can be resized to 7453 MB. I have found that you need to add 5% onto the size of the partition in order for it to be successful in a known way. If it is not successfully, don't worry, it won't destroy your partition; it'll just error out and you'll have to try a larger size.

To resize it, just feed that value back to ntfsresize using the -s flag:

After running ntfsresize -s, you'll have a ntfs filesystem that is much smaller than the partition, and when you clone it, it'll allow you to restore it onto volumes that are the same size or larger that this new volume size.

Now we can go ahead and clone the partition using ntfsclone:

The "--save-image" flag tells ntfsclone to do a special image format that doesn't backup empty data blocks. "-o" specifies the output file, in this case "-", which is standard output. We'll feed that to gzip to compess it. I have seem compression ratios from 5:1 to 2.5:1.

CHKDSK and -f

Now for a note about chkdsk and all those -f flags.

When using ntfsresize, it will mark the filesystem as needing a file system check. The next time you boot into Windows, it will run a chkdsk and verify that there aren't any problems, and then reboot. However, the process is NOT complete. You have to boot back to the Windows partition in order for Windows to write the new journal file.

If the disk needs to be checked, the ntfsprogs will warn you. If it also detects that the volume was unmounted uncleanly, it will warn you as well. If you are sure that you did not unmount the volume uncleanly (like a force restart or power failure), and a chkdsk needs to be run due to running ntfsresize, you can ignore these warnings with the "-f" flag. The "-f" flags are cumulative, so providing two of them will result in ntfsclone (or ntfsresize for that matter) ignoring all the warnings. Handy for scripting.

Restoring

Now that we have a backup file, and our bootstrap code, we can restore Windows. Just do a ntfsclone:

and then resize if back to fill up the entire partition:

If you do not specify any options with ntfsresize, it'll resize the filesystem to fill up the partition. Handy.

If you are doing it onto the machine that you backed it up from, the process is simple since the disk geometry didn't change, the bootstrap code already exists, and it is all ready to go.

The interesting part comes in when you try to restore it onto a Intel Mac that you did not back it up from. Somethings will be different:

1. There is not a Windows partition
2. There is no bootstrap code
3. The Windows partition will not be marked bootable, so the bootstrap code will have no idea of which partition to boot from.
4. The start of the Windows partition will most likely be at a different place on the hard drive

To create a Windows partition, you could re-run BootCamp on the new machine. However, you could also use the shiney new diskutil resizeVolume command:

There is a couple of thinkgs to note here. The disk must have only 1 partition, and it must be HFS+ Journaled. There are other options using resizeVolume, but for this example, you have 1 partition, and it is HFS+J. The 100G number is the new size of the HFS+J volume, and the 10G is the size of the Windows partition. The last number (10G) is not really used, as the resizeVolume command will take the size of the drive, subtract the HFS+J size (100G) and resize the last partition to that size.

Once you run the diskutil resizeVolume command, you'll need to reboot so that all of Mac OS X recoginizes the new partitions (and the new size of the HFS+J partition).

Bootstrap code:

To restore the boot loader code, you use fdisk and the boot loader code file we backed up with dd earlier.

-u : update boot loader code, without distrubing partition table
-f : boot loader code file

We could have used dd, but dd requires that volume not be in use to write to it, and since we are writing to /dev/disk0 (the boot disk), we'd have to boot off of another device. fdisk will just warn you.

Note that we needed to specify the "raw"device file, which is just the device name with a "r" in front of the "disk" label.

Marking the Partition as bootable

If you don't mark the Windows partition as bootable, it will not show up in the OS Chooser menu when starting up, and the boot loader code will not be able to find it. To mark it as bootable:

You can do this all on one like by piping the commands to fdisk (this was a pain to figure out):

Location of the start of the NTFS partition

NTFS also writes the location of the ntfs partition to the start of the NTFS filesystem. Yes, read that again. For some reason, NTFS needs to know in the filesystem where it is located on the drive. If this is not modified, Windows will not boot if the geometry has change Ick.

To make it even worse, you have do some byte swapping. First, let's get the start of the NTFS partition:

As you can see, the ntfs partition starts at 213469792. We need to convert that to hex:

We take those 4 hex numbers and reverse their order: 0C B9 4A 60 becomes 60 4A B9 0C.

We can now write out this value to the NTFS filesystem:

xxd -r -p will convert the hex numbers to binary

dd writes out the values to bytes 28, 29,30,31 of the NTFS fileysystem.

The process is now complete. You can reboot into windows, and it should start up after doing a chkdisk.

Cloning Vista

The second issue was due to the change in BCD. In XP (and prior versions of Windows), the boot menu referenced a file called boot.ini. With Vista, this info moved to a binary file called “BCD” in the Boot folder at the top level. This BCD contains the block where the partition that contains Vista starts so that it knows where to find the proper files to boot. This is great if you are starting to boot from one partition, but want to present a menu of other OSs on multiple partitions. However, if you clone the NTFS filesystem that has Vista (and this BCD file), the BCD file will now contain wrong information about where the NTFS partition starts (unless you clone it back to the original partition). There are a couple of ways around this. You can use the bcdedit command on the Windows command line to set up the BCD file to reference the partition that the boot menu is presented from. This is the easiest way to do it. However you must do the following steps prior to cloning. To do this:

1. Boot into vista.
2. enter in cmd in search.
3. right click on it and run as administrator
4. run the following commands exactly as shown:

bcdedit /set {bootmgr} device boot
bcdedit /set {default} device boot
bcdedit /set {default} osdevice boot

If you do preparations to your Windows partition prior to cloning, you can just add the steps above. If not, just check the box in Winclone and it will do it for you.

The final issue is a bit more convoluted, and goes back to the days when a hard drive over 8 GB was thought as too big to be realized in our lifetime. In those days, a block on the hard disk was referenced by using “CHS”, which stands for Cylinder, Head, Sector. You could reference 1024 cylinders, 255 heads and 63 sectors, or a total of ~ 8GB (1024*255*63*512). The limit on the CHS values was due to the bytes allocated for the values at the start of the partition. When hard drives got bigger than this, it was switched over the LBA, or logical block addressing. However, since the CHS values still existed at the start of the partition, a CHS value of 1023 254 63 is used as a placeholder and the LBA value is used.

When Vista sees a cylinder value of 1023, it uses the LBA value. Or at least partially. However, it seems very sensitive to having correct head and sector values even if the cylinder value is at the maximum of 1023 (and therefore incorrect). Windows XP does not seem to have this issue if the partition it resides on is > ~ 8GB, but requires a correct CHS value if the start of the partitions is less than that. This is not usually an issue if the HFS+ partition resides on the same disk before it, as it is usually > 8GB. However, if you have Windows XP on its own partition, this issue will arise when cloning. Why doesn’t the issue happen when using bootcamp? Well, when you install Windows XP or Vista, Windows rewrites the CHS values to be what it wants them to be and everything works.


Note that this issue is usually resolved on a PC by going into BIOS and selecting LBA mode for the drive. It appears that the EFI BIOS layer is set to CHS mode, but Windows recognizes CHS values that are incorrect and uses the LBA values.

So how do we set these CHS values so that Windows boots? Usually you use fdisk to write these out, but fdisk on Mac OS X Tiger does not write out these values when changed. It allows you to change them, but when writing them out, reverts back to 1023 254 63. Ick. Since they are just bits on the disk, we could write them out directly. However, the data to write is dependent on what the GPT partition table says is the start and and end of the partition. In order to write the correct CHS values, we need to reference the GPT. I wrote a c program called gptrefresh to do this. The name is a bit of a misnomer, as it does not refresh the gpt but rather refreshes the partition table located on block 0 on the disk. It also does a few other things (like setting the partition that is bootable, write out the MBR, and other low level “write bits to disk” operations). Note that the program WILL change blocks in the MBR on the disk, and DOES muck with the partition table, so it can completely mess up all data on a disk. I have been testing it for a while and it does seem to work well, but YMMV. Use with caution, and never, ever, on a computer with data that can’t be overwritten. You have been warned.

If you use the steps above, you can successfully clone Vista. However, you need to keep the architecture as close as possible between machines when cloning. For example, have an image for iMacs, an image for Mac Pros, etc. I have successfully created an image on a core 2 Duo iMac and restored it onto a Mac Book Pro that has a Core 2 Duo processor, but Vista had install a bunch of drivers (which happened pretty much automatically). The same image on a Mac Pro failed to boot. My guess is that this had to do with the different CPU in the Mac Pro tower.