dd2vmdk relocated into the cloud

A while ago I wrote a tool to convert flat disk images (which we commonly call dd images) to VMWare .vmdk disk images (the original blog post on the tool, called dd2vmdk is posted here). I have in the mean time ceased development of it, but despite its relatively archaic nature, some still find it of use.

Today I relocated hosting of it to Google’s new cloud web application service, Google App Engine. My motivations to do this were to reduce adminitrative burden, to eliminate server costs, and to increase long term availability (the Java App Engine is Beta, so there might be some short term availability issues).

The relocation took roughly one hour of my time, was dead simple, and required no code modification, just the simple addition of a new text configuration file.

dd2vmdk is still available via my personal website.


New tool – CERT/CMU Live View

I am in Lafayette, Indiana this week at DFRWS2006. A gent from CERT was present and demonstrating an excellet tool called “Live View” which, from first impressions to be a p2v GUI that automates running dd images in vmware. It appears that the features of it are far beyond what dd2vmdk does in some respects: you appear to point it at an image upon which it:
* generates a vmware vmdk
* generates a corresponding virtual machine definition
* fixes up the driver boot problem
* optionally lets one set the time to a different value.
* automatically boots up the image in vmware

On the downside, it doesnt appear to handle disk images, just partition images. This introduces further complications such as having to specify the OS used, and remapping of drive letters, which they do however handle. I am not convinced that their insistence of replacing the MBR is necessary either.

When I get back from DFRWS I will be testing if it does handle disk images, and if it does, how it copes with geometry problems and LDM.


dd2vmdk – dd Image to VMWare Virtual Disk converter

While performing the last set of investigations, I have produced a simple web based application for automating the conversion of dd images into VMWare Virtual Disks. I have called this tool dd2vmdk – it is accessable at http://www.bschatz.org/2006/p2v/index.html

Currently the tool carves up the image into a virtual disk composed of a number of files, where partitions are contained individual files. The next version of the tool will support directly modifying the partition table and NTFS boot record in-situ within the image file.


P2V – Will the 2K MBR boot up a non cylinder aligned partition?

I left my last post unsure whether of not a PC can boot into a partition that is not aligned with the beginning of a cylinder boundary. I devised a quick test, employing the same image that I have been using for the last two posts.

In this case, I left the partition table unmodified, but went into the NTFS boot record and adjusted its conception of the hard drive to reflect the Virtual Drive’s geometry. I got this:

Error Loading Opertating System.

If the bootable partition was the first on the hard drive, there may have been a chance for it to work, but since it is 73440 sectors in, the CHS values will be all wrong.

Recalling the original partition table as taken from fdisk is:

Disk /dev/ida/c0d0: 8711 cylinders, 255 heads, 32 sectors/track
Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System
/dev/ida/c0d0p1 32 73439 73408 12 Compaq diagnostics
/dev/ida/c0d0p2 * 73440 20555039 20481600 42 SFS
/dev/ida/c0d0p3 20555040 71081759 50526720 42 SFS
/dev/ida/c0d0p4 0 – 0 0 Empty

We regenerate the new partition table with sfdisk, setting the correct CHS values for the new virtual disk. (sfdisk recalculates the right CHS values for the start and end of the partitions based on geometry specified on the command line, and the sector offsets and sizes):

sfdisk -uS -C 4436 -H 255 -S 63 -f c0d0.dd << EOF
32,73408,12
73440,20481600,42,*
20555040,50526720,42
EOF

The image now boots correctly.

So, to answer my initial question. Yes, the MBR boot code WILL boot up a non cylinder aligned partition. Inside the OS the dynamic disk shows up as healthy and exactly as it was in the original physical machine.


P2V – hard drive geometry problems

I have been trying to convert a physical Windows 2000 server running on SCSI RAID to run inside a virtual machine. Given my interest in digital evidence, I was interested in achieving the conversion (which is popuarly referred to as Physical To Virtual or P2V conversion) from first principles.

A while ago I came across the Windows Dynamic Disk partitioning scheme (also called Dynamic Disk or LDM). It’s support under linux is slowly gaining momentum, but still remains a bugbear for manipulating disks. So just to complicate things, I decided to convert the physical host’s drive to using Dynamic Disks.

My proposed methodology was as follows:
1. Use a live CD (helix) to acquire the source drive to a external SATA drive. This included recording the partition and disk geometry using the sfdisk program, recording the LDM partition information using the ldminfo tool from the Linux NTFS project, and imaging the drive using dcfldd.
2. Load the image onto the Linux host running VMWare Server.
3. Generate a VMWare Virtual Disk file which works with the image.
4. Create and configure a VM using the new virtual disk.
5. Fixup the drivers inside the VM using the Ultimate P2V method.

In practise this didn’t work as planned. What I was presented with was the following:

A blank screen with a non flashing cursor on boot.

Looking into the problem in more detail it appeared that the boot problem was due to a difference in disk geometry between the SMART2 RAID controller present in the the physical host and VMWare’s emulated SCSI hard drives. As it turns out VMWARE will only emulate a disk with 63 sectors per track, and 255 heads (the number of cylinders from this perspective is irrelevant). My RAID controller however presented a disk with 32 sectors per track, and the 255 heads. Fdisk showed me this:

Disk /dev/ida/c0d0: 8711 cylinders, 255 heads, 32 sectors/track
Units = sectors of 512 bytes, counting from 0

Device Boot Start End #sectors Id System
/dev/ida/c0d0p1 32 73439 73408 12 Compaq diagnostics
/dev/ida/c0d0p2 * 73440 20555039 20481600 42 SFS
/dev/ida/c0d0p3 20555040 71081759 50526720 42 SFS
/dev/ida/c0d0p4 0 – 0 0 Empty

Running sfdisk inside the virtual machine with this image resulted with a number of warnings about partitions not being aligned on cylinder boundaries.

It appeared that this problem was related to the differences in geometry between the two drives. On boot, the boot code in the MBR of the hard drive is looking inside the image at the wrong point. It appears that the Boot code can only boot partitions that are aligned on cylinder boundaries, ie. the start address of partitions must be fully divisible by heads*sectors. The differences between these means that the boot code is looking in the wrong place inside my image.

[Update: I am not sure about this after researching this some more... it appears that some old versions of DOS required that partitions be cylinder aligned, but the MBR boot code appears like it is independent of cylinder alignment.]