Switching a Windows 8 Consumer Preview from a VHD to Hyper-V

By jay at March 04, 2012 13:39 Tags: , ,

Now that Windows 8 Consumer Preview (CP) is out the doors, I'll start publishing a series of articles about Windows 8 development in various areas, but primarily focused on development.

I'll start today with some tips and tricks that allow to run Windows 8 natively from a VHD, but also run that same VHD from Hyper-V, within Windows 8 Consumer Preview.

The common scenario is simple, you want to try out stuff that worked in the Developer Preview (DP) inside Win8 CP, or run a two instances of the Win8 CP for testing purposes. Chances are you installed the DP on a VHD, from the excellent blog post from Scott Hanselman.

A few things about Hyper-V

Hyper-V’s been here for a while on the Server side, but chances are you never looked at it. Hyper-V will most likely completely replace Virtual PC, as it now supports 64 Bits virtual machines and the very useful dynamic memory feature. Dynamic memory basically add or removes physical memory assigned to the virtual machines depending on the actual demand. This allows to have higher virtual machine density on servers, but on client machines this is also pretty useful because you may not have 32GB of ram, and still want to use it for your primary OS.

This is why this tutorial suggests to have a machine starting with 512MB of RAM, so that it can grow to the appropriate amount, and not arbitrarily reserve 2 or 4GB of ram, just in case.


Running the VHD from Hyper-V

So now that Hyper-V is now included natively in Windows 8, and even though the final SKU in which it will be included is unknown, we can use it from the CP.

If you’re like me, you may have installed the CP directly on a SSD to make it pretty darn fast. Now, here’s how you install Hyper-V :

    1. Open the start menu, type “Feature” and open “Turn Windows Features on and off
    2. Select “Hyper-V” and OK. Reboot. (If the Hyper-V infrastructure is off, then either your CPU does not support hardware virtualization, or it is disabled in your bios)
    3. Launch the “Hyper-V Manager
    4. In the right hand side panel, select “Virtual Switch Manager
    5. Click on “Create a Virtual Switch
    6. Set a relevant network or network adapter name
    7. Select your network card, could be either a wireless or wired network card, then OK.
    8. In the right hand side panel, select “New” and “Virtual Machine
    9. Give it 512MB of RAM and check the “Dynamic Memory” box, then next
    10. Select the virtual network switch you previously created
    11. Select “Use an existing Virtual Hard Disk” and select the Windows 8 DP VHD or VHDX file
    12. Click next twice.
    13. Now in the list of machines in the Hyper-V Manager, right click on your machine and start.


Chances are that your machine will not boot and ask you to replace the disk with a bootable one.


Here’s how to fix that :

    1. Double click on the virtual machine line to open a Display Console
    2. In the Media menu, select “Insert Disk” and select the Windows 8 CP iso file
    3. Reboot the virtual machine using the reset button in the toolbar
    4. Once on the Windows 8 install welcome screen, type Shift+F10, this will show a command line window
    5. Run Diskpart
    6. Type “list volume
    7. Type “select volume 1
    8. Type “list partition” and find the “System Boot” partition or the primary partition
    9. Type “select partition 1” (replace the 1 with the partition listed at the previous step)
    10. Type “active
    11. Then type “exit
    12. Back at the command line, you should be on the X: drive
    13. Type the following “bcdboot c:\windows /s c:” (you may need to replace the second “c:” with the drive that is the System Boot, when installing on a physical machine)
    14. Once done, close the installer window to exit the installation and reboot
    15. You’re done !


Note that this section can be used to register pre-installed Windows 7 or 8 VHD to an existing Windows 7 or 8 physical machine. But in this case, you may need to find the actual hidden boot partition that the Windows Installation creates automatically. You’ll find that partition with the “list partition” command.

Happy Windows 8 CP’ing !

A bit of IT in developer's world: services.exe high CPU usage

By jay at March 24, 2011 12:36 Tags: , , ,

One of the advantages of virtualization is the P2V (Physical to Virtual) process: Converting an "old" build machine to a VM so it can be moved around with the load as-is, snapshotted, backed-up and so on.

This is particularly useful when say, you have a build machine that's been there for a very long time, has a lot of dependencies over old third party software, has been customized by so many people (that have long left the company) that if you wanted to rebuild that machine from scratch, it would literally take you weeks of tweaking to get it to work properly. And that machine is running out of very old hardware that may break at any time. And that the edition of Windows that does not migrate easily to new hardware because of HAL or Mass Storage issues, requiring a reinstallation. That a lot of "ands".

That's the kind of choice you do not need to make: You just take the machine and virtualize it using SCVMM 2008 R2.

But still, even virtualized, the machine been there that long, and things have started falling apart, like having the services.exe process taking 100% of the CPU. And I did not want to have to rebuild that machine just because of that strange behavior.

If you read scott hanselman's blog, you've been recalled that Windows Server 2008 and later has the resource monitor that gives a wealth of information about the services running under services.exe. But if you're out of luck, like running under Windows Server 2003, you can still use Process Explorer. This will give you the similar kind of insight in the Windows Services that are running.

For my particular issue, this was actually the Event Log service that was taking all the resources.


How about I get my CPU back ?

After some digging around, I noticed that :

  • All Event Viewer logging sections in the MMC snap-in were all displaying the same thing, which was actually a mix of all the System, Application and Security logs.
  • Displaying any of these logs was taking a huge amount of time to display.
  • The HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\System\Source was containing something like "System System System System System System System" a hundred of times, the same thing for the keys for the other event logs
  • A whole bunches (thousands) of interesting sources named like some .NET application domain created by the application being built on this machine

To fix it, a few steps in that order :

  • Disable the Event Log service and reboot. You won't be able to stop it, but at the next reboot it will not start.
  • In the C:\WINDOWS\system32\config folder, move the files *.evt to a temporary folder, so they don't get picked up by the service when it'll restart
  • In the registry, for each HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\[System|Security|Application]\Source, replace the content with the one found on the same key on another very similar Windows Server 2003 machine. You can install a brand new machine and pick up the content.
  • If you have, like I did, a whole bunch of sources that look familiar and should not be there under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\[System|Security|Application], remove their keys if you removed them from the "Source" value.
  • Set the Event Log service to "Automatic" and reboot.


The interesting part about the virtualization of that build machine is like in many other occasions, the snapshots, where you can make destructive changes and go back if they were actually too destructive.


What's with the event log "interesting sources" ?

The application being built and tested is running tests on the build machine, and it makes use of application domains and log4net. Log4net has an EventLogAppender that allows the push of specific content to the Windows Event Log. Log4net defaults the name of the source to the application domain name, if there is no entry assembly.

Those tests were actually using a default configuration, and were logging Critical messages to the event log, but the domains were created using a new GUID to avoid supposed name collisions. This is something that did actually more harm than good in the long run, because each new appdomain that was logging to the event log was creating a new event source.

And the build system has been there for a long time. Hence the thousands of "oddly named" event sources.


Virtual Machines, Snapshots, Automated Tests and Machine Trust Account

By jay at December 14, 2010 21:47 Tags:

During the development of a project, you may need at some point to automate the testing of your whole application. Virtual Machines make that very easy, especially with Visual Studio Lab Management in the loop.

You can have a step in your nightly build to have your application installed silently, and then have some acceptance tests running on it.

To make all this repeatable and mostly predictable, you can use snapshots of a clean environment and restore that environment before starting your test scenarios. That way, any previous test run does not affect the new run, as long as the tests can run confined in a single machine. For multiple machines tests, like a web front end and DB backend, you may need to synchronize all of the snapshots, but this is definitely doeable.

The case of the VM part of a Domain

Your tests may also need to have a domain account to run. To do this, you join your VM to the domain, then make a snapshot.

Your automated tests run fine for about a month, then start to fail for reasons like :

The trust relationship between this workstation and the primary domain failed.
Or something a bit different like :
This computer could not authenticate with \\MYDC, a Windows domain controller for domain MYDOMAIN, and therefore this computer might deny logon requests.
Which means that the machine account registered with the domain is out of sync.
For a bit of background, the machine account is used by windows subsystem, but also the "NT AUTHORITY\System" account to communicate with the domain to apply GPOs, for instance. This account is also used for many other things, like in SCVMM, to ensure that the server has access to a host without requiring a "real" account that has administrative access to that host.

Password Renewal


That machine trust account has a password like a normal user account, even though it is not accessible to the users. That password is changed periodically, every 30 days or so, and that change is initiated by a request of the machine tied to this account. It is designed in such a way that it allows machines to be offline for long periods and not get out of sync because the DC would have changed the password unilaterally.
At this point, you're probably seeing why that account information can be out of sync when using VMs and reverting to snapshots.
When joining the domain, the account is in sync, and it is possible to revert to that snapshot until that 30 days window is reached. Then, the live snapshot of the machine asks for password renewal of the trust account, and then you revert to your original snapshot. This is where the problem occurs, the machine cannot authenticate on the domain anymore because it is using the previous password.


Keeping the password in sync


There are a few techniques to keep that password in sync, the first being a leave and join domain sequence. This is fairly easy to do, but there are some caveats.
When you leave and re-join, you get a new SID for that machine, which means that if you gave access rights to the previous machine account on an other machine, you're forced to update your ACLs on that other machine.
But there is a lesser known technique based on the netdom command, to reset that trust account password : 
netdom resetpwd /server:MYDC /userd:MYDOMAIN\myuser /passwordD:* /securepasswordprompt​

This needs to be run on the target machine, using an account that has the rights to update machine accounts, which is most probably the same account you used to add your machine to the domain.

Making it permanent


You still don't want to have to reset that password every two weeks, so you can use the DisablePasswordChange registry setting for that, which will disable the update of the machine account password.
This will make your VM a bit more vulnerable to password based attacks to hijack your machine account, just so you know.

Hyper-V VM Mover on CodePlex

By jay at June 29, 2010 19:37 Tags: , ,

I've decided, after a long time, to publish the source code of my little utility on CodePlex : http://vmmove.codeplex.com

It that allows to perform attach and detach operations of Hyper-V VMs.

I discussed a while back the origin of Hyper-V VM Mover, and as of now, Microsoft still has no official method for attaching and detaching VMs without export and import operations.

Feel free to submit updates and comments on the tool !

Thoughts on Migrating from WSS 3.0 to SharePoint Foundation 2010

By Admin at June 03, 2010 21:31 Tags: ,

Cet article est disponible en francais.

I upgraded recently a WSS 3.0 Farm to Sharepoint Foundation 2010, I tought I’d share some of notes and pitfalls I found during the upgrade.

My setup is built around two Windows Server 2008 R2 64 Bits VMs hosted on Hyper-V Server R2, one VM for the frontend, one for the Database (SQL Server 2008 SP1 64 Bits) and the Search Server Express.


Hyper-V Assisted Upgrade

Having the setup built on Hyper-V saved me a great deal of time, primarily by the use of snapshots taken at the same time on both machines. This helped a lot to be able to experiment directly on the production system during an expected downtime for the users.

The snapshots allow a trial and error process that lead to a somehow “perfect” environment where mistakes can be reversed pretty easily. Ugrading using this snapshot technique is however pratical only with enough disk space and a reasonable content database size depending on the physical hardware.



Here are the steps I followed to perform the upgrade :

  • Made clones of both machines in a VM library, just to be safe in case the Hyper-V messed up the VMs because of the snapshots (you never know)
  • Upgraded WSS 3.0 with latest Cumulative Updates (KB978396)
  • Downloaded Sharepoint Foundation and Search Server Express packages
  • Installed the prerequisites of Sharepoint Foundation on both VMs (not Search Servers prerequisistes, which do not install properly, and are seemingly the same as the SPF package)
  • Installed SQL Server 2008 SP1 Cumulative Updates KB970315 and KB976761 (in this order)

That’s the easy part, where updates do not impact the running farm.

I took a snapshot of both VMs at this point.


Upgrading SharePoint and Search Server

You may want to read information on this on technet, which is very extensive.

Now, the Sharepoint upgrade :

  • Put a site lock in place (just in case a user might try to update stuff he’d probably lose)
    • stsadm -o setsitelock -url http://site -lock readonly
  • Detached the content databases using : (See later for the explanation of this step)  :
    • stsadm.exe -o deletecontentdb -url http://site –databasename WSS_Content
  • Backup the content DB so they can be upgraded on an freshly installed SPF2010 setup.
  • Executed the Search Server Express setup on both machines, without configuring it
  • Upgraded Sharepoint Foundation setup, without configuring it
  • On the frontend VM (so the admin site can track the update jobs), ran the Sharepoint Configuration wizard to perform the upgrade. I selected the Visual Upgrade so the site collections templates would use the new visual style (Ribbon powered !)
  • After the configuration ended on the frontend machine, ran the same wizard on the database VM.
  • Let the jobs run and finish properly.
  • On a temporary empty SPF 2010 setup on a spare VM, mounted the backed up content DB and ran this powershell command :
    • Mount-SPContentDatabase -Name WSS_Content -DatabaseServer db.server.com -WebApplication http://site –Updateuserexperience
    • You may require to install the templates used on your production environment to perform the upgrade properly.
  • After the content db upgrade ended, detached the content database using the admin site (beware of the dependencies between content DBs if you have more than one)
  • Backuped up the content DB.
  • On the production Farm, dropped and recreated the appropriate Web Application without any site collection. I did this step to make sure the AppPools and sites were configured properly.
  • Restored and attached the content DB on the production server using the SPF admin site.

Now, I did perform the “Attach/detach” procedure because it can be performed in parallel on multiple farms if the content databases are huge, and on my production setup, the in-place upgrade did not work properly. The images libraries where not upgraded properly (images where not displayed), and the default pages did not render properly for some obscure reason.


A few additional gotchas

  • I had a few other issues with the search SSP, where I needed to remove completely the Search SSP and recreate it to avoid this error :
    • CoreResultsWebPart::OnInit: Exception initializing: System.NullReferenceException
  • The search SSP needs uses of the Security Token Service Application, which by default does use the “Extended Security” setting, which needs to be turned off in IIS.
  • Since I use Search Server Express, Content Databases must not select the search provider named “Sharepoint Foundation Search Server” for the search to work properly.


Upgraded Wikis

You’ll find the the wiki editor has been greatly enhanced, and you’ll find it even more powerful when you select the “Convert to XHTML” option in the html menu of the ribbon. The original pages were using the very loose HTML 4.0, which does not seem to work very well.

Other than that, everything else works very fine in this area.


Upgraded Discussion Boards

I had a few discussion boards that had issues with the view of individual conversations using the threaded view. The conversation were defaulted to the default “subject” view, which is not applicable to view single threads. To fix this, make a new “subject” view and delete the previous one the normal behavior will come back.


Happy SharePointing ! Now I can go back to my Immutability and F# stuff :)

Hyper-V, CPU Load and System Clock Drift

By jay at October 14, 2009 20:39 Tags:

Cet article est disponible en francais.

Using Hyper-V Server, you may find that the time is drifting a lot from the actual time, especially when Guest Virtual Machines are using CPUs heavily. The host OS is also virtualized, which means that the load of the host is also making the clock drift.

How to prevent the clock from drifting

  1. Disable the Time Synchronization in the Integration Services. (Warning, this setting is defined per snapshot)
  2. Import the following registry file :

    Windows Registry Editor Version 5.00


    Note: If you are using notepad, make sure to save the file using an Unicode encoding.

  3. If the guest OS (and the server OS) is not on a domain, type in the following to set the time source :

    w32tm /config /manualpeerlist:"time.windows.com,0x01 1.ca.pool.ntp.org,0x01 2.ca.pool.ntp.org,0x01" /syncfromflags:MANUAL /update

    Note: Hosts FQDN are separated by spaces.
  4. Run the following command to force a time synchronization

    w32tm /resync
  5. Check that the clock is not drifting anymore by using this command :

    w32tm /monitor /computer:time.windows.com


A bit of background ...

The system I’m currently working on is heavily based on time. It relies a lot on timestamps taken from various steps of the process. If for some reason the system clock is unstable, that means the data generated by the system is unreliable. It sometimes generates corrupt data, and this is not good for the business.

I was investigating a sequence of events stored in the database in order that could not have happened, because the code cannot generate it this way.

After loads of investigation looking for code issues, I stumbled upon something rather odd in my application logs, considering that each line from the same thread should be time stamped later than the previous :

2009-10-13T17:15:26.541T [INFO][][7] ...
2009-10-13T17:15:26.556T [INFO][][7] ...
2009-10-13T17:15:24.203T [INFO][][7] ...
2009-10-13T17:15:24.219T [INFO][][7] ...
2009-10-13T17:15:24.234T [INFO][][7] ...

All the lines above were generated from the same thread, which means that the system time changed radically between the second and the third line. From the application point of view, the time went backward of about two seconds and that also means that during that two seconds, there were data generated in the future. This is not very good...


The Investigation

Looking at the Log4net source code, I confirmed that the time is grabbed using System.DateTime.Now call, which excludes any code issues.

Then I looked at the Windows Time Service utility, and by running the following command :

w32tm /stripchart /computer:time.windows.com

I found out that the time difference from the NTP source was very different, something like 10 seconds. But the most disturbing was not the time difference itself, but the evolution of that time difference.

Depending on the load of the virtual machine, the difference would grow very large, up to a second behind in less than a minute. Both the host and the guest machines were exposing this behavior. Since Hyper-V Integration Services are by default synchronizing the clock of all the virtual machines on the guest OS, that means that the load of a single virtual machine can influence the clock of all other virtual machines. The host machine CPU load can also influence the overall clock rate, because it is also virtualized.


Trying to explain this behavior

To try and make an educated guess, the time source used by windows seems to be the TSC of the processor (by the use of the RDTSC opcode), which is virtualized. The preemption of the CPU by other virtual machines seems to have an negative effect on the counter used as a reference by windows.

The more the CPU is preempted, the more the counter drifts.


Correcting the drift

By default, the Time Service has a “phase adjustment” process that slows down or speeds up the system clock rate to match a reliable time source. The TSC counter on the physical CPU is clocked by the system Quartz (If it is still like this). The “normal” drift of that kind of component is generally not very important, and may be related to external factors like the temperature of the room. The time service can deal with that kind of slow drift.

But the default configuration does not seem to be a good fit for a time source that drifts this quickly and is rather unpredictable. We need to shorten the process of phase adjustment.

Fixing this drift is rather simple, the Time Service needs to correct the clock rate more frequently, to cope with the load of the virtual machines that slow down the clock of the host.

Unfortunately, the default parameters on Hyper-V Server R2 are those of the default member of a domain, which are defined here. The default polling period from a reliable time source is way too long, 3600 seconds, considering the drift faced by the host clock.

A few parameters need to be adjusted in the registry for the clock to stay synchronized :

  • Set the SpecialInterval value to 0x1 to force the use of SpecialPollInterval.
  • Set SpecialPollInterval to 10, to force the source NTP to be polled every 10 seconds.
  • Set the MaxAllowedPhaseOffset to 1, to force the maximum drift to 1 second before the clock is set directly, if adjusting the clock rate failed.

Using these parameters will not mean that the clock will stay perfectly stable, but at the very least it will correct itself very quickly.

It seems that there is a hidden boot.ini parameter for Windows 2003, /USEPMTIMER, which forces windows to use the ACPI timer and avoid that kind of drift. I have not been able to confirm this has any effect at all, and I cannot confirm if the OS is actually using the PM Timer or the TSC.

A tool to move an Hyper-V Virtual Machine without exporting it

By jay at February 20, 2009 20:26 Tags:

Cet article est disponible en francais.

Hyper-V is a wonderful tool, and provides great performance and stability. But on the administration side, available tools are a bit scare, and even though most general operations are available, some are a bit hard to use. One can guess that this will improve in Windows Server 2008 R2.

But for now, the administration tools do not provide any mean to import a VM that has not been previously exported. Exporting a VM can only be done on the original host server while it is still running. In the case of a crashed server, exporting a VM becomes a bit more complex.

Some techniques do exist, here and there, that explain by means of mklink and icacls, how to recreate symbolic links and file permissions for the VM configuration files. But that stays a particularly complex task, mostly because all files must be modified, and a specific order must be respected for all modifications. And this is especially true for the case of a running host server.

After having dug in Hyper-V symlinks and WMI interface, I created a GUI tool that allows to attach and detach VM that have not been previously epoxted.

Some thougts on this tool :

  • A VM can only be detached if it is in the "Saved" or "Stopped" state.
  • It is not necessary to stop the Hyper-V service and all modifications are detected live by the service.
  • A VM can only be imported if it contains at least on HDD on the IDE 0 controller.
  • All the VM files must be under the same directory, HDD and snapshots.
  • All files that are modified are backed-up next to the original files; All other files are not modified nor moved.
  • .NET 3.5 must be installed.

I'll provide the sources for this tool in a near future, as well as a console version.

Of course, there will be bugs, and do not hesitate to report them to me. I may not be able to do anything about it because it is a tool that performs a operation that is (I assume) not supported by Microsoft.

You can download the tool here.

How to convert a (big) VMWare VMDK into an Hyper-V VHD

By Jay at October 16, 2008 19:36 Tags:

Cet article est disponible en francais ici.

I've been playing a lot with Hyper-V lately and quite frankly, I'm very pleased with it.

VMs a very responsive, the IO is not the bottleneck it was before, the impact of VMs on the host is far lower than with VPC or VMWare, it supports snapshots and fine grained ACLs on each VMs.The performance part is subjective as always, but I find it faster, and better integrated in the OS that other products. And now, there is a free version of Windows 2008 Server named Hyper-V Server, which allows to run a bare minimum text-mode only version of windows Just Hyper-V and the VMs.

By the way, I partially agree with Paul Thurrott on the "GUI user experience" of Hyper-V which is not very elegant, involving some scripting and information found on blogs, but hey, this is a server product. This is definitely not for the average joe that does not know a bit of what he is doing.

I've had recently to port a VMWare VM to Hyper-V and the main disk was created as a fixed length (flat) -- disk of 80GB, which is obviously a VMDK file of 80GB.

First, I tried converting the file with the VMDK to VHD converter, which unfortunately does not seem to support big flat disks. I already tried converting disks with this tool, I know for a fact that is does work, so it must be the size of the file.

Then I tried using the VMWare Virtual Disk Manager to convert the flat VMDK to a multiple 2GB VMDK spanned file. After the conversion, the VMDK to VHD converter worked perfectly by converting my spanned VMDK to a flat VHD disk compatible with Hyper-V.

This does not end here however, because mass storage drivers installed by Windows at install time for VMWare VMs are not compatible with the one Hyper-V is using. This leads to a nice BSOD saying INACCESSIBLE_BOOT_DEVICE, described here and by the KB314082.

There's two ways to fix this : Create the reg file from the KB article and merge it when the VM is running under VMWare, or mount the VHD disk into an other Hyper-V VM and merge the reg file in the SYSTEM hive of the target OS.

In my case the first possibility was out of the question; Moving the disk again to another machine would have been a waste of time.

That left me with the registry hive mounting solution. Here's how to integrate the registry file :

  • Use another VM to mount the target VHD, to be able to see the SYSTEM hive file.
  • Using the regedit, load the system hive from System32\config\system into HKLM\temp.
  • Modify the KB reg file replace every reference to "SYSTEM\CurrentControlSet" by "temp\ControlSet001", and import it to update the default boot configuration
  • Modify again the KB reg file to replace every reference to "SYSTEM\CurrentControlSet" by "temp\ControlSet002", to modify the "Last Known Good Configuration", and import it , just in case.
  • Unload the hive.
  • stop the current VM.
  • Boot the new VM using the converted VHD disk, and voilà !
This is time consuming, but worth the trouble, the VM is now working properly under Hyper-V.

About me

My name is Jerome Laban, I am a Software Architect, C# MVP and .NET enthustiast from Montréal, QC. You will find my blog on this site, where I'm adding my thoughts on current events, or the things I'm working on, such as the Remote Control for Windows Phone.