VMGuru.com - Home of the Virtualization Gurus

Updated to fix a minor bug on April 16, 2010

I took it as a personal challenge this week to leverage PowerShell, PowerCLI and the vSphere API to track the amount of data that changes in a VM over a regular interval window.  It turns out VMware does make it quite simple to query what blocks in a VMDK file have changed and what the length of data is as long as you know how to structure the API call.

The following script can be run as a scheduled task in Windows on a regular interval such as every 15 or 60 minutes, or even just once a day (depending on how granular you want your data) against a single VM.  This is perfect for profiling the amount of data a particular VM will need to send (uncompressed) over a network connection for backup or replication purposes. 

There are a few quick things to note about this script:

1. It requires vSphere and CBT to be enabled for the selected virtual machine.  This also means the VM Hardware version must be set to 7.  There is an optional flag of "-enableCBT $true" that can be set as a script parameter to enable CBT for the specified VM.  This executes an additional function in the script to turn CBT on.  If CBT is already enabled, the function simply exits cleanly, no harm, no foul.
2. I don't recommend running this script against a VM if there is potential overlap with other regular processes such as backup or replication that add snapshots to virtual machines.  I don't have advanced snapshot debugging in here and simply don't know how all scenarios of snapshots being added/removed from multiple programs against a single VM will play out.
3. I have it limited so a single instance of the script can be run against a single VM only.  I started off with this being a simple proof-of-concept script, and haven't built in proper multi-VM intelligence yet.  If there is enough demand, I'll definitely consider it.

You should create a standalone directory somewhere on an available hard drive of a Windows system.  Create a new PS1 file using the built in script editor in Virtualization EcoShell. Copy and paste the following code block and save the file as "CBT_Tracker.ps1" in the newly created directory:

#(c)2010 Scott Herold - VMGuru.com
#Quest Software
Param (
    [string] $vmName,
    [string] $vimHost,
    [string] $vimUser,
    [string] $vimPass,
    [Boolean] $enableCBT = $false

)
#Use the optional $enableCBT paramater at input to enable CBT on
#a compatible virtual machine (HW Version 7). In order for CBT flags to take
#effect, a Snapshot must be added and removed.
function Add-CBTFlag {
    if ($vmv.Config.ChangeTrackingEnabled){
        "CBT is already enabled on $vm"
        return
    }
    $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $vmConfigSpec.changeTrackingEnabled = $true
    $vmv.ReconfigVM($vmConfigSpec)
    sleep 3
    $vm | New-Snapshot -Name "Enable CBT"
    sleep 5
    $vm | Get-Snapshot -Name "Enable CBT" | Remove-Snapshot -Confirm:$false
    "Enabled CBT for $vm"
}

#Add a snapshot specific to the purposes of this script
function Add-CBTSnap {
    $newSnap = $vm | New-Snapshot -Name "CBT Change Tracker"
    return $newSnap
}

#Remove the snapshot created for the purpose of this script
function Remove-CBTSnap {
    param (
        $vmSnap
    )
    $vmSnap | Remove-Snapshot -Confirm:$false
}

#Captures the ChangeID of the snapshot, which is required for comparisson
#tracking purposes.  The current run of the script will always compare
#current state against the previously logged ChangeID for differences.
function Get-ChangeID {
    param (
        $vmSnap
    )
    $vmSnapv.Config.Hardware.Device | where {($_.GetType()).Name -eq "VirtualDisk"} | ForEach-Object {
        $vzCBT = New-Object PSObject
        $vzCBT.PSObject.TypeNames.Clear()
        $vzCBT.PSObject.TypeNames.Insert(0,"vzCBTTracker")
        $vzCBT `
            | Add-Member -MemberType NoteProperty -Name VMName -Value $vm.Name -PassThru `
            | Add-Member -MemberType NoteProperty -Name DeviceID -Value $_.Key -PassThru `
            | Add-Member -MemberType NoteProperty -Name ChangeID -Value $_.Backing.Get_ChangeID()
        [array] $vzCBTColl += $vzCBT
    }
    return $vzCBTColl
}

#Writes the historic ChangeID and VMDK device information to a tracking file
#This file is read on each iteration of the script to capture the previos
#iterations ChangeID inforation.
function Write-TrackerLog {
    $output = $null
    $cid | ForEach-Object {
        $output += $_.VMName + "," + $_.DeviceID + "," + $_.ChangeID + "`n"
    }
    $output | Out-File "$vm-CBTTracker.cfg"
}

#Reads previous state information of the VM from the Tracker.cfg file
#Captures the list of changed blocks for the specified VM for each VMDK file
#calculates the total size in bytes of changed block data across all VMDK
#files associated with the VM.
function Get-ChangedBlockSize {
    $chSize = 0
    Get-Content "$vm-CBTTracker.cfg" | ForEach-Object {
        if ($_ -ne "") {
            $TrackerRef = $_.split(",")
            $changes = $vmv.QueryChangedDiskAreas($vmSnapv.MoRef,$TrackerRef[1],0,$TrackerRef[2])
            $changes.ChangedArea | ForEach-Object {
                $chSize += $_.Length
            }
        }
    }
    $chSize = $chSize/1024/1024
    return $chSize
}

#Writes the actual iterations change data out to a CSV file
function Write-TrackerCSV {
    $longDate = Get-Date
    $shortDate = $longDate.toShortDateString() + " - " + $longdate.toShortTimeString()
    Add-Content "$vm-CBTTracker.csv" "$vm,$shortDate,$amtChanged"
}

#Loads VMware's PowerCLI PowerShell Cmdlets if necessary
[void](Get-PSSnapin VMWare.VimAutomation.Core -ErrorVariable getVmwareSnapinErr 2> $null)
if ($getVmwareSnapinErr.Count -gt 0) {    Add-PSSnapin VMware.VimAutomation.Core }

#Establish a connection to the VI Server based on input parameters
Connect-VIServer $vimHost -User $vimUser -Password $vimPass

#Load some VM variables
$vm = Get-VM $vmName
$vmv = $vm | Get-View

#We can only run against a single VM at this point in time.  This makes sure
#the object being passed is a single VirtualMachineImpl object, and not a
#collection.
if (($vm.GetType()).Name -ne "VirtualMachineImpl"){
    "Script can only run against a single VM at a time"
    return
}

#Checks to make sure VM Hardware version is 7.  Terminates if not.
if ($vmv.Config.Version -ne "vmx-07"){
    "The Virtual Machine $vm must be VM Hardware Version 7 to support CBT"
    return
}

#If the "-enableCBT $true" flag is set as an input parameter, enables CBT
#for specified VM if necessary
if ($enableCBT){
    Add-CBTFlag
}

#If all other checks fail, this is a final failsafe to ensure CBT is enabled
#on the VM before continuing the script.
if ($vmv.Config.ChangeTrackingEnabled -eq $false){
    "CBT is not enabled on $vm.  Script cannot proceed."
    return
}

#Load up snapshot variables
$vmSnap = Add-CBTSnap
$vmSnapv = $vmSnap | Get-View
$amtChanged = 0

#Checks to see if the script has previous written out a CBTTracker.cfg file. This
#would indicate file deletion, or first script run.  If cfg file doesn't exist,
#initializes CSV headings.  If this is an iterative run, Gets the amount of
#data that has changed since previous iteration.  Writes to CSV file.
if (Test-Path "$vm-CBTTracker.cfg") {
    $amtChanged = Get-ChangedBlockSize
    Write-TrackerCSV
}
else {
    Add-Content "$vm-CBTTracker.csv" "VMName,Time,ChangedMB"
}

#Capsture the current ChangeID to use against next iteration, Remove Snapshot
#Write cfg data.
$cid = Get-ChangeID ($vmSnap)
Remove-CBTSnap ($vmSnap)
Write-TrackerLog
 

With the script file created, use notepad or other simple text editor to create a new CMD file called "CBT_Tracker.cmd".  This is what you will execute within Windows Task Scheduler.  Paste the following line in the CMD file and edit with the proper parameters for your environment.  Make sure you copy exactly, including the quotes.

powershell -command "& '.\CBT_Tracker.ps1' -VMName VMName -vimHost vCenter.domain.com -vimuser domain\user -vimpass Passw0rd"

You can now fire up Windows Task Scheduler and point to the CMD file.  After the script runs for the first time, you will see 2 new files per VM that you set this to run against.  $VMName-CBT_Tracker.cfg is used for tracking previous snapshot history for reference in the current script iteration.  $VMName-CBT_Tracker.csv is the actual data file that contains the VMName, Timestamp, and Amount of data changed.

The CSV/Excel combination will let you do some creative things with the data output.

I've captured quite a bit of attention with my first nook eBook Reader and PDF Documents article. With my creative use of metadata and article title, I've captured more than just my normal VERY casual virtualization audience. I have actually received a few follow-up questions that I can address with a second review, which digs a bit deeper into bookmarks and scanned OCR documents.

The first request that came across was around bookmarks and highlighting and taking notes. When it comes to the PDF functionality of the nook, there are several notable missing features. First the "Highlights and notes" option is simply missing in action. Granted, I have yet to use this feature in a standard eBook, but with technical documentation or research PDFs, I see this almost as a vital option. Also noted as missing, is the "Look up word" feature.

Next, I wanted to dig deeper into the bookmark capabilities. In a regular eBook, the bookmarks are stored in the nook as their Chapter and page number. I'm currently re-reading Stephen King's The Stand, which unfortunately doesn't seem to have proper chapter marks so the whole book is one single (and super long) chapter simply called "Cover". For this book, my current bookmark is called "Cover: p. 165". Easy to understand if the eBook was properly rendered.

In my PDF, I get nothing even close to as friendly. My book is properly rendered for chapters as a part of the PDF I constructed. I can select any particular chapter from the "Go To" option in the menu. The major problem is that when I add a bookmark, it uses some fancy PDF markup, which makes absolutely no sense. In my case, I bookmarked a page describing VirtualCenter licensing on page 100. When I went to return to this bookmark, the nook instead threw out "5,#pdfloc(2506,99) at me. TECHNICALLY, my PDF page numbering seems to be off by 1. Page 99 of the PDF is displayed as Page 100 in the nook. It still doesn't explain what the heck the first portion of information is that it is throwing at me.

The final question thrown at me was around scanned OCR PDF documents. In the case and samples provided to me, it appears to be college research. I was presented with 2 documents of similar quality, one that was just a seemingly normal 1 column PDF with no highlighting. The second document was 2 columns and had Acrobat highlighting and annotations embedded in the document. The standard single column document rendered extremely well, as can be seen in the sample photos below. Paragraph breaks were properly aligned and the only weirdness I ran into was when one PDF page ended in the middle of a sentence, the nook would show that as cut off on half a page and continued on the next full page. Not a big deal and still quite readable.

Where the wheels fell off the bus is when I tried to view the 2-Column PDF that had highlights and annotations. I do not know if it was the highlighting at annotations, or if it was the multi-column rendering of the nook, but it was largely unreadable. On some pages, the document scaled down to fit both columns onto a single page, and on others, the nook properly rendered multiple pages and scaled a readable version for a few pages before flipping back to a scaled multi-column view. On the scaled pages, there was also an instance of random shadowing behind some of the text, which made it literally unusable. In the first image below, you see the situation where the document scaled onto the page. The second instance is zoomed in to see the shadow text. I had to use a slow shutter speed to low light, but you should be able to see the difference between my horrible camera hand and the text shadowing.

As a last-ditch effort, I decided to try to use the highly touted Calibre to convert the documents from PDF to ePub. By converting the PDF's to ePub, I again gained access to the "Highlights and Notes" and "Look up word" features within the documents. The problem was the formatting was garbage. There were random paragraph breaks that would split paragraphs mid-sentence and all kinds of page break issues. Calibre also had NO idea what to do with the multi-column OCR document and simply turned the document into a 3-Page document, which was the text leading up to the first multi-column page. I admittedly need to dig deeper into Calibre before stating the program was to blame, but I was better off from a readability standpoint letting the nook internally convert the PDF documents, albeit with missing features.

Again, Jeff hasn't stepped up and offered me a Kindle just yet, so can't really do any comparisons against what the Kindle can do with PDFs with its latest updates. Anyone willing to test, I'm more than willing to share source documents to get a good subjective review of both devices.