Since installing Backup Exec FP3 Hyper-V backup jobs leave a checkpoint against the virtual machine

Problem

Before installing Backup Exec 15 Feature Pack 3 (FP3) , a Hyper-V backup job created a checkpoint (usually visible in the Hyper-V manager). This checkpoint results in the creation of  an AVHDX file against each VHDX file inside the storage containing the virtual machine (VM). However at the end of the backup job the checkpoint and associated AVHDX files are merged and removed/deleted.

Since installing FP3, it can be seen that Hyper-V backups now create a checkpoint (and associated AVHDX files), however even when the backup job finishes, a checkpoint now remains linked to the VM although the checkpoint will be updated during subsequent backup processes.

NOTE: Backup Exec 15 Feature Pack 4 (FP4) and later updates or later Backup Exec versions may also see such symptoms, although the default setting for FP4 and later depends on which Feature Pack (FP) was installed prior to installing FP4 and if it was installed over an FP that already includes the choice, what the current setting for Hyper-V processing was prior to updating.

Cause

In the newer versions of Hyper-V, Microsoft have introduced a potentially faster method of performing incremental backups and Backup Exec 15 FP3 implements this new method as a server default (for Hyper-V 2012 and later installations) . This means all Hyper-V jobs (not just newly created jobs) on a given Backup Exec server will use the new settings immediately after FP3 is installed.

The basic concept of the new method is:

    •    A child checkpoint (snapshot) is created to collect the changes.
    •    On incremental backup, a new child checkpoint is created for current changes and then the parent checkpoint is backed up.
    •    When the backup is complete, the child and parent checkpoint is merged leaving only the new combined checkpoint.
    •    This combined checkpoint remains on the system and is used as the new parent for the next backup.
    •    A full backup will backup the current complete state of the VM and not just the parent checkpoint


Extra information:

Within Backup Exec this new method is known as the Faster processing method for Incremental Hyper-V backups with the original method being called the Standard processing method

Backup Exec has supported this new method since Backup Exec 2014 was released however Backup Exec 15 FP3 enabled it as the default method where prior to this, to enable it required the use of Powershell Commands (scripts) which are no longer needed against the FP3 (or later) updates.

FP4 (and later) set the default Hyper-V processing Standard when installing over FP2 and earlier update levels, but honors the current setting when installing over FP3 (or later) update levels. 

The backup job log will clearly show which method was used to perform the backup operation.

With the faster method, a checkpoint is created and remains in place against the VM even if the backup strategy is to only run full backups.

It is possible to revert to the Standard method (see information in the Solution section below), however, it should be noted that if reverting to the Standard method where at least one backup job with the Faster method has already run, then Powershell Commands will be needed to remove the Backup Checkpoints. Typically Hyper-V 2012 R2 installations will not remove the Backup Checkpoints when changing method, but Hyper-V 2012 installations usually will remove the Backup Snapshots. This is because Microsoft changed the behaviour (and the terminology) between these two Operating System versions.

The AVHDX file is often referred to as a differencing disk

The very first time a Faster method backup is performed against a VM, it will result in the Hyper-V-VMMS log on the host containing events 15070 and 10150,  reporting  errors of being unable to remove  a checkpoint. These errors can be ignored unless they happen during every backup operation.

A restore from a backup set created using one method will still work if the server settings have been changed to the other method.

If more than 1 backup checkpoint is present (when a backup job is not running) or more than 2 backup checkpoints present (when a backup job is running) then an issue has occurred that has blocked the removal of a checkpoint. As Backup Exec is calling Microsoft processes it is likely the cause is within the Operating System and not Backup Exec (unless you can see service stability issues against Backup Exec at the time of the backup.)
 

Solution

Backup Admins managing Hyper-V environments should choose between electing to continue using the default Faster method (in which case no further action is necessary) or reverting back to the Standard method. To revert to the Standard method perform the following steps:

• In the Backup Exec console click the Backup Exec button (yellow button top left) and select Configuration and Settings —> Backup Exec Settings.
• In the left sidebar of the Settings screen, click on Virtual Machines
• Change the radio button in the Hyper-V, Incremental Backup Settings to “Use the standard processing method” and click OK
• For Hyper-V Windows 2012 R2 installation then go to the Hyper-V host and open Powershell

To merge and remove all snapshots use the command
Get-VMSnapshot -VMName 'VM' -ComputerName 'HOST' | Remove-VMSnapshot
although if there are manual checkpoints against the VM that need retaining the to remove just the Backup Checkpoints use the command
Get-VMSnapshot -VMName 'VM' -ComputerName 'HOST' -SnapshotType Recovery | Remove-VMSnapshot

Where VM represents the actual name of your virtual machines as seen in the Hyper-V Manager on the host and HOST represents the name of the Hyper-V host

Note: these commands are case sensitive and can also be used to remove backup checkpoints that due to operating system errors have not been removed correctly. However, customers with these types of issue  should also review the Hyper-V-VMMS logs to understand what may have caused the condition.