There are some differences in setting up the CommVault Simpana NDMP iDataAgent in combination with NetApp Clustered Data OnTap (CDOT) compared to a 7-mode filer.
- Commvault Sp19
- Commvault V11 Sp12 Hotfixes
- Commvault Sp20
- Commvault V11 Sp12 Documentation
- Commvault Sp15
- Commvault Sp16
This post explains the configuration process to start backing up snapshots stored on a SnapVault NetApp running CDOT by using two-way NDMP.
Install CommVault V11 SP12 to replace existing Veeam backup solution for multi-location infrastructure utilizing Exagrid for the back end target. Implemented AppAware backups for MS Exchange,. We're using CommVault 10 (SP12). We're currently seeing 63MB/s restore speeds and about half that for backup speeds. I understand that it's a 'how long is a piece of string question', however just looking for some general input for CV users who may have a simliar setup. The CommVault recommended sizing for the Index server is significant. Based on up to 500 million messages (of approx 150KB each) the following servers specifications are advised for index server without content indexing: 16 cores; Index storage space of 3 to 5 percent of application size; 48GB RAM. New Metallic VM & Kubernetes backup solution broadens Commvault’s comprehensive container storage and protection portfolio Tinton Falls, NJ – November 17, 2020 – Today Commvault (NASDAQ: CVLT), a recognized global enterprise software leader in the management of data across cloud and on-premises environments, announced new offerings that provide customers enterprise-grade data protection.
In case you are using three-way NDMP to protect the data stored on the filer, some of the steps are not required. I will list those throughout the procedure.
If you need a better understanding about two-way NDMP vs. three-way NDMP, please refer to my blogpost called: “Lingo Explained: 2-way NDMP vs. 3-way NDMP”.
Before you start, ensure the CommVault Simpana software is running on a supported version. My environment is running CommVault Simpana 10 Service Pack 12 when I performed this integration. More information about the CommVault configuration prerequisites and – requirements can be found on CommVault BOL. The configuration process of NDMP on NetApp CDOT is outlined in a document called “Clustered Data OnTap 8.3 NDMP Configuration Express Guide“.
GENERAL RECOMMENDATIONS:
- Try to use controller-based protocols (such as CIFS & NFS) as much as possible. The filer is able to see the contents in these folders, which results in an increased flexibility in the restore process (compared to FC – or iSCSI LUN emulation).
- Create one NDMP subclient per volume instead of using one subclient containing all volumes. By splitting a higher level of performance can be achieved as data restores require less tapes to be mounted resulting in an increased recovery (limitation of the way NDMP works).
- NDMP backup data cannot be used in combination with Content Indexing (End User Search). If this is a business requirement, consider to use a pseudo client.
- Data archiving and stubbing is not supported on NDMP subclients. If this is a business requirement, consider to use a pseudo client.
PREREQUISITES:
- Before executing the steps outlined in this procedure, verify if all required DNS-entries are created.
- In case you are using 2-way NDMP, make sure the tape devices are presented to all controllers within the cluster before executing the procedure outlined below.
CONFIGURATION:
STEP 1: Configure an NDMP user on the filer(s) which will be used to access the volumes and perform the backup process. The account needs to be configured in the cluster object with the following permissions: “ontapi, password, admin, ssh & backup“. Or refer to the YouTube video below for additional guidance.
STEP 2: To ensure the device file names do not change after a reboot or device rescan, the following option needs to be configured on all NDMP filers requiring data protection: “options -option-name tape.reservations -option-value persistent“.
STEP 3: Open the CommVault Console and let’s add the NetApp cluster as an NDMP client. On the tab “Tools” and choose “New Client > NAS“. In the pop-up fill in the following parameters:
- Server Hostname: FQDN of the NetApp Cluster;
- Login account: the account name;
- Password: the pasword generated by the NDMP process (this password is not the same as the NDMP account password!);
- Listen port: default port 10000.
STEP 4: consult the NetApp Cluster NDMP Client properties. Make sure “Physical File Server/Cluster File Server” is defined as “Client Type“.
STEP 5: I mentioned above there is a difference in configuring NDMP with NetApp 7-mode and NetApp Clustered Data OnTap. In an NDMP configuration with NetApp 7-mode, this step is not required.
In the CommVault Console, navigate to the “Storage” tab and click on “Array Management“. Click “Add” to configure the NetApp Cluster as an array in the CommVault Simpana software.
Commvault Sp19
STEP 6 (2-way NDMP config): The physical nodes are not required to be added in 3-way NDMP as the tape write-out is performed by a Media Agent (or remote NDMP). The credentials to add the physical nodes are the same as the ones from the cluster object.
In the CommVault Simpana Console, navigate to the tab “Tools” and choose “New Client > NAS“. In the pop-up fill in the following parameters:
- Server Hostname: FQDN of the NetApp Controller which is part of the added cluster;
- Login account: the account name;
- Password: the NDMP password from the NetApp cluster.
- Listen port: default port 10000.
Follow this procedure to add all nodes of the NetApp cluster.
STEP 7 (2-way NDMP config): Open the properties of the recently added physical nodes and verify if the controller is listed as “Physical File Server/Cluster File Server” as “Client Type“.
STEP 8 (optional): To ease out management I renamed my clients with a prefix:
- “NOD_” to identify the physical nodes
- “CLU_” to identify the cluster
STEP 9 (2-way NDMP config): It’s time to discover the tape devices! In the CommVault Simpana Console, navigate to the tab “Storage” and click on “Library and Drive“. In the Media Agent selection, select the Media Agent which shall host the NDMP drive pool of the library.
Click on “Start > Detect/Configure Devices…“. In the Detect Library window, select the option “NDMP Devices” and select the Media Agent which will host the NDMP Drive Pool. Click “OK” to initiate the device discovery process.
Remark: Device discovery and – configuration is only required on the physical controllers. Devices do not need to be configured on the cluster object. The Cluster-Aware-Backup extension will take care of the routing through the hardware virtualization layer.
STEP 10 (2-way NDMP config): After successful device discovery, the library needs to be re-configured. Right-click on the library and choose “Configure“. Once prompted, select the option “Library and all drives“. The device configuration process can take some time, depending on the number of media agents and devices to be configured.
STEP 11 (2-way NDMP config): Disable the SCSI reservation in the library properties as this can cause some device lock errors. As all the NDMP backup operations are instructed by the CommVault Simpana software, there is no need to keep those enabled. SCSI reservation can be disabled in the “Library Properties” on the tab “Drive” in the section “SCSI Reservation“.
STEP 12 (2-way NDMP config): It’s not advised to re-use an existing Storage Policy to protect NDMP data. Instead, create a new storage policy and assign this to an NDMP subclient. To create a new storage policy, right-click on “Storage Policies“, choose “New Storage Policy” and configure the storage policy settings as per business requirement. To force the filer to perform the dump itself, right-click on the “storage policy copy” and choose “All Tasks > Change Data Path“.
In the data path configuration, select the “Library“, “Master Drive Pool” and “Scratch Pool“. In the “Drive Pool” dropdownlist, select the “NDMP drive pool” on the Media Agent. The easiest way to find the drive pool identification, is by consulting the “Data Paths” on the tape devices in “Library and Drive“. NDMP drive pools are defined with the prefix “NDMP:MediaAgent“.
Once the data path has been changed to NDMP, add the remaining data paths for the other filers (physical controllers only) in the “Storage Policy > Data Paths“. There is no need to change the “Data Path Configuration“, the default configuration “Use preferred data path” is optimal!
STEP 13: Before we are able to configure the first NDMP backup, we need to discover the Virtual Machines on the NetApp CDOT Cluster. In the CommVault Simpana console, navigate to the “NetApp Cluster Client” in the “Client Computers” section. In the Client Properties on the tab “Storage Virtual Machine“, click on “Detect“. Once the detection process is complete, add the discovered “Storage Virtual Machines” to the “Associated Storage Virtual Machines“. Click on “OK” to save the configuration.
STEP 14: The CommVault software and environment has been configured properly. It’s now time to define the first NDMP SubClient and associate it with the Storage Policy created in STEP 12. Create a new subclient on the NetApp Cluster Client with the following content: “/StorageVirtualMachine/Volume/.snapshot/SnapshotName“, where:
- StorageVirtualMachine is the SVM hostname hosting the volume;
- Volume is the volume name;
- SnapshotName is the specific snapshot file to be backed up.
STEP 15: Execute a manual full backup of the NDMP subclient to verify if it completes successfully. Upon completion, open an RDP session to the media agent hosting the NDMP drive pool. On the media agent open the logfile “NasBackup.log” and verify if the Cluster-Aware-Backup (CAB) Extension is being used to perform a node-scope NDMP backup.
STEP 16: Last but not least, perform a restore of the volume to verify the restore process. Upon completion, navigate to the media agent again and check the logfile “NasRestore.log” to verify if the restore is performed by the filer hosting the restore volume and not by another filer or media agent.
Thanks for reading!
-Ruben
There are some differences in setting up the CommVault Simpana NDMP iDataAgent in combination with NetApp Clustered Data OnTap (CDOT) compared to a 7-mode filer.
This post explains the configuration process to start backing up snapshots stored on a SnapVault NetApp running CDOT by using two-way NDMP.
In case you are using three-way NDMP to protect the data stored on the filer, some of the steps are not required. I will list those throughout the procedure.
If you need a better understanding about two-way NDMP vs. three-way NDMP, please refer to my blogpost called: “Lingo Explained: 2-way NDMP vs. 3-way NDMP”.
Before you start, ensure the CommVault Simpana software is running on a supported version. My environment is running CommVault Simpana 10 Service Pack 12 when I performed this integration. More information about the CommVault configuration prerequisites and – requirements can be found on CommVault BOL. The configuration process of NDMP on NetApp CDOT is outlined in a document called “Clustered Data OnTap 8.3 NDMP Configuration Express Guide“.
GENERAL RECOMMENDATIONS:
- Try to use controller-based protocols (such as CIFS & NFS) as much as possible. The filer is able to see the contents in these folders, which results in an increased flexibility in the restore process (compared to FC – or iSCSI LUN emulation).
- Create one NDMP subclient per volume instead of using one subclient containing all volumes. By splitting a higher level of performance can be achieved as data restores require less tapes to be mounted resulting in an increased recovery (limitation of the way NDMP works).
- NDMP backup data cannot be used in combination with Content Indexing (End User Search). If this is a business requirement, consider to use a pseudo client.
- Data archiving and stubbing is not supported on NDMP subclients. If this is a business requirement, consider to use a pseudo client.
Commvault V11 Sp12 Hotfixes
PREREQUISITES:
- Before executing the steps outlined in this procedure, verify if all required DNS-entries are created.
- In case you are using 2-way NDMP, make sure the tape devices are presented to all controllers within the cluster before executing the procedure outlined below.
CONFIGURATION:
STEP 1: Configure an NDMP user on the filer(s) which will be used to access the volumes and perform the backup process. The account needs to be configured in the cluster object with the following permissions: “ontapi, password, admin, ssh & backup“. Or refer to the YouTube video below for additional guidance.
STEP 2: To ensure the device file names do not change after a reboot or device rescan, the following option needs to be configured on all NDMP filers requiring data protection: “options -option-name tape.reservations -option-value persistent“.
STEP 3: Open the CommVault Console and let’s add the NetApp cluster as an NDMP client. On the tab “Tools” and choose “New Client > NAS“. In the pop-up fill in the following parameters:
- Server Hostname: FQDN of the NetApp Cluster;
- Login account: the account name;
- Password: the pasword generated by the NDMP process (this password is not the same as the NDMP account password!);
- Listen port: default port 10000.
STEP 4: consult the NetApp Cluster NDMP Client properties. Make sure “Physical File Server/Cluster File Server” is defined as “Client Type“.
STEP 5: I mentioned above there is a difference in configuring NDMP with NetApp 7-mode and NetApp Clustered Data OnTap. In an NDMP configuration with NetApp 7-mode, this step is not required.
In the CommVault Console, navigate to the “Storage” tab and click on “Array Management“. Click “Add” to configure the NetApp Cluster as an array in the CommVault Simpana software.
STEP 6 (2-way NDMP config): The physical nodes are not required to be added in 3-way NDMP as the tape write-out is performed by a Media Agent (or remote NDMP). The credentials to add the physical nodes are the same as the ones from the cluster object.
In the CommVault Simpana Console, navigate to the tab “Tools” and choose “New Client > NAS“. In the pop-up fill in the following parameters:
- Server Hostname: FQDN of the NetApp Controller which is part of the added cluster;
- Login account: the account name;
- Password: the NDMP password from the NetApp cluster.
- Listen port: default port 10000.
Follow this procedure to add all nodes of the NetApp cluster.
STEP 7 (2-way NDMP config): Open the properties of the recently added physical nodes and verify if the controller is listed as “Physical File Server/Cluster File Server” as “Client Type“.
STEP 8 (optional): To ease out management I renamed my clients with a prefix:
- “NOD_” to identify the physical nodes
- “CLU_” to identify the cluster
STEP 9 (2-way NDMP config): It’s time to discover the tape devices! In the CommVault Simpana Console, navigate to the tab “Storage” and click on “Library and Drive“. In the Media Agent selection, select the Media Agent which shall host the NDMP drive pool of the library.
Commvault Sp20
Click on “Start > Detect/Configure Devices…“. In the Detect Library window, select the option “NDMP Devices” and select the Media Agent which will host the NDMP Drive Pool. Click “OK” to initiate the device discovery process.
Remark: Device discovery and – configuration is only required on the physical controllers. Devices do not need to be configured on the cluster object. The Cluster-Aware-Backup extension will take care of the routing through the hardware virtualization layer.
STEP 10 (2-way NDMP config): After successful device discovery, the library needs to be re-configured. Right-click on the library and choose “Configure“. Once prompted, select the option “Library and all drives“. The device configuration process can take some time, depending on the number of media agents and devices to be configured.
STEP 11 (2-way NDMP config): Disable the SCSI reservation in the library properties as this can cause some device lock errors. As all the NDMP backup operations are instructed by the CommVault Simpana software, there is no need to keep those enabled. SCSI reservation can be disabled in the “Library Properties” on the tab “Drive” in the section “SCSI Reservation“.
Commvault V11 Sp12 Documentation
STEP 12 (2-way NDMP config): It’s not advised to re-use an existing Storage Policy to protect NDMP data. Instead, create a new storage policy and assign this to an NDMP subclient. To create a new storage policy, right-click on “Storage Policies“, choose “New Storage Policy” and configure the storage policy settings as per business requirement. To force the filer to perform the dump itself, right-click on the “storage policy copy” and choose “All Tasks > Change Data Path“.
Commvault Sp15
In the data path configuration, select the “Library“, “Master Drive Pool” and “Scratch Pool“. In the “Drive Pool” dropdownlist, select the “NDMP drive pool” on the Media Agent. The easiest way to find the drive pool identification, is by consulting the “Data Paths” on the tape devices in “Library and Drive“. NDMP drive pools are defined with the prefix “NDMP:MediaAgent“.
Once the data path has been changed to NDMP, add the remaining data paths for the other filers (physical controllers only) in the “Storage Policy > Data Paths“. There is no need to change the “Data Path Configuration“, the default configuration “Use preferred data path” is optimal!
STEP 13: Before we are able to configure the first NDMP backup, we need to discover the Virtual Machines on the NetApp CDOT Cluster. In the CommVault Simpana console, navigate to the “NetApp Cluster Client” in the “Client Computers” section. In the Client Properties on the tab “Storage Virtual Machine“, click on “Detect“. Once the detection process is complete, add the discovered “Storage Virtual Machines” to the “Associated Storage Virtual Machines“. Click on “OK” to save the configuration.
STEP 14: The CommVault software and environment has been configured properly. It’s now time to define the first NDMP SubClient and associate it with the Storage Policy created in STEP 12. Create a new subclient on the NetApp Cluster Client with the following content: “/StorageVirtualMachine/Volume/.snapshot/SnapshotName“, where:
- StorageVirtualMachine is the SVM hostname hosting the volume;
- Volume is the volume name;
- SnapshotName is the specific snapshot file to be backed up.
STEP 15: Execute a manual full backup of the NDMP subclient to verify if it completes successfully. Upon completion, open an RDP session to the media agent hosting the NDMP drive pool. On the media agent open the logfile “NasBackup.log” and verify if the Cluster-Aware-Backup (CAB) Extension is being used to perform a node-scope NDMP backup.
STEP 16: Last but not least, perform a restore of the volume to verify the restore process. Upon completion, navigate to the media agent again and check the logfile “NasRestore.log” to verify if the restore is performed by the filer hosting the restore volume and not by another filer or media agent.
Thanks for reading!
Commvault Sp16
-Ruben