AUTOMATION OF VPLEX REPLICATION USING REST API
Vijay GadwalSenior Application Developer II EMC Global Services
Terence JohnySolutions Architect EMC Global Services
2014 EMC Proven Professional Knowledge Sharing 2
Table of Contents The need for REST API Integration ............................................................................................ 3
Architecture of the solution ......................................................................................................... 5
VPLEX Element Manager API .................................................................................................... 6
Overview ................................................................................................................................ 6
VPLEX Commands Used ....................................................................................................... 6
Replication Automation Script .................................................................................................... 8
Modules implemented ............................................................................................................ 8
Perl Script Design ................................................................................................................... 8
Bibliography ..............................................................................................................................10
Disclaimer: The views, processes, or methodologies published in this article are those of the
authors. They do not necessarily reflect EMC Corporation’s views, processes, or
methodologies.
2014 EMC Proven Professional Knowledge Sharing 3
The need for REST API Integration
VPLEX® features such as non-disruptive data mobility across heterogeneous arrays, data
mobility across sites, and highly advanced cache coherency algorithms dissolve physical
barriers in and between data centers and open a whole new approach to data center design.
At the same time, customer investments in array-based storage replication play a very important
role in ITs new approach. While VPLEX as a product is designed to work with the existing array-
based replication tools—with VPLEX acting as the Virtualization layer—best practices and
special considerations need to be accounted for when the existing replication scripts need to be
leveraged.
This problem relates to a key requirement while using VPLEX with array-based storage
replication; the step to invalidate the read cache.
Since VPLEX leverages its data caching and distributed caching features, it becomes important
to invalidate the read cache for target volumes before new replication operations can be
performed on them. Performing replication operations on VPLEX virtual volumes directly can
potentially cause data corruption. This is due to the fact that replication operations are directly
performed on the backend array outside the VPLEX I/O path, causing the read cache of the
VPLEX to differ from the data on the physical disk on the array.
Expansion of this requirement comes into play when existing replication scripts need to be
leveraged and additional cache invalidation operations need to be run on the VPLEX virtual
volumes. Since the cache invalidation steps would need to be run separately on the VPLEX CLI,
the existing replication process would NOT continue to be a fully automated process, requiring
manual intervention for the VPLEX part.
The replications scripts also have to be modified to include the read cache invalidation steps
and require running on the VPLEX management servers at the specific points of the replications
scripts.
Now, consider developing these scripts and modifications on multiple machines and even at
multiple customer sites. This would involve investing in resource and time to understand each
environment, develop custom scripts for each site, and then run through a phase of test and
acceptance to ensure the proper operation of the complete replication cycle.
2014 EMC Proven Professional Knowledge Sharing 4
This would then mean an additional investment from EMC into all sites where we deploy VPLEX
and follow the steps of discovery, design, and testing to make array-based replication in a
VPLEX based environment work.
We have devised a very simple and elegant solution that will leverage the REST API interfaces
already supported on VPLEX and that would not require modification to the existing customer
array-based replication automation scripts.
In this article, we describe the architecture of the solution and explain in detail the technical
components such as the operations supported using REST APIs, the steps necessary for the
array-based replication, various modules used, and the technologies used for the scripting.
Our solution has been successfully designed and implemented in a number of Fortune 500
companies and has saved several hours of work. The level of automation provided has an
immense impact on total customer experience (TCE) and requires low or no learning curve to
adopt this solution. The simplicity and innovative method we used can lead to wider use of this
solution to rapidly integrate array-based replication scripts for VPLEX environments.
2014 EMC Proven Professional Knowledge Sharing 5
Architecture of the solution
The key operation for array-based replication in a VPLEX environment is the invalidation of the
VPLEX read cache.
This is achieved by executing these three steps.
1. Remove the Virtual Volumes (Replication Targets) from the VPLEX Storage View
- This will invalidate the read cache for the corresponding virtual volume.
2. Perform the Replication Operations
- Leverage existing array-based replication scripts.
3. Add Virtual Volumes to the VPLEX Storage View
- Maintains previous HLU information
These steps are fully automated with the help of VPLEX REST API Integration.
As part of the VPLEX 5.2 release, a new command has been supported to invalidate the read
cache directly instead of using the 3-step process outlined above for VPLEX 5.1.
Virtual-volume cache-invalidate
The solution has been implemented using a Perl script that leverages the VPLEX REST API
framework to execute remote commands on the VPLEX.
2014 EMC Proven Professional Knowledge Sharing 6
VPLEX Element Manager API
Overview
VPLEX Element Manager API uses the Representational State Transfer (REST) software
architecture for distributed systems such as the World Wide Web. It allows software developers
and other users to use the API to create scripts to run VPLEX CLI commands.
The Element Manager API supports most VPLEX CLI commands apart from session aware
commands (i.e. pushd, popd), commands that manage files (i.e. getsysinfo), and commands
that manage the security aspects of the product.
VPLEX supports the following RESTful requests:
HTTPS GET - Performs an ls of a context and shows the context attributes and sub-
contexts
HTTPS PUT - Sets values for a writeable attribute
HTTPS POST - Executes a VPLEX CLI command
Command options are sent and responses are received as JSON strings.
URIs are used to identify VPLEX objects and contexts.
HTTP POST requests are used to execute the commands to modify the storage views in the
VPLEX (adding/removing virtual volumes).
VPLEX Commands Used
The following VPLEX CLI commands will be used in the solution, and will be executed over the
REST API requests.
ls -f --attribute/clusters/<cluster-name>/exports/storage-views/<storage-view-
name>::virtual-volumes
o Lists the virtual volumes that are part of a storage view
export storage-view removevirtualvolume –v <storage-view-name> -o <virtual-volume-
name> -f
o Removes the specified virtual volume from the storage view
export storage-view addvirtualvolume –v <storage-view-name> -o <virtual-volume-
name> -f
2014 EMC Proven Professional Knowledge Sharing 7
o Adds the specified virtual volume from the storage view. The HLU number will
also be specified along with the virtual volume name.
virtual-volume cache-invalidate –v clusters/<cluster-name>/virtualvolumes/<virtual-
volume-name>
o Invalidates the cache of the specified virtual volume.
2014 EMC Proven Professional Knowledge Sharing 8
Replication Automation Script
Modules implemented
VPlexRestApiScript.pl -- the main script which in turn calls the modules listed below based on
user parameters.
VplexListVirtualVolumes
VplexRemoveVirtualVolumes
VplexAddVirtualVolumes
VplexInvalidateVirtualVolumes (Only for VPLEX 5.2)
VplexListVirtualVolumes
This module will list all the virtual volumes that are part of a particular storage view in the
VPLEX, along with their HLU numbers. This can be used to generate a list of virtual volumes to
later filter them and retain only the virtual volumes that are participating in the replication
process.
VplexRemoveVirtualVolumes
This module will remove a list of virtual volumes from a particular storage view in the VPLEX. It
can either read a filtered list of virtual volumes from a file (generated by using the
VplexListVirtualVolumes module) or remove all the virtual volumes that are part of the storage
group.
VplexAddVirtualVolumes
This module will add a list of virtual volumes back into a particular storage view in the VPLEX. It
will read the list of virtual volumes to be added from a file that is generated while using the
VplexRemoveVirtualVolumes module. This module will retain the HLU numbers of the virtual
volumes while adding them back to the storage view.
VplexInvalidateVirtualVolumes
This module will invalidate the cache of all virtual volumes belonging to a particular storage view
in the VPLEX. It will read the list of virtual volumes to be invalidated using another command.
Perl Script Design
The perl script used to perform the automation uses the perl modules shown below to achieve
specific tasks.
2014 EMC Proven Professional Knowledge Sharing 9
Net::HTTPS and Net::SSL
These modules are used to send HHTP request using SSL security to the VPLEX. The HTTP
requests are comprised of commands to be executed on the VPLEX.
JSON
This module is used to encode/decode data being passed/retrieved to/from the VPLEX. Data to
be sent for the VPLEX commands over the HTTP requests is sent in JSON format. This is a
standard format used along with REST API requests. The perl module encodes the data before
passing it to the VPEX and decodes from JSON format to normal perl format once the VPLEX
responds with the requested data.
Sending a HTTP request to the VPLEX
The code snippet shown below indicates the approach used to send the HTTP commands to
the VPLEX.
Reading the HTTP response from the VPLEX
The code snippet shown below indicates the approach used to decode and retrieve the data
from the REST API response obtained from the VPLEX.
2014 EMC Proven Professional Knowledge Sharing 10
Bibliography
VPLEX – Powerlink documentation – http://powerlink.emc.com
VPLEX central – http://one.emc.com/clearspace/community/active/vplex_central
Perl – http://www.perl.org/
VPLEX Element Manager API – Element Manager API Guide P/N 302-000-281-01
REST API – http://en.wikipedia.org/wiki/Representational_state_transfer
EMC believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION
MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO
THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an
applicable software license.