product description...2018/07/12 · 2 functions dws provides you with various methods of using...

Data Warehouse Service

Product Description

Issue 03

Date 2018-07-30

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2018. All rights reserved.No part of this document may be reproduced or transmitted in any form or by any means without prior writtenconsent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei and thecustomer. All or part of the products, services and features described in this document may not be within thepurchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information,and recommendations in this document are provided "AS IS" without warranties, guarantees orrepresentations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd.Address: Huawei Industrial Base

Bantian, LonggangShenzhen 518129People's Republic of China

Website: http://e.huawei.com

Issue 03 (2018-07-30) Huawei Proprietary and ConfidentialCopyright © Huawei Technologies Co., Ltd.

i

http://e.huawei.com

Contents

1 DWS Overview.............................................................................................................................. 1

2 Functions......................................................................................................................................... 2

3 Application Scenarios...................................................................................................................6

4 Related Services............................................................................................................................. 9

5 Basic Concepts..............................................................................................................................115.1 DWS Management Concepts .......................................................................................................................................115.2 DWS Database Concepts.............................................................................................................................................. 11

6 Accessing and Using DWS........................................................................................................ 136.1 How to Access and Use DWS...................................................................................................................................... 136.2 Requirements and Limitations......................................................................................................................................156.3 DWS Charging .............................................................................................................................................................156.4 User Permissions.......................................................................................................................................................... 166.5 Creating Access Keys (AK and SK).............................................................................................................................17

Data Warehouse ServiceProduct Description Contents


ii

1 DWS Overview

Data Warehouse Service (DWS) is an online data processing database that uses the publiccloud infrastructure and platform to help you mine and analyze massive volumes of data.

AdvantagesDWS database kernel uses Huawei GAUSS200 OLAP, which is a self-developed databasekernel engine based on PostgreSQL 9.2.4. With the database kernel engine, DWS databasesare developed from a single OLTP database to an enterprise-class MPP-based OLAPdistributed database.

Compared with conventional data warehouses, DWS excels in hyper-scale data processingand general platform management and delivers the following benefits:

l With its massively parallel processing-based (MPP-based) architecture and row-basedand column-based storage, DWS is capable of processing massive amounts of data.

l Its optimized query model makes complex query operations easier than ever.l DWS makes full use of computing resources through the distributed and parallel

computing modes.l With complete monitoring and management capabilities and a proficient and efficient

service management platform, DWS enables you to quickly create a data warehousecluster, launch your service, and reduce your O&M workload so that you can spend moretime and money on your service and application development.

Data Warehouse ServiceProduct Description 1 DWS Overview


1

2 Functions

DWS provides you with various methods of using this service, such as the DWS managementconsole, DWS client, and REST APIs. This section describes the main functions of DWS.

Cluster Management

A data warehouse cluster contains nodes with the same flavor in the same subnet. Thosenodes jointly provide services. DWS provides a professional, efficient, and centralizedmanagement console, allowing you to quickly apply for clusters, easily manage datawarehouses, and focus on data and services.

Main functions of cluster management are described as follows:

l Creating clustersYou can specify the node quantity and node type based on service requirements toquickly create a cluster or purchase prepaid nodes and create a cluster.

l Managing snapshotsA snapshot is a complete backup that records point-in-time configuration data andservice data of a data warehouse cluster. A snapshot can be used to restore a cluster at acertain point in time. You can manually create snapshots for a cluster or enable automaticsnapshot creation (periodic). Automatic snapshots have a limited retention period. Youcan copy automatic snapshots to generate manual snapshots for long-term retention.When you restore a cluster from a snapshot, the system creates a new cluster with thesame flavor and node quantity as the original one, and imports the snapshot data.You can delete snapshots that are no longer needed to release the storage space.

l Scaling out clustersAs the service volume increases, the current scale of a cluster may not meet servicerequirements. In this case, you can scale out the cluster by adding compute nodes to it.Services are not interrupted during the scale-out.

l Restarting clustersRestarting a cluster may cause data loss in running services. If you have to restart acluster, ensure that there is no running service and all data has been saved.

l Deleting clustersYou can delete a cluster when you do not need it. Deleting a cluster is risky and maycause data loss. Therefore, exercise caution when performing this operation.

Data Warehouse ServiceProduct Description 2 Functions


2

DWS allows you to manage clusters and snapshots in either of the following ways:

l Management consoleUse the management console to access data warehouse clusters. After you haveregistered a public cloud account, log in to the management console and choose DataWarehouse Service.For more information about cluster management, see section Managing Clusters.

l REST APIsUse REST APIs provided by DWS to manage clusters. If you need to integrate DWSinto a third-party system for secondary development, use APIs to access the service.For details, see the Data Warehouse Service API Reference.

Enterprise-level Data Warehouses and Compatibility with Standard SQLAfter a data warehouse cluster is created, you can use the SQL client to connect to the clusterand perform operations such as creating a database, managing the database, importing andexporting data, and querying data.

DWS provides PB-level high-performance databases with the following features:

l MPP computing framework, hybrid row-column storage, and vectorized execution,enabling response to billion-level data correlation analysis within seconds

l Optimized in-memory computing based on Hash Join of Bloom Filter, improving theperformance by 2 to 10 times

l Optimized communication between large-scale clusters based on telecommunicationtechnologies, improving data transmission efficiency between compute nodes

l Cost-based intelligent optimizers, helping generate the optimal plan based on the clusterscale and data volume to improve execution efficiency

DWS has comprehensive SQL capabilities:

l Supports SQL 92 and SQL 2003 standards, stored procedures, GBK and UTF-8character sets, and SQL standard functions and OLAP analysis functions.

l Compatible with the PostgreSQL ecosystem and supports interconnection authenticationwith mainstream database ETL and BI tools provided by third-party vendors.

For details about the SQL syntax and database operation guidance, see the Data WarehouseService Database Development Guide.

Diverse Data Import ModesDWS supports efficient data import from diverse sources. The following lists typical dataimport modes. For details, see the Importing and Exporting Data.

l Object Storage Service (OBS) data import and exportl Using GDS to import and export datal Importing data from MapReduce Service (MRS) to DWSl Importing stream data from Data Ingestion Service (DIS) to DWSl Using Cloud Data Migration (CDM) to migrate data to DWSl Using the Migration Tool to migrate the Teradata/Oracle scripts to DWS

In addition, DWS supports data import using mainstream third-party ETL tools.



3

https://support.huaweicloud.com/en-us/mgtg-dws/en-us_topic_0056326003.html

https://support.huaweicloud.com/en-us/api-dws/en-us_topic_0048802577.html

https://support.huaweicloud.com/en-us/devg-dws/index.html


https://support.huaweicloud.com/en-us/mgtg-dws/dws_01_0113.html

APIs

You can invoke standard APIs, such as JDBC, ODBC, Python, and third-party psycopg2 toaccess databases in a cluster.

For details, see sections Using the JDBC and ODBC Drivers to Connect to the Cluster andUsing the Third-Party Function Library psycopg2 of Python to Connect to a Cluster.

High Reliabilityl Supports instance and data redundancy, ensuring zero single points of failure (SPOF) in

the entire system.l Supports multiple data backups, and all data can be manually backed up to OBS.l Automatically isolates the faulty node, uses the backup to restore data, and replaces the

faulty node when necessary.l Combines automatic snapshot creation and OBS storage, implementing cross-AZ DR.

Security Managementl Isolates tenants and controls access permissions to protect the privacy and data security

of systems and users based on the network isolation and security group rules and securityhardening measures.

l Supports SSL Internet connections, user permission management, and passwordmanagement, ensuring data security at the Internet, management, application, and systemlayers.For details, see section Configuring Cluster Security Settings.

Monitoring and Auditingl Monitoring a Cluster

DWS integrates with Cloud Eye, allowing you to monitor compute nodes and databasesin the cluster in real time.

l Notifying EventsDWS interconnects with Simple Message Notification (SMN) so that you can subscribeto events and view events that are triggered.

l Audit Logs– DWS integrates with Cloud Trace Service (CTS), allowing you to audit operations

performed on the management console and API invocation operations.– DWS records all SQL operations, including connection attempts, query attempts,

and database changes.

Multiple Database Tools

DWS provides the following self-developed tools. You can download related tools from theDWS management console. For details about how to use the tools, see the Data WarehouseService Tool Guide.

l gsql toolgsql is a command line SQL client tool running on the Linux OS. It is used to connect tothe database in a data warehouse cluster and operate and maintain the database.



4







https://support.huaweicloud.com/en-us/tg-dws/dws_07_0001.html

https://support.huaweicloud.com/en-us/tg-dws/dws_07_0001.html

l Data Studio toolData Studio is a Graphical User Interface (GUI) SQL client tool running on the WindowsOS. It is used to connect to the database in a data warehouse cluster, manage thedatabase and database objects, edit, run, and debug SQL scripts, and view the executionplans.

l GDS toolGDS is a data service tool provided by DWS. It works with the foreign table mechanismto implement high-speed data import and export.The GDS tool package needs to be installed on the server where the data source file islocated. This server is called the data server or the GDS server.

l Migration ToolMigration Tool is used to migrate the SQL scripts in Teradata or Oracle databases toSQL scripts applicable to DWS to restore the database on DWS.It is a command line tool running on the Linux OS. You can download and use it withoutinstallation.



5

3 Application Scenarios

DWS has features and characteristics that are useful (if not essential) for enterprises workingin different industries. It is perfect for but not limited to the following online analysisscenarios:

l Enhanced ETL + Real-time BI analysis

Figure 3-1 ETL+BI analysis

The data warehouse is the pillar of the BI system for collecting, storing, and analyzingmassive amounts of data. It provides powerful business analysis support for IoT, mobileInternet, gaming, and O2O industries.Advantages of DWS are as follows:– Data migration: efficient and real-time data import in batches from multiple data

sources– High performance: cost-effective PB-level data storage and second-level response

to correlation analysis of trillions of data records– Real-time: real-time consolidation of service data for timely optimization and

adjustment of operation decision-making

Data Warehouse ServiceProduct Description 3 Application Scenarios


6

l E-commerce

Figure 3-2 E-commerce

Huawei Vmall leverages DWS as its database engine for data analysis. Data of onlineretailers is mainly used for marketing recommendation, operating and customer analysis,and full text search.Advantages of DWS are as follows:– Multi-dimensional analysis: analysis from commodities, users, operation, and

regions– Scale-out as the business grows: on-demand cluster scale-out as the business

grows– High reliability: long-term stable running of the e-commerce system

l IoT



7

Figure 3-3 IoT

DWS helps you analyze massive amounts of data from Internet of Things (IoT) in realtime and perform optimization based on the results. It is widely used in industrial IoT,O2O service system, and IoV solutions.Advantages of DWS are as follows:– Real-time archiving of stream data: importing stream data from IoT devices and

the gateway to DWS using HUAWEI CLOUD DIS– Device monitoring and prediction: device monitoring, control, optimization,

supply, self-diagnosis, and self-healing based on data analysis and prediction– Information recommendation: tailed recommendation based on data of users'

connected devices



8

4 Related Services

IAM

DWS uses Identity and Access Management (IAM) for authentication and authorization.

Users who have the DWS administrator permissions can fully utilize DWS. To apply forDWS administrator permissions, contact a user having the security administrator permissionsor create a user with the permissions.

ECS

DWS uses an Elastic Cloud Server (ECS) as a node in the cluster.

VPC

DWS uses the Virtual Private Cloud (VPC) to provide a network topology for clusters toisolate clusters and control access.

OBS

DWS uses OBS to convert cluster data and external data, satisfying the requirements forsecure, reliable, and cost-effective storage.

MRS

DWS uses MRS to migrate data from MRS to data warehouse clusters so that DWS cananalyze and query the data after the data has been processed by Hadoop.

CDM

DWS uses CDM to migrate multiple data sources to DWS.

DIS

DIS implements real-time access of multiple data sources to DWS.

Data Warehouse ServiceProduct Description 4 Related Services


9

Cloud EyeDWS uses Cloud Eye to monitor cluster performance metrics, delivering status information ina concise and efficient manner. Cloud Eye supports alarm customization so that you arenotified of the exception immediately.

CTSDWS uses CTS to audit your non-query operations on the management console to ensure thatno invalid or unauthorized operations are performed, enhancing service security management.

SMNDWS uses SMN to actively push notification messages according to your event subscriptionrequirements, so that you can immediately receive a notification when an event occurs (forexample, a key cluster operation). For more information, see the Simple MessageNotification User Guide.

TMSDWS uses TMS to provide cross-region and cross-service centralized tag management andresource classification functions, allowing you to customize tags for resources and implementresource classification so that you can search for resources using tags.

Data Warehouse ServiceProduct Description 4 Related Services


10

https://support.huaweicloud.com/en-us/smn_gls/index.html

https://support.huaweicloud.com/en-us/smn_gls/index.html

5 Basic Concepts

5.1 DWS Management Concepts

Cluster

A cluster is a server group that consists of multiple nodes. DWS is organized using clusters. Adata warehouse cluster contains nodes with the same flavors in the same subnet. Those nodesjointly provide services.

Node

Each data warehouse cluster contains at least three nodes, all of which support data storageand analysis.

Flavor

You need to specify the node flavors when you create a data warehouse cluster. CPU,memory, and storage resources vary depending on node flavors.

Snapshot

You can create a snapshot to back up data warehouse cluster data so as to restore the clusterdata from the snapshot. A snapshot is retained until you delete it on the management console.Automatic snapshots cannot be manually deleted. Snapshots will occupy your OBS quotas.

5.2 DWS Database Concepts

Database

A data warehouse cluster is an analysis-oriented relational database platform that supportsonline analysis.

Data Warehouse ServiceProduct Description 5 Basic Concepts


11

OLAPOLAP is a major function of the data warehouse cluster. It supports complex analysis,provides decision-making support tailored to analysis results, and delivers intuitive queryresults.

MPPOn each node in the data warehouse cluster, memory computing and disk storage systems areindependent. With MPP, DWS distributes service data to different nodes based on thedatabase model and application characteristics. Nodes are connected through the network andcollaboratively process computing tasks as a cluster and provide database services that meetservice needs.

Shared Nothing ArchitectureShared Nothing Architecture is a distributed computing architecture. Each node isindependent so that nodes do not compete for resources, which improves working efficiency.

Database VersionEach data warehouse cluster has a specific database version. You can check the clusterversion when creating a data warehouse cluster.

Database ConnectionYou can use a client to connect to a data warehouse cluster in the public cloud and over theInternet.

Database UserYou can add users and control users who can access the database of the data warehousecluster by assigning specific permissions to users. The database administrator generated whenyou create a data warehouse cluster is the default database user.

Data Warehouse ServiceProduct Description 5 Basic Concepts


12

6 Accessing and Using DWS

6.1 How to Access and Use DWSFigure 6-1 shows how to use DWS.

Figure 6-1 Process of using DWS

Accessing a Cluster

DWS provides a web-based management platform. You can manage data warehouse clustersusing the management console or HTTPS-compliant APIs.l Management console

Data Warehouse ServiceProduct Description 6 Accessing and Using DWS


13

Use the management console to access data warehouse clusters. After you haveregistered a public cloud account, log in to the management console and choose DataWarehouse Service.For more information about cluster management, see section Managing Clusters.

l REST APIsUse REST APIs provided by DWS to manage clusters. If you need to integrate DWSinto a third-party system for secondary development, use APIs to access the service.For details, see the Data Warehouse Service API Reference.

Accessing Databases in a ClusterDWS supports database access using the following methods:l DWS client

You can use the DWS client to access the cluster's database. For details, see sectionConnecting to a Cluster.

l Open-source PostgreSQL clientThe following lists compatible open-source clients:– PostgreSQL psql 9.2.4 or later

For more information, see https://www.postgresql.org/.– pgAdmin

For more information, see https://www.pgadmin.org/.– dbeaver

For more information, see https://dbeaver.jkiss.org/download/.l API invocation methods such as JDBC and ODBC

You can invoke standard APIs, such as JDBC, ODBC, Python, and third-party psycopg2to access databases in a cluster.For details, see sections Using the JDBC and ODBC Drivers to Connect to theCluster and Using the Third-Party Function Library psycopg2 of Python toConnect to a Cluster.

End-to-end Data Analysis ProcessDWS has been seamlessly integrated with other services of the public cloud, helping yourapidly deploy end-to-end data analysis solutions.

Figure 6-2 describes the end-to-end data analysis processes and service in use during eachprocess.



14


https://support.huaweicloud.com/en-us/api-dws/en-us_topic_0048802577.html


https://www.postgresql.org/

https://www.pgadmin.org/

https://dbeaver.jkiss.org/download/





Figure 6-2 End-to-end data analysis process

6.2 Requirements and Limitationsl Currently, DWS is only available for users in CN North-Beijing1 and CN East-

Shanghai2. However, the team plans to roll DWS out in other regions soon.l You can manage clusters only and cannot directly access nodes in the cluster. To access

DWS, you need to access the IP address and port of the cluster corresponding to thedatabase.

l You cannot modify the flavor of an existing cluster. If you need nodes with higherflavors, create a new cluster.

l When you use a client to connect to a data warehouse cluster, the subnet of the VPCmust be the same as that of the cluster.

l If you copy commands from the Data Warehouse Service User Guide to the executionenvironment, the texts wrap automatically, causing command execution failures. Tosolve the problem, delete the line break.

6.3 DWS ChargingDWS provides two charging modes, the pay-per-use pricing (hourly) and prepaid node pricing(monthly/yearly). The longer you use, the less you pay.

l Pay-per-use pricing (hourly): In this charging mode, you can enable or disable DWS asyou like. You are charged by use duration on an hourly basis (the time is accurate tominutes) and the system generates a bill every natural hour and deducts fees from youraccount balance. This mode is applicable to customers who need to perform preliminaryoperation tests and Proofs-of-Concept verifications, and short-term users.

l Prepaid node pricing (monthly or yearly): You can choose this mode and pay the feesat a time when purchasing DWS if you plan to use this service for a long time. Thismode offers you significant discounts over pay-per-use pricing and saves you 35% to45% of costs. This mode is applicable to customers who plan to use DWS for a long timeafter the tests and verifications.The prepaid node is a charging concept only. You can regard it as a coupon. Purchasingprepaid nodes will not have the system to create a cluster. When you are using a cluster,the billing system automatically compares the number of valid prepaid nodes youpurchased with the number of nodes in the same region and flavor you are using. The



15

nodes within the prepaid node quantity range will not be charged and the extra nodes arecharged based on pay-per-use pricing.Prepaid nodes are not bound to clusters. For details, see the following descriptions:

a. After you purchase prepaid nodes, the system will not create a cluster automatically.If you have not run any clusters, go to the Data Warehouse Service console tocreate a cluster with the corresponding region, node type, and node quantity.

b. If your cluster is charged based on pay-per-use pricing, the billing systemautomatically identifies prepaid nodes and gives you discounts when generating thebill after you purchased prepaid nodes with the corresponding region, flavor, andquantity.

c. If you want to scale out your cluster and you have valid prepaid nodes in the region,the billing system automatically identifies prepaid nodes and gives you discountswhen generating the bill. If you do not have any available prepaid nodes, the newlyadded nodes will be charged based on pay-per-use pricing.

d. If you delete an existing cluster and create a new one, your prepaid nodes can stillbe used in the new cluster with the same region and node type.

NOTICEPrecautions for prepaid nodes:l Prepaid nodes are bound to a specific region. The purchased prepaid nodes can be

used in the bound region only.l Prepaid nodes are bound to a specific node flavor. Nodes with different flavors have

different prepaid prices.l Prepaid nodes take effect as of the date when they are purchased and automatically

end when the validity period expires. For example, if you purchase a package withthree nodes on January 1 at a year and the package duration is one year, then thepackage automatically ends at the end of December 31 in the year. Please note that ifyou do not run the cluster within the validity period, the validity period will not beextended and the fee cannot be refunded. Therefore, to save your costs, you areadvised to create a cluster immediately after purchasing prepaid nodes, or purchaseprepaid nodes after creating a cluster.

For details about the charging standards, see the Data Warehouse Service Price Description.

6.4 User PermissionsUser permissions are classified into the following types:

l User management: refers to the management of users, user groups, and user group rights.l Resource management: refers to the control operations that can be performed by users on

cloud service resources.For details about user and resource management permissions, see Permissions.

l Database user management: refers to the permission used to connect databases, accessdatabase objects, and execute SQL statements.For details about database user permission management, see User PermissionManagement in the Data Warehouse Service Database Development Guide.



16

https://console.huaweicloud.com/dws

https://support.huaweicloud.com/en-us/price-dws/dws_06_0001.html

https://support.huaweicloud.com/en-us/usermanual-permissions/en-us_topic_0063498930.html

https://support.huaweicloud.com/en-us/devg-dws/user_management_0001.html

https://support.huaweicloud.com/en-us/devg-dws/user_management_0001.html

https://support.huaweicloud.com/en-us/devg-dws/preface.html

6.5 Creating Access Keys (AK and SK)When users who have registered with the public cloud call APIs or use SDKs to access OBS,access keys (AK/SK) are required for identity authentication and request encryption. Thismechanism ensures the confidentiality and integrity of a request, and the correctness of theidentities of both parties.

Detailed explanations about AK and SK are as follows:

l An AK is an Access Key ID on OBS. One AK maps to only one user but one user canhave multiple AKs. OBS recognizes users by their AKs.

l An SK is the Secret Access Key on OBS, which is required as the key to access OBS.You can generate authentication information based on SKs and request header fields.SKs and AKs are in one-to-one mapping.

In the following scenarios, you need to connect to the DWS database through a client or aJDBC/ODBC application to access OBS. Therefore, you must obtain the access key (AK andSK) required for accessing OBS.

l Importing Data from OBS to a Data Warehouse Clusterl Exporting Data from a Data Warehouse Cluster to OBSl SQL on OBS

For details about the preceding features, see the following sections in the Data WarehouseService Database Development Guide:

Creating Access Keys (AK and SK)Before creating the AK/SK pair, ensure that your public cloud account (used to log in to themanagement console) has passed the real-name authentication.

To create an AK and an SK in OBS Console, perform the following steps:

Step 1 Log in to the DWS management console at https://console.huaweicloud.com/dws.

Step 2 Move the cursor to the username in the upper right corner and choose My Credential fromthe drop-down list.

Step 3 On the My Credential page that is displayed, click Access Keys.

If an AK/SK pair has been created on the Access Keys page, you can directly use it.

NOTE

Each user can create a maximum of two valid access keys. If more access keys need to be created, deleteexisting ones and create another.

Step 4 Click Add Access Key.

Step 5 In the Add Access Key dialog box that is displayed, enter the password and its verificationcode and click OK.

NOTE

l If you have not bound an email address or mobile number, you need to enter only the password.

l If you have bound an email address and a mobile number, you can use either the email address ormobile number for verification.



17

https://support.huaweicloud.com/en-us/devg-dws/import_from_obs_0001.html

https://support.huaweicloud.com/en-us/devg-dws/export_to_obs_0001.html

https://support.huaweicloud.com/en-us/devg-dws/sql_on_obs_0001.html



https://auth.huaweicloud.com/authui/login?service=https%3A%2F%2Fconsole.huaweicloud.com%2Fdws%2F&locale=en-us#/login

Step 6 In the Download Access Key dialog box that is displayed, click OK to save the access keysto your browser's default download path.

NOTE

l To prevent the AK from being leaked, keep it secure.

l If you click Cancel in the dialog box, the AKs will not be downloaded, and you cannot downloadthem later. In this case, re-create an AK.

Step 7 Open the downloaded credentials.csv file to obtain the access keys (AK and SK).

----End

PrecautionsIf you find that your AK/SK pair is abnormally used (for example, the AK/SK pair is lost orleaked) or will be no longer used, delete your AK/SK pair in the access key list or contact theadministrator to reset your AK/SK pair.

When deleting the access keys, you need to enter the login password and either an email ormobile verification code.

NOTE

Deleted AK/SK pairs cannot be restored.



18

product description...2018/07/12 · 2 functions dws provides you with various methods of using...

Documents