transwarp data cloud · 2019-12-05 · 02 the data age has witnessed the transformation of...

Whitepaper

Transwarp Data Cloud

Transwarp Technology (Shanghai) Co., Ltd.

Stable, Efficient, and Intelligent Data Application Cloud Platform

01


Integration of Big Data + Cloud + AI

Big data and cloud computing technologies have entered the second decade of rapid growth. As artificial intelligence technology develops and prospers, the integration of the three technologies has become a popular exploration trend. The growth of container and virtualization technology has brought big data and AI with infinite resources and computing capability. Big data and AI have extended our understanding of data scope and depth, and they also extend the application of clouds.

Recently, most Fortune 50 companies have published their plans to migrate applications to clouds. A number of start-ups have already built their IT architecture on the cloud since the very beginning and realized cloud-native applications as well. At the same time, well-known public cloud servers at home and abroad have made a deeper exploration into intelligent scenarios. They rely on big data and AI to provoke the next round of business revolution and are making every attempt to improve the way in which enterprises process big data with the support of cloud.

Against the backdrop of data cloudization, Transwarp Technology, as the leader in the development of big data technology productization, has been studying the integration of cloud computing, big data, and AI. With rich experience in big data technology, Transwarp has developed a container platform specially designed for big data. Now, a data application cloud platform product, Transwarp Data Cloud, has come into being. This platform integrates big data + cloud + AI and renders an ecosystem of data + application.

02

The data age has witnessed the transformation of enterprises, represented by Google, Facebook, and Amazon, from IT magnates to DT magnates. With technological advantages in big data, cloud computing, and AI, these enterprises are way ahead in developing data business, data assetization, and enterprise operation datafication, which speeds up the transformation of business value as well. As a result, they not only become tech leads in the industry but also achieve great business success. These big companies have evolved and transformed in several years, which can be summarized as the following.

In this phase, enterprises need to build a flexible technology platform to support large data volumes, ultra-huge data dimensions, and diverse data types. Data unification consists of building a unified computing export platform, unified metadata management and data standard, as well as data integration into the platform.

When data are unified, data integration and assetization need to be realized through data analysis. At the same time, data quality and validity should be ensured by effective data quality management. The more high-quality data accumulated on the platform, the more developers would be attracted to start the data assetization work according to the data features, which includes connecting data and business glossary as well as data management process. As a result, primitive data are converted into valuable data assets.

When data are unified and assetized, an enterprise would have a powerful computing capability and rich data assets, which makes it convenient to build data business. Currently, typical data businesses that can produce huge values are in the fields of operation datafication, intelligent applications, and online data services. These businesses integrate big data and AI technology effectively so as to quickly extract the values from massive data.

In this phase, the enterprise has created a unified platform for data, computing, and business, and a larger number of developers can conduct self-service business development on this platform. Meanwhile, businesses produce new data and assets, which attracts developers to build new business. In this sense, data, business, and developers have formed positive feedback in a complete data ecosystem.As the technologies of big data, cloud, and AI develop rapidly, although enterprises may not exactly follow the four phases above in the process of business evolution and each phase may overlap, repeat, or iterate, the technological evolution based on the four phases will surely become more mature in terms of technology and business and best suit the data-intensive strategy for enterprises.

Data Unification

Data Assetization

Data Business

Data Ecologicalization

•

•

•

Data Integration

Data Quality Management

Assetization & Measurement

Data Assetization

Operation Datafication

Intelligent Applications

•

•

• Online Data Services

Data Business

•

•

•

Closed-Loop Data Domain & Business

Operational Data

Service & Application Sharing

Data Ecologicalization

DataUnification

How does the data business of big companies evolve?

•

•

•

Unified Data Processing

Unified Metadata

Unified Computing Platform


Introduction to Transwarp Data CloudIn order to help enterprises with the evolution of data business, Transwarp Technology makes use of its technological advantages in big data (big data platform TDH), container cloud platform (cloud operating system TCOS), and AI (AI platform Sophon) to develop a new-generation intelligent big data cloud platform Transwarp Data Cloud (referred to as TDC hereinafter), which provides enterprises with an efficient infrastructure technology platform, empowers department-oriented business, and helps enterprises with digital transformation.

TDC integrates three kinds of PaaS platforms, Analytics PaaS, Database PaaS, and Application PaaS; the underlying layer shares the resources on the IaaS platform. It helps enterprises to overcome the difficulties in collaboration data analysis, application development normalization, stock application governance, efficiency management, and solves the problems of data management chaos and resource conflict.

TDC Analytics PaaS meets the internal and external requirements for data analysis business, allowing data engineers and scientists to work on the platform at the same time, which facilitates team collaboration. TDC provides multiple analysis platforms across datasets, so that analysts do not need to re-define policies when using different tools or platforms. In this sense, it saves time for re-configuration and parameter adjustment when conducting analysis on different platforms among team members. It also supports systematic and efficient management for models.

03

Analytics PaaS

Finance

Analytics PaaS

Analytical PortalBig Data

Dev ToolkitTranswarp Studio

Data Catalog Application Market Security Center

Dev / Test DevOps

Service GovernanceGraphDatabaseStellarDB

Flash-BasedDatabaseArgoDB

AnalyticalDatabaseInceptor

Real-TimeComputing Engine

Slipstream

Data SearchingSearch

NoSQL DatabaseHyperbase

TransactionalDatabase

KunDB

Data SciencePlatformSophon

Container Service

ImageRepo

Network Service

StorageService

VirtualMachine

Container Orchestration System

Reporting ToolPilot

Database PaaS

Transwarp Cloud Operating System

Application PaaS CMP

CRM OA ERP Analysis Mobile APP

Data Interface Security Interface Resource Interface Microservice Interface

Operation Center

Metrics Monitor

Billing Service

Ticket Tracking

Disaster Recovery

TDC Database PaaS solves the problem of disperse data and isolation and avoids exchange barriers, which enables unified storage of all kinds of data assets and realizes data service open-up as well as data exchange. Data Catalog is provided for comprehensive data governance.

At the same time, TDC Database PaaS consists of a full set of data development tools. It provides a well-equipped data development environment, which lowers the technical requirements, simplifies data development procedures, overcomes the challenges in data development, and addresses efficiency issues, which altogether improve the data development capability.

04

TDC Application PaaS provides plentiful intermediary components and application development platforms, thus solving the efficiency problem in application development, deployment, maintenance, and governance. It helps to remove the barriers in the development and management of all kinds of applications.TDC provides essential mainstream application development tools, microservice framework, and DevOps tools, which effectively standardizes the application development procedures, improves efficiency, and controls quality. At the same time, TDC supports service deployment and service governance and features resource resilient scalability, resource isolation, and fault-tolerance.

Application PaaS

The underlying IaaS platform of TDC solves the efficiency problem in resource management and achieves unified management of system resources horizontally and vertically.It packs all kinds of resources such as computing, storage, and network to services, provides efficient configuration and scheduling, maintains the lifecycle of resources, and monitors the overall allocation conditions. It also provides accurate and reasonable billing services, which improves application maintenance efficiency and speeds up resource scheduling.

IaaS Platform

Procedures to build an enterprise data application center

Old Monolithic Business System Microservice Platform

Application Center Designing Phase

Development & Maintenance Phase

Designing Microservices

Studying Monolithic Applications

Designing Microservices Developing Microservices

Operating Microservices

Platform Publishing

Microservice Split

Planning & Research

111101101101010101011

1111011011010101010110

1111011011010101010110

11110110110101010101

1111011011010101010110

Database PaaS


05

TDC covers the functions of Analytics Cloud, Database Cloud, and Application Cloud, rendering the integration of multiple platforms. TDC connects different cloud platforms and can meet the requirements to build all three kinds of cloud platforms on the same platform. It breaks the boundaries between applications and data, and resource sharing is realized on the underlying IaaS layer. As a result, unified account management is achieved, which avoids departments to build a cloud platform vertically and realizes ecological communication within the enterprise system.

Integrating Analytics Cloud + Database Cloud + Application Cloud

One platform features the capability of three cloud platforms, which avoids different departments to build business systems separately and realizes the communication between applications and data. It helps to promote innovative business and empower the construction of ecosystem.

Integrating Three Clouds

It realizes a unified platform and provides consistent network, security, and maintenance management, so that communication barriers across platforms can be avoided, and there is no need to switch between different platforms in development.

Unifying Platforms

Since the underlying layer adopts a unified container supporting platform, three cloud platforms can share resources. Resources are resilient and can be borrowed from different platforms, which improves the resource usage efficiency greatly.

Resource Resilience

TDC provides unified resource and maintenance management for the three cloud platforms. No additional third-party cloud management platform needs to be introduced, which reduces the IT costs.

Easy Maintenance

Why Transwarp Data Cloud?

Cloud Platform Portal

Infrastructure as a Service

AnalyticsCloud

DatabaseCloud

ApplicationCloud

Network Communication

DataTransmission

PermissionControl

Security Management

Integration & Communication Between Clouds

Security Perm

ission

ResourceM

aintenance

06

Meeting the Requirements for Different Roles in the Company

TDC provides one-stop cloud services covering IaaS+PaaS+SaaS and is a leading cloud platform in the industry which features comprehensive service capability. From tenant & resource management to big data, AI applications, and data analysis, to microservice development, TDC can manage all types of applications and services, which meets the requirements of different roles in a company.

TDC solves the problem of application island and data island, builds a unified application center for data entry and permission control as well as a big data and data asset center, which helps managers to control business and data from a more comprehensive perspective.

For Enterprise Managers

TDC has a built-in application center, which supports self-service installation and deployment of any applications on the cloud, including reporting tool, daily office applications, ERP, and other SaaS services.

For Enterprise Clerks

TDC provides a full set of services from application development, testing, publishing, and maintenance, which consists of a development/testing platform, DevOps tools, intelligent & agile publishing, and metrics monitoring.

For Developers

TDC provides out-of-the-box data development services, which is scalable based on needs. And the graphical development interface frees the engineers from the architecture details, which speeds up business implementation.

For Data Engineers

TDC provides diverse visualized data analysis tools, which helps online data exploration and modeling in business analysis. All models support iteration, collaboration, and publishing.

For Analysts

TDC realizes comprehensive service governance, which ensures system stability and business continuousness and at the same time lessens the workload. It also provides course tracking of maintenance service chain for trouble-shooting.

For Maintenance Staff

BusinessLayer

Data Analysis Platform

Analytical Database NoSQL Database Real-Time Computing

Transactional Database

DevOps Service

Data Storage

Service Governance Center Web Middleware

Business Business Business Business Business Development & TestingPortal

Business Analysis Portal

Data Catalog

AI PlatformSophon

Object StorageObject Storage

DistributedStorageHDFS

Node.jsApplication

Jenkins Gitlab Tomcat

ContainerService

StorageServiceBilling Service Image Market Web Service Virtual Machine Bare Metal

Security / Monitoring /Management / Maintenance

VM-Based SolutionContainer Cloud Platform

ServiceGateway

KONG

ServiceTracking

Zipkin

APP PerformanceMgmt

Pinpoint

LogMgmtMilano

ApplicationServer

ConfigCenter

Hamurapi

Metrics MonitorPrometheus

GoApplication

DataVisualization

Pilot

Flash-BasedDatabaseArgoDB

Data AnalysisAnalytical

Portal

AnalyticalDatabaseInceptor

Search EngineSearch

Real-Time Computing Engine

Slipstream

DistributedDatabase

KunDB

Message Queue Kafka

HA DatabaseTxSQL

In-Memory KVKV Storage

GraphDatabaseStellarDB

NoSQLDatabase

Hyperbase

Big DataDevelopment

Studio

Model Market APP Market Data Service Layer PaaS Interface Layer

Maintenance Portal

InterfaceLayer

DevelopmentService

DataService

ResourceService

SaaS

PaaS

IaaS


07

Features of TDCBig Data Product

Artificial Intelligence Product

TDC Database PaaS provides all kinds of big data services that can be deployed by one-click. These services are supported by the basic components in Transwarp big data ecosystem, which have logics built within automatically to render all functions for common scenarios in data processing.

Build one-stop data warehouses, providing a full set of services such as data integration, processing and analysis, with the purpose of building the core of data.【Application Scenarios】 Data lake, batch processing, enterprise data warehouse, logical data warehouse.

Data Warehouse

Oriented at department-level data analysis businesses, provide OLAP Cube engine and reporting & scheduling tools; support performing interactive analysis and building automated reporting applications.【Application Scenarios】 Self-service reporting, multidimensional analysis, interactive analysis.

Data Mart

PB-level high-speed full-text search, supporting high concurrency, hot/cold data isolation, accurate/fuzzy retrieval, and quick statistics.【Application Scenarios】 Industry search engine, knowledge sharing platform, information retrieval.

Information Retriever

A stream processing platform on cloud for real-time data collection and processing, building real-time data warehouses and applications and discovering the value of data streaming.【Application Scenarios】 Real-time data analysis, online anti-fraud, sensor network analysis, intelligent device detection and failure prediction.

Real-Time Computing

Featuring scalability, high concurrency, and high availability, providing a common solution for building modern databases, and supporting business platforms of all industries.【Application Scenarios】 E-commence platform, financial distributed system, business aggregation center.

Distributed Transactional Database

A database product designed for the future all-flash servers, with delicate storage structure and efficient algorithms designed for the underlying layer.【Application Scenarios】 Public security center, financial analysis platform, targeted marketing system.

Distributed Flash-Based Database

Build an enterprise RDB to process OLTP business, supporting complex SQL query and guaranteeing high stability, scalability, and strong consistency.【Application Scenarios】 Online trading system.

Relational Database

Build a unified enterprise-level AI application platform for data cleansing, data analysis and mining, machine learning, deep learning, model management, API deployment, and workflow scheduling, thus helping enterprises with the innovation and revolution of business in the age of AI.【Application Scenarios】 Data mining and modeling, graphical modeling and feature engineering.

Artificial intelligence Platform

Application Development & Management Platform

08

TDC Application Market has all kinds of built-in applications, including intermediary components as well as tools for visualized analysis, development, and website building, which handles all tasks such as business development, data analysis, efficiency improvement, and effective manage-ment. Users can install applications according to their needs. These out-of-the-box applications do not occupy local resources and enable service sharing with others, which facilitates team collaboration and saves resources.

In addition, as an open platform, TDC Application Market supports developers to release applications at any time, so that all users with granted permissions can get informed in time. In this way, it shortens the application release cycle and improves the efficiency in the process from application design to implementation.

Application Market

TDC Development Platform provides a one-stop application publishing service for application development, release, and management. Users can complete the entire procedures from development to release on the same platform, which shortens the application release cycle. It also supports the management of application lifecycle.

DevOps Tools

Automatic deployment of DevOps tools, including code management, auto-testing, publishing, and deployment, which enables agile development, continuous integration, and continuous delivery. In this sense, a unified development and maintenance process is realized.

Application Publishing

For DevOps tools, self-built applications on other platforms, and open-source services, developers only need to do some simple configuration on the publishing platform so as to release them on the market. TDC supports scientific version management. Applications can have multiple versions and can be maintained individually. Users can feel the version change online in real time after updates.

My Applications

Display a list of all applications created, monitor the release status of all versions, control the release conditions, manage application deployment, and provide log analysis tools for trouble-shooting.

Development Platform


Tenant

…Project 1

…

Project M

…

Tenant

Role1UserGroup Project 1

Data Catalog

Management

09

TDC has a built-in Data Catalog service, which provides reliable, convenient, and intelligent support for enterprise data governance. It helps enterprises to create data map, unify data standard, mark data location, analyze data relationship, manage model change, and improve data quality, so that they can extract and make use of the value more effectively, realize accurate, efficient analysis and decision, promote system change management, and reduce project risks.

TDC management platform focuses on the projects, tenants, and users, achieving reasonable division and management of permissions and resources, and multi-tenancy is supported on a unified management platform. Tenant administrator has the highest tenant management permission, who is responsible for permission management. Product instances are managed and organized in each project as a unit, so as to achieve clear and rational permission management in terms of granularity.

Multi-Tenancy Management

08

07

10

With container technology, TDC features the isolation of applications, data, resources, and execution among multi-tenancy. Tenants running on the same platform are completely transparent to each other, as if they were operating on different infrastructures. In addition, for enterprise private cloud, TDC can provide unified data management for internal tenants. Shared data are placed in the public area while high-value and sensitive data in the sensitive area, thus ensuring unified metadata management and data quality control, supporting unified data lifecycle management, and promoting data assetization.

TDC adopts an accurate billing architecture, which ensures data accuracy and security. At the same time, this architecture features high availability and scalability, and is able to conduct near real-time data computing.TDC platform brings up a reasonable and clear billing standard for users. Pricing items are categorized into four types, which are hardware resource, big data software, data service, and third-party application. At the same time, a variety of billing units are supported so as to form a fair and convenient billing model.

In addition, TDC provides powerful fee management functions, including detailed tenancy bill, unified operation analysis report, account write-off and reconciliation, tenant quota setting, charge items, price setting, and discount rule customization, to release full control of finance management to platform managers.

Accurate Billing

Business Billing Description

Computing resources + storage resources Package Billing: set several billing stalls, charge by the month, and the exceeding parts are billed on demand Accurate Billing: charge by operational events on physical resources

Resource Billing

Software Service Billing

Data Billing

Application Billing

Charge by data services of Transwarp big data and AI platform Pricing: Basic price × Advanced/Standard/Lite configuration

Each data type can be priced individually or collectivelyPackage Billing: data volume VS update frequencyAccurate Billing: provided by applications or services

Sell third-party applications at a fair price Charge by the billing agreement

稳定、高效、智能的大数据云平台

TDC has built-in maintenance & management services, which realizes unified maintenance of the entire platform and internal management for tenants.

TDC's maintenance & management system has distinguished performance in throughput, security, and availability, which is capable of providing high-quality maintenance services for the cloud platform.Full-Course High Throughput

Maximum TB-level throughput. The number of logs collected per second on a single node reaches tens of thousands; a three-node cluster can collect up to 2 billion logs in one day.Full-Course Security

Log data is encrypted by Kerberos. Logs of different tenants are isolated from each other. Security authentication is needed when users try to access and analyze logs.Full-Course High Availability

Status monitoring is enabled within the system to ensure the high availability of data, prevent data loss and duplication.

Unified Graphical Maintenance Monitoring

Alert Message

Visualized StatusAnalysis

Resource Mgmt& Monitoring

Log Analysis &Retrieval


11

Health Check

Log TimePortrait

Business SolutionsData Sharing Center

稳定、高效、智能的大数据云平台

TDC can be used to build a data sharing center for multi-tenancy scenarios, which provides fine-grained tenant management. The head office can build a unified platform with all its departments as individual tenants. With the resources and technological capability of the head office, it can realize self-service big data and application development. TDC will conduct unified scheduling and governance for resources and services. TDC supports horizontal and vertical data flow. The head office has a global overview of all the data from lower divisions, and data sharing is enabled across departments. They can obtain the desired data in a self-service manner, which enables more comprehensive and valuable production in data platform construction.

TDC can build a unified Internet big data platform, integrating massive network and user data resources, and realizing unified development & management for applications. TDC Application Aggregation Platform provides development/testing support for developers and the underlying resources are under unified management. According to application requirements, custom services are provided with data in the storage service, intermediary and result data on the mining & analysis platform, and open data on the distribution platform. This platform can deploy third-party applications flexibly, support calling other services via API, collect and manage applications on a large scale, which altogether achieves the maximum data value.

Application Aggregation Platform

08

07

12

Data Collection

Real-TimeData Analysis Platform

Big Data Mining Platform

Typical Applications

Big Data Storage & Management Platform

Big Data Distribution Platform

All Internal/ExternalApplications

Big Data Application Aggregation Platform

Basic Information Resource

Platform PortalOpen Big Data Platform Collaborative Governance

Platform

Oriented at the general public: Oriented at enterprises & governments:

PopulationInformation Library

Legal EntityInformation Library

Basic Information Resource Service platform

Data Sharing Cloud Platform

Department 1 Department 2 Department 3 Department 4 Department 5

Spatial GeographicInformation Library

CreditInformation Library

E-LicenseInformation Library

About UsTranswarp is a world leading big data and AI platform provider, focusing on enterprise level cloud computing on

container, big data and AI core platform research and services. Transwarp is based in Shanghai, with regional

headquarters in Beijing，Guangzhou, Singapore, and support centres in Nanjing, Zhengzhou and Chengdu. Transwarp

also has numerous domestic offices in different provinces and overseas branches in the United States and Canada.

After years of R&D, Transwarp establishes major product lines with many patents: Transwarp Data Cloud (TDC), a

container-based intelligent big data cloud platform; Transwarp Data Hub (TDH), a one-stop big data platform; AI

platform Transwarp Sophon; and Hyper-Converged big data server TxData Appliance. Meanwhile, Transwarp attaches

great importance to knowledge exporting and training on big data and AI, founded the Transwarp University, which

focus on training product R&D, professional certification and training services providing.

In 2016, Transwarp was appraised by Gartner as the world's most visionary vendor in data warehouse and data

management. In 2017, Transwarp was recognized by IDC as the leader of the big data market in China. In 2018,

Transwarp became the first database vendor worldwide to complete TPC-DS testing and pass official audits over the

past 12 years. Transwarp Technology has secured its series D1 funding which is led by TCL Capital and CICC Capital.

Core Technology

Enterprise-grade data warehouse and data mart based on one-stop big data platform;

High-performance and scalable distributed database;

Streaming engine with low-latency event-driven approach and complex batch processing programming model;

AI platform with complete algorithms and practice models for statistics, machine learning and deep learning;

Container based operating system to support big data at production level;

Multi-tenant PaaS platform based on container technology.

Industry Focus

Products of Transwarp have been widely used in more than ten industries, such as finance, government, trans-

portation, manufacturing, telecommunications, logistics, education and so on. Transwarp has more than 1000

clients, including Ministry of Finance of the People's Republic of China, Bank of China, China Post, China

National Petroleum Corporation (CNPC), Airbus, China Unicom, etc.

Selected Customers

Version 2.0

11F&12F&15F, Block B, 9F, Block A, 88 Hongcao Road, Xuhui，Shanghai

86-4007-676-098

www.transwarp.io

@transwarp_data

@[email protected]

transwarp data cloud · 2019-12-05 · 02 the data age has witnessed the transformation of...

Documents