service overview · simply set instance information on the dms for kafka console, submit your...

Distributed Message Service for Kafka

Service Overview

Issue 01

Date 2020-07-31

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 01 (2020-07-31) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 What is DMS for Kafka?.........................................................................................................1

2 Product Advantages................................................................................................................ 2

3 Application Scenarios............................................................................................................. 4

4 Specifications............................................................................................................................7

5 Comparing Kafka Instances and DMS Advanced Queues............................................10

6 Comparing Kafka and RabbitMQ...................................................................................... 12

7 Comparing DMS for Kafka and Open-Source Kafka..................................................... 15

8 Notes and Constraints..........................................................................................................17

9 Related Services.....................................................................................................................19

10 Basic Concepts..................................................................................................................... 20

11 Permissions Management................................................................................................. 22

12 Billing..................................................................................................................................... 25

Distributed Message Service for KafkaService Overview Contents

Issue 01 (2020-07-31) Copyright © Huawei Technologies Co., Ltd. ii

1 What is DMS for Kafka?

Apache Kafka is distributed message middleware that features high throughput,data persistence, horizontal scalability, and stream data processing. It adopts thepublish-subscribe pattern and is widely used for log collection, data streaming,online/offline system analytics, and real-time monitoring.

Distributed Message Service (DMS) for Kafka is a message queuing service basedon Apache Kafka. This service provides Kafka premium instances. The computing,storage, and bandwidth resources used by an instance are exclusively occupied bythe user. You can apply for instances as required and customize partitions andreplicas for the topics in the instances. The instances can be used right out of thebox, taking off the deployment and O&M pressure for you so that you can focuson developing your services.

Readers' GuideThis documentation introduces DMS for Kafka and its differences from ApacheKafka. You will learn about the detailed information about the specifications,console operations, API calling, and client access to instances of HUAWEI CLOUDDMS for Kafka.

For more information about the basic knowledge of Kafka or technical detailsabout creating and retrieving messages, please go to the official Apache Kafkawebsite.

Distributed Message Service for KafkaService Overview 1 What is DMS for Kafka?

Issue 01 (2020-07-31) Copyright © Huawei Technologies Co., Ltd. 1

https://kafka.apache.org/

https://kafka.apache.org/

2 Product Advantages

HUAWEI CLOUD DMS for Kafka provides easy-to-use message queuing based onApache Kafka. Services can be quickly migrated to the cloud without any change,reducing maintenance and usage costs.

● Rapid deploymentSimply set instance information on the DMS for Kafka console, submit yourorder, and a complete Kafka premium instance will be automatically createdand deployed.

● Service migration without modificationsDMS for Kafka is compatible with open-source Kafka APIs and supports allmessage processing functions of open-source Kafka.If your application services are developed based on open-source Kafka, youcan easily migrate them to HUAWEI CLOUD DMS for Kafka after specifying afew authentication configurations.

NO TE

DMS for Kafka is compatible with Apache Kafka 1.1.0. Clients later than version 0.10are supported. Version 1.1.0 is recommended (which is consistent with the serverversion).

● SecurityOperations on Kafka premium instances are recorded and can be audited.Messages can be encrypted before storage.In addition to SASL, Virtual Private Clouds (VPCs) and security groups alsoprovide security controls on network access.

● Data reliabilityKafka premium instances support data persistence and replication. Messagescan be replicated synchronously or asynchronously between replicas.

● High availabilityKafka runs in clusters, enabling failover and fault tolerance so that servicescan run smoothly.Kafka premium instances can be deployed across AZs to further enhanceservice availability.

● Simple O&M

Distributed Message Service for KafkaService Overview 2 Product Advantages


HUAWEI CLOUD provides a whole set of monitoring and alarm services,eliminating the need for 24/7 attendance. A set of Kafka premium instancemetrics are monitored and reported, including the number of partitions,topics, and accumulated messages. You can configure alarm rules and receiveSMS or email notifications on how your services are running in real time.

● Massive accumulation and auto scalingKafka features high scalability because it runs in a distributed system, orcluster. You can configure up to 20 partitions for a topic. The storage spacecan be also expanded. This means that billions of messages can beaccumulated, suitable for scenarios requiring high concurrency, highperformance, and large-scale access.

● Flexible specificationsYou can customize the bandwidth and storage space for the instance and thenumber of partitions and replicas for topics in the instance.

Distributed Message Service for KafkaService Overview 2 Product Advantages


3 Application Scenarios

Kafka is popular message-oriented middleware that features highly reliable,asynchronous message delivery. It is widely used for transmitting data betweendifferent systems in the enterprise application, payment, telecommunications, e-commerce, social networking, instant messaging, video, Internet of Things, andInternet of Vehicle industries.

Asynchronous Communication

Non-core or less important messages are sent asynchronously to receivingsystems, so that the main service process is not kept waiting for the results ofother systems, allowing for faster responses.

For example, Kafka can be used to send a notification email and SMS messageafter a user has registered with a website, providing fast responses throughout theregistration process.

Figure 3-1 Serial registration and notification

Figure 3-2 Asynchronous registration and notification using message queues

Traffic Control

In e-commerce systems or large-scale websites, there is a processing capabilitygap between upstream and downstream systems. Traffic bursts from upstreamsystems with high processing capabilities may have a large impact on downstreamsystems with lower processing capabilities. For example, online sales promotionsinvolve a huge amount of traffic flooding into e-commerce systems. Kafka

Distributed Message Service for KafkaService Overview 3 Application Scenarios


provides a three-day buffer by default for hundreds of millions of messages, suchas orders and other information. In this way, message consumption systems canprocess the messages during off-peak periods.

In addition, flash sale traffic bursts originating from frontend systems can behandled with Kafka, keeping the backend systems from crashing.

Figure 3-3 Traffic burst handling using Kafka

Log Synchronization

In large-scale service systems, logs of different applications are collected for quicktroubleshooting, full-link tracing, and real-time monitoring.

Kafka is originally designed for this scenario. Applications asynchronously send logmessages to message queues over reliable transmission channels. Othercomponents can read the log messages from message queues for further analysis,either in real time or offline. In addition, Kafka can collect key log information tomonitor applications.

Log synchronization involves three major components: log collection clients, Kafka,and backend log processing applications.

1. The log collection clients collect log data from a user application service andasynchronously send the log data in batches to Kafka clients.

Kafka clients receive and compress messages in batches. This only has aminor impact on the service performance.

2. Kafka persists logs.

3. Log processing applications, such as Logstash, subscribe to messages in Kafkaand retrieve log messages from Kafka. Then, the messages are searched forby file search services or delivered to big data applications such as Hadoop forstorage and analysis.

Figure 3-4 Log synchronization process



NO TE

Logstash is for log analytics, ElasticSearch is for log search, and Hadoop is for big dataanalytics. They are all open-source tools.



4 Specifications

Kafka Premium Instance SpecificationsKafka premium instances are compatible with open-source Kafka 1.1.0. Theinstance specifications are classified based on bandwidth, namely, 100 MB/s, 300MB/s, 600 MB/s, and 1200 MB/s.

NO TE

● The number of brokers varies according to the underlying resources, and the underlyingresources vary from region to region. The following table lists the specifications for theCN North-Beijing4 region.

● In the following table, transactions per second (TPS) are calculated assuming that thesize of a message is 1 KB.

Table 4-1 TPS and maximum number of partitions supported by different instancespecifications

Bandwidth

Brokers

UnderlyingResourceType

I/OType

TPS (High-Throughput)

TPS(SynchronousReplication)

MaximumPartitions

100MB/s

3 c6_2vCPUs |4GB

High I/O 100,000 60,000 300

c6_2vCPUs |4GB

Ultra-high I/O

100,000 80,000 300

300MB/s

3 c6_4vCPUs |8GB

High I/O 300,000 150,000 900

c6_4vCPUs |8GB

Ultra-high I/O

300,000 200,000 900

600MB/s

4 c6_8vCPUs |16GB

Ultra-high I/O

600,000 300,000 1800

1200MB/s

8 c6_8vCPUs |16GB

Ultra-high I/O

1.2 million 400,000 1800

Distributed Message Service for KafkaService Overview 4 Specifications


Bandwidth SelectionThe bandwidth of a Kafka instance refers to the maximum read or writebandwidth. You are advised to select a bandwidth 30% higher than what isrequired.

● 100 MB/sRecommended for up to 3000 client connections, 60 consumer groups, and 70MB/s of service traffic.

● 300 MB/sRecommended for up to 10,000 client connections, 300 consumer groups, and210 MB/s of service traffic.



Storage Space SelectionKafka premium instances support storage with 1 to 3 replicas. The storage space isspace consumed by all replicas. When creating an instance, specify its storagespace based on the expected service message size and the number of replicas.

For example, if the estimated message size is 100 GB, the disk capacity must be atleast: 100 GB x Number of replicas + 100 GB (reserved).

The storage space can be expanded as your service grows.

Topic QuantityThere is no limit on the topic quantity, but there is an upper limit on theaggregate number of partitions in the topics. When the partition quantity limit isreached, you can no longer create topics.

The number of topics is related to the maximum number of partitions allowed(see Table 4-1) and the specified number of partitions in each topic (see Figure4-1).



Figure 4-1 Setting the number of partitions

For example, the maximum number of partitions for a 100 MB/s instance is 300.

If the number of partitions of each topic in the instance is 3, the number of topicsis 300/3 = 100.

If the number of partitions of each topic in the instance is 1, the number of topicsis 300/1 = 300.



5 Comparing Kafka Instances and DMSAdvanced Queues

Both Kafka premium instances and DMS advanced queues are compatible withApache Kafka. However, they differ in the following aspects.

Open-Source Compatibility● DMS advanced queues:

Kafka 0.10.2.1● Kafka premium instances:

Kafka 1.1.0With each version upgrade, Apache Kafka introduces new features, improvesAPIs, and updates producer and consumer configuration files. To checkwhether your application features and APIs are compatible with your Kafkaclients, see the upgrade notes on the official Apache Kafka website.

Purchase● DMS advanced queues:

An advanced queue (equivalent to a topic) is created on the DMS console.You do not need to configure the storage space or the bandwidth becausethese resources are allocated by the system.

● Kafka premium instances:A Kafka premium instance is created on the DMS for Kafka console. Beforecreating a Kafka premium instance, determine the required bandwidth andstorage space based on your service expectations for the next one or twoyears. You also need to prepare a VPC and security group for the instance.After the instance has been created, you must create topics in the instanceand configure the number of partitions and replicas for the topics.

Usage● DMS advanced queues:

Advanced queues are compatible with Kafka APIs. DMS provides SDKs in Java,Python, Lua, C, and Go languages. For details, see the DMS Kafka DeveloperGuide.

Distributed Message Service for KafkaService Overview

5 Comparing Kafka Instances and DMS AdvancedQueues


https://kafka.apache.org/documentation.html#upgrade_1_1_0

https://support.huaweicloud.com/en-us/devg-dms/en-us_topic_0089834634.html

https://support.huaweicloud.com/en-us/devg-dms/en-us_topic_0089834634.html

To use open-source Kafka clients, see Usage of the Enhanced Java SDK. Addthe enhanced Kafka Java SDK provided by DMS to the directory of the open-source client package and then pass the security authentication.

● Kafka premium instances:DMS for Kafka is fully compatible with open-source Kafka. You can accessKafka premium instances and topics using open-source Kafka clients. If SASLaccess is enabled, you must use the SSL certificate provided by DMS forKafka.

Performance● DMS advanced queues:

There are two queue modes: high-throughput and high-reliability. In the high-throughput mode, messages are flushed to disk asynchronously, ensuring highconcurrency.

● Kafka premium instances:Compute, bandwidth, and storage resources are physically isolated for eachinstance. Determine the required bandwidth and storage space when creatingan instance. For storage space, you can choose Ultra-high I/O, whichindicates that messages are stored on SSDs.

Other DimensionsYou can customize the number of partitions and replicas for a Kafka premiuminstance. Each topic can have 1 to 20 partitions and 1 to 3 replicas.

By default, an advanced queue has three partitions and three replicas. If this doesnot meet your requirement, submit a service ticket.

NO TE

Divide a topic into a certain number of partitions so that messages can be evenlydistributed to partitions, enabling load balancing and horizontal scalability. Differentconsumers can retrieve messages from one or more partitions, improving messageprocessing performance.With more replicas come higher reliability. However, synchronizing messages betweenreplicas consumes bandwidth and offsets compute performance.

Distributed Message Service for KafkaService Overview

5 Comparing Kafka Instances and DMS AdvancedQueues


https://support.huaweicloud.com/en-us/devg-dms/dms-devg-180626001.html

https://cwiki.apache.org/confluence/display/KAFKA/Clients

https://dms-demo.obs.myhuaweicloud.com/dmskafkasasl.zip

6 Comparing Kafka and RabbitMQ

Kafka is pull-based and provides higher throughput. It is suitable for collecting anddelivering large volumes of data, such as collecting and analyzing logs. RabbitMQdoes not provide as high throughput as Kafka, but it offers more message queuingfunctions.

The following is a comparison analysis on the performance, data reliability, serviceavailability, and functions of Kafka and RabbitMQ.

PerformanceThe performance of message-oriented middleware is measured by throughput.While RabbitMQ provides tens of thousands of QPS, Kafka provides millions.

However, if idempotency and transactions are enabled for Kafka, its performancewill be compromised.

Data ReliabilityBoth Kafka and RabbitMQ provide the replication mechanism to ensure high datareliability.

Service AvailabilityKafka runs in clusters and has partitions and replicas. Therefore, single-nodefailure does not affect services and the capacity of Kafka can be linearly scaled up.

DMS for RabbitMQ provided by HUAWEI CLOUD also supports clustereddeployment with multiple node quantity options.

FunctionsBoth Kafka and RabbitMQ are popular open-source message-oriented middleware.They differ mainly in the functions, which are listed in the following table.

Distributed Message Service for KafkaService Overview 6 Comparing Kafka and RabbitMQ


Table 6-1 Function differences between Kafka and RabbitMQ

Function Kafka 1.1.0 RabbitMQ 3.7.0

Priorityqueue

Not supported Supported. It is recommended thatthe priority be set to 0–10.

Delayedqueue

Not supported Supported

Deadletterqueue


Retry Not supported Not supported

Retrievalmode

Pull-based Pull-based and push-based

Messagebroadcasting

Supported Supported

Messagetracking

Supports offset andtimestamp tracking.

Not supported. Once a messageretrieval has been acknowledged,RabbitMQ will be notified that themessage can be deleted.

Messageaccumulation

Supports higheraccumulation performancethan RabbitMQ thanks tohigh throughput.

Supported

Persistence

Supported Supported

Messagetracing

Not supported Supported by the firehose feature orthe rabbitmq_tracing plugin.However, rabbitmq_tracing reducesperformance and should be usedonly for troubleshooting.

Messagefiltering

Supported Not supported, but can beencapsulated.

Multi-tenancy


Multi-protocol

Only Apache Kafka issupported.

RabbitMQ is based on AMQP andsupports MQTT and STOMP.

Multi-language

Kafka is written in Scala andJava and supports clients inmultiple programminglanguages.

RabbitMQ is written in Erlang andsupports clients in multipleprogramming languages.



Function Kafka 1.1.0 RabbitMQ 3.7.0

Throttling Supports throttling onproducer or consumerclients and users.

Supports credit-based throttling onproducers, a mechanism thattriggers protection from within.

Orderedmessagedelivery

Supports partition-levelFIFO.

Supports FIFO only for single-threaded message queuing withoutadvanced features such as delayedqueues or priority queues.

Security Supports SSL and SASLauthentication and read/write permissions control.

Similar to Kafka.

Idempotency

Supports idempotency for asingle producer session.

Not supported

Transactionmessages

Supported Supported

NO TE

The comparison is made between open-source Kafka and RabbitMQ.DMS for Kafka and DMS for RabbitMQ maintain open-source compatibility whilesupporting or enhancing features in the open-source versions.



7 Comparing DMS for Kafka and Open-Source Kafka

DMS for Kafka is compatible with open-source Kafka and has customized andenhanced Kafka features. In addition to the advantages of open-source Kafka,DMS for Kafka provides more reliable and useful features.

Table 7-1 Differences between DMS for Kafka and open-source Kafka

Category

Item DMS for Kafka Open-source Kafka

Ease ofuse

Readilyavailable

Instances can be createdintuitively within minutesand used right out of thebox with visualizedoperations and real-timemonitoring.

Preparing server resources andinstalling and configuring thesoftware is time-consumingand prone to mistakes.

APIs Instances can be managedeasily by calling RESTfulAPIs.

N/A

Costs On-demand use

Multiple specifications areavailable to suit differentneeds. The instancebandwidth and disk spacecan be expanded withoutdowntime.

Expenses are incurred forsetting up a message serviceand occupying underlyingresources.

Fullymanaged

Services are readilyavailable without requiringadditional hardwareresources or expenses.

Users must prepare hardwareresources and set up theservice by themselves, and bearhigh usage and maintenancecosts.

Distributed Message Service for KafkaService Overview 7 Comparing DMS for Kafka and Open-Source Kafka


Category

Item DMS for Kafka Open-source Kafka

Provensuccess

Mature DMS has been deployed inHuawei products andproven successful in largee-commerce events suchas the Vmall 11.11Shopping Festival. It is alsoused in the clouds ofcarrier-grade customersacross the world, andmeets strict carrier-gradereliability standards. DMSclosely follows up withcommunity updates tocontinuously fix knownopen-source vulnerabilitiesand add support for newfeatures.

Using open-source softwarerequires lengthy self-development and verificationand has had few successfulcases.

Feature-rich

While maintaining 100%open-source compatibility,DMS further optimizesopen-source code toimprove performance andreliability, and providesmessage querying,dumping, tracing(available soon), andmany other features.

Functionality is limited andrequires self-development.

Reliability

Highlyavailable

DMS supports cross-AZdeployment to improvereliability. In addition,automatic fault detectionand alarms ensure reliableoperations of key services.

High availability requires self-development or open-sourcecode implementation, whichare costly and cannotguarantee reliability.

SimpleO&M

O&M is entirelytransparent to tenantswith a full set ofmonitoring and alarmfunctions. O&M personnelwill be informed of anyexceptions, eliminating theneed for 24/7 attending.

Users need to develop andoptimize O&M functions,especially alarm notificationfunctions. Otherwise, manualattendance is required.

Secure DMS uses VPC isolationand SSL channelencryption.

Security must be hardened byusers themselves.

Distributed Message Service for KafkaService Overview 7 Comparing DMS for Kafka and Open-Source Kafka


8 Notes and Constraints

DMS for Kafka has the following constraints, as listed in Table 8-1.

Table 8-1 Kafka usage restrictions

Item Constraint Description

Kafka Zookeeper Not exposed externally DMS ZooKeeper doesnot provide servicesexternally. It is only usedto serve Kafka instances.

Version 1.1.0 Clients later than version0.10 are supported.Version 1.1.0 isrecommended (which isconsistent with theserver version).

Message size 10 MB The message size cannotexceed 10 MB.Otherwise, the messagecreation will fail.

Logging in to the VMwhere the Kafka brokersreside

No supported N/A

Distributed Message Service for KafkaService Overview 8 Notes and Constraints


Item Constraint Description

Partition quantity Limited Kafka managesmessages by partition. Ifthere are too manypartitions, messagecreation, storage, andretrieval will befragmented, affectingthe performance andstability. If the totalnumber of partitions oftopics reaches the upperlimit, you cannot createmore topics.

Automatic topic creation Supported Configurable duringinstance creation.If it is enabled, a topicwill be automaticallycreated with 3 partitionsand 3 replicas when amessage is created in orretrieved from a topicthat does not exist.

Creating consumergroups, consumers, andproducers

Not required Consumer groups,consumers, andproducers are generatedautomatically when youuse the instance.

Decreasing partitionquantity

Not supported The partition quantitycannot be decreased dueto the limitations ofApache Kafka.

Distributed Message Service for KafkaService Overview 8 Notes and Constraints


9 Related Services

● CTSCloud Trace Service (CTS) generates traces to provide you with a history ofoperations performed on cloud service resources. The traces include operationrequests sent using the management console or open APIs, as well as theoperation results. You can view all generated traces to query, audit, andbacktrack performed operations.For details about the operations recorded by CTS, see Operations That CanBe Recorded by CTS

● VPCKafka premium instances run in VPCs and use the IP addresses and bandwidthof VPC. Security groups of VPCs enhance the security of network access to theKafka premium instances.

● Cloud EyeCloud Eye is an open platform that provides monitoring, alarm reporting, andalarm notification for your resources in real time.

NO TE

The values of all Kafka instance metrics are reported to Cloud Eye every minute.

Distributed Message Service for KafkaService Overview 9 Related Services


https://support.huaweicloud.com/en-us/usermanual-kafka/kafka-ug-180418002.html

https://support.huaweicloud.com/en-us/usermanual-kafka/kafka-ug-180418002.html

10 Basic Concepts

DMS for Kafka of HUAWEI CLOUD uses Kafka as the message engine. This chapterpresents explanations of basic concepts of Kafka.

TopicA topic is a category for messages. Messages are created, retrieved, and managedin the form of topics.

Topics adopt the publish-subscribe pattern. Producers publish messages intotopics. One or more consumers subscribe to the messages in the topics. Theproducers and consumers are not directly linked to each other.

ProducerA producer publishes messages into topics. The messages are then delivered toother systems or modules for processing as agreed.

ConsumerA consumer subscribes to messages in topics and processes the messages. Forexample, a monitoring and alarm platform (a consumer) subscribing to logmessages in certain topics can identify alarm logs and then send SMS or emailalarm notifications.

BrokerA broker is a Kafka process in a Kafka cluster. Each process runs on a server, so abroker includes the storage, bandwidth, and other server resources.

PartitionMessages in a topic are distributed to multiple partitions to achieve scalability andfault tolerance.

ReplicaA replica is a redundant copy of a partition in a topic. Each partition can have oneor more replicas, enabling message reliability.

Distributed Message Service for KafkaService Overview 10 Basic Concepts


Messages in each partition are fully replicated and synchronized, preventing dataloss if one replica fails.

Each partition has one replica as the leader which handles the creation andretrievals of all messages. The rest replicas are followers which replicate theleader.

Topics and partitions are logical concepts, while replicas and brokers are physicalconcepts. The following diagram shows the relationships between partitions,brokers, and topics in messages streaming.

Figure 10-1 Kafka message streaming

Distributed Message Service for KafkaService Overview 10 Basic Concepts


11 Permissions Management

If you need to assign different permissions to employees in your enterprise toaccess your DMS resources, Identity and Access Management (IAM) is a goodchoice for fine-grained permissions management. IAM provides identityauthentication, permissions management, and access control, helping you secureaccess to your HUAWEI CLOUD resources.

With IAM, you can use your HUAWEI CLOUD account to create IAM users, andassign permissions to the users to control their access to specific resources. Forexample, some software developers in your enterprise need to use DMS resourcesbut should not be allowed to delete the resources or perform any other high-riskoperations. In this scenario, you can create IAM users for the software developersand grant them only the permissions required for using DMS resources.

If your HUAWEI CLOUD account does not require individual IAM users forpermissions management, skip this section.

IAM can be used free of charge. You pay only for the resources in your account.For more information about IAM, see the IAM Service Overview.

NO TE

Kafka permissions policies are based on DMS. Therefore, when assigning permissions, selectDMS permissions policies.

DMS Kafka Permissions

By default, new IAM users do not have permissions assigned. You need to add auser to one or more groups, and attach permissions policies or roles to thesegroups. Users inherit permissions from the groups to which they are added andcan perform specified operations on cloud services based on the permissions.

DMS is a project-level service deployed and accessed in specific physical regions.To assign DMS permissions to a user group, specify the scope as region-specificprojects and select projects for the permissions to take effect. If All projects isselected, the permissions will take effect for the user group in all region-specificprojects. When accessing DMS, the users need to switch to a region where theyhave been authorized to use this service.

You can grant users permissions by using roles and policies.

Distributed Message Service for KafkaService Overview 11 Permissions Management


https://support.huaweicloud.com/en-us/productdesc-iam/iam_01_0026.html

● Roles: A type of coarse-grained authorization mechanism that definespermissions related to user responsibilities. This mechanism provides only alimited number of service-level roles for authorization. When using roles togrant permissions, you need to also assign other roles on which thepermissions depend to take effect. However, roles are not an ideal choice forfine-grained authorization and secure access control.

● Policies: A type of fine-grained authorization mechanism that definespermissions required to perform operations on specific cloud resources undercertain conditions. This mechanism allows for more flexible policy-basedauthorization, meeting requirements for secure access control. For example,you can grant DMS users only the permissions for managing instances. Mostpolicies define permissions based on APIs. For the API actions supported byDMS, see Permissions Policies and Supported Actions.

Table 1 lists all the system-defined roles and policies supported by DMS.

Table 11-1 System-defined roles and policies supported by DMS

Role/PolicyName

Description Type Dependency

DMS FullAccess Administrator permissionsfor DMS. Users grantedthese permissions canperform all operations onDMS.

System-definedpolicy

None

DMS UseAccess Common user permissionsfor DMS, excludingpermissions for creating,modifying, deleting, scalingup instances, and dumping.


None

DMSReadOnlyAccess

Read-only permissions forDMS. Users granted thesepermissions can only viewDMS data.


None

DMSAdministrator

Administrator permissionsfor DMS logical tenancy.Users granted thesepermissions can perform alloperations in DMS QueueManager.

System-defined role

None

NO TE

● Currently, DMS uses only the DMS Administrator role.● DMS FullAccess, DMS UseAccess, and DMS ReadOnlyAccess policies are used to

control the operation permissions for DMS for Kafka and DMS for RabbitMQ.● The DMS FullAccess, DMS UseAccess, and DMS ReadOnlyAccess policies contain OBS

actions. Due to data caching, the policy will take effect five minutes after it is attachedto a user, user group, or project.



https://support.huaweicloud.com/en-us/api-kafka/api-grant-policy.html

Table 2 lists the common operations supported by each system-defined policy orrole of DMS for Kafka. Select the policies or roles as required.

Table 11-2 Common operations supported by each system-defined policy or roleof DMS

Operation DMS FullAccess DMS UseAccess DMSReadOnlyAccess

Creatinginstances

√ × ×

Changing thebilling modefrom pay-per-use to yearly/monthly

√ × ×

Modifyinginstances

√ × ×

Deletinginstances

√ × ×

Modifyinginstancespecifications

√ × ×

Enablingdumping

√ × ×

Creatingdumpingtasks

√ √ ×

Restartinginstances

√ √ ×

Queryinginstanceinformation

√ √ √

Helpful Links● IAM Service Overview● Creating User Groups, Users, and Granting DMS permissions● Permissions Policies and Supported Actions



https://support.huaweicloud.com/en-us/productdesc-iam/iam_01_0026.html

https://support.huaweicloud.com/en-us/usermanual-kafka/CreateUserAndGrantPolicy.html

https://support.huaweicloud.com/en-us/api-kafka/api-grant-policy.html

12 Billing

DMS for Kafka supports two billing modes: pay-per-use and yearly/monthly. Fordetails, see Pricing Details.

Billing Items

DMS for Kafka is billed based on Kafka instance specifications and storage space.

Table 12-1 DMS for Kafka billing

Billing Item Description

Instance ● Kafka instances are billed based ontheir bandwidth. To ensure stableservice running, you are advised tochoose a bandwidth 30% higherthan the actual throughputexpected for read or write traffic,whichever is higher. You should alsoconsider other parametersdescribed in Table 12-2.For example, if the maximum writetraffic is 70 MB/s and the maximumread traffic is 200 MB/s, thebandwidth to be chosen should beat least 200 MB/s x (1 + 30%) =260 MB/s.

● Kafka instances can be billed inyearly/monthly or pay-per-use(hourly) mode.

● If you enable the dumping function,there will be additional charges.For example, if you enable dumpingfor a 100 MB/s instance (3 brokers),2 additional nodes will be createdfor dumping and you need to payfor them.

Distributed Message Service for KafkaService Overview 12 Billing


https://www.huaweicloud.com/en-us/pricing.html?tab=detail#/kafka

Billing Item Description

Storage space ● Queues are billed based on thestorage space. For each type ofinstance specification, you canchoose the common I/O, high I/O,or ultra-high I/O disk type to meetyour service requirements.You can specify the number ofreplicas. For example, if therequired disk size to store messagedata is 500 GB and there are threereplicas, the disk capacity should beat least: 500 GB x 3 = 1500 GB.

● Table 12-2 describes the storagespace options in 100 GB increments.

● Queue storage space can be billedon a yearly/monthly basis or pay-per-use (hourly) basis.

Table 12-2 Kafka instance specifications

Specification(Bandwidth)

MaximumPartitions

MaximumConsumerGroups

MaximumClientConnections

StorageSpace

100 MB/s 300 60 3000 600 GB–90,000 GB

300 MB/s 900 300 10,000 1,200 GB–90,000 GB

600 MB/s 1800 600 20,000 2,400 GB–90,000 GB

1200 MB/s 1800 600 20,000 4,800 GB–90,000 GB

Billing ModesTwo billing modes are available, allowing you to pay less by using more.

● Yearly/Monthly: Provides a larger discount than pay-per-use mode and isrecommended for long-term users.

● Pay-per-use (hourly) mode: More flexible, enabling you to start and stopservices anytime. You pay only for what you use. The minimum time unit isone hour. Less than an hour is recorded as an hour.



Changing Configurations● You can change the bandwidth for a Kafka instance. After the change, you

will be billed based on the new specifications.● You can also change the storage space of Kafka queues. You will be billed

based on the new storage space after the storage space increase. Storagespace can only be increased, and cannot be decreased. The minimum increaseincrement is 100 GB.

RenewalYou can renew a resource package upon its expiration, or you can set auto-renewal rules for a resource package. For more information about renewingresource packages, see Renewal Management



https://support.huaweicloud.com/en-us/usermanual-billing/renewals_topic_10000000.html

service overview · simply set instance information on the dms for kafka console, submit your...

Documents