service overview - huawei · complete rabbitmq instance will be automatically created and deployed....

Distributed Message Service

Service Overview

Date 2020-03-06

Contents

1 DMS Overview..........................................................................................................................1

2 Functions................................................................................................................................... 5

3 Application Scenarios............................................................................................................. 8

4 RabbitMQ Instances..............................................................................................................10

5 Kafka Premium Instances and Advanced Queues......................................................... 115.1 What is DMS for Kafka?..................................................................................................................................................... 115.2 Product Advantages............................................................................................................................................................. 125.3 Application Scenarios.......................................................................................................................................................... 135.4 Differences Between Kafka Premium Instances and Advanced Queues........................................................... 155.5 Specifications.......................................................................................................................................................................... 165.6 Usage Restrictions and Quotas........................................................................................................................................ 185.7 Basic Concepts....................................................................................................................................................................... 18

6 Differences Between Kafka and RabbitMQ.................................................................... 21

7 Related Services.....................................................................................................................24

8 Usage Restrictions and Quotas.......................................................................................... 25

9 Basic Concepts........................................................................................................................27

Distributed Message ServiceService Overview Contents

2020-03-06 ii

1 DMS Overview

Distributed Message Service (DMS) is a message middleware service using thedistributed, high-availability clustering technology. It provides reliable, scalable,and fully managed queues for sending, receiving, and storing messages.

DMS queues serve as an interchange station to transmit messages amongapplication components. With DMS, components that send messages andcomponents that receive messages do not have to be available at the same time.

Figure 1-1 illustrates the process of creating and retrieving a message.

Distributed Message ServiceService Overview 1 DMS Overview

2020-03-06 1

Figure 1-1 Flowchart for creating and retrieving a message

Table 1-1 Lifecycle of a message

Step

Description

1 The producer sends message M to a queue.Message M is redundantly distributed in the queue.

2 Consumers retrieve message M from the queue.While message M is being retrieved, it remains in the queue. It cannot beretrieved again within 30s since the start of retrieval. If message M is notacknowledged within this period, DMS determines the message retrievalto be unsuccessful, and message M can be retrieved again.


2020-03-06 2

Step

Description

3 Once message M is acknowledged, it can no longer be retrieved byconsumers from the same consumer group.However, it can still be retrieved by consumers from other consumergroups. It remains in the queue for at least 72 hours (unless the queue isdeleted) and will be deleted after this period.

If messages are retrieved using APIs, they must then be acknowledged using the ACK API. Ifmessages are retrieved using the DMS console, they are automatically acknowledged.

As shown in Figure 1-2, DMS stores messages of a queue on different servers, andeach message has multiple replicas, achieving high reliability and availability.

Figure 1-2 Distribution of messages in a queue

Message queues have the following characteristics:

● Ordered message deliveryStandard queues support partition-level and global FIFO.Partition-level FIFO queues provide higher concurrency, but cannot guaranteethat messages are retrieved in the sequence they are received. If you want topreserve the message sequence, you are advised to put sequence informationin each message, so that messages can be sequenced after they are received.


2020-03-06 3

Global FIFO queues support FIFO message delivery and are applicable toscenarios with high sequence requirements.

● At-least-once deliveryOne of the servers storing message replicas may be unavailable when theclient requests or deletes messages. If this happens, the message replicas willnot be deleted from that server and may be sent again when the connectionis restored.This process is called "at-least-once" delivery. Ensure that your application canprocess a message repeatedly while producing the same result.


2020-03-06 4

2 Functions

DMS is a fully managed, high-performance message queuing service that enablesreliable, flexible, and asynchronous communication between distributedapplications. It provides standard, FIFO, and advanced queues compatible withHTTP and TCP. DMS enables you to create message queues for transmittingmessages among application components. With DMS, application componentsthat send or receive messages do not have to remain available at the same time.

Queue Types and Access ProtocolsDMS is compatible with native Kafka queues, enabling you to migrate yourmessage services without any modifications.

● Multiple queue typesDMS provides standard, FIFO, and advanced queues.

● Multi-protocol accessDMS supports access using HTTP RESTful APIs, the TCP SDK, and the KafkaSDK.

Queue FeaturesDMS supports auto scaling, queue sharing, and authorization.

● Auto scalingQueue throughput can scale automatically as required. The use of queues isbilled by queue throughput.

● Dead letter queueMessages failed to be processed are sent to a dead letter queue for analysisand separated processing.

● Queue sharing and authorizationQueues can be shared and authorized between tenants or users created bythe same tenant.

Message FeaturesWith just a few clicks, queue throughput can scale to the required level withoutaffecting upper-layer services.

Distributed Message ServiceService Overview 2 Functions

2020-03-06 5

● Message filteringConsumers can use labels to filter the messages they want to retrieve fromthe chosen queue.

● Message trackingConsumers can specify the time or position at which messages will beretrieved again.

● Intentional delivery delayMessages can be delivered after a specified delay time.

● Message broadcastingThe same message can be delivered to all consumers in the same consumergroup.

● Message redeliveryMessages that will not be immediately retrieved can be delivered back toqueues. Consumers can retrieve the messages when they are ready to do so.

High ReliabilityDMS features data reliability of up to 99.999999999% and service availability ofup to 99.95%.

● Data reliabilityData replication and synchronous flushing to disks ensure data reliability ofup to 99.99999999%.

● Service availabilityClustered and cross-AZ deployments ensure service availability of up to99.95%.

High PerformanceDMS supports massive accumulation of messages, auto scaling of queues, andhigh concurrency.

● Massive accumulationA single queue can hold hundreds of millions of messages withoutcompromising queue performance.

● High concurrencyQueue throughput reaches up to 100,000 concurrent messages per second.Higher concurrency can be achieved simply by adding queues.

● Low latencyMessage delivery time is accurate to the millisecond.

SecurityOperations are traceable and messages are encrypted before being stored.

● TraceabilityDMS interworks with Cloud Trace Service to record and audit tenantmanagement operations.


2020-03-06 6

● Message encryptionEncrypted message storage protects against unauthorized access to messages.


2020-03-06 7

3 Application Scenarios

DMS can be used in various fields, such as enterprise applications, onlinepayments, telecommunications, e-commerce, logistics, marketing, socialnetworking, instant messaging (IM), mobile gaming, video, Internet of Things(IoT), and Internet of Vehicles (IoV).

DMS can be applied to the following scenarios:

● Service decouplingDMS provides message notifications for supplementary services that aredependent on other systems. It decouples supplementary services from keyservices, allowing key services to proceed without waiting for other systems.For example, the order processing (OP) system of an e-commerce websiteputs order information in DMS message queues during promotional activities.The inventory and delivery management systems will read the orderinformation from the queues later.

● Eventual consistencyIn trading or payment systems, the transaction status (success or failure)must be consistent across subsystems or modules. Reliable messagetransmission is required between subsystems or modules to ensure servicecontinuity. DMS provides highly reliable data transmission betweensubsystems or modules to ensure transaction statuses consistency at lowercosts.For example, if a bank customer buys wealth management products by usingthe deposit, the gains of wealth management products may not be includedin the customer's deposit account. This is because a wealth managementsystem usually processes transactions at the end of each day. To avoidreconciliation inconsistency between the banking system and the wealthmanagement system, purchasing and payment data of wealth managementproducts can be stored in DMS, ensuring the eventual consistency betweenthe deposit balance and the wealth management gains.

● Off-peak traffic controlIn e-commerce systems or other large-scale websites, there is a processingcapability gap between upstream and downstream systems. Traffic burstsfrom upstream systems with high processing capabilities may have a largeimpact on downstream systems with low processing capabilities. For example,online sales promotions involve a huge amount of traffic flooding into e-

Distributed Message ServiceService Overview 3 Application Scenarios

2020-03-06 8

commerce systems. DMS can help to buffer orders and other information,relieving pressure on downstream systems. It provides a three-day buffer forhundreds of millions of messages, allowing message consumption systems toprocess them during off-peak periods.

● Log synchronizationApplications asynchronously send log messages to DMS over reliabletransmission channels. Other components can read log messages frommessage queues for further analysis, either in real time or offline. In addition,DMS can collect required key log information to monitor applications.DMS's log synchronization process includes the following steps:

a. The log collection client collects log data from a user application serviceand writes the log data to message queues.

b. Message queues receive, store, and forward the log data.c. Log processing applications subscribe to and consume log messages

stored in message queues.

Figure 3-1 Log synchronization process

Scribe, Fluent, Flume, Logstash, and Rsyslog are mainstream open-source logcollection tools. Logstash, ElasticSearch, and Kibana are open-source log analysis tools.

Distributed Message ServiceService Overview 3 Application Scenarios

2020-03-06 9

4 RabbitMQ Instances

RabbitMQ instances are developed by the HUAWEI CLOUD DMS team. They arephysically isolated and exclusively occupied by each tenant. DMS is fullycompatible with open-source RabbitMQ and supports deployment of RabbitMQinstances with just a few clicks.

Features● Rapid deployment

Set instance information on the DMS console and then submit your order. Acomplete RabbitMQ instance will be automatically created and deployed.

● Usability out of the boxA RabbitMQ instance can be accessed once it is created.

● Service migration without modificationsDMS is fully compatible with open-source RabbitMQ, enabling you to migrateservices that use open-source RabbitMQ clients for creating and retrievingmessages without modifying the service logic.

● Exclusive experienceRabbitMQ instances are isolated from each other.

● Pay per useRabbitMQ instances are free of charge during the open beta test (OBT)period.

● Multiple specifications availableSelect from various types of specifications to meet your service requirements.

● Simple O&MDMS requires minimal O&M, relieving you from service maintenance and thussaving a lot of costs.

Distributed Message ServiceService Overview 4 RabbitMQ Instances

2020-03-06 10

5 Kafka Premium Instances and AdvancedQueues

5.1 What is DMS for Kafka?Apache Kafka is distributed message middleware that features high throughput,data persistence, horizontal scalability, and stream data processing. It adopts thepublish-subscribe pattern and is widely used for log collection, data streaming,online/offline system analytics, and real-time monitoring.

DMS for Kafka is a message queuing service based on Apache Kafka. This serviceprovides Kafka premium instances. The computing, storage, and bandwidthresources used by an instance are exclusively occupied by the user. You can applyfor instances as required and customize partitions and replicas for the topics in theinstances. The instances can be used right out of the box, taking off thedeployment and O&M pressure for you so that you can focus on developing yourservices.

Advanced QueuesAdvanced queues are fully managed by DMS. You can create queues andconsumer groups on the DMS console, and then create and retrieve messages.

Regardless of whether your services are deployed in the cloud, you can use thequeues as the message channel right after the queues are created, without havingto concern about Kafka cluster deployment and O&M. Advanced queues featurelow costs and are billed by the number of times the messages are called.

Kafka Premium InstancesKafka premium instances provide better flexibility. You can customize theresources for your instances and configure the number of partitions and replicasfor topics in the instances.

Readers' GuideDue to the limited space, this documentation introduces DMS for Kafka and itsdifferences from Apache Kafka. You will learn about the detailed information

Distributed Message ServiceService Overview 5 Kafka Premium Instances and Advanced Queues

2020-03-06 11

about DMS for Kafka specifications and how to use the console, call APIs, andconnect to instance using clients.

For more information about the basic knowledge of Kafka or technical detailsabout creating and retrieving messages, please go to the official Apache Kafkawebsite.

5.2 Product AdvantagesDMS for Kafka provides easy-to-use message queuing based on Apache Kafka.Services can be quickly migrated to the cloud without any change, reducingmaintenance and usage costs.

● Rapid deploymentKafka premium instances: Simply set instance information on the console,submit your order, and a complete Kafka premium instance will beautomatically created and deployed.Advanced queues can be used right out of the box. You can connect to aqueue and publish or subscribe to its messages once it has been created onthe DMS console.

● Service migration without modificationsDMS for Kafka is compatible with open-source Kafka APIs and supports allmessage processing functions of open-source Kafka.If your application services are developed based on open-source Kafka, youcan easily migrate them to DMS for Kafka after specifying a fewauthentication configurations.

Advanced queues and Kafka premium instances are compatible with Apache Kafka0.10.2 and 1.1.0, respectively.

● SecurityOperations are recorded and can be audited. Messages can be encryptedbefore storage.In addition to SASL, VPCs and security groups also provide security controls onnetwork access.

● Data reliabilityBoth advanced queues and Kafka premium instances support data persistenceand replication. Messages can be replicated synchronously or asynchronouslybetween replicas.

● High availabilityKafka runs in clusters, enabling failover and fault tolerance so that servicescan run smoothly.Kafka premium instances can be deployed across AZs to further enhanceservice availability.

● Simple O&MHUAWEI CLOUD provides a whole set of monitoring and alarm services,eliminating the need for 24/7 attendance. A set of metrics are monitored andreported, including the number of partitions, topics, and accumulated


2020-03-06 12

http://kafka.apache.org/

http://kafka.apache.org/

messages. You can configure alarm rules and receive SMS or emailnotifications on how your services are running in real time.

● Massive accumulation and auto scalingKafka features high scalability because it runs in a distributed system, orcluster. You can configure up to 20 partitions for a topic. The storage spacecan be also expanded. This means that billions of messages can beaccumulated, suitable for scenarios requiring high concurrency, highperformance, and large-scale access.

● CustomizationYou can choose between advanced queues and Kafka premium instances.Advanced queues are fully managed and can be used immediately aftercreation. You can customize the bandwidth and storage space for a Kafkapremium instance and the numbers of partitions and replicas for topics in theinstance.

5.3 Application ScenariosKafka is popular message-oriented middleware that features highly reliable,asynchronous message delivery. It is widely used for transmitting data betweendifferent systems in the enterprise application, payment, telecommunications, e-commerce, social networking, instant messaging, video, Internet of Things, andInternet of Vehicle industries.

Asynchronous CommunicationNon-core or less important messages are sent asynchronously to receivingsystems, so that the main service process is not kept waiting for the results ofother systems.

For example, Kafka can be used to send a notification email and SMS messageafter a user has registered with a website, providing fast responses throughout theregistration process.

Figure 5-1 Serial registration and notification

Figure 5-2 Asynchronous registration and notification using message queues

Traffic ControlIn e-commerce systems or other large-scale websites, there is a processingcapability gap between upstream and downstream systems. Traffic bursts from


2020-03-06 13

upstream systems with high processing capabilities may have a large impact ondownstream systems with low processing capabilities. For example, online salespromotions involve a huge amount of traffic flooding into e-commerce systems.DMS can help to buffer orders and other information, relieving pressure ondownstream systems. It provides a three-day buffer for hundreds of millions ofmessages, allowing message consumption systems to process them during off-peak periods.

In addition, flash sale traffic bursts originating from frontend systems can behandled with Kafka, keeping the backend systems from crashing.

Figure 5-3 Traffic burst handling using Kafka

Log Synchronization

In large-scale service systems, logs of different applications are collected for quicktroubleshooting, full-link tracing, and real-time monitoring.

Kafka is originally designed for this scenario. Applications asynchronously send logmessages to message queues over reliable transmission channels. Othercomponents can read the log messages from message queues for further analysis,either in real time or offline. In addition, Kafka can collect key log information tomonitor applications.

1. The log collection clients collect log data from a user application service andasynchronously send the log data in batches to Kafka clients.Kafka clients receive and compress messages in batches, which only has aminor impact on the service performance.

2. Kafka persists logs.3. Log processing applications, such as Logstash, subscribe to messages in Kafka

and retrieve log messages from Kafka. Then, the messages are searched forby file search services or delivered to big data applications such as Hadoop forstorage and analysis.

Figure 5-4 Log synchronization process


2020-03-06 14

Logstash is for log analytics, ElasticSearch is for log search, and Hadoop is for big dataanalytics. They are all open-source tools.

5.4 Differences Between Kafka Premium Instances andAdvanced Queues

Both Kafka premium instances and advanced queues are compatible with ApacheKafka. However, they differ in the following aspects.

Open-Source Compatibility● Advanced queues:

Kafka 0.10.2.1● Kafka premium instances:

Kafka 1.1.0With each version upgrade, Apache Kafka introduces new features, improvesAPIs, and updates producer and consumer configuration files. To checkwhether your application features and APIs are compatible with your Kafkaclients, see the upgrade notes on the official Apache Kafka website.

Purchase● Advanced queues:

An advanced queue (equivalent to a topic) is created on the DMS console.You do not need to configure the storage space or the bandwidth becausethese resources are allocated by the system.

● Kafka premium instances:A Kafka premium instance is created on the DMS for Kafka console. Beforecreating a Kafka premium instance, determine the required bandwidth andstorage space based on your service expectations for the next one or twoyears. You also need to prepare a VPC and security group for the instance.After the instance has been created, you must create topics in the instanceand configure the number of partitions and replicas for the topics.

Usage● Advanced queues:

Advanced queues are compatible with Kafka APIs. DMS provides SDKs in Java,Python, Lua, C, and Go languages. For details, see the DMS Kafka DeveloperGuide.To use open-source Kafka clients, see Usage of the Enhanced Java SDK. Addthe enhanced Kafka Java SDK provided by DMS to the directory of the open-source client package and then pass the security authentication.

● Kafka premium instances:Kafka premium instances are fully compatible with open-source Kafka. Youcan access Kafka premium instances and topics using open-source Kafka


2020-03-06 15

http://kafka.apache.org/documentation.html#upgrade_1_1_0

https://support.huaweicloud.com/intl/en-us/devg-dms/en-us_topic_0082178499.html

https://support.huaweicloud.com/intl/en-us/devg-dms/en-us_topic_0082178499.html

https://support.huaweicloud.com/intl/en-us/devg-dms/dms-devg-180626001.html

https://cwiki.apache.org/confluence/display/KAFKA/Clients

clients. If SASL access is enabled, you must use the SSL certificate providedby DMS.

Performance● Advanced queues:

There are two queue modes: high-throughput and high-reliability. In the high-throughput mode, messages are flushed to disk asynchronously, ensuring highconcurrency.

● Kafka premium instances:

Compute, bandwidth, and storage resources are physically isolated for eachinstance. Determine the required bandwidth and storage space when creatingan instance. For storage space, you can choose Ultra-high I/O, whichindicates that messages are stored on SSDs.

Other Dimensions

You can customize the number of partitions and replicas for a Kafka premiuminstance. Each topic can have 1 to 20 partitions and 1 to 3 replicas.

By default, an advanced queue has three partitions and three replicas. If this doesnot meet your requirement, submit a service ticket.

Divide a topic into a certain number of partitions so that messages can be evenlydistributed to partitions, enabling load balancing and horizontal scalability. Differentconsumers can retrieve messages from one or more partitions, improving messageprocessing performance.

With more replicas comes higher reliability. However, synchronizing messages betweenreplicas consumes bandwidth and offsets compute performance.

5.5 Specifications

Advanced Queue Specifications

The network, storage, and computing resources of advanced queues are managedby DMS. The resources allocated to each queue should be capable of meeting theconcurrency and storage requirements of most services. If you have higherconcurrency requirements, submit a service ticket for technical support.

To occupy isolated network, storage, and computing resources, use Kafka premiuminstances instead.

Kafka Premium Instance Specifications

Kafka premium instances are compatible with open-source Kafka 1.1.0. Theinstance specifications are classified based on bandwidth, including 100 MB/s, 300MB/s, 600 MB/s, and 1200 MB/s.


2020-03-06 16

https://cwiki.apache.org/confluence/display/KAFKA/Clients

https://res-static1.huaweicloud.com/content/dam/cloudbu-site/archive/commons/web_resoure/cms/static/upload/files/dms/dmskafkasasl.zip

Table 5-1 TPS and maximum number of partitions supported by different instancespecifications

Bandwidth I/O Type TPS (High-Throughput)

TPS(SynchronousReplication)

MaximumPartitions

100MB/s High I/O 100,000 60,000 300

Ultra-high I/O 100,000 80,000 300

300MB/s High I/O 300,000 150,000 900

Ultra-high I/O 300,000 200,000 900

600MB/s Ultra-high I/O 600,000 300,000 1800

1200MB/s Ultra-high I/O 1.2 million 400,000 1800

Bandwidth SelectionThe bandwidth of a Kafka instance refers to the maximum read or writebandwidth. You are advised to select a bandwidth 30% higher than what isrequired.

● 100MB/sRecommended for up to 3,000 client connections, 60 consumer groups, and70 MB/s of service traffic.




Storage Space SelectionKafka premium instances support storage with 1 to 3 replicas. The storage space isspace consumed by all replicas. When creating an instance, specify its storagespace based on the expected service message size and the number of replicas.

For example, if the estimated message size is 100 GB, the disk capacity must be atleast: 100 GB x Number of replicas + 100 GB (reserved).

The storage space can be expanded as your service grows.


2020-03-06 17

5.6 Usage Restrictions and Quotas

Advanced Queue

You can create a maximum of 30 advanced queues in each region.

There are no network access restrictions on advanced queues. If the client isdeployed in HUAWEI CLOUD, access a queue using its intranet domain name andport number. If the client is deployed outside of HUAWEI CLOUD, access a queueusing its internet domain name and port number. Log in to the DMS console toview the domain names and port numbers of a queue.

If you want to change the default restrictions, see Quotas.

Kafka Premium Instance

You can buy a maximum of 100 Kafka premium instances in each region. Themaximum bandwidth of an instance is 1200 MB/s.

An instance runs in a VPC. A client accesses an instance using the subnet IPaddresses and port numbers of the instance. Therefore, ensure that the client andinstance are in the same VPC and subnet and have correct security group rules.For details, see How to Select and Configure a Security Group? If the client andthe instance are in different VPCs, establish a VPC Peering Connection.

If public access has been enabled for the instance, the client can access theinstance using the public network IP address and port number of the instance. Inthis case, there are no restrictions on the VPC but correct security group rules stillmust be configured for the instance, allowing the port to be accessed throughpublic networks.


5.7 Basic ConceptsThis chapter presents explanations of basic concepts of Kafka.

Topic

A topic is a category for messages. Messages are created, retrieved, and managedin the form of topics.

Topics adopt the publish-subscribe pattern. Producers publish messages intotopics. One or more consumers subscribe to the messages in the topics. Theproducers and consumers are not directly linked to each other.

Producer

A producer publishes messages into topics. The messages are then delivered toother systems or modules for processing as agreed.


2020-03-06 18

https://support.huaweicloud.com/intl/en-us/usermanual-iaas/en-us_topic_0040259342.html

https://support.huaweicloud.com/intl/en-us/dms_faq/dms-ug-180604024.html

https://support.huaweicloud.com/intl/en-us/dms_faq/dms-ug-180604022.html


ConsumerA consumer subscribes to messages in topics and processes the messages. Forexample, a monitoring and alarm platform (a consumer) subscribing to logmessages in certain topics can identify alarm logs and then send SMS or emailalarm notifications.

BrokerA broker is a Kafka process in a Kafka cluster. Each process runs on a server, so abroker includes the storage, bandwidth, and other server resources.

PartitionMessages in a topic are distributed to multiple partitions to achieve scalability andfault tolerance.

ReplicaA replica is a redundant copy of a partition in a topic. Each partition can have oneor more replicas, enabling message reliability.

Messages in each partition are fully replicated and synchronized, preventing dataloss if one replica fails.

Each partition has one replica as the leader which handles the creation andretrievals of all messages. The rest replicas are followers which replicate theleader.

Topics and partitions are logical concepts, while replicas and brokers are physicalconcepts. The following diagram shows the relationships between partitions,brokers, and topics in messages streaming.


2020-03-06 19

Figure 5-5 Kafka message streaming


2020-03-06 20

6 Differences Between Kafka andRabbitMQ

Kafka is pull-based and provides higher throughput. It is suitable for collecting anddelivering large volumes of data, such as collecting and analyzing logs. RabbitMQdoes not provide as high throughput as Kafka, but it offers more message queuingfunctions.

The following is a comparison analysis on the performance, data reliability, serviceavailability, and functions of Kafka and RabbitMQ.

Performance

The performance of message-oriented middleware is measured by throughput.While RabbitMQ provides tens of thousands of QPS, Kafka provides millions.

However, if idempotency and transactions are enabled for Kafka, its performancewill be compromised.

Data Reliability

Both Kafka and RabbitMQ provide the replication mechanism to ensure high datareliability.

Service Availability

Kafka runs in clusters and has partitions and replicas. Therefore, single-nodefailure does not affect services and the capacity of Kafka can be linearly scaled up.

The RabbitMQ service provided by HUAWEI CLOUD also supports clustereddeployment.

Functions

Both Kafka and RabbitMQ are popular open-source message-oriented middleware.They differ mainly in the functions, which are listed in the following table.

Distributed Message ServiceService Overview 6 Differences Between Kafka and RabbitMQ

2020-03-06 21

Table 6-1 Function differences between Kafka and RabbitMQ

Function Kafka 1.1.0 RabbitMQ 3.6.10

Priorityqueue

Not supported Supported. It is recommended thatthe priority be set to 0–10.

Delayedqueue

Not supported Supported

Deadletterqueue


Retry Not supported Not supported

Retrievalmode

Pull-based Pull-based and push-based

Messagebroadcasting

Supported Supported

Messagetracking

Supports offset andtimestamp tracking.

Not supported. Once a messageretrieval has been acknowledged,RabbitMQ will be notified that themessage can be deleted.

Messageaccumulation

Supports higheraccumulation performancethan RabbitMQ thanks tohigh throughput.

Supported

Persistence

Supported Supported

Messagetracing

Not supported Supported by the firehose feature orthe rabbitmq_tracing plugin.However, rabbitmq_tracing reducesperformance and should be usedonly for troubleshooting.

Messagefiltering

Supported Not supported, but can beencapsulated.

Multi-tenancy


Multi-protocol

Only Apache Kafka issupported.

RabbitMQ is based on AMQP andsupports MQTT and STOMP.

Multi-language

Kafka is written in Scala andJava and supports clients inmultiple programminglanguages.

RabbitMQ is written in Erlang andsupports clients in multipleprogramming languages.


2020-03-06 22

Function Kafka 1.1.0 RabbitMQ 3.6.10

Throttling Supports throttling onproducer or consumerclients and users.

Supports credit-based throttling onproducers, a mechanism thattriggers protection from within.

Orderedmessagedelivery

Supports partition-levelFIFO.

Supports FIFO only for single-threaded message queuing withoutadvanced features such as delayedqueues or priority queues.

Security Supports SSL and SASLauthentication and read/write permissions control.

Similar to Kafka.

Idempotency

Supports idempotency for asingle producer session.

Not supported

Transactionmessages

Supported Supported

The comparison is made between open-source Kafka and RabbitMQ.DMS for Kafka and DMS for RabbitMQ maintain open-source compatibility whilesupporting or enhancing features in the open-source versions.


2020-03-06 23

7 Related Services

● Cloud Trace ServiceCloud Trace Service (CTS) generates traces to provide you with a history ofoperations performed on cloud service resources. The traces include operationrequests sent using the management console or open APIs as well as theoperation results. You can view all generated traces to query, audit, andbacktrack performed operations.For details about DMS operations recorded by CTS, see OperationsSupported by CTS.

● Virtual Private CloudDMS RabbitMQ and Kafka premium instances run in a Virtual Private Cloud(VPC) and use the IP addresses and bandwidth created in the VPC. Securitygroups of the VPC ensure secure access to the RabbitMQ and Kafka premiuminstances.

● Cloud EyeCloud Eye is an open platform that provides monitoring, alarm reporting, andalarm notification for your resources in real time.For details about DMS metrics supported by Cloud Eye, see DMS Metrics.

The values of all DMS metrics are reported to Cloud Eye every minute.Advanced queues have only one metric: Accumulated Messages.

Distributed Message ServiceService Overview 7 Related Services

2020-03-06 24

https://support.huaweicloud.com/intl/en-us/usermanual-dms/dms-ug-180418002.html



8 Usage Restrictions and Quotas


Item Default Restriction Modifiable orNot

Queues A maximum of 30 queues can becreated under a project.

Yes

Consumer groups A maximum of 3 consumer groups canbe created in a queue.

Yes

RabbitMQinstances

A maximum of 100 RabbitMQinstances can be created under aproject.

Yes

Kafka platinuminstances

A maximum of 100 Kafka platinuminstances can be created under aproject.

Yes

Whether you can access DMS over a public network depends on the DMS queuetype and access mode.● RESTful API

Public access is supported. You can access standard and advanced queues overa public network by using a RESTful API.

● TCP APIPublic access is not supported. To access DMS by using a TCP API, you mustdeploy your message client on an Elastic Cloud Server (ECS) to which noelastic IP address is bound.

● Open-source Kafka APIPublic access is supported. Alternatively, you can access DMS from an ECS towhich no elastic IP address is bound. Note that the DMS endpoints aredifferent in these two access modes.

Distributed Message ServiceService Overview 8 Usage Restrictions and Quotas

2020-03-06 25


For details on how to access DMS by using a RESTful API, see the DMS APIReference. For details on how to access DMS by using the other two types of APIs,see the DMS Developer Guide.

Distributed Message ServiceService Overview 8 Usage Restrictions and Quotas

2020-03-06 26

9 Basic Concepts

● MessageMessages are JavaScript object notation (JSON) objects used for transmittinginformation. They can be sent one by one or in batches. Sending messages inbatches can be achieved only through calling DMS APIs.Each message can contain up to 512 KB of data.

● AttributesEach message can optionally have one or more attributes. Each messageattribute consists of a name and a value in the key-value format. Attributesare part of the message and are sent and retrieved along with it.Attributes allow consumers to decide how to process the messages beforeprocessing the message body.

● Message queueA message queue is a container that receives and stores message files. Bydefault, 30 queues can be created under a project. Different messages in onequeue can be retrieved by multiple consumers at the same time.– Standard queue: The queue mode can be partition-level first in first out

(FIFO) or global FIFO. Partition-level FIFO queues have higherconcurrency performance than global FIFO queues, but do not guaranteethe FIFO order of all messages.

– Advanced queue: Open-source Kafka clients can be used to create andretrieve messages. The queue mode can be high throughput or highreliability. In high throughput mode, all message replicas are flushed to adisk asynchronously to ensure high performance. In high reliability mode,all message replicas are flushed to a disk synchronously to ensuremessage reliability.

– RabbitMQ queue: RabbitMQ queues are created in RabbitMQ instances,which are physically isolated. You can customize the computingcapabilities and storage space of RabbitMQ instances based on servicerequirements.

● Dead letter messageDead letter messages are messages that cannot be correctly retrieved.DMS can process messages that are not correctly retrieved. If a message failsto be retrieved for a preset number of times, it will be sent to the dead letter

Distributed Message ServiceService Overview 9 Basic Concepts

2020-03-06 27

queue and retained for at least 72 hours. You can then retrieve the messagefrom the dead letter queue.Messages in the dead letter queue are specific to individual consumer groups.Consumers in a consumer group cannot retrieve dead letter messages ofother consumer groups.Dead letter messages from standard queues in global FIFO mode are sent tothe dead letter queue in the FIFO order.

● ProducerA producer, also called a message sender, sends messages to specific queues.

● ConsumerA consumer, also called a message receiver, retrieves messages from queues.After acknowledging a message, a consumer cannot retrieve the samemessage again.

● Consumer groupConsumer groups classify different types of message consumers. A maximumof three consumer groups can be created in each queue.Messages in a queue can be retrieved once by each consumer group.Messages acknowledged by a consumer group are no longer available to thatconsumer group but still available to other consumer groups.Consumers in the same consumer group can retrieve different messages froma queue at the same time.

● Resetting the message retrieval start positionYou can reset the position from which to sequentially retrieve messages sentfrom producers to DMS. You can choose to start retrieving messages from thebeginning of a queue, from the end of a queue, or from a specified point intime within the message retention period (by default, 72 hours for standardqueues).

● Message filteringDMS allows you to use labels to filter the messages you want to retrieve fromthe chosen queue.Messages can be sent to the chosen queues regardless of whether themessages have labels or not.If the Message Labels field in the Retrieve Message dialog box is leftunspecified, all types of messages in the chosen queue will be retrieved. If thisfield is assigned a non-empty value, only messages with the specifiedmessage labels will be retrieved.

● Delay message deliveryMessages can be delivered after a specified delay time.

Distributed Message ServiceService Overview 9 Basic Concepts

2020-03-06 28

service overview - huawei · complete rabbitmq instance will be automatically created and deployed....

Documents