openstack internal messaging at the edge: in-depth evaluation€¦ · openstack internal messaging...
TRANSCRIPT
![Page 1: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/1.jpg)
OpenStack internal messaging at the edge: In-depth evaluation
Ken GiustiJavier Rojas Balderrama
Matthieu Simonin
![Page 2: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/2.jpg)
Who’s here?
Javier Rojas BalderramaMatthieu Simonin
Ken Giusti
Fog Edge and Massively Distributed Cloud Working Group (a.k.a FEMDC)Wed. 5:30pm - 6:10pm
![Page 3: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/3.jpg)
Challenges at the edge
Core Network
DC1
DC2
Regional sitesLocal sites
Edge sitesConceptual challenges
● Scalability
● Locality
● Placement● Resiliency● ...
![Page 4: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/4.jpg)
● Scalability increase the number of communicating agents● Locality
○ Keep control traffic in the same latency domain (site) as much as possible○ Mitigate control traffic over WAN:
■ APIs■ Database state accesses and internal management
(Thursday, 9AM: Keystone in the context of Fog/Edge MDC)■ Remote Procedure Calls
➢ Scope: Openstack’s Remote Procedure Calls in a massively distributed context
Messaging challenges at the edgeConceptual challenges—the messaging perspective
![Page 5: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/5.jpg)
What’s next ?
● Oslo.messaging● Scalability evaluation● Locality evaluation● Lessons learnt
![Page 6: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/6.jpg)
Openstack Internal Messaging
![Page 7: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/7.jpg)
Openstack Interprocess Communications
BusRemote Procedure Call
(RPC)
![Page 8: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/8.jpg)
Openstack Interprocess Communications● Oslo.messaging
○ Part of the OpenStack Oslo Project
(https://wiki.openstack.org/wiki/Oslo)
○ APIs for messaging services
○ Remote Procedure Call (RPC)
■ Inter-project control messaging in OpenStack
○ Abstraction - hides the actual message bus implementation
■ Opportunity to evaluate different messaging architectures
![Page 9: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/9.jpg)
oslo.messaging RPC● Remote Procedure Call (RPC)
○ Synchronous request/response pattern
○ Three different flavors:
■ Call - typical request/response
■ Cast - request/no response expected
■ Fanout - multicast version of Cast
○ How does the message get to the proper server???
Client Server
Client
Client Server
Server
Server
![Page 10: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/10.jpg)
oslo.messaging Addressing (Targets)● Service Addressing
○ Project assigns servers a well known address
■ Example: “Service-A”
○ Server subscribes to that address on message bus
○ Clients sends requests to “Service-A”
○ Represented by a Target Class in the API
○ Unique to a particular Server
■ Direct messaging
○ Or Shared among Servers
■ Load balancing/Multicast
Client
Server
Server
BUS
To:Service-A
Service-A
Service-B
![Page 11: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/11.jpg)
oslo.messaging Alternative Message Buses● Supports multiple underlying messaging implementations:
○ RabbitMQ Broker (based on AMQP 0-9.1 prototype)
○ Apache Qpid Dispatch Router (AMQP 1.0 ISO/IEC 19464)
■ Barcelona Summit Router Presentation: https://bit.ly/2Iuw6Pu
BrokerRPC
ServerRPC Client
RPC Client
RPC Server
![Page 12: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/12.jpg)
● Brokered RPC Messaging (RabbitMQ)○ Centralized communications hub (broker)
○ Queues “break” the protocol transfer
○ Non-optimal path
Broker and brokerless approaches to RPC
CoreNetwork
Broker
RegionalSite
Local Site
RPC Client
RPC Server
A
RPC Server
B
![Page 13: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/13.jpg)
Broker and brokerless approaches to RPC● Brokered RPC Messaging (RabbitMQ)
○ Centralized communications hub (broker)
○ Queues “break” the protocol transfer
○ Non-optimal path
CoreNetwork
Broker
RegionalSite
Local Site
RPC Client
RPC Server
A
RPC Server
B
![Page 14: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/14.jpg)
Broker and brokerless approaches to RPC
CoreNetwork
Broker
RegionalSite
Local Site
RPC Client
RPC Server
A
RPC Server
B
● Brokered RPC Messaging (RabbitMQ)○ Centralized communications hub (broker)
○ Queues “break” the protocol transfer
○ Non-optimal path
![Page 15: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/15.jpg)
Broker and brokerless approaches to RPC
CoreNetwork
RegionalSite
Local Site
RPC Client
RPC Server
A
RPC Server
B
● Brokerless (Apache Qpid Dispatch Router)○ Deployed in any Topology
○ Dynamic Routing Protocol (Dijkstra)
○ Least Cost Path between RPC Client & Server
![Page 16: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/16.jpg)
Broker and brokerless approaches to RPC
CoreNetwork
RegionalSite
Local Site
RPC Client
RPC Server
A
RPC Server
B
● Brokerless (Apache Qpid Dispatch Router)○ Deployed in any Topology
○ Dynamic Routing Protocol (Dijkstra)
○ Least Cost Path between RPC Client & Server
![Page 17: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/17.jpg)
Broker and brokerless approaches to RPC
CoreNetwork
RegionalSite
Local Site
RPC Client
RPC Server
A
RPC Server
B
● Brokerless (Apache Qpid Dispatch Router)○ Deployed in any Topology
○ Dynamic Routing Protocol (Dijkstra)
○ Least Cost Path between RPC Client & Server
![Page 18: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/18.jpg)
The Tools used for Testing
![Page 19: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/19.jpg)
Oslo messaging benchmarking toolkit
RPCs RPCs
Test BUS
CommandsApplication metrics ombt controller
Oslo Messaging Benchmarking Toolombt: https://git.io/vp2kX
Deploys
System
metrics
oo controller
Ombt orchestratoroo: https://git.io/vp2kh
![Page 20: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/20.jpg)
Scalability evaluation
![Page 21: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/21.jpg)
Evaluation● Test Plan http://bit.do/os-mdrpc
○ Six scenarios (synthetic / operational)○ Two drivers
■ Router■ Broker
○ Three comm. patterns■ Cast■ Call■ Fanout
○ Grid’5000 testbed
Methodology
![Page 22: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/22.jpg)
Scalability evaluation
● Single (large) shared distributed target○ Global shared target○ Scale the # of producers, constant throughput○ How big a single target can be ?
● Multiple distinct distributed targets○ Many targets running in parallel○ Scale the # of targets○ How many targets can be created ?
● Single (large) distributed fanout○ Scale the # of consumers
○ How large a fanout can be ?
Synthetic scenarios (TC1, TC2, TC3)
Target
Target
Target
![Page 23: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/23.jpg)
Scalability evaluationParameters
● # of calls, # of agents (producers, consumers), call types● Bus topologies
○ RabbitMQ cluster of size 1, 3, 5○ Complete graph of routers of size 1, 3, 5○ Ring of routers up to size 30 (inc. latency between routers)
● Bus load○ Light: 1K msgs/s○ Heavy: 10K msgs/s
> oo test_case_2 --nbr_topics 100 --call-type rpc-call --nbr_calls 1000
> oo campaign --incremental --provider g5k --conf conf.yaml test_case_1
![Page 24: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/24.jpg)
Scalability evaluationMemory consumption: Single shared target (TC1) - rpc-call
>25GB
~12GB each
<10GB each
<5GB~2GB each <2GB each
max # of supported agents
![Page 25: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/25.jpg)
Scalability evaluationCPU consumption: Single shared target (TC1) - rpc-call
>20 cores
~3 cores~2 cores each ~1 core each
>15 cores each
<10 cores each
![Page 26: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/26.jpg)
Scalability evaluationLatency: Single shared target (TC1) - rpc-call - 8K clients
Target
![Page 27: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/27.jpg)
Scalability evaluationLatency: Multiple distinct targets (TC2) - rpc-call - 10K Targets
Target
![Page 28: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/28.jpg)
Scalability evaluationLatency: single large fanout (TC3) - rpc-fanout - 10K consumers
Target
![Page 29: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/29.jpg)
Wrap up● All test cases
○ Routers are lightweight (CPU, memory, network connections)● Single shared distributed target:
○ Implicit parallelism is observed only with routers (single queue in brokers)○ Scale up to 10K producers
● Multiple distinct distributed targets:○ Similar behaviour for both drivers because of short buffering○ Scale up to 10K targets (20K agents)
● Single distributed fanout:○ Router is less sensitive to the size of broadcast○ Scale up to 10K consumers
![Page 30: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/30.jpg)
Locality evaluation
![Page 31: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/31.jpg)
Multisite : producers and consumers spread over different distant locations
● Scalability
● Locality○ Keep traffic in the same latency domain as much as possible
Locality evaluationConceptual challenges: reminder
![Page 32: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/32.jpg)
Locality evaluationStrategies: Centralized message bus
● Producer in site 1○ Need to break the symmetry of the consumers
■ Give less rabbit_qos_prefetch_count/
rpc_server_credit to remote consumers¶¶
■ Effects depends on ● Latency● Actual workload
● Producer in site 2■ Sub-Optimal data path
➢ Bad locality in the general case
ConsumerProducer
Consumers
LAN latency
WAN latency
Site 1 Site 2
![Page 33: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/33.jpg)
Locality evaluationStrategies: Sharded Message bus
● E.G : Nova CellsV2
● Strict locality ○ A shard can be a latency domain○ Traffic remains in a shard
● Caveat : ○ routing requests (consumer index,
inter-shard communication),
➢ Routing is deferred to the application
Top Shard (orchestration)
Shard 1
ConsumersProducers
Shard 2
ConsumersProducers
Shard 3
ConsumersProducers
![Page 34: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/34.jpg)
Locality evaluationStrategies: Alternative to sharding
● A tree of routers
➢ Routing is transparent to the application
➢ How locality is ensured ?
Shard 1
ConsumersProducers
Shard 2
ConsumersProducers
Shard 3
ConsumersProducers
r1 r2 r3
r0
![Page 35: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/35.jpg)
Two levels of locality
● Strict locality:○ Closest mode○ Local consumers are picked over remote○ Caveat: local consumers backlog
● Load-sharing○ Balanced mode○ Cost is associated with every consumer○ Consumer with the lower cost is picked○ Cost is dynamic
Locality evaluationStrategies: Decentralized bus (amqp 1.0 only)
r3 r2
r1
Producers Consumer 1(C1)
(C2)Consumer 2
(C3)Consumer 3
50ms100ms
Site 1
Site 2 Site 3
![Page 36: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/36.jpg)
➢ Cost is dynamic➢ Load sharing is locality aware
Locality evaluationDecentralization of the message bus
Up to n=30 sites (ring)
Increase the message rateIncrease the intersite latency
Evaluate the locality in message delivery
99%Local
66%Local
66%Local
95%Local
Increasing load
Increasing inter-site latency
n=3
![Page 37: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/37.jpg)
Locality evaluationDecentralization of the message bus
From High Availability OpenStack to High Locality OpenStack
r2
r1
Site 3C + N + cpts
Site 2
C + N + cp
ts
r1
Control (C)
Network (N)
Compute node 1
Compute node n
Site 1C + N + cpts
r3
● One single Openstack● Shared bus (router mesh)
➢ RPCs locality only
In the future● APIs locality● Database locality
➢ Join the FEMDC !
![Page 38: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/38.jpg)
Lessons learnt
![Page 39: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/39.jpg)
Lessons learnt● Communication layer of OpenStack
○ Two implementations : AMQP 0-9.1 / rabbitmq and AMQP 1.0 / qpid-dispatch-router
● Centralized deployments○ Similar scalability○ Router are lightweight and achieve low latency message delivery esp. under high load
● Decentralized Deployments○ A mesh of routers offers guarantees in the locality of the messages○ 2 levels of locality : strict / locality-aware load sharing
● Toward a high-locality OpenStack for multisite deployment○ Leveraging a router mesh○ Same ideas need to be applied to APIs and database
● FEMDC working group is studying different options○ Wed. 4:40pm - 5:20pm
![Page 40: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/40.jpg)
OpenStack internal messaging at the edge: In-depth evaluation
[email protected]@inria.fr
http://bit.do/oo-jupyter-tc1http://bit.do/oo-jupyter-tc2http://bit.do/oo-jupyter-tc3http://bit.do/oo-tc1-ring30
![Page 41: OpenStack internal messaging at the edge: In-depth evaluation€¦ · OpenStack internal messaging at the edge: In-depth evaluation Ken Giusti Javier Rojas Balderrama Matthieu Simonin](https://reader036.vdocument.in/reader036/viewer/2022071112/5fe7f672861b567d6f79f442/html5/thumbnails/41.jpg)