better living through messaging - leveraging the hornetq message broker at shopzilla
DESCRIPTION
Internally, some projects at Shopzilla have recently started to leverage the HornetQ messaging system to meet performance and scalability requirements. In this talk, Mark Lui and Josh Long review the basic principles of messaging and distributed communication. They demonstrate how loosely coupled, asynchronous communication can improve performance, scalability and reliability and finally touch on Shopzilla-specific use cases for messaging.TRANSCRIPT
![Page 1: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/1.jpg)
- 1 -
Better Living Through Messaging -
Leveraging the HornetQ Message
Broker at Shopzilla
Mark Lui ( Merchant Services)
and
Josh Long (Inventory)
![Page 2: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/2.jpg)
- 2 -
What is Messaging
![Page 3: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/3.jpg)
- 3 -
What is Messaging?
Store and Forward
Enables asynchronous communication
• Reduces latency for information transfer
![Page 4: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/4.jpg)
- 4 -
What is Messaging?
Loosely coupled distributed communication (Data or Command)
Exchange of messages between software components
Two forms
• Point-to-Point (Queue)
• Publish/Subscribe (Topic)
![Page 5: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/5.jpg)
- 5 -
Enterprise Application Integration
Integration of data and services between applications
Communication between information silos
Approaches
• File Transfer
• Shared Database
• Remote Procedure Call
• Messaging
![Page 6: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/6.jpg)
- 6 -
Integration Patterns
Patterns
• Data Synchronization
• Workflow
• Portal
![Page 7: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/7.jpg)
- 7 -
Enterprise Application Integration
• Approaches integration using messaging
• Enables Event Driven Architecture
• Multiple users of information
![Page 8: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/8.jpg)
- 8 -
Message Queue Features
Reliable
Fast
…
Secure
Duplicate message detection
Flow control – Slow/Fast consumer handling
Wild card destinations
Durable subscription
Scheduled Delivery
![Page 9: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/9.jpg)
- 9 -
Message Queue Features
Redelivery handling
• DLQ/Discard
• Delay redelivery
Message Ordering on Topics
Client side load balancing options
• Random
• Round Robin
Message prioritization
![Page 10: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/10.jpg)
- 10 -
Introducing HornetQ
New ultra high performance enterprise grade messaging system
from JBoss
Apache V2 License
HornetQ 2.0.GA obtained scores up to 307% higher than
previously published SPECjms2007 benchmark results
“Throughput of 800K small non persistent messages per second
on a single 4 x 2.5GHz core server” – Tim Fox
![Page 11: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/11.jpg)
- 11 -
HornetQ – Secret Sauce
LibAIO
• Linux Kernel 2.6 Subsystem
• Able to submit I/O requests without waiting for completion
HornetQ uses an append-only journal using LibAIO
ActiveMQ 6 will support LibAIO
![Page 12: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/12.jpg)
- 12 -
ActiveMQ Issues
Difficult to configuration correctly
Prone to hang
Poor persistence options
• KahaDB – better scalability, shorter recovery time
• AMQ Message Store – faster
• JDBC database – too slow
![Page 13: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/13.jpg)
- 13 -
High Availability - About Resiliency
Active/Passive replication
HornetQ Example
![Page 14: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/14.jpg)
- 14 -
High Availability - About Resiliency
Live Server
![Page 15: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/15.jpg)
- 15 -
High Availability - About Resiliency
Backup Server
![Page 16: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/16.jpg)
- 16 -
Clustering - About Scalability
Client side load balancing
![Page 17: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/17.jpg)
- 17 -
About performance
Offer Bidding Service Performance Data
![Page 18: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/18.jpg)
- 18 -
Spring Framework
Connection Options
![Page 19: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/19.jpg)
- 19 -
Spring Framework
Alternative Connection Options
![Page 20: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/20.jpg)
- 20 -
Spring Framework
JMS Template
![Page 21: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/21.jpg)
- 21 -
Spring Framework
Message Driven POJOs : MessageListener
![Page 22: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/22.jpg)
- 22 -
Spring Framework
Message Driven POJOs : MessageListener
![Page 23: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/23.jpg)
- 23 -
Spring Framework
Message Driven POJOs : SessionAwareMessageListener
![Page 24: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/24.jpg)
- 24 -
Spring Framework
Message Driven POJOs : SessionAwareMessageListener
![Page 25: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/25.jpg)
- 25 -
Spring Framework
Spring Integration
![Page 26: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/26.jpg)
- 26 -
Spring Framework
Spring Integration
![Page 27: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/27.jpg)
- 27 -
Messaging at Shopzilla
Merchant Service
• Offer Bidding Service – Using to scale
Inventory
• Offer Service
Dampening request to Coherence
Dampening request to Oracle for write behind
![Page 28: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/28.jpg)
- 28 -
Merchant Service’s Order Bidding Service
Demo
![Page 29: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/29.jpg)
- 29 -
Inventory’s Offer Service
Demo
![Page 30: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/30.jpg)
- 30 -
HornetQ – Lessons Learned
Use the right Connection Factory
Use Spring 2.5.6 or 3.0
For High Availability disable Spring reconnection logic
Be aware of the High Availability restart sequence
Paging is turned off by default
JMX is your friend
![Page 31: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/31.jpg)
- 31 -
Possible Future Opportunities
SLA compliance
Unified messaging
Real-time, event driven architectures
![Page 32: Better Living Through Messaging - Leveraging the HornetQ Message Broker at Shopzilla](https://reader033.vdocument.in/reader033/viewer/2022060116/557eba8cd8b42ad27b8b4c56/html5/thumbnails/32.jpg)
- 32 -
Conclusion
Q and A?
Don't forget to tip your waiter!