deploying immutable infrastructures with rabbitmq and solr
TRANSCRIPT
![Page 1: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/1.jpg)
Deploying Immutable Infrastructures
RabbitMQ and Solr
@jordillonch@eloipoch
March 2017
Xavier Sanchez
≥
![Page 2: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/2.jpg)
Who are we?
Eloi Poch Jordi LlonchXavier Sanchez
![Page 3: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/3.jpg)
Agenda
• What is an immutable infrastructure?
• How do we deploy a RabbitMQ cluster?
• How do we deploy a Solr cluster?
• Conclusions
![Page 4: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/4.jpg)
What is an immutable infrastructure?
![Page 5: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/5.jpg)
ChangeChange
Change
Change
Change
Change
Change
![Page 6: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/6.jpg)
once you instantiate something you never
change it
![Page 7: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/7.jpg)
replace instances with other ones with the desired changes already applied
![Page 8: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/8.jpg)
State A
State B
State E
State D
State F
State C
![Page 9: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/9.jpg)
Key benefits• no differences between servers (eventually)
• lower failures & easier troubleshooting
• no downtime
• more confident update (up, running & tested)
• simple update & rollback
![Page 10: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/10.jpg)
...but this is a simplification
![Page 11: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/11.jpg)
we should be aware that infrastructure is divided into “data”
and “everything else”
![Page 12: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/12.jpg)
How do we deploy a RabbitMQ cluster?
![Page 13: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/13.jpg)
What is RabbitMQ?
![Page 14: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/14.jpg)
M M
M
M
M
M
M M
MMM
M
M M M
M
M
![Page 15: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/15.jpg)
RabbitMQ @ Wallapop
![Page 16: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/16.jpg)
Domain Event Bus
• preferred system of communication between services
• avoids coupling between origin (publisher) and destiny/destinies (subscribers)
• failure tolerant architecture
![Page 17: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/17.jpg)
Characteristics
• transient data
• no control over the publishers or consumers
• sync publish and async consumption
• soft/near realtime
![Page 18: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/18.jpg)
How do we migrate to the new cluster?
![Page 19: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/19.jpg)
Initial status
0
![Page 20: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/20.jpg)
Great Service
Bored ServiceWTF Service
RabbitMQ Cluster
Crazy Service
Suicidal Service
Mainstream Service
![Page 21: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/21.jpg)
Create new cluster
1
![Page 22: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/22.jpg)
Great ServiceBored ServiceWTF Service
RabbitMQ Cluster
Crazy ServiceSuicidal Service Mainstream Service
RabbitMQ Cluster
new
![Page 23: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/23.jpg)
Copy definitions
2
![Page 24: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/24.jpg)
Great ServiceBored ServiceWTF Service
RabbitMQ Cluster
Crazy ServiceSuicidal Service Mainstream Service
RabbitMQ Cluster
newconfigconfig
![Page 25: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/25.jpg)
Create federated queues
3
![Page 26: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/26.jpg)
Queue A
Queue B
Queue C
Queue A
Queue B
Queue C
message flow
message flow
message flow
Upstream Downstream
Only when a downstream queue has idle consumers
pull messages
pull messages
pull messages
![Page 27: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/27.jpg)
Great ServiceBored ServiceWTF Service
RabbitMQ Cluster
Crazy ServiceSuicidal Service Mainstream Service
RabbitMQ Cluster
newconfig
Federated Queues
config
![Page 28: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/28.jpg)
Change DNS
4
![Page 29: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/29.jpg)
Great ServiceBored ServiceWTF Service
RabbitMQ Cluster
Crazy ServiceSuicidal Service Mainstream Service
RabbitMQ Cluster
new
Federated Queues
configconfig
![Page 30: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/30.jpg)
Close connections
5
![Page 31: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/31.jpg)
Great ServiceBored ServiceWTF Service
RabbitMQ Cluster
Crazy ServiceSuicidal Service Mainstream Service
RabbitMQ Cluster
new
Federated Queues
configconfig
![Page 32: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/32.jpg)
Great ServiceBored ServiceWTF Service
RabbitMQ Cluster
Crazy ServiceSuicidal Service Mainstream Service
RabbitMQ ClusterFederated Queues
oldconfigconfig
![Page 33: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/33.jpg)
Delete federated queues
6
![Page 34: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/34.jpg)
Great ServiceBored ServiceWTF Service
RabbitMQ Cluster
Crazy ServiceSuicidal Service Mainstream Service
RabbitMQ Cluster
oldconfigconfig
Federated Queues
![Page 35: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/35.jpg)
Delete old cluster
7
![Page 36: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/36.jpg)
Great Service
Bored ServiceWTF Service
RabbitMQ Cluster
Crazy Service
Suicidal Service
Mainstream Service
![Page 37: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/37.jpg)
How do we deploy a Solr cluster?
![Page 38: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/38.jpg)
What is Apache Solr?
![Page 39: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/39.jpg)
• Solr is an open source enterprise search platform built on Apache Lucene providing advanced full-text search capabilities
• distributed search
• faceted search
• real-time indexing
• index replication
• hit highlighting
• suggesters
• ...
![Page 40: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/40.jpg)
Why Solr?• NRT indexing
• Fast replication
• Handle large volumes of data (>25 millions docs)
• Handle high traffic rates
• Easy to scale
• Geospatial search
• ...
![Page 41: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/41.jpg)
Solr @ Wallapop
![Page 42: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/42.jpg)
Multiple uses• Wall index
• General Search index
• Specialized verticals indexes
• Search suggesters
• Upload suggesters
• Filter suggesters
• ...
![Page 43: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/43.jpg)
![Page 44: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/44.jpg)
Characteristics
• Non-volatile data
• Total control over the publishers and consumers
• Sync/async publish and sync consumption
• Near real-time
![Page 45: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/45.jpg)
How do we deploy a Solr cluster?
![Page 46: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/46.jpg)
• Combining live (sync) indexing and async reindexing
• Approach used in our NRT clusters
• Search service is not affected by the process
• Fast and easy roll-back
![Page 47: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/47.jpg)
Why reindexing?• Changes in the index format and schema require reindexing the
whole corpus!!!
• New features might require changes in schema
• Changes in data type
• Changes in tokenization and analysis chain
• Add new fields or update fields to already indexed documents
• Solr/Lucene version upgrades require reindexing
• We need seamless tools for reindexing
![Page 48: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/48.jpg)
Initial status
0
![Page 49: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/49.jpg)
![Page 50: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/50.jpg)
Create new cluster
1
![Page 51: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/51.jpg)
![Page 52: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/52.jpg)
Activate double NRT indexing
2
![Page 53: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/53.jpg)
![Page 54: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/54.jpg)
Reindex old data
3
![Page 55: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/55.jpg)
![Page 56: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/56.jpg)
Switch search traffic
4
![Page 57: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/57.jpg)
![Page 58: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/58.jpg)
Deactivate double NRT indexing
5
![Page 59: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/59.jpg)
![Page 60: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/60.jpg)
now we can kill our old cluster...
![Page 61: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/61.jpg)
... or maybe we can use both!
![Page 62: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/62.jpg)
Search traffic doubling: Functional testing
• Duplicate search requests
• Send sync request to main cluster
• Send async request to alternative cluster
• Check that everything works correctly
• use real traffic without serving to real users.
![Page 63: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/63.jpg)
Traffic segmentation• A/B testing
• relevance tests
• new ranking algorithms
• changes in tokenization/analysis chain
• new search components
• Performance testing
• Changes in machine configuration
• Changes in Solr configuration
• Changes in java configuration
• Changes in index/schema format
![Page 64: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/64.jpg)
Much more than a traffic split
• DNS traffic splitting has some limitations
• Unable to send different search queries to each cluster
• Splitting traffic at the API level allows us to change Solr queries
• Removing barriers for testing new Solr features.
![Page 65: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/65.jpg)
Conclusions
• there is no silver bullet
• actually know your service
• you are not the only one
• repeat it (even when not necessary)
![Page 66: Deploying Immutable infrastructures with RabbitMQ and Solr](https://reader034.vdocument.in/reader034/viewer/2022042707/58d1a39b1a28ab6f6b8b5a3f/html5/thumbnails/66.jpg)
Thanks!We are hiring!
Mobile Test Automation Engineer Web Test Automation Engineer
Functional QA Software Engineer Junior - Mobile iOS
Android Engineer