how we cooked elasticsearch, consul, haproxy and dns-recursor
TRANSCRIPT
How we cooked Elasticsearch, Consul,Haproxy and DNS-recursor
In comic strip
Authors:Shcherbakov S.
Sin A.Tokarev O.
DNS
I resolve my elasticsearch location by DNS
And send request to it
Gimme my data!
Whe
re is
my
Ela
stic
sear
ch?
What happens if Elasticsearch down?
DNSW
here
is m
y E
last
icse
arch
?
Ok. I will hide my Elasticsearch(es) behind HAproxy!!!
Data request
Service monitoring
DNS
Ok. Now I can access my data even if one of elasticsearch is down!
Gimme my data!
Whe
re is
my
Ela
stic
sear
ch?
DNSW
here
is m
y E
last
icse
arch
?
^#%$*&#%*%@*&%##*&$%399493
Ok. Let's add extra balancer for redundancy!
DNS
Ok. Now DNS points to one of balancer.But what will happen
if one of balancer down??
Gimme my data!
{OR}
Whe
re is
my
Ela
stic
sear
ch?
DNSW
here
is m
y E
last
icse
arch
?
Gimme my data!
DNS
Whe
re is
my
Ela
stic
sear
ch?
Gim
me
my
data
!
DNSW
here
is m
y E
last
icse
arch
?
Gimme my data!
It makes me crazythat DNS continues give me
An address of brokenserver
How to make DNS forget broken balancer serverASAP?!?
Make Consul cluster monitor your balancers healthand inform your NS about IPs of health ones
(Yep, Consul can speak in DNS language)
DNS
Forward zone
Consul cluster
Ok! Now if one of my balancer downConsul will return IPs of health ones only
Consul is authoritative for a zonewith my critical services
Not enough thermometers???
DNS
Consul clusterNotify Admin!
Retrieve services IPs
Three safety loops
Node level
Disaster notifications
(zabbix)
Application lo-level
Automated application cluster load balancing
(HAproxy)
Application hi-level
Automated application address discovering
(Consul, DNS recursor)
Do you thing that it is the DNS server that is the Single Point of Failure?
ZABBIX!!!
But it does not affect any component of production system directlyNeither customers are affected
As for DNS servers they can be readily deployed in required amountwith resolv.conf updated with new servers IPs
DNS
Still works..NOT BAD!!
HAproxy
/etc/haproxy/haproxy.cfg
frontend elasticsearch bind *:9200 default_backend elasticsearch
backend elasticsearch server es01 123.123.123.1:9200 check inter 1000 rise 10 fall 5 server es02 123.123.123.2:9200 check inter 1000 rise 10 fall 5 server es03 123.123.123.3:9200 check inter 1000 rise 10 fall 5 option httpchk GET /_cluster/health http-check expect ! string "status":"red"
Consul agent (on HAproxy nodes)/opt/consul/config/srv-es-lb.json
{ "service": { "name": "es-lb", "port": 9200, "checks": [ { "interval": "5s", "http": "http://localhost:9200", "timeout": "1s" } ] }}
Consul looks like as NS from outside
[root@consul01 ~]# dig @consul01 -p 8600 es-lb.service.martyshka +short185.66.251.35185.66.251.34[root@consul01 ~]#
PDNS recursor
/etc/pdns-recursor/recursor.conf
packetcache-ttl=0
/etc/pdns-recursor/conf.d/forward-zones.conf
forward-zones=martyshka=consul01:8600;
Resolver (all nodes)
/etc/resolv.conf
search openprovider.nl hosting-concepts.nl nameserver <recursor 1 IP>nameserver <recursor 2 IP>nameserver <provider NS 1 IP>nameserver <provider NS 2 IP>nameserver 8.8.8.8
DNS
Consul clusterNotify Admin!
Retrieve services IPs
● Cloud● Cluster
● Fail-over● Redundancy
● Fault tolerance
“fashionable stylish for youth”