aws meet-up: logging at scale on aws
TRANSCRIPT
Logging @ Scale on AWS
Visualization > Plaintext
Who am I?
• ChrisRiddell• TechCo-Founderofsomestartups,SeniorSoftwareEngineer• BigDataguy– Redshift,S3,EC2,EMR,Hive,Spark,Dynamoandmanyothers….• ImanagebigAWSinfrastructurefromsoftwaretoarchitecturetoDevOps (startuplife...)• Java(+others)/AWS/geek• AWSProfessionallyCertifiedSolutionArchitect(askmehow!)• Notaloggingexpert- liketothinkI’mgettingcloseJ
Logging @ Scale on AWS
• Usecases•Whattolog• Commercialoptions• Commontools• Plausiblearchitectures• Demo!EC2w/Fluentd ->KinesisFirehose ->Elasticsearch
Why log?
• Getvisibilityintoapphealth,centrallyaccessibleandsearchable• Alerts&fasterroreventresponse– Agility!• Nothavingtologintoindividual instancesandtail• Contextualtracking e.g.userbehaviour• Findingcodeoptimisations• …andmanymanyotherreasons!
First have common language
•WhatisDEBUG,INFO,WARNandERRORusedforinyourorganisation?• Havecommonlanguageforwhatshouldbeloggedwhere• Badlevelingmessesupyourstorage• E.g.DEBUGlogsgoingtoyourexpensiveElasticsearch store,whentheyneverneedtobesearched
• Oneguy’sopinion:http://stackoverflow.com/a/8021604/3843660
Centralising the logs
• Let’sgetthemoffthehost• BasicDIY:• Syslog-ng,rsyslogd,nxlog
• AdvancedDIY:• Splunk forwarder,Logstash,Flume,Fluentd
• Thirdparty• SaaS
Commercial options: SaaS
• LogEntries,SumoLogic, Loggly,Splunk Cloud,PaperTrails,AWSCloudWatch Logs…• TypicallyRESTful JSONlogdumpAPIs• Search&visualizationsarecorefeatures•Mosthaveafreetier•Manylibrariesavailableforvariouslanguagesand/orpackagedversions• Costsgoupwithdatasize,retentionperiodandusercount• Nicetohave:Userdefinedalerts;S3archival…..
Self-hosted solutions
• Splunk – Enterprisesolutionconsistingofforwarders,searchheads,andindexers.Licensed.• ELK/EFKstack=Elasticsearch,Logstash orFluentd, Kibana• Today=Fluentd (forwarder),KinesisFirehose,Elasticsearch,Kibana• FKFEKstack?
Elasticsearch?
• Forsearch!• Indexes• Shards- Distributed&scalesout• Replicas• JSONRESTAPI• ApacheLucene• Kibana isanElasticsearch pluginthatprovidesaniceinterfacetothesearchdata*withvisualizations*
Logstash & Fluentd agents
• Packagedinstall• Inputandoutputlogs• Centraliseyourinstancelogs• Oftenusedasasyslogtail’er orasalocalHTTPlogendpoint• Parse/transform/filter/tag• StoreorForward• “Logstash emphasizesflexibilityandinteroperability whereas Fluentd prioritizessimplicityandrobustness” - http://goo.gl/f5I4cL
Lo
Architecture – Pre October 2015
Today’s Solution
Lo
Syslog/FluentdonEC2
KinesisFirehose Elasticsearch
Demo: Set up EC2 and Fluentd
•WespinupadefaultAWSAMIEC2instancewithrolepermissiontopushdatatoFirehose,accessviaSSH&HTTP)•WeSSHinandinstallFluentd• curl -L https://td-toolbelt.herokuapp.com/sh/install-redhat-td-agent2.sh | sh• /usr/sbin/td-agent-gem install fluent-plugin-kinesis #installAWSFHplugin• Thenconfigurefluentd topushoursyslog’s
Demo: Fluentd config (/etc/td-agent/td-agent.conf)## Syslog reader. Configure port 42185 to send events to in rsyslog config<source>
type syslogport 42185bind 0.0.0.0tag system
</source>
## Filters to transform records and add metadata<filter **>
type record_transformerenable_ruby<record>
@timestamp ${require 'time'; Time.now.utc.iso8601}</record>
</filter>
## Output to Firehose using the instance role<match **>
@type kinesis_firehoseregion us-west-2 delivery_stream_name logsflush_interval 2s
</match>
Demo: restart log agents and serve HTTP
# After /etc/td-agent/td-agent.conf has been setup# Send syslog to fluentd listenerecho "*.* @127.0.0.1:42185" | sudo tee /etc/rsyslog.d/22-fluent.conf
sudo service td-agent restartsudo service rsyslog restart
# Let’s make a web server for you to push your own logs!mkdir webcd webecho 'Hello!' > index.htmlsudo python -m SimpleHTTPServer 80 |& logger -t httpsvr &
Demo: Setting up Elasticsearch
•WesetupAWSElasticsearch Service:Somenotes:• Dedicatedmaster- performsclustermanagementtasks,doesn’tholddata• Metrics:Theusualstuff.NotetheJVMMemoryPressuremetric.Amazonrecommendsscaleup/outif>85%• Clusterstatusisyellowonsinglenodebecausereplicascannotbeassigned.Addanodeorchangethesetting• Clusterhasit’sownaccesspolicy.Ifyouchooseinstanceroleaccesscontrol,youmustsignallrequeststoES(useAWSSDKs).YouwillnotbeabletoaccessKibana onthissetting• Checkwhatsizetheinstancestoreisonyourselectedinstancetype,oruseEBS
Demo: Setting up Kinesis Firehose
•WecreatealogsdeliverystreamwithElasticsearch asthetarget• Firehose:Somenotes:• Apipelinetopushdatainathighscale• Dumpdatain,anditbuffersrecrods (logsinourcase)thembeforepushingtoElasticsearch andoptionallyS3(Redshiftalsosupported)• PayperGBofingestion$0.035USD(eachrecordroundedtonearest5kb)• Differentdestinations(e.g.WARN/ERRORtoElasticsearch butrestonlytoS3)wouldneeddifferentFirehose deliverystreams
Limitations? Further features?
• AWS’sElasticsearch islimitedonplugins,butverygoodoutoftheboxsettings• Fastwaytogethighthroughput,highscalelogsintoastoredindexwithS3backups,andvisualisation!• FurtherfeaturesusingAWSLambdaasglue:
• Alerts• CloudTrial/S3logsingestiontoFH/Elasticsearch,• Deletionofoldindexesataspecifiedperiod(wewantthelastXdaysonly)• andmore…
• S3bucketpolicyforstorageoptimisation (eg oldstufftoglacier)• Yourcustomapplications:PushdirectlytoFluentd’s HTTPendpoint,notviasyslog(moreflexibilityandtagging)• Furthercustomisations toFluentd config
The end!
• Thanks!•@ChrisJRiddell•@ParrotAnalytics• HiringIntermediate/SeniorJavaEngineersJ• Upcoming:Webdev &moreengineers• https://parrot-analytics.workable.com/ toapply