powerdns technical deep-dive

PowerDNS Technical Deep Dive Dynamic Filtering for Malware & Parental Control

Pieter Lexis, Peter van Dijk, Bert Hubert, Alexander ter Haar, Andrea Tosatto

2 | PowerDNS Platform

Agenda: Technical Deep DiveOctober 2016

•  PowerDNS (re)introduction•  Why (malware) filtering? How effective is it?•  How does it work, challenges•  Recursor 4.0 relevant features: Lua & RPZ•  Sources of security data•  Platform implementation: IP address tracking, user

preferences, help desk panel, spotting infected users, query logging

•  Demo time!


PowerDNS introduction


1999

Company introduces database driven DNS and geographical load balancing,

2002-2006

PowerDNS Nameserver and PowerMail go open source; PowerDNS Express launched for EU, US markets

2007-2013

P o w e r D N S Authoritative, Recursor open source products l a u n c h e d ; 2 4 / 7 migration, installation, i n t e g r a t i o n , consolidation services & support

2015

PowerDNS merges with Open-Xchange: target audience and instal led base are amongst the largest Telcos globally

PowerDNSBe open or be history

2016

PowerDNS 4.0•  Malware protection•  Parental control

•  Reporting


Market ShareLarge ISPs and Telcos use PowerDNS

•  Authoritative:•  30%+ of all hosted domains (40-50% in Europe)•  75% - 95% of all hosted DNSSEC domains•  Hundreds of millions of phone numbers (call routing, number portability)•  PowerDNS is default choice for very large scale hosting deployments

•  Recursor: •  150 million+ users served by PowerDNS Recursor •  Shipped with all major Linux & BSD distributions

•  PowerDNS Products have over 150k+ deployed instances•  1315 profiles on LinkedIn mention PowerDNS experience

Who are you?

6


PowerDNS Core TechnologiesWhat we do: more than just name serversPowerDNS Authoritative Server: Up to extremely large scale domain hosting, fully automated DNSSEC, database backed, error checking API drivenPowerDNS Recursor: Resolves domain names, robust, focus on customer experience, security, (per-subscriber) statistics, dynamic domain redirection, very flexiblednsdist: highly DoS and DNS aware load balancer and firewallPowerDNS Tooling: powerful tools to visualize & study DNS problems and measure performance

•  Platform: •  fully graphical, monitored, GUI controllable, High Availability environment for Authoritative

and Recursor•  Recursor platform including support for (selectively) filtered DNS (Malware detection and

parental control, long term query logging & user statistics (malware)


PowerDNS AuthoritativeThe gold standard for large scale hosting

Standard & compliant serving of DNS information from all relevant databases:

•  MySQL, PostgreSQL, LDAP, SQLite, MS SQLServer

•  Text files, dynamic scripts •  Native support for legacy BIND zonefiles

Leading DNSSEC implementation worldwide•  Hosting over 75% of DNSSEC domains•  “1 click DNSSEC”

Scales to millions of domains per server

Powerful dynamic features:•  Geographical load balancing•  Content redirection, smart failover


PowerDNS RecursorFast & Flexible

Standards compliant resolution of domain names•  Strive for maximum resolution percentage

•  At highest speed•  With least operator intervention

•  or conversely: least customer complaints!•  DNSSEC, RPZ

•  Powerful dynamic capabilities•  Query & answer modification for security & filtering•  Dynamic–aware cache


DNSDISTDNS and DoS aware load balancing•  DNS benefits from special load balancing policies not

frequently found in existing load balancing solutions

•  Example is “query concentration”, leading to a few very busy servers with extremely high cache hit rates

•  Customers may also be abusing DNS for tunneling purposes, or otherwise irregular use

•  Infected users generate harmful traffic, which dnsdist filters & reports (at very high query rates)

•  dnsdist delivers complete flexibility in routing and measuring of DNS traffic, even on non-PowerDNS platforms


DNSDISTDNS and DoS aware load balancing

•  Per subscriber rate-limiting•  “Abusive queries pool” for difficult customers•  DoS defence by detection of:

•  Timeout generation•  Servfail generation•  NXDOMAIN overloading•  Random subdomain attacks•  Botnets

•  Kernel based many gigabit/s filltering•  DNS tunneling detection/blocking•  Known bad domain detection & shunting•  UDP to TCP forcing to fend off spoofing attacks•  Extensive statistics on ”right now” query traffic


PowerDNS PlatformFull featured DNS solution

•  Management of DNS infrastructure to deliver high performance resolution and always-on availability

•  Even legacy servers•  Granular level graphing and

analysis of performance and subscriber behaviour

•  Protection from DoS aimed at the nameservers

•  Protection of subscribers from malware, phishing and malicious websites

•  Per user content control for subscribers to prevent access to undesirable websites

•  Subscriber metadata storage & search


Product lineup

Authoritative Platform •  Management interfaces •  Report & Analytics •  Automation •  Load Balancing •  DOS Protection +Basic Support Services

Recursor Platform •  Management interfaces •  Report & Analytics •  Automation •  Load Balancing •  DOS Protection +Basic Support Services

PowerDNS Recursor

Opt

iona

l M

odul

es

PowerDNS Authoritative Server

Parental Control

Malware Filtering

DNS Dist DNS Dist

OX PowerDNS for Internet Service

Providers

OX PowerDNS for Hosting providers

Plat

form

Long term query logging

ENUM

Long term query logging


Security Challenges•  Old software, old phones, old anti-virus•  You may be on up to date OS, up to date

browser•  Many of your users are not!•  Windows XP is still out there. Old Android

phones •  Old = 1 year

•  Goal: do something for security from the network


Parental Control•  In some countries, governments demand “safe internet” browsing

•  For kids•  For .. Husbands?•  A bit like “18+ movies” which must be labelled

•  Some parents also just want this, because the internet can be a scary place

•  Can install app on every tablet, computer, phone, tv in the house•  Or.. The network can filter


DNS based (malware) filtering

1.  Check if user wants / should get filtering, and what kind of filtering

2.  Check DNS lookups against reputation, categorization, malware supplier databases

3.  Compare with filtering requirements•  Some people WANT malware!

4.  Either answer DNS query as normal, or, fake in IP address of “sorry” page

•  And keep statistics for user feedback


DNS Filtering: does it work? Is it right? What do you think?•  Malware, Botnets, Phishing, Parental Control•  Evasion (8.8.8.8)•  Non-DNS malware•  Speed of list updates•  Ethics

•  Opt-in•  “Double opt-in”•  Opt-out

•  Network neutrality


PowerDNS Filtering: Open platformAn open platform for detecting and preventing subscriber infection

•  PowerDNS Filtering is an open platform •  Integrates with all major categorization / threat list providers

?


PowerDNS Open Source Features Relevant for filtering•  Available for 10 years: Lua based question/answer

modification•  Synchronous•  Asynchronous lookups (!)

•  New in 4.0: •  RPZ support

•  Modifiable from Lua•  Protobuf based logging of all queries


RPZ: Response Policy Zone

•  Innovation by ISC, Paul Vixie, Vernon Schryver•  Describes how to treat content matched by:

•  A domain name•  A response IP address•  A nameserver (potentially) used in resolution

•  Transferred via IXFR•  Updates every few seconds if needed•  Many RPZ feeds are available•  Support in: BIND and PowerDNS


Challenges for per-user (malware) filtering

•  Can’t do 100% mandatory filtering for everyone•  Not legally, and there are always people that want access to malware•  For parental control: not everyone is a parent or cares

•  Per-user settings are nice, but name server sees IP addresses, not users•  And users may not be circuit-ids or MAC addresses or IMSIs!•  1M users, 5 hours lease time: 55 updates/second•  Or: 1 update/minute -> 3000 people get wrong settings

•  Needs to be 100% reliable and low-overhead•  Needs UI for users, customer support and (re)categorization


PowerDNS Infrastructure

•  Lua support•  Determine status of user (CDB, Redis)•  Determine status of domain (custom modules per provider)

•  Or: configure RPZ flags•  PowerDNS:

•  Consult the right cache (filtering, non-filtering)•  If miss, do the right lookup or provide the A-record of the sorry page•  Store answer in the right cache


Malware FilteringSafe Guard your Subscribers against malware

PowerDNS Malware Filtering offers possibility to:•  Prevent infection•  Detect & warn infected users

•  retroactively detect infection •  Investigate suspicious traffic

For ISP’s:•  Offer to all or some of your customers •  Detect problems to better help subscribers (i.e. ‘slow internet complaints’)•  Enabled / disabled ‘globally’ or ‘per user’ (as an upsell)


Parental ControlSafe Guard your Customers with Multi-Level Access Control

PowerDNS offers unique Multi-Level Control for Browsing:•  Safe Browsing •  Easy to use Web Control Panel•  Supports categories and time-windows•  Both white lists and black lists•  Per-device/per-user parental control

•  CPE assistance required

Architecture

27


PowerDNS Platform Components•  Stock PowerDNS Recursor, dnsdist•  Lua modules that take decisions for filtering•  Nginx server that hosts “sorry” page, and proxies URL-level filters•  Sniproxy for TLS termination•  User-interface for subscribers/customers

•  Database to store it•  Helpdesk interface to (re)set customer preferences•  Full DNS traffic logging (dstore)•  Malware analysis of logged traffic•  IP/User listener (Radius)•  Reporting module•  Redis distribution of IP/User/Preferences setting•  Deployment script


Dstore: Query logging & searchingOn commodity hardware•  Store all queries for days, weeks or months

•  Response codes•  Response latency•  Response records

•  Used to:•  Investigate customer/domain complaints (‘x doesn’t resolve for me’)•  Determine source and target of DoS attacks•  Comply with Lawful Intercept / Data retention regulations•  Find/flag infected subscribers / devices•  Find sources of spam without using DPI

•  Potentially fully anonymized

31

Recursor Recursor Recursor Recursor

dnsdist dnsdist

dstore dstore

dstore dstore

Dgateway

Raw packets


IP/User matching listener•  Receive IP address (IPv4, IPv4:port, IPv6) mappings

•  Radius•  DHCP•  “tail –f”

•  Highly redundant•  Multiple receivers

•  To protect against state loss•  Distributed to every resolver

•  Knows about multiple level mappings: circuit-id to user to IP


Parental Control: Fine-grained control over Parental Control filter


Malware Filtering: Analysis•  Analysis of Per-query, per user results. •  Shows detailed user data for advanced troubleshooting


What is an infected user?Security application

•  Many users click on bad links from time to time•  Does not make you infected

•  Large wifi at school will have many infected laptops, but whole school can not be flagged as infected

•  Detection is in fact a dynamic process that needs to be tuned and monitored•  Impact of wrongly flagging a user as infected is huge•  PowerDNS Platform Security Solution therefore offers:

•  Modular flagging •  Potential for manual verification in interface

•  Note: customer care processes can benefit greatly from knowing user’s infected status!


Query logging & searchingFunctionality

•  Search via API, command line or attractive web interface•  Output as JSON, XML or HTML

•  Example scenario: 1 million qps, 1 week retention, 5 small storage servers, 200TB of data total

•  Rapid queries keyed on: source IP, query name, response content•  Few second response times worst case

•  Scanning queries based on time window at 25 million queries/s•  In other words, scan an hour of traffic in 2 minutes

•  Delivers exact queries, error codes, responses, drops and response times•  No dependencies beyond regular server hardware, works on rotating media•  FULL ANONIMYZATION MODULE


Query logging & searchingOther notable features

•  Easy rotation/archiving of old data•  Split out per day/week

•  “Hot data” can live on SSD/NVMe and copied over to near-line storage for slower but still rapid retrieval

•  Can be configured for various scenarios (long term low cost lower performance, short term, low cost, higher performance etc)

•  Data sources:•  PCAP (vendor neutral), •  Port mirror (vendor neutral) •  native from PowerDNS


Query logging & searchingSecurity application

•  Combines with malware filtering to store status of query•  Blocked•  Flagged

•  Delivers lists of (recently) infected users•  Combines with subscriber communications for notifying infected users•  Detecting which users are infected, or simply clicked on the wrong link, is a

customizable process. •  Rules will depend on business logic and risk appetite


Reporting and insight


Dashboard for real-time informationLive display:•  Plot statistics

•  Overview of Query rate and pattern •  historical and 'here and now'.

•  Keeps NOC informed on current DNS performance. •  Live display for technical personnel


Dashboard: live security panelLive display for real-time continuous information•  Shows attacks currently in progress, and •  Which IP addresses and domain names are being shielded


Weekly Automated ReportsAutomatically generated Reports •  Overview of DNS performance•  Mailed to relevant staff•  Current metrics + comparison with past•  Gives management overview

Per server quality & volume metrics•  CPU utilization, Peak memory use•  Allows resource management


Highly Scalable solutionLow latency, high resilience againt DoSInternet traffic and related DNS queries grow:•  Number of DNS queries grows 30% per year•  With LTE, mobile internet looks more like fixed internet•  500kqps = 5 million subscribers!

PowerDNS benefits from special load balancing policies not frequently found in existing load balancing solutions:

•  Example is “query concentration”, leading to a few very busy servers with extremely high cache hit rates

•  dnsdist delivers complete flexibility and protection in routing and measuring of DNS traffic, even on non-PowerDNS platforms


Demo time!


The PowerDNS Demo APFull locally hosted setup

•  The full PowerDNS Platform stack•  Nameserver, •  DHCP/Radius tracker•  Statistics,•  Logging, queries, reporting•  Filtering: malware & parental•  User control panel•  Customer care control panel•  Hardware: one i7 Intel NUC with a number of virtual machines•  This setup would support millions of internet users


Join the PowerDNS Demo APAnd be filtered

• Join the “PowerDNS Demo” AP• Password: PowerDNS

• You will get a highly dynamic IP address• To change your settings and find out your

PowerDNS Name, go to:•  http://filter-user.demo.powerdns.com/

• Turn on some filtering!


Join the PowerDNS Demo APAnd be filtered

•  BE CAREFUL

•  Your DNS lookups will appear on the big screen!!!•  With your PowerDNS Name

•  If you have malware, we’ll also see that•  Suggested test domain: hollandcasino.nl which is blocked as ‘gambling’ and ‘games’.•  Please be careful testing adult sites!

Germany

Open-Xchange AGRollnerstrasse 1490408 NurembergTel.: +49-2761-8385-0

Netherlands

PowerDNS Herengracht 38B2511 EJ Den HaagTel.: +31-15-785-0372

USA, California

Open-Xchange530 Lytton AvenuePalo Alto, CA 94301Tel.: +1-408-500-0768

Spain

Open-XchangeCamino del Cerro de los Gamos 28224, MadridTel.: +34 91-79-012-26

Contact us

www.powerdns.com www.open-xchange.com


Questions/discussion

powerdns technical deep-dive

Software