are bot operators eating your lunch?
TRANSCRIPT
![Page 1: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/1.jpg)
Are bot operators eating your lunch?
![Page 2: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/2.jpg)
Agenda
Bots 101The growing bot problemHow bots are eating your lunch!Hayneedle case studySelection criteria for a bot detection solutionQ & A
![Page 3: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/3.jpg)
Good BotsSearch Engine Crawling Power APIsCheck system connectivity and status
Bad BotsSteal contentScan for vulnerabilitiesPerform fraudetc.
The Basics of Bots
A “Bot” is an automated program that runs on the internet
Traffic Distribution by Type, 2016
![Page 4: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/4.jpg)
High Profile Web Scraping in the Ecommerce Industry
QVC is an American television home shopping network and online ecommerce site.
Aggressive price and inventory scraping by shopping aggregator app resulted in the following repercussions for QVC
● Two day website outage● Loss of $2M in revenue● Highly publicized lawsuit● Damage to QVC Brand
![Page 5: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/5.jpg)
Traffic by Size, Ecommerce Sites, 2014 vs 2015
Small and medium ecommerce sites saw about a 100% increase in bad bots between 2014 and 2015
![Page 6: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/6.jpg)
Majority of Bots are Advanced Persistent Bots (APBs)
APBs have one or more of the following abilities:
AdvancedMimick human behaviorLoad JavaScriptLoad external resourcesSupport cookiesBrowser automation (Selenium, PhantomJS)
Persistent Dynamic IP rotationDistribute attacks across IP addressesHide behind anonymous and peer-to-peer proxies
2016 Distil Bad Bot Report
![Page 7: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/7.jpg)
Why the Massive Increase in APBs?
Online data has increased in valuePricing information, product availability, product descriptions, and vendor reviews are changing daily and highly valuable to competitors
Anyone can get in the gameCheap or free virtual servers, bandwidth, easy-to-use tools, and scrapers for hire
Bots no longer tied to IP addressesBots cycle through random IP addresses Bots hide behind anonymous proxies Consumer IPs now infected with bot traffic too
![Page 8: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/8.jpg)
Loading Assets & Bots Mimicking Humans
% of bots able to load external assets (e.g. JavaScript) % of bots able to mimic human behavior
These bots skew marketing tools such as (Google Analytics, A/B testing,
conversion tracking, etc.)These bots fly under the radar of most
security tools
![Page 9: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/9.jpg)
That Majority of Bad Bots Now Use Multiple IP Addresses
Bots which dynamically rotate IP addresses, or distribute attacks are significantly harder to detect and mitigate
![Page 10: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/10.jpg)
Bad Bots Cause the Majority of Website Problems
19% of Traffic Causes the Following Problems
![Page 11: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/11.jpg)
How Bots Eat Your Lunch
How bots are eating your lunch!
![Page 12: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/12.jpg)
How Bots Eat Your Lunch
LOST PROFITS
Decreased Customer Loyalty
Reduce Findability
Lost Cross/Upsell Opportunities
Decreased Customer Satisfaction
Increased Costs
Increased Fraud
![Page 13: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/13.jpg)
Bots and Competitive Data Mining
Duplicating your Product PortfolioBots can easily gather product and supplier listsfor replication elsewhere
Undermining your PricesBots monitor your prices, ensuring competitorscan undercut with lower price listings
Availability TrackingIdentifying when your supply has been exhausted provides competitors a unique opportunity to raise the price of their goods.
![Page 14: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/14.jpg)
Negative SEO Attacks Damage Relevancy
Bots steal content, product lists, and prices for duplication elsewhere on the Internet
Duplicated content reduces your company’s uniqueness and thus quality score
SEO damage may result, especially if○ Your prices are undercut○ The content is repurposed on a more popular site
Duplicate Content Results in Diminished SEO
![Page 15: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/15.jpg)
Common hacking tools like network mappers and vulnerability scanners are automated programs
Once a victim’s network has been mapped, automated vulnerability scanning can be used to find security flaws that can be exploited
These bots let hackers scale their operations
Vulnerability Scanning and Target Exploitation
![Page 16: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/16.jpg)
Bots Make Large Scale Account Takeover Possible
Over 1 billion usernames, passwords combinations exist in the wild
Bot operators create bots to test millions of username/password combinations from breaches at other websites to find the credentials that work on your site
Newly compromised accounts are then used for various forms of fraud/theft
![Page 17: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/17.jpg)
Automated Stolen Credit Card Testing Enables Fraud
“Carding” uses micro-transactions on stolen credit cards against e-commerce sites to test their validity
Carding results in poor user experiences and lots of expensive chargebacks
![Page 18: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/18.jpg)
Bots Plant Malicious Links in Fake Comments
Comment spam is frequently used to redirect users to malicious websites
Malicious Site
![Page 19: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/19.jpg)
Hayneedle Case Study
Hayneedle Case Study
![Page 20: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/20.jpg)
About Hayneedle
Leading online retailer for indoor and outdoor home furnishings and decor
1,000s of top brands - including Hayneedle exclusive designs - and millions of products for every space, style, and budget
![Page 21: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/21.jpg)
Hayneedle Bad Bot Challenges
Bad Bot Challenges Business Impact
Competitive price scraping Competitors attempt to undercut pricing
Automated CVV guessing games
Fraudsters use stolen credit cards in carding attacksTime investigating and reporting the problem
Bot traffic competing with real customers Web performance and the user experience
Skewed analytics Conversion funnel optimizationA/B testing
Inefficient DIY approach“Battle-of-the-bots” ate up 20% of team resourcesOnly 30% effective (at best)Quality of life issues
![Page 22: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/22.jpg)
Hayneedle Bad Bot Challenges
Constant game of bad bot “Whack-a-mole”
Log file analysis and performance monitoring
Agent-string analysis
IP blocking
Traffic redirects
Tarpits
...but the bad bots keep changing their identities, scripts, and IP addresses
![Page 23: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/23.jpg)
Hayneedle Bot Selection Criteria
Bot Detection and Mitigation Solution Requirements
No impact on human visitors
“Self tuning” for defending against emerging and unknown threats
Crowd-sourced threat intelligence model
Seamlessly co-exist with existing solutions(CDN, WAF, etc.)
No “black boxes”
![Page 24: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/24.jpg)
Traffic Overview Report
On August 7th bad bot traffic:
● Spiked ~10x ● Was 4x human traffic
![Page 25: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/25.jpg)
Total Traffic vs CAPTCHAs Served
CAPTCHAs served was 73% of overall traffic served that day!
![Page 26: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/26.jpg)
CAPTCHA Failed Attempts and Solved
Out of 17,000,000+ CAPTCHAs served, only 78 were solved
![Page 27: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/27.jpg)
How to Manage Transactional Traffic
Best Practices and Lessons Learned
Monitor (don’t CAPTCHA) traffic on your checkout and account subdomainsReview Threats by OrganizationUnderstand the rationale of scrapersSelectively Block nefarious organizations
![Page 28: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/28.jpg)
Blocking Nefarious Organizations
Can probably block traffic coming from this data center, especially when 70% of the traffic is from Automated Browsers and/or Known Violators
![Page 29: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/29.jpg)
Hayneedle Results with Distil Networks
Eliminated competitive data miningIntercepting bot traffic with negligible false positivesClean analytics for funnel optimization and A/B testingDistil is a key piece of our fraud detection and prevention suite of toolsUpstream HTTP Errors Report highlighted an issue with our CDN providerWeb infrastructure dedicated to serving humans Boosted team morale!
![Page 30: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/30.jpg)
The Only Easy and Accurate Way to Protect Web Applications from Bad Bots, API Abuse, and Fraud
![Page 31: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/31.jpg)
Browser ValidationDetects all known browser automation tools, such as Selenium and Phantom JS
Protects against browser spoofing by validating each incoming request as self reported
Advanced Bot Detection Increases Accuracy
Behavioral Modeling and Machine LearningMachine-learning algorithms pinpoint behavioral anomalies specific to your site’s unique traffic patterns
Self optimizing algorithms improve bot detection and mitigation without manual configuration
![Page 32: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/32.jpg)
Sticky Bot Tracking With No Impact On Real Users
Device FingerprintingFingerprints stick to the bot even if it attempts to reconnect from random IP addresses or hide behind an anonymous proxy or peer-to-peer network
Tracks distributed attacks that would normally fly under the radar
Without Distil
With Distil
Without Impacting Users Sharing the Same IPAvoids blocking residential users or organizations that might share the same NAT as the bot or botnet
![Page 33: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/33.jpg)
Threat Intelligence From All Distil-Protected Sites
Known Violators DatabaseReal-time updates from the world’s largest Known Violators Database, which is based on the collective intelligence of all Distil-protected sites
Distil customers are automatically protected against new threats discovered anywhere on the network
![Page 34: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/34.jpg)
Automated Attackers Leverage APIs as an Attack Vector
Web Applications
API Endpoints
When blocked from a website, Bots frequently use APIs as an attack vector
APIs tend to have access to the same content, but without as many security controls
![Page 35: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/35.jpg)
○ Install on virtualized or bare metal appliance(s)○ High availability configurations with failover
monitoring○ Heartbeat up to Distil Cloud ○ Deploys in days
Flexible Deployment Options
○ Automatically compresses and optimizes content for faster delivery
○ 17 global datacenters automatically fail over when a primary location goes offline
○ Automatically increases infrastructure and bandwidth to accommodate spikes
○ Deploys in hours
Physical or Virtual Appliances
Content Delivery Network
![Page 36: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/36.jpg)
Dedicated Analyst Team
Fully Managed Service (aka High Touch Services)
![Page 37: Are Bot Operators Eating Your Lunch?](https://reader036.vdocument.in/reader036/viewer/2022070601/587ed2781a28abdb198b559f/html5/thumbnails/37.jpg)
So, is a bot
eating your
lunch?