proactive bot defense: a deep dive - f5.com · prefix of the fictive url that is used in the...
TRANSCRIPT
![Page 1: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/1.jpg)
![Page 2: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/2.jpg)
PRESENTED BY:
![Page 3: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/3.jpg)
Bots Rule the Internet
30% Malicious
53% Automated Traffic
53%
reference
30%
![Page 4: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/4.jpg)
ASMPR
OAC
TIVE BOT
DEFEN
SEBO
T SIG
NATU
RES
BEHAVIOR-BASEDWeb Scraping, Credential Stuffing, Human Detection
CAPABILITIES CHALLENGE
JS CHALLENGE
IP/DOMAIN VALIDATION
BOT SIGNATURESSIMPLE BOTScURL, ApacheBench, Nikto, NESSUS
IMPERSONATING BOTSGoogleBot???, Safari???, FireFox???
COOKIE/JS-ENABLED BOTSPhantonJS, SlimerJS, Selenium, HTMLUnit
AUTOMATED BROWSERS
![Page 5: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/5.jpg)
![Page 6: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/6.jpg)
![Page 7: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/7.jpg)
••
![Page 8: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/8.jpg)
![Page 9: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/9.jpg)
••
••••
![Page 10: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/10.jpg)
••
![Page 11: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/11.jpg)
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
GoogleSource: 66.249.66.1 Bypass PBD
![Page 12: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/12.jpg)
Bot Signature PBD Bypass
1. Signature has domain for validation2. Source IP/Domain is ACTUALLY validated3. Signature Category is set to “Report”
(This is the default for Search Engine category)
OR….1. Signature is in a “Benign” category2. Signature Category is set to “Report”3. DB variable dosl7.proactive_defense_exclude_benign_bots is set to
“enabled” (default is disabled).
All three conditions must be met!
![Page 13: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/13.jpg)
![Page 14: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/14.jpg)
![Page 15: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/15.jpg)
security dos bot-signature "/Common/ADmantX Platform Semantic Analyzer" {category /Common/Crawlerrisk narule "headercontent:\"ADmantX Platform Semantic Analyzer\"; useragentonly; nocase;"user-defined false
}
security dos bot-signature "/Common/Google Keyword Suggestion" {category /Common/Crawlerdomains { .google.com }risk narule "headercontent:\"Google Keyword Suggestion\"; useragentonly; nocase;"user-defined false
}
![Page 16: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/16.jpg)
•• •
![Page 17: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/17.jpg)
![Page 18: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/18.jpg)
•
• Block Suspicious Browsers
•
•
• Cross-Domain Requests
•
![Page 19: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/19.jpg)
USER
BRO
WSER
SERVER
BIGIP
Initial Web Page AccessHTTP Request (no cookie)
Client-Side JS Challenge
Resend Request (with cookie)
Send Original HTTP Request
HTTP Response (main page)HTTP Response (main page)
Request objects (with cookie)
Requests for objects
Object ResponsesHTTP Response (main page)
Display page to user
![Page 20: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/20.jpg)
![Page 21: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/21.jpg)
••••
•••••
![Page 22: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/22.jpg)
•••
•••••
••••
![Page 23: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/23.jpg)
•••
••••••
Redirect challenge - Can client follow redirect AND maintain cookie stateJS-free test - HTTP/1.1, Keep-Alive, Language, other Header check
![Page 24: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/24.jpg)
BIGIP
Site1
Site2
Browser
Set Cookie For Site1
Set Cookie For Site2
![Page 25: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/25.jpg)
BIGIP
Site1
Site4
Browser
Set Cookie For Site1
Set Cookie For Site2
Site3
Site2
Set Cookie For Site3
Set Cookie For Site4
![Page 26: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/26.jpg)
BIGIP
Site1
Site2
Browser
Set Cookie For Site1
Set Cookie For Site2
![Page 27: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/27.jpg)
sys db variable Default Description
dosl7.proactive_defense_fictive_url /TSPD/ Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required.
dosl7.proactive_defense_cookie_name TSPD_101 Name of signed cookie marking the request as validated. This cookie is global (not per VS), and the cookie name is as-is: not a prefix
dosl7.proactive_defense_prefix TSPD In some cases, some intermediate cookies and parameters are set. This is the prefix of their names.
dosl7.proactive_defense_validate_ip enable Allows disabling the validation of the client IP address in the cookie
dosl7.proactive_defense_validation_percent 100 Percentage of requests on which the signed cookie is validated (may be lowered to improve performance)
dosl7.proactive_defense_excluded_headers (empty) Comma-separated list of request headers that cause the Proactive Bot Defense to be bypassed
dosl7.proactive_defense_exclude_benign_bots disable Exclude bots from benign categories from PBD challenges
![Page 28: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/28.jpg)
sys db variable Default Description
dosl7.browser_legit_min_score_captcha 60 Minimum score at which suspicious browser challenge will challenge with CAPTCHA (if enabled)
dosl7.browser_legit_min_score_drop 120 Minimum score at which suspicious browser challenge will block (if JS-challenge is possible)
dosl7.browser_legit_min_score_jsfree_drop 100 Minimum score at which suspicious browser challenge will block (if JS-challenge is NOT possible. i.e. cross-domain non-html resource)
dosl7.cors_ajax_urls None a comma-separated list of wildcard-supported URLs. The URLs in this list are HTML pages from which CORS AJAX requests could be sent.
dosl7.cors_font_urls None a comma-separated list of wildcard-supported URLs. The URLs in this list are CORS AJAX-requested fonts. (i.e. /t/cors/font/style.css,/t/cors/font/font.otf)
![Page 29: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/29.jpg)
sys db variable Default Description
dosl7.max_normalization_cycles 2 The amount of normalization cycles done on uri before matching attack signatures.
dosl7.max_lookup_length 255 The maximum length in characters in which signature is searched for.Applies to both URLs and User-Agent strings.
dosl7.max_user_agent_occurrences 1 The maximum number of User-Agent header occurrences in whichthe signatures is searched.
dosl7.max_num_headers 50 Maximum number of headers in which the User-Agent string is looked for.
![Page 30: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/30.jpg)
![Page 31: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/31.jpg)
•
•
•
•
1.
2.
3.
![Page 32: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/32.jpg)
sys db variable Default Description
dosl7d.shun_list enable Whether to use the shun list to block IP addresses
dosl7d.min_challenge_success_ratio 10 The minimum percentage of good transactions per IP address (or else add it to the shun list).
dosl7d.min_challenge_rps 10 The minimum requests per second before the system will apply shun mitigation
dosl7d.shun_prevention_time 120 The time in seconds to keep the IP address in the shun list.
![Page 33: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/33.jpg)
![Page 34: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/34.jpg)
![Page 35: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/35.jpg)
![Page 36: Proactive Bot Defense: A Deep Dive - f5.com · Prefix of the fictive URL that is used in the Proactive Bot Defense. The slashes at the beginning and end are required. dosl7.proactive_defense_cookie_name:](https://reader030.vdocument.in/reader030/viewer/2022020109/5d2b4d5f88c993140a8dc53e/html5/thumbnails/36.jpg)