![Page 1: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/1.jpg)
![Page 2: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/2.jpg)
![Page 3: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/3.jpg)
Windows Defender
ATP
Signals from hundreds of
millions of customers
Microsoft Edge and
Internet Explorer
8B internet downloads
Office 365
400B emails analyzed
Bing
18B web pages scanned
WHY MACHINE LEARNING? BILLIONS OF SIGNALS
![Page 4: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/4.jpg)
WHY MACHINE LEARNING? THREAT LANDSCAPE
96%of malware are seen
only once
1 2 3 4 5 6 7 8 9 10 11 12Do
wn
load
att
em
pts
Hours after first encounter
Malicious downloads encountered
more than once
SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017
SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017
55%
6% 6% 7%
26%
1 2-10 11-100 101-1000 1001+Perc
en
t to
tal en
cou
nte
rs
Number of client encounters per threat
Customer impact of unique and
prevalent threats
![Page 5: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/5.jpg)
WHY MACHINE LEARNING? SCALE
![Page 6: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/6.jpg)
WHY MACHINE LEARNING? PRECISION
![Page 7: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/7.jpg)
WHY MACHINE LEARNING? HUMAN BIAS
![Page 8: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/8.jpg)
![Page 9: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/9.jpg)
MODELS ARE AN ABSTRACT REPRESENTATION OF REALITY
Multidimensional model…The one true earth… Two-dimensional model
![Page 10: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/10.jpg)
H o w w e m e a s u r e m a c h i n e l e a r n i n g m o d e l s
By definition*machine learning models are imperfect
*else they would not be a “model”
CONFUSION TABLE
---------------------------------------
||===============================||
PREDICTED || positive | negative | Recall||
TRUTH ||===============================||
positive || 65,975 | 277,058 | 0.1923||
negative || 48,608 | 11,179,862 | 0.9957||
||===============================||
Precision || 0.5758 | 0.9758 | ||
||===============================||
OVERALL 0/1 ACCURACY: 0.971856
---------------------------------------
ACCURACY, PRECISION, AREA UNDER THE CURVE
---------------------------------------
AUC: 0.828116 (0.0000)
Accuracy: 0.971856 (0.0000)
Positive precision: 0.575783 (0.0000)
Positive recall: 0.192328 (0.0000)
Negative precision: 0.975817 (0.0000)
Negative recall: 0.995671 (0.0000)
Log-loss: 0.155248 (0.0000)
Log-loss reduction: 19.396277 (0.0000)
F1 Score: 0.288342 (0.0000)
AUPRC: 0.323377 (0.0000)
---------------------------------------
![Page 11: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/11.jpg)
![Page 12: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/12.jpg)
Our Approach
Retrospectively measure…FNs - false negatives (misses)FPs - false positives (incorrect detections)
Impact to consumers Are people more likely to switch from Windows Defender Antivirus to another product after an FN or FP event? (We call this switch customer churn.)
Source: Consumer Windows Defender Antivirus customers on Windows 10 who used the Microsoft Malicious Software Removal Tool, Jan.-Apr. 2017
Measuring FNsThreat active upon detection
Classifier, threat report or researcher later marked file or behavior as malicious and client sent telemetry-only report (did not block)
Measuring FPs
Classifier or researcher later marked file or certificate as clean and reported as threat
![Page 13: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/13.jpg)
![Page 14: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/14.jpg)
45%non-Microsoft
antivirus
another non-Microsoft
antivirus
![Page 15: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/15.jpg)
91.4%not correlated with an FN or FP
8.6%
![Page 16: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/16.jpg)
2.1%2.3%
3.4%
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
4.0%
Control Group
(no FN or FP)
FN Experience FP Experience
Ch
urn
Rate
1.1x
1.5x
![Page 17: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/17.jpg)
![Page 18: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/18.jpg)
1
10
100
1,000
10,000
100,000
1,000,000
10,000,000
100,000,000
1,000,000,000
Very low Low Moderate High
Clean File Min and Max Prevalence by
Prevalence Category
Percent of clean files in ecosystem 84% 12% 3.7% 0.4%
2.1%
3.4%
4.0%
3.4%3.1%
2.2%
Control group Any fp High prevalence
fps
Moderate
prevalence fps
Low prevalence
fps
Very low
prevalence fps
Percent churned
![Page 19: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/19.jpg)
Region Control
churn
Fp
churn
Increased likelihood
of churn after FP
Argentina 0.2% 3.2% 18.7
Colombia 0.2% 3.1% 12.6
Indonesia 1.2% 4.7% 4.1
United States 2.8% 10.1% 3.6
United Arab Emirates 1.0% 3.3% 3.5
Poland 3.4% 11.1% 3.3
![Page 20: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/20.jpg)
![Page 21: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/21.jpg)
2.1%2.3%
2.7% 2.6%2.4% 2.4% 2.3%
2.1%
Control group Any fn Software
bundlers
Trojans Password
stealers
Viruses Ransomware Support scams
Percent churned
![Page 22: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/22.jpg)
Region Control
churn
Fn
churn
Increased likelihood
of churn after FN
Argentina 0.2% 2.2% 13.2
Colombia 0.2% 3.1% 12.5
Israel 0.7% 2.1% 3.3
United Arab Emirates 1.0% 2.5% 2.6
Poland 3.4% 7.3% 2.2
![Page 23: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/23.jpg)
![Page 24: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/24.jpg)
![Page 25: Windows Defender ATP Next-gen protection and …...SOURCE: WINDOWS DEFENDER ANTIVIRUS, AUGUST 2017 SOURCE: WINDOWS DEFENDER ANTIVIRUS, Q1 2017 55% 6% 6% 7% 26% s 1 2-10 11-100 101-1000](https://reader034.vdocument.in/reader034/viewer/2022042802/5f3b09e718397611c4743f75/html5/thumbnails/25.jpg)