ISA 673 Operating Systems’ Security

Topic: Malware Reverse Engineering Unpacking Binary Packers

Arnur Tokhtabayev, George Mason University!

Malware types (source: Securelist.com)

¨  Viruses and Worms (by propagation: net, e-mail, p2p, IM, IRC, USB)

¨  Trojan programs (by payload: backdoor, rootkit, clicker, keylogger - spyware, dropper, FakeAV, ransom, DDoS, downloader)

¨  Malware tools (by functionality: constructor, packer, hacktool)

¨  Riskware (by purpose: net-tool, monitor, proxy, fraud-tool, adware)

Malware reverse engineering scheme

1.  Create isolated experimental environment (host only VM)

2.  Submit malware to existing sandboxes (Anubis, ThreatExpert) •  Inspect its high level behavior •  Watch file droppers and created processes

3.  Examine its imports, exports and strings n  Based on import API, guess malware type (e.g.

SetWindowsHook)

4.  Identify packer and unpack (manual or auto, PEiD)

Malware RE scheme (2)

5.  Disassemble/decompile malware n  Trace API usage (context, constant attributes) n  Deduce variables, create structures, name them properly (semantics) n  Deduce function semantics (what does it do, approx.) n  Trace sensitive data path

6.  Debug binary n  Trigger conditions n  Resolve implicit jumps (control flow) n  Reveal actual values variables (dynamic API attributes)

7.  Patch binary (optional) n  Execute malicious payload (addresses limitations of step 2)

Malware self-defense technologies

Source: Kaspersky (2007)

PE packer technologies

Code

Virtualization

Anti-crack (prevent debugging/dumping, resist

unpacking tools)

PE packers (multilayer packing) UPX, ASPack, PECompact, Boxed App

Armadillo, ASProtect, Themida

VMProtect, Enigma, Themida

Packer Distribution in Malware

Source: CSIS research team, CS, GMU 2012 (analyzed 7000 malware samples)

45%

22%

9%

6%

3%

2% 1% 1%

9%

Armadillo

UPX

BobSoft

ASPack

ASProtect

PECompact

Themida/WinLicense

PE File Format

Program code

Imported functions

Data (constants, strings, etc.)

Section offsets

UPX packing scheme (basic)

Compress

Copy resources

Empty space for an unpacked image

Compressed data

Unpacking code

Resources

Headers (new) Entry point Original executable

Packed executable

PE Viewer Demo (packed vs. original)

UPX operation scheme (basic)

Compress

Copy resources

Empty space for an unpacked image

Compressed data

Unpacking code

Resources

Headers (new) Entry point Original executable

Packed executable

Dynamic Unpacking Scheme Tool chain

Identify packer • PEiD

Trace (debug) unpacking

routine • OllyDbg

Define Original Entry

Point

• Signatures (semantics)

Dump unpacked

image • OllyDump

Rebuild import table

(section) • ImportRec

Anti-Debugging in Packers Debugger detection

¨  System API: check for the existence of a debugger using system information (IsDebugerPresent(), CheckRemoteDebuggerPresent()).

¨  System data: check debugger presence in Process/Thread global structures (NtGlobalFlag, PEB.BeingDebugged).

¨  Other: code integrity check (INT 3h instruction detection), SEH (INT 3h exception check), hardware breakpoints, execution latency

Anti-Debugging in Packers: Example (Source: http://www.openrce.org/reference_library/anti_reversing)

Note: FS – segment register (points to various items in TIB (Tread Information Block)) Example: FS[0] -> SEH FS[30h] -> Address of PEB (Process Environment Block) structure

CALL IsDebuggerPresent ; invoke API CMP EAX,1 JE @DebuggerDetected

Ssytem API:

MOV EAX,DWORD PTR FS:[18h] ; get TID entry address MOV EAX,DWORD PTR DS:[EAX+30h] ; get PEB entry address MOVZX EAX,BYTE PTR DS:[EAX+2h] CMP EAX,1 JE @DebuggerDetected

System data:

typedef struct _PEB { BOOLEAN InheritedAddressSpace; BOOLEAN ReadImageFileExecOptions; BOOLEAN BeingDebugged; BOOLEAN Spare; …

Experiments: Malware Packing

¨  Packers (UPX, ASPack) ¨  Packer view ¨  Packer robustness

Experiments manual unpacking Malware generation tool, PEiD, OllyDbg

OllyDbg

Plugins: http://www.openrce.org/downloads/

Unpacked Code Dumping to New PE

Contact info

Dr. Arnur Tokhtabayev, Center for Secure Information Systems George Mason University Computer Science Department Research I, Rm 435

E-mail: ¨  atokhtab@gmu.edu ¨  arnur78@gmail.com