bait your hook a novel detection technique for keyloggers

Bait your HookBait your HookA Novel Detection Technique for A Novel Detection Technique for KeyloggersKeyloggers

Stefano Ortolani, Cristiano Giurida, and Bruno Crispo

RAID 2010 Sep.

1

OUTLINEOUTLINEIntroductionOur ApproachArchitectureEvaluationConclusion

2

Introduction – Introduction – threat?threat?Malware with keylogging

functionalities

3

Introduction – Introduction – threat?threat?

4

Introduction - Introduction - KeyloggerKeyloggerWin32 API

◦ GetKeyState()◦ GetAsyncKeyState()◦ SetWindowsHookEx()

X 11◦ gdk_window_add_filter()

Source: http://www.securityfocus.org - 20075

CountermeasureCountermeasureAnti-keylogging software

◦ Signature-based◦ APIs tracing and detection

Virtual keyboardOne-time password

6

Countermeasure(cont.)Countermeasure(cont.)An ideal approach should be:

◦ Unprivileged◦ Reliable◦ Portable

Many keyloggers, one behavior

7

Our approachOur approachThe correlation between keystroke

events and I/O activity

Bait your Hook!◦ Inject keystrokes◦ Record the bytes output

8

ArchitectureArchitecture

The prototype's architecture divided in components and domains.9

InjectorInjectorInject the input stream

◦ Mimicking the behavior of a simulated userRely on unprivileged API calls

◦ Win32: SendInput◦ X11: XTestFakeKeyEvent

The resulting keystroke events should:◦ No different than keystroke generated by a

user at the keyboard

10

MonitorMonitorApplication-level statistics of I/O

activitiesRely on unprivileged performance

counter◦ Win32: WriteTransferCount

Total number of bytes the process wrote since its creation

◦ Linux: Analogous performance counterMinimal overhead

11

Pattern TranslatorPattern TranslatorTransform an AKP(Abstract Keystroke

Pattern) into a stream and viceversa◦ A pattern P is a set of samples Pi

◦ Each sample Pi is the normalized amount of data measured at the time interval i

◦ The normalization scales all the samples within the range [0,1]

12

Pattern Translator(cont.)Pattern Translator(cont.) N - the number of samples in the pattern. T - the constant time interval between any two

successive samples. Kmin - the minimum predefined number of keystrokes

per sample allowed. Kmax - the maximum predefined number of keystrokes

per sample allowed. Ri – the average keystroke rate

13

DetectorDetectorInfer a cause-effect relationship

between input/output AKP formOperates on a single process at a

timeCalculate PCC(Pearson

correlation coefficient)

14

Detector(cont.)Detector(cont.)Linearity: scale and location invariant.

◦ PCC(P,Q) = PCC(a + P*c,Q)Immune to data normalization such as

encryption or ignored keystrokes.Buffering?

◦ Focus-based buffering◦ Time-based buffering

15

Pattern GeneratorPattern GeneratorIt must NOT resemble any pattern

exhibited by legitimate processes.It must be easily identifiable in the

output.SYSmark 2004 SE suite

16

Pattern Generator(alg.)Pattern Generator(alg.)Random

◦ random within the range [0,1]Random Fixed Range

◦ random permutation of uniformly distributed samples

Sine Wave◦ discrete sine wave oscillating between 0 and 1

Impulse◦ pattern composed of alternated 0 and 1.

Workload Aware◦ maximally uncorrelated to the actual workload

17

ParametersParametersHow the AKP’s parameters influence the PCC？◦ A proof-of-concept keylogger

18

Parameters(cont.)Parameters(cont.)Time interval

19

False NegativesFalse NegativesEffect of random padding

20

False PositivesFalse PositivesWorkload generated by the “Office” suite

21

ResultResultDetection results for the keyloggers used in

the evaluation. PCC's threshold set to 0.80

New threshold and parameters◦ Pattern=RFR, PCC=0.6, N=60, T=1000, Kmin=1,

Kmax=50

Top monitoring free software list - http://www.keylogger.org

22

LimitationLimitationA Keylogger actively performing I/O

activities◦ Higher inject rates?

Aggressive buffering◦ Monitor the memory accesses (privileged

API)

23

ConclusionConclusionAn unprivileged approach for

keyloggers.Effective against real keyloggers in a

realistic scenario.The proposed architecture is OS

independent.The resulting tool (GPL licensed) will

be soon made public.

24