introduction to software security malwaresecuresw.dankook.ac.kr › iss19-1 ›...

70
Seong-je Cho Spring 2019 Computer Security & Operating Systems Lab, DKU Introduction to Software Security Malware (Chapter 11)

Upload: others

Post on 25-Jun-2020

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Seong-je Cho

Spring 2019

Computer Security & Operating Systems Lab, DKU

Introduction to Software Security

Malware(Chapter 11)

Page 2: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 2 -

Sources / References

Textbook

N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Nicholas Weaver, Computer Science 161: Computer Security, Berkeley

Myrto Arapinis, Computer Security: INFRA10067, University of Edinburgh

UFL_EDU 2015, Ch04-Malware: “Malware: Malicious Software”

Please do not duplicate and distribute

Computer Security & OS Lab, DKU

Page 3: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 3 -

Contents

Malware

Malicious Software = Malicious Code

Types of Malware

Virus

Worm

Malware Detection

Signature/Change/Anomaly based Detection

New Trends of Malware

Do You Trust Software?

Smartphone app, Anti-malware software, DBMS, Compiler, OS/Kernel, Boot loader…

Computer Security & OS Lab, DKU

Page 4: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 4 -

Malicious Software

Malware

any software program designed to cause harm to a computer system or designed to frustrate the user experience.

Fred Cohen’s initial virus work in 1980’s Used viruses to break the MLS (multi-level security) systems of the time

Types of malware (lots of overlap) Virus passive propagation

Worm active propagation

Trojan horse unexpected functionality

Trapdoor/backdoor unauthorized access

Rabbit exhaust system resources can implemented by virus, warm …

Computer Security & OS Lab, DKU

Page 5: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 5 -

Viruses, Worms, Trojans, Rootkits

Malware can be classified into several categories, depending on

propagation and concealment

Propagation

Virus: human-assisted propagation (e.g., open email attachment)

Modifies existing file, program, library, boot block, etc.

Worm: automatic propagation without human assistance

Stand-alone program

Concealment

Rootkit: modifies operating system to hide its existence

Trojan: provides desirable functionality but hides malicious operation

Various types of payloads, ranging from annoyance to crime

Computer Security & OS Lab, DKU

Page 6: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 6 -

Malware Timeline

In 1982, high-school student Rich Skrenta wrote first virus released in the wild: Elk Cloner, a boot sector virus

First academic use of term virus by PhD student Fred Cohen in 1984, who credits advisor Len Adleman with coining it

(c)Brain, by Basit and Amjood Farooq Alvi in 1986, credited with being the first virus to infect PCs Brain virus

Morris worm (1988)

Code Red worm (2001)

SQL Slammer worm (2004)

Future of malware?

Computer Security & OS Lab, DKU

Page 7: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 7 -

Computer Viruses

• A virus is an obligate parasite

• A computer virus is computer code that can replicate itself by modifying other files or programs to insert code that is capable of further replication.

• Self-replication into existing files/programs is what distinguishes computer viruses from other kinds of malware, such as logic bombs.

• Another distinguishing property of a virus is that replication requires some type of user assistance, such as clicking on an email attachment or sharing a USB drive – the infected file is opened.

Computer Security & OS Lab, DKU

Page 8: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 8 -

Brain Virus

First appeared in 1986 More annoying than harmful

Brain did nothing malicious Not much reaction by users

A prototype for later viruses

What it did1. Placed itself in boot sector (and other places)2. Screened disk calls to avoid detection3. Each disk read, checked boot sector to see if boot sector

infected; if not, goto 1

Screen: = check (적절한지)확인하다[거르다], 가려내다. I use my answerphone to screen my phone calls.

Computer Security & OS Lab, DKU

Page 9: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 9 -

Virus Phases

• Dormant phase. During this phase, the virus just exists—the virus is laying low and avoiding detection.

• Propagation phase. During this phase, the virus is replicating itself, infecting new files on new systems.

• Triggering phase. In this phase, some logical condition causes the virus to move from a dormant or propagation phase to perform its intended action.

• Action phase. In this phase, the virus performs the malicious action that it was designed to perform, called payload. – This action could include something seemingly innocent, like displaying a

silly picture on a computer’s screen, or something quite malicious, such as deleting all essential files on the hard drive.

Computer Security & OS Lab, DKU

Page 10: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 10 -

Where do Viruses live?

Boot sector Take control before anything else

Memory resident Stays in memory – Rebooting system can remove the virus out

Applications, macros, data, etc.

Library routines

Compilers, debuggers, virus checker, etc. These are particularly nasty!

Computer Security & OS Lab, DKU

Page 11: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 11 -

Virus Infection Types

• Overwriting

– Destroys original code

• Pre-pending

– Keeps original code, possibly

compressed

• Infection of libraries

– Allows virus to be memory

resident

– E.g., kernel32.dll

• Macro viruses

– Infects MS Office documents

– Often installs in main document

template

Computer Security & OS Lab, DKU

original code

virus

compressedvirus

virus

original code

Page 12: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 12 -

Degrees of Complication• Viruses have various degrees of complication in how they can insert

themselves in computer code.

Computer Security & OS Lab, DKU

Page 13: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 13 -

Concealment

• Encrypted virus

– Decryption engine + encrypted body

– Randomly generate encryption key

– Detection looks for decryption engine

• Polymorphic virus (다형성)

– Encrypted virus with random variations of the decryption engine (e.g.,

padding code)

– Detection using CPU emulator

• Metamorphic virus (변성)

– Different virus bodies

– Approaches include code permutation and instruction replacement

– Challenging to detect

Computer Security & OS Lab, DKU

Page 14: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 14 -

Polymorphic Malware

The first responses of virus writers to avoid signature detection

Polymorphism loosely means: “change the appearance of”

Polymorphic malware is (usually) encrypted Data appending / Data pre-pending can be used too

New key is used each time malware propagates

The purpose of encryption is for masking

The encryption is weak (repeated XOR)

Malware body has no fixed signature

Malware must include code to decrypt itself

Signature detection searches for decrypt code

Detectable by signature-based method

Though more challenging than non-polymorphic…

That is, there are limitations:

The decrypted code is essentially the same in each case, thus memory based signature detection is possible

Computer Security & OS Lab, DKU

Page 15: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 15 -

Metamorphic Malware

“automatically recodes itself each time it propagates or is distributed”

Simple techniques include:

Adding varying lengths of NOP instructions

Permuting use registers

Adding useless instructions and loops within the code segments

Advanced techniques include:

Function reordering

Program flow modification

Static data structure modification

Reordering structures

Inserting unused data types

Computer Security & OS Lab, DKU

Page 16: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 16 -

Metamorphic Malware

Limitations

Identification of Morphing Engine

Code semantics

Behavior

Automated code identification and analysis of memory snapshots or analysis of swap space remnants

Computer Security & OS Lab, DKU

Morphing Engine ComponentsMetamorphic Structure

Page 17: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Insider Attacks, Backdoors, Logic Bombs

Computer Security & OS Lab, DKU

Page 18: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 18 -

Insider Attacks & Backdoors

Insider Attacks

An insider attack is a security breach that is caused or facilitated by someone who is a part of the very organization that controls or builds the asset that should be protected.

In the case of malware, an insider attack refers to a security hole that is created in a software system by one of its programmers.

Backdoors, Logic bombs, …

Backdoors

A backdoor, which is also sometimes called a trapdoor, is a hidden feature or command in a program that allows a user to perform actions he or she would not normally be allowed to do.

But if the hidden feature is activated, the program does something unexpected, often in violation of security policies, such as performing a privilege escalation.

Benign example: Easter Eggs in DVDs and software

Computer Security & OS Lab, DKU

Page 19: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 19 -

Logic Bombs

• A logic bomb is a program that performs a malicious action as a result of a certain logic condition.

• The classic example of a logic bomb is a programmer coding up the software for the payroll system who puts in code that makes the program crash should it ever process two consecutive payrolls without paying him.

• Another classic example combines a logic bomb with a backdoor, where a programmer puts in a logic bomb that will crash the program on a certain date (i.e., a time bomb).

Computer Security & OS Lab, DKU

Page 20: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 20 -

The Omega Engineering Logic Bomb

• An example of a logic bomb that was actually triggered and caused damage is one that programmer Tim Lloyd was convicted of using on his former employer, Omega Engineering Corporation.

• On July 31, 1996, a logic bomb was triggered on the server for Omega Engineering’s manufacturing operations, which ultimately cost the company millions of dollars in damages and led to it laying off many of its employees.

The Omega Bomb Code

The Logic Behind the Omega Engineering Time Bomb included the following strings:

• 7/30/96 - Event that triggered the bomb

• F: - Focused attention to volume F, which had critical files

• F:\LOGIN\LOGIN 12345 - Login a fictitious user, 12345 (the back door)

• CD \PUBLIC - Moves to the public folder of programs

• FIX.EXE /Y F:\*.* - Run a program, called FIX, which actually deletes everything

• PURGE F:\/ALL - Prevent recovery of the deleted files

Computer Security & OS Lab, DKU

Page 21: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 21 -

Logic Bomb / Time Bomb

In 1986 Donald Gene Burleson told employer to stop withholding taxes from his paycheck

His company refused

He planned to sue his company He used company computer to prepare legal docs

Company found out and fired him

Burleson had been working on a malware…

After being fired, his software “time bomb” deleted important company data

Company was reluctant to pursue the case

So Burleson sued company for back pay!

Then company finally sued Burleson

In 1988 Burleson fined $11,800

Took years to prosecute

Cost thousands of dollars to prosecute

Resulted in a slap on the wrist

One of the first computer crime cases

Many cases since follow a similar pattern

Companies often reluctant to prosecute

Computer Security & OS Lab, DKU

Page 22: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 22 -

Defenses against Insider Attacks

• Avoid single points of failure.

• Use code walk-throughs.

• Use archiving and reporting tools.

• Limit authority and permissions.

• Physically secure critical systems.

• Monitor employee behavior.

• Control software installations.

Computer Security & OS Lab, DKU

Page 23: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Computer Worms

Computer Security & OS Lab, DKU

Page 24: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 24 -

Computer Worms• A computer worm is a malware program that spreads copies of itself without

the need to inject itself in other programs, and usually without human interaction.

• Thus, computer worms are technically not computer viruses (since they don’t infect other programs), but some people nevertheless confuse the terms, since both spread by self-replication.

• In most cases, a computer worm will carry a malicious payload, such as deleting files or installing a backdoor.

• Early History

• First worms built in the labs of John Shock and Jon Hupp at Xerox PARC in the early 80s

• The first internet worm was the Morris Worm, written by Cornell student Robert Tappan Morris and released on November 2, 1988

Computer Security & OS Lab, DKU

Page 25: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 25 -

Worm Development

Computer Security & OS Lab, DKU

• Identify vulnerability still unpatched

• Write code for

– Exploit of vulnerability

– Generation of target list

• Random hosts on the internet

• Hosts on LAN

• Divide-and-conquer

– Installation and execution of payload

– Querying/reporting if a host is infected

• Initial deployment on botnet

• Worm template

– Generate target list

– For each host on target list

• Check if infected

• Check if vulnerable

• Infect

• Recur

• Distributed graph search

algorithm

– Forward edges: infection

– Back edges: already infected or

not vulnerable

Page 26: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 26 -

Morris Worm – 1/4

• Robert Morris – Cornell U. grad student (escape 1988.11.02)

• Find target hosts

– /etc/hosts file

– .rhost

– hosts.equiv

• at random, Gain access to machine using

– symmetry of trust (.rhost, hosts.equiv)

– common user accounts/passwords

– password guessing/cracking

– fingerd buffer overflow

– sendmail misconfiguration (DEGUG mode)

• In debug mode, the worm can send a command string, in place of dest address

• The Worm

- Led to resource exhaustion

- Adverse effect was like a so-called rabbit

Computer Security & OS Lab, DKU

Page 27: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 27 -

Morris Worm – 2/4

The worm (consists of two parts)

a 99-line bootstrap loader written in the C language, plus

a large relocatable object file that comes in VAX and Sun-3 flavors

Launch Grappling Hook (bootstrap loader)

99 lines of C code transferred and compiled on target (victim)

could run on many kinds of hosts

given one-time password used to authenticate itself to source machine

Victim even authenticated the sender!

fetch rest of worm from source machine to target -

remove traces if any errors

compile, link, load and execute

Could only run on DEC Vax/4BSD and Sun-3 machines

Computer Security & OS Lab, DKU

Page 28: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 28 -

Morris Worm – 3/4

Hide (How to remain undetected?)

use of one-time password for grappling hook

encrypt memory-resident copy

Code was encrypted when downloaded

delete all files once in memory

If transmission of the worm was interrupted, all code was deleted

Downloaded code was deleted after decrypting and compiling

change name of program periodically

change PID periodically

When running, the worm regularly changed its name and process identifier (PID)

exit before running for too long

(note: this make cracking attempts limited per worm)

Computer Security & OS Lab, DKU

Page 29: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 29 -

Morris Worm – 4/4

Essentially shut down many university computing facilities (Vax and Sun-3s) (“estimate” of 6000), cost estimation by GAO of $100,000 - $10,000,000

Ad hoc response teams formed (CMU, Purdue, etc)

First prosecution under 1986 Computer Fraud and Abuse Act

Lead to formation of CERT (Computer Emergency Response Team)

Failures of Morris Worm

Re-infected same host multiple times

Clogged up process table and network

Failures of Internet admins and users

Admins did not patch fingerd

Sendmail configuration interface was horrible

No secure login capabilities available

User passwords were pretty weak

No mechanisms in place for coordination, etc.

Computer Security & OS Lab, DKU

Page 30: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 30 -

Worm Propagation

Worms propagate by finding and infecting vulnerable hosts.

They need a way to tell if a host is vulnerable

They need a way to tell if a host is already infected.

Computer Security & OS Lab, DKU

initial infection

Page 31: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 31 -

Code Red Worm

Appeared in July 2001

Infected more than 250,000 systems in about 10 ~ 15 hours

In total, infected 750,000 out of 6,000,000 susceptible systems

To gain access to a system, exploited buffer overflow in Microsoft IIS server software

Then monitored traffic on port 80 looking for other susceptible servers

What it did

Day 1 to 19 of month: tried to spread infection

Day 20 to 27: distributed denial of service (DDOS) attack on www.whitehouse.gov

Later versions (several variants)

Included trapdoor for remote access to infected system

Rebooted to flush worm, leaving only trapdoor

Has been claimed that Code Red may have been “beta test for information warfare”

Computer Security & OS Lab, DKU

Page 32: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 32 -

SQL Slammer Worm

Infected 250,000 systems in 10 minutes!

Code Red took 15 hours to do what Slammer did in 10 minutes

At its peak, Slammer infections doubled every 8.5 seconds

Slammer spread too fast

“Burned out” available bandwidth

Computer Security & OS Lab, DKU

Why was Slammer so successful?

Worm fit in one 376 byte UDP packet

Firewalls often let small packet thru, assuming it could do no harm by itself

Then firewall monitors the connection

Expectation was that much more data would be required for an attack

Slammer defied assumptions of “experts”

Y 축: 트래픽, 초당 1백만패킷

Page 33: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Other Malware

Computer Security & OS Lab, DKU

Page 34: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 34 -

Trojan Horses – 1/3

A Trojan horse (or Trojan) is a malware program that appears to perform some useful task, but which also does something with negative consequences (e.g., launches a keylogger).

Trojan horses can be installed as part of the payload of other malware but are often installed by a user or administrator, either deliberately or accidentally.

Computer Security & OS Lab, DKU

Page 35: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 35 -

Trojan Horses – 2/3

A Trojan has unexpected function

Prototype of Trojan for the Mac

Example

File icon for freeMusic.mp3:

For a real mp3, double click on icon

iTunes opens

Music in mp3 file plays

But for freeMusic.mp3, unexpected results…

Double click on freeMusic.mp3

iTunes opens (expected)

“Wild Laugh” (probably not expected)

Message box (unexpected)

A wolf in sheep’s clothing

Computer Security & OS Lab, DKU

Page 36: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 36 -

Trojan Horses – 3/3

How does freeMusic.mp3 trojan work?

This “mp3” is an application, not data!

This Trojan is harmless,

but… Trojan could have done anything user can do delete files, download files, launch apps, etc.

Computer Security & OS Lab, DKU

Page 37: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 37 -

Definitions of Terms in 2003

[source] Separating Trojan Horses, Viruses, and Worms - A Proposed Taxonomy of Software Weapons, Martin Karresand

Virus:

A self-replicating program.

Some definitions also add the constraint that it has to attach itself to a host program to be able to replicate

Worm:

Also a self-replication program, which does not need another program to be able to replicate but instead is a stand alone program.

The difference between a worm and a virus is often said to be the way they replicate, worms replicate over network connections, while viruses replicate on the host computer.

Trojan horse:

A program performing for the user unknown and unwanted actions, while at the same time posing as a legitimate program.

Computer Security & OS Lab, DKU

Page 38: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 38 -

Definitions of Terms in 2003

[source] Separating Trojan Horses, Viruses, and Worms - A Proposed Taxonomy of Software Weapons, Martin Karresand

Computer Security & OS Lab, DKU

Page 39: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 39 -

Rootkits

• A rootkit modifies the operating system to hide its existence

– E.g., modifies file system exploration utilities

– Hard to detect using software that relies on the OS itself

• RootkitRevealer

– By Bryce Cogswell and Mark Russinovich (Sysinternals)

– Two scans of file system

– High-level scan using the Windows API

– Raw scan using disk access methods

– Discrepancy reveals presence of rootkit

– Could be defeated by rootkit that intercepts and modifies results of raw scan

operations

Computer Security & OS Lab, DKU

Page 40: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 40 -

Malware Zombies Malware can turn a computer in to a zombie, which is a machine that is

controlled externally to perform malicious attacks, usually as a part of a botnet.

Computer Security & OS Lab, DKU

Botnet Controller (Attacker)

Victim

Botnet:

Attack Commands

Attack Actions

Page 41: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 41 -

Professional Malware

Growth in professional cybercrime

and online fraud has led to demand

for professionally developed

malware

New malware is often a custom-

designed variations of known

exploits, so the malware designer

can sell different “products” to

his/her customers.

Like every product, professional

malware is subject to the laws of

supply and demand.

Recent studies put the price of a

software keystroke logger at $23 and

a botnet use at $225.Computer Security & OS Lab, DKU

Image by User:SilverStar from http://commons.wikimedia.org/wiki/File:Supply-demand-equilibrium.svg used by permission under the Creative Commons Attribution ShareAlike 3.0 License

Page 42: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 42 -

Adware

Computer Security & OS Lab, DKU

Adware software payload

Adware engine infects a user’s computer

Computer user

Adware agent

Adware engine requests advertisementsfrom adware agent

Advertisers

Advertisers contract withadware agent for content

Adware agent deliversad content to user

Page 43: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 43 -

Spyware

Computer Security & OS Lab, DKU

Spyware software payload

1. Spyware engine infects a user’s computer.

Computer user

Spyware data collection agent

2. Spyware process collectskeystrokes, passwords, and screen captures.

3. Spyware processperiodically sendscollected data tospyware data collectionagent.

Page 44: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Malware Detection

Malware Countermeasures

Computer Security & OS Lab, DKU

Page 45: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 45 -

Malware Detection

Three common methods

Signature detection 흔적탐지 (패턴매칭)

Change detection 변경탐지

Anomaly detection 비정상탐지 (이상탐지)

For signature detection & change detection, cryptographic hash functions can be used

We’ll briefly discuss each of these

And consider advantages and disadvantages of each

Computer Security & OS Lab, DKU

Page 46: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 46 -

Signatures: A Malware Countermeasure

• Scan compare the analyzed object with a database of signatures

• A signature is a string of bits found in software (or could be a hash value)

• A signature is a virus (etc.) fingerprint

– E.g., a string with a sequence of instructions specific for each virus (etc.)

– Different from a digital signature

• A file is infected if there is a signature inside its code

– Fast pattern matching techniques to search for signatures

• All the signatures together create the malware database that usually is proprietary

Suppose that a virus has signature 0x23956a58bd910345

We can search for this signature in all files

Computer Security & OS Lab, DKU

Page 47: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 47 -

Malware Signature

Brain Virus

The virus marks the six disk sectors as faulty, so that OS may not use them.

Signature: in 5th and 6th bytes of the file, it stores 1234 ( HEX ).

Action : with every disk read, it examines the file for its signature.

If it is not there, it infects the file.

Signature of another malware

Computer Security & OS Lab, DKU

Page 48: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 48 -

Malware Signature

Example of simple code obfuscation

Computer Security & OS Lab, DKU

Page 49: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 49 -

Signatures Database

• Common Malware Enumeration (CME)

– aims to provide unique, common identifiers to new virus threats

– Hosted by MITRE

– http://cme.mitre.org/data/list.html

• Digital Immune System (DIS)

– Automatically create new signatures

Computer Security & OS Lab, DKU

Page 50: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 50 -

Signature Detection

If we find the signature, are we sure we’ve found the virus?

No, same signature could appear in other files

But at random, chance is very small: 1/264

Software is not random, so probability is higher

Advantages Effective on “traditional” malware

Minimal burden for users/administrators

Disadvantages Signature file can be large (10,000’s)…

…making scanning slow

Signature files must be kept up to date

Cannot detect unknown viruses

Cannot detect some new types of malware

By far the most popular detection method!

Computer Security & OS Lab, DKU

Page 51: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 51 -

Hash Values Comparison Example

Computer Security & OS Lab, DKU

Source:

Check against contamination of critical information

http://i.cs.hku.hk/cisc/projects/va/contam_index.html

Page 52: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 52 -

Change Detection

Viruses must live somewhere on system

If we detect that a file has changed, it may be infected

How to detect changes? Hash files and (securely) store hash values

Re-compute hashes and compare

If hash value changes, file might be infected

Advantages

Virtually no false negatives

Can even detect previously unknown malware

Disadvantages

Many files change and often

Many false alarms (false positives)

Heavy burden on users/administrators

If suspicious change detected, then what?

Might still need signature-based system

Computer Security & OS Lab, DKU

Page 53: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 53 -

Tripwire Maintain database of cryptographic hashes for

Operating system files

Popular applications

Known infected files

Compute hash of each file

Look up into database

Need to protect the integrity of the database

Computer Security & OS Lab, DKU

Page 54: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 54 -

Anomaly Detection – 1/3

Monitor system for anything “unusual” or “virus-like” or potentially malicious

What is unusual?

Files change in some unusual way

System misbehaves in some way

Unusual network activity

Unusual file access, etc.

But must first define “normal”

And normal can change!

Computer Security & OS Lab, DKU

Page 55: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 55 -

Anomaly Detection – 2/3

Steps

• Feature selection – what to

measure

• Feature reduction – space too big

• Training – what is “normal”?

• Training – what is known bad?

• Classification – separate bad from

good

• Anomaly detection – NOT known

to be good

Computer Security & OS Lab, DKU

Issues

• Feature space is too big

• Getting clean training sets

• What is normal changes over time

• False positive rate

Behavior ofknown attacks

Normalobserved

Behavior space

unobserved

Page 56: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 56 -

Anomaly Detection – 3/3

Advantages Chance of detecting unknown malware

Disadvantages Unproven in practice

Attacker can make anomaly look normal

Must be combined with another method (such as signature detection)

Also popular in intrusion detection (IDS)

A difficult unsolved (unsolvable?) problem! As difficult as AI?

Computer Security & OS Lab, DKU

Page 57: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 57 -

Quiz

What are the similarities and differences between virus and worm?

What kinds of executable file formats are there?

MS Windows: ?, Linux: ?, Android: ?, iOS: ?

What is malware analysis?

What is static analysis technique for malware detection?

What is dynamic analysis technique for malware detection?

What are the main differences between static analysis and dynamic analysis?

How do you create a unique signature for malware detection?

Is it OK to run a malicious code on bare machine?

What are the common evasive techniques used by malware to avoid anti-malware detection?

What do you do to be a malware analyst?Computer Security & OS Lab, DKU

Page 58: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 58 -

Heuristic Analysis

• Useful to identify new and “zero day” malware

• Code analysis

– Based on the instructions, the antivirus can determine whether or not the program is

malicious, i.e., program contains instruction to delete system files,

• Execution emulation

– Run code in isolated emulation environment

– Monitor actions that target file takes

– If the actions are harmful, mark as virus

• Heuristic methods can trigger false alarms

Computer Security & OS Lab, DKU

Page 59: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 59 -

Static vs. Dynamic Analysis

Computer Security & OS Lab, DKU

Static Analysis

• Checks the code without trying to execute it

• Quick scan in white list

• Filtering: scan with different antivirus and check if they return same result with different name

• Weeding: remove the correct part of files as junk to better identify the virus

• Code analysis: check binary code to understand if it is an executable, e.g., PE (www.heaventools.com)

• Disassembling: check if the byte code shows something unusual

Dynamic Analysis

• Check the execution of codes inside a

virtual sandbox

• Monitor

– File changes

– Registry changes

– Processes and threads

– Networks ports

Page 60: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Anti-Malware SW

Anti-Virus SW (AV SW):

V3, ALYac, ViRobot, Kaspersky, Norton, …

Computer Security & OS Lab, DKU

Page 61: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 61 -

Anti-malware

What is anti-malware?

How does anti-malware work?

Why should we know types of malware?

Why should anti-virus program distinguish the unique types of malware from each other?

Why is malware family classification important?

Computer Security & OS Lab, DKU

Page 62: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 62 -

Shield vs. On-demand

Computer Security & OS Lab, DKU

• Shield– Background process

(service/daemon)

– Scans each time a file is touched (open, copy, execute, etc.)

On-demand• Scan on explicit user request

or according to regular schedule

• On a suspicious file, directory, drive, etc.

Performance test of scan techniqueso Comparative: check the number of already known viruses that are

found and the time to perform the scan

o Retrospective: test the proactive detection of the scanner for unknown

viruses, to verify which vendor uses better heuristics

Anti-viruses are ranked using both parameters:http://www.av-comparatives.org/

Page 63: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 63 -

Online vs Offline Anti Virus Software

Computer Security & OS Lab, DKU

Online

• Free browser plug-in

• Authentication through third

party certificate (i.e. VeriSign)

• No shielding

• Software and signatures update

at each scan

• Poorly configurable

• Scan needs internet connection

• Report collected by the company

that offers the service

Offline

• Paid annual subscription

• Installed on the OS

• Software distributed securely by

the vendor online or a retailer

• System shielding

• Scheduled software and

signatures updates

• Easily configurable

• Scan without internet connection

• Report collected locally and may

be sent to vendor

Page 64: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 64 -

Machine Learning for Malware Detection (Kaspersky)

Computer Security & OS Lab, DKU

Page 65: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 65 -

The Best Antivirus Protection of 2018- https://www.pcmag.com/article2/0,2817,2372364,00.asp

Computer Security & OS Lab, DKU

Page 66: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 66 -

Detection Performance

Computer Security & OS Lab, DKU

Source: https://www.pcmag.com/article2/0,2817,2481367,00.asphttps://en.wikipedia.org/wiki/F1_score

Page 67: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 67 -

Open-Source Antivirus / Free Antivirus

ClamAV

OpenAV

Armadito AV

Computer Security & OS Lab, DKU

The Best Free Antivirus Protection of 2018 by PC magazine

VirusTotal Analyze suspicious files and URL to detect types of malware

https://www.virustotal.com/#/home/upload

YARA The pattern matching swiss knife for malware researchers

https://virustotal.github.io/yara/

YaraRules Project: http://yararules.com/

Page 68: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

Summary

• Numerous attacks involve software

• Can you ever trust software?

Computer Security & OS Lab, DKU

Page 69: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 69 -

Trusting Software

Can you ever trust software? See Reflections on Trusting Trust

Consider the following thought experiment

Suppose C compiler has a virus When compiling login program, virus creates backdoor (account with known

password)

When recompiling the C compiler, virus incorporates itself into new C compiler

Difficult to get rid of this virus!

Suppose you notice something is wrong

So you start over from scratch

First, you recompile the C compiler

Then you recompile the OS Including login program…

You have not gotten rid of the problem!

In the real world Attackers try to hide viruses in virus scanner

Imagine damage that would be done by attack on virus signature updates

Computer Security & OS Lab, DKU

Page 70: Introduction to Software Security Malwaresecuresw.dankook.ac.kr › ISS19-1 › ISS_2019_06_malware.pdf · 2019-04-02 · N. Vlajic, CSE 3482: Introduction to Computer Security, Yorku

- 70 -

Summary, Q & A

Malware

Malware family classification

Malware Detection

Signature/Anomaly-based Detection

Anti-malware software, Vaccine = Anti virus,

Most Common Evasive Techniques used by Malware

Antivirus evasion technique,

50 Open Source Tools to Replace Popular Security Software https://www.datamation.com/osrc/article.php/3882711/50-Open-Source-Tools-To-Replace-Popular-Security-Software.htm

Computer Security & OS Lab, DKU

• What types of threat are related to this lecture?

‒ Consider the STRIDE model !