“Big” data (analytics) and Cyber-security

จักรพงศ์ นาทวิชัย

มหาวิทยาลัยเชียงใหม่

3

Objectives

• Data systems in brief: from transactional data to “big” data (analytics)

• Linkage of “data” with the criteria

• The brief concepts of cyber-security

4

4.1a(1)

• HOW do you track data and information on daily operations and overall organizational PERFORMANCE?

- MtBnB service, has a platform to manage their own six apartments in Chiang Mai area. Some are in condominium (of cause, violent the regulation!), some are house.

5

4.1a(1)

Booking

ApartmentID GuestID BookingID StatusCode StartDate EndDate PerNight Insurance

1 5 1 3 6/10/2018 10/10/2018$100 $20

2 1 2 3 1/9/2018 30/10/2018$300 $100

1 3 3 4 3/10/2018 6/10/2018$100 $20

3 1 4 1 30/10/2018 5/11/2018$105 $20

6 2 5 2 15/10/2018 17/10/2018$120 $30

Apartment

ApartmentID ApartmentType BuildingID Room Number BedroomNo BathroomNo Note

1 1 1 1801 1 1 -

2 2 2 3 2 No pet allows

3 1 1 1802 1 1 Smart lock with biometric security

5 3 3 2 2 No pet allows

6 4 5 - 3 2 Two-storeys house

Apartment Type

ApartmentTypeCode Description

1 Studio

2 Penthouse

3 Two-bedroom

4 Full-house

ApartmentFacility

ApartmentID FacilityID

1 1

1 4

1 6

3 1

3 4

3 6

2 7

2 1

2 3

2 5

2 6

5 6

6 5

Building

BuildingID Name Manager Address

1 New Deak Mor Jeff 239 Huay Kaew Rd

2 The Coloniel David Nimman Soi 3

3 Avenue 25 Pual 25 Suthep Rd

5 - David 10/23 Suthep Rd

Facility

FacilityID Description

1 LED TV

2 TV

3 Sofa

4 Free Internet

5 Kitchen

6 Aircondition

7 Cable TV

Guest

GuestID Gender Firstname Lastname DoB

1 M Michael Young 1/5/1965

2 M Charlie Wu 28/10/1980

3 F Astrid Leong 10/11/1981

5 F Rachel Chu 18/1/1985

BookingStatus

StatusCode Description

1 Issue

2 Confirm

3 Stay

4 Leaved

6

4.1a(1)

• ลูกค้าคนใดบ้างที่จะ Checkout จาก Apartment หลังจากวันท่ี 28 October 2018 ให้ระบุช่ือสกุล?

• Astrid Leong เคยพักท่ี Avenue 25 หรือไม่?• ถ้าต้องจ้างแม่บ้านดูแล Apartment ที่ The Coloniel โดยจ่ายเหมา 2

เดือน $400 ท่านยอมจ่ายหรือไม?่• หากต้องการรายไดเ้ฉลี่ยจากแต่ละ Apartment ท าได้หรือไม?่ รายได้

เฉลี่ยแต่ละ Apartment แยกตามเพศผู้เข้าพักท าได้หรือไม่? แยกตามรายได้ท าได้หรือไม่?

7

4.1a(1)

• HOW do you track data and information on daily operations and overall organizational PERFORMANCE?

• ถ้าองค์กรมีระบบท่ีเก็บข้อมูลตามท่ีแสดง ท่านให้ 4.1a(1) Band ใด?– Daily operation “ตอบโจทย์” หรือไม?่ ท าอะไรได้บ้าง?

– Overall organizational PERFORMANCE ท าได้หรือไม?่

8

4.1a(1)

• From rental database, if we add one more guest, how many table is affected?

• What if a new guest register? Then book an apartment?

• Check-out?

• Any meaningful data touching wrt. a business activity is called a transaction. <OLTP Online Transaction Processing>

9

4.1a(1)

• Data Systems: “On the most fundamental level, a database needs to do two things: when you give it some data, it should store the data, and when you ask it again later, it should give the data back to you” – Martin Kleppmann

10

4.1a(1)• Linkedin data• Arrows represent

“relation” between data

• Relational databases E. F. Codd in 1970

• SQL: SELECT first_name FROM users table WHERE user_id=‘251’

11

4.1a(2)

• 4.1a(2) Comparative Data HOW do you select comparative data and information to support fact-based decision making?– หากจะตัดสินใจลงทุนท า Apartment เพิ่ม ส าหรับลูกค้าที่มีก าลงัจ่ายสงู

– ถ้ายอมลงทุนซื้อฐานข้อมูลของคู่แข่งใน Segment ดังกล่าวมาใช้ร่วมกับฐานข้อมูลขององค์กร?

William H. Inmon, 1992

12

4.1a(2)

• 4.1a(2) Comparative Data HOW do you select comparative data and information to support fact-based decision making?

S.No

Product Name

Category Product Cost

1 25 Avenue Full-house xxxx

S.No

Product Name

Category Product Cost

7 Empire Studio room Xxxx

9 Noble House Xxxx

ETL (Data Cleaning and Integration)

Data Warehousing

<OLAP Online

Analytic

Processing>

William H.

Inmon, 1992

13

4.1a(3)

• 4.1a(3) Measurement Agility HOW do you ensure that your PERFORMANCE measurement system can respond to rapid or unexpected organizational or external changes and provide timely data?

William H. Inmon, 1992

14

4.1a(3)

• Note: Agility in your measurement system might be needed in responseto regulatory changes, other changes in the political environment, innovations in organizational processes or business models, new competitor offerings, or productivity enhancements. Responses to such changes might involve, for example, adopting different performance measures or adjusting the intervals between measurements.– จากตัวอย่าง รัฐบาลมีนโยบาย Vat refund ส าหรับค่าที่พักคืนแรกส าหรับชาวต่างชาติ?

อยากเห็นตัววัดใน Dashboard ที่แสดงรายได้ที่มากขึ้นจากนโยบายนี้แบบ Quarter-over-quarter?

– คิดว่า IT จะหายไปนานไหม?– Data platform technology เช่น Data-lake, visualization tools อาจตอบโจทย์

15

Tableau

16

4.2b(1)

• 4.2b(1) Knowledge Management HOW do you build and manage organizational knowledge? HOW do you– collect and transfer WORKFORCE knowledge;– blend and correlate data from different sources to build new

knowledge;– transfer relevant knowledge from and to CUSTOMERS,

suppliers, PARTNERS, and COLLABORATORS; and– assemble and transfer relevant knowledge for use in your

INNOVATION and strategic planning PROCESSES?

17

4.2b(1)

• Note4.2b(1). Blending and correlating data from different sources may involve handling big data sets and disparate types of data and information, such as data tables, video, and text. Furthermore, organizational knowledge constructed from these data may be speculative and may reveal sensitive information about organizations or individuals that must be protected from use for any other purposes.

18

4.2b(1)

• In 2000, a sale guy in Walmart tried to boost the sales by bundle (at least) two products and offered discount. – Jam and Bread?

https://www.forbes.com/sites/bernardmarr/2017/01/23/really-big-data-at-walmart-real-time-insights-from-their-40-petabyte-data-cloud/#24aa3c476c10

19

4.2b(1)

• Diapers are probably too heavy for recently pregnant women so they ask their husbands to pick them up coming home from work and since hubby is off the clock and ready to get his drink on, he also picks up beer.

• A diaper emergency occurs fairly late in the evening and the husband is sent out while the new mother cares for the baby. Being annoyed, he also picks up a 12 pack to relax.

20

4.2b(1)

• Walmart – the world’s biggest retailer with over 20,000 stores in 28 countries, is in the process of building the world’ biggest private cloud, to process 2.5 petabytes of data every hour.

https://www.forbes.com/sites/bernardmarr/2017/01/23/really-big-data-at-walmart-real-time-insights-from-their-40-petabyte-data-cloud/#24aa3c476c10

3 from 5 transactions

21

4.2b(1)

• ถ้าจะหาว่ามีสินค้าคู่ไหนที่มีคนซื้อพร้อมกันเยอะๆ ยากไหม?

22

4.2b(1) - “Big” Data definition

• According to Gartner (2001), the data is not just scaling but also more complex.

• The ‘3V’: Volume, Velocity, Variety, of Big Data–Market Basket Analysis -> Big Volume ขนาดใหญ่มาก ค านวณ

ซับซ้อน

23

4.2b(1)

• Tweet stream

“Today, Twitter generates 500 million tweets/day, each about 3 kilobytes including metadata. While this figure is beginning to plateau, a projected logarithmic growth rate would suggest a 2.4-fold growth by 2025, to 1.2 billion tweets per day, 1.36 petabytes/year.” - Stephens, D.Z et al Big Data: Astronomical or Genomical?, PLoS Biol. 2015

24

NLP Natural Language Processing 101

25

Example Text

• “Xiaomi Community is a place for Xiaomi Fans to ask and answer questions, discuss Xiaomi products and get the latest news from Xiaomi and its ecosystem partners. The website is your daily source of information on all the things that happen within Xiaomi.

26

Tokenization

• “A process to divide a sequence of entities of a written language into entities”

• It can be applied with both articles and sentences.

• Article ===> sentences

• Sentence ===> words

27

Tokenization

28

Tokenization

Is it different from “thing”?

29

Text Lemmatization and Stemming

• Stemming “A process to reduce into their root form by cutting off the end or the beginner of the word.”

• The goal of the stemming is to by removing the prefix of suffix of the word.

• The result is not guarantee to match with the root form.

• Different algorithms produce a different result.

30

Text Lemmatization and Stemming

• Stemming

Studying

Studies

Banks

Study

Studi

Bank

31

Text Lemmatization and Stemming

• Lemmatization

Is, am, are, been,was, were

Have, had, has

be

have

Car, cars, car’s car

32

Bag of words

• Machine learning can not work with the string.

– It works with numerical data.

• To apply the machine learning with the string, we need to convert the string into computable data.

• Bag-of-word is a representation of a string based on frequency of entities.

33

Bag of wordsJohn buys a cat.Jane gets a dog.James has a dog.

John Jane James Friends Buy Buys Gets Has Cat Dog

1 1 1

1 1 1

1 1 1

1 1 1 1

Friends buy a cat and a dog.

34

Bag of words

• We can use lemmatization and stemming.

• We can group similar words together.

Human Have Cat Dog

1 1 1

1 1 1

1 1 1

1 1 1 1

35

Cosine Similarity

𝜃

𝐴

𝐵

• A and B denotes two vectors.

𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦 = cos 𝜃 =𝐴 ∙ 𝐵

𝐴 𝐵

36

Cosine Similarity

doc 0 : "John buys a cat.",doc 1 : "Jane gets a dog.",doc 2 : "James has a dog.",doc 3 : "Friends buy a cat and a dog."

doc 0 doc 1 doc 2 doc 3

doc 0 0 0 0.71

doc 1 1 0.71

doc 2 0.71

doc 3

37

Insight finding from customer tweets

• องค์กรเคลมว่าประเมินความพึงพอใจ ความไม่พึงพอใจจาก Twitter?• หา Tweet ที่มีช่ือบริษัทของเรา ยากไหม?• เอาเฉพาะที่บ่นการบริการ {แย่, ช้า, ไม่ดี, ....} ยากไหม?

– “Today, Twitter generates 500 million tweets/day, each about 3 kilobytes including metadata. While this figure is beginning to plateau, a projected logarithmic growth rate would suggest a 2.4-fold growth by 2025, to 1.2 billion tweets per day, 1.36 petabytes/year.” - Stephens, D.Z et al Big Data: Astronomical or Genomical?, PLoS Biol. 2015

38

The Big ‘Velocity’

• Drinking from the firehose!

39

4.2b(1)

• 4.2b(1). Blending and correlating data from differentsources may involve handling big data sets and disparate types of data and information, such as data tables, video, and text. Furthermore, organizational knowledge constructed from these data may be speculative and may reveal sensitive information about organizations or individuals that must be protected from use for any other purposes.

40

Big Variety

41

Big Variety

• ก ากับพนักงานให้มีพฤติกรรมท่ีเหมาะสม

• สังเกตพฤติกรรมลูกค้าในร้านค้า

• ปรับปรุง Productivity ของ Workforce

42

Cyber Security in Brief

• Phishing - is the fraudulent attempt to obtain sensitive information such as usernames, passwords and credit card details by disguising oneself as a trustworthy entity.– จัดการอย่างไร?

• Malware - is any software intentionally designed to cause damage to a computer, server, client, or computer network.– เกี่ยวกับ System/Information Availability จัดการอย่างไร?

ประสิทธิผลวัดอย่างไร?

43

Cyber Security in Brief

• Cyberattack is any attempt to expose, alter, disable, destroy, steal or gain unauthorized access to or make unauthorized use of an asset.– Keystroke logging: record key struck on a keyboard

– Denial-of-service attack

– Buffer overflow

44

Cyber Security in Brief

• Firewall

45

Cyber Security in Brief• Pen-test (Penetration test) is an authorized simulated

cyberattack on a computer system, performed to evaluate the security of the system.

• https://sth.sh

46

Cyber Security in Brief – Pentest phase1. Reconnaissance - The act of gathering important information on a target

system.

2. Scanning - Uses technical tools to further the attacker's knowledge of the system. For example, Nmap can be used to scan for open ports.

3. Gaining Access - Using the data gathered in the reconnaissance and scanning phases, the attacker can use a payload to exploit the system.

4. Maintaining Access - Maintaining access requires taking the steps involved in being able to be persistently within the target environment in order to gather as much data as possible.

5. Covering Tracks - The attacker must clear any trace of compromising the victim system, any type of data gathered, log events, in order to remain anonymous.

47

Cyber Security in Brief – GDPRGeneral Data Protection Regulation

48

Cyber Security in Brief - GDPRThe UK’s data watchdog has announced plans to fine the airline British Airways a record £183 million over last year’s data breach. The Information Commissioner’s Office (ICO) said that “poor security arrangements” at the company lead to the breach of credit card information, names, addresses, travel booking details, and logins for around 500,000 customers.