testing machine learning, learning machine testing. eurostar 2016 rik marselis

Testing Machine Learning,

Learning Machine Testing

Rik Marselis, Sogeti Nederland

rik.marselis@sogeti.com

1980

2007 2012 20122008

Workgroup-member

20142009

Rik Marselis@rikmarselis

Who uses a robot today?

You will use multiple within just a few years!!

And please realise: these are robots too!

Chatbot Self-driving mini-bus

WePod: self-driving bus

This minibus is allowed on the road now in Wageningen (the Netherlands) andwas tested by mySogeti-colleaguesDragan, Hugo andRemco.

Robots, ok, but in less than 10 years????

Well;

Did you understand this

trafficsign 10 years ago?

That’s how fast technology

progresses!!

What is Robotics?

• Internet of Things (IoT)

• Big data analytics

• Cloud technology

• Machine learning

• Artificial Intelligence

Robotics

Definition: A robot is a machine that gathers information about its environment by input of sensors and based on this input changes its behavior. Combined with machine learning and artificial intelligence the robot’s reactions over time get more and more adequate. The use of Internet of Things, Bid Data Analytics and Cloud technology make a robot versatile. Robots may appear in a physical shape (e.g. a tin man) or non-physical (e.g. chatbot).

Traditional: Robots for dull & dangerous work

New possibilities, just 2 examples

Exo-skeleton

Healthcare disabled peopleTransport drone delivery

New risks

The bull doesn’t likethe manure robot andconstantly pushes it off its track. (to much frustration of the farmer who needs to cometo rescue the robot ;-)

New angles of quality, aimed at roboticsMechanical

Electrical

Information Processing

Machine Intelligence

Business Impact

Social Impact

e.g. Ethics

e.g. Embodiment

e.g. Empathy

Part 1:

Testing machine learning

Machine Learning in the 80’s

Machine learning today: demonstrating

Machine learning today: demonstrating

Learning by creating rules

How can we find the rules that self-drivingcars must use?

Ask you!

moralmachine.mit.edu

Some

results

Machine learning or machine training?

• A learning machine consists of a complex network

of algorithms

• The learning machine has very many “dials and

switches”

• Making sure the machine learns the right things is

just like ordinary training (well, in a different way…)

Testing Machine Intelligence

• Human: “Dear robot please get me a softdrink”

• Robot: “Dear human here is your cola”

• Is this the correct output?

• No!

In Sweden I want:

Testing Machine Intelligence

• Human: “Dear robot please get me a softdrink”

• Robot: “Dear human it’s Saturday

evening now, so here is your beer”

• Is this the correct output?

• After studying your behavior,

the robot knows your patterns

Testing traditional IT vs. machine intelligence

Traditional (example):

• 1 + 1 = ?

• There’s only one correct output

With Machine Intelligence (example):

• Prepare a meal

• When is the meal “good enough”?

• Is it always the same meal?

2

Two sides of testing machine intelligence

• Testing the output

This we know from traditional testing,

but it’s hard to predict outputs, the

result may be different every time.

• Testing the input

If the machine gets the right input

it will learn and do the right things.

To control the results of AI:

control the inputs…

Helpful chatbots?

• There’s a very thin line between a robot that is not helpful,

a robot that is helpful and a robot that’s too helpful

Helpful chatbots?

The uncanny valley:

What is the role of the tester?

• The tester must see whether the intelligent

machine will maintain the intended behavior in live

operation

• In a way the tester is the psychologist of the

intelligent machine

• A tester must ask even more questions than

before !!

Note: I picked up the idea of the tester being the psychologist of robots from James Bach.

It’s no longer about right or wrong

• Testing intelligent machines is not an easy check

whether the result is right or wrong

• It becomes an appraisal of a specific response in a

certain situation with a certain amount of

knowledge for a certain audience

• If these parameters change the right behavior will

also change

After finishing the test…

• The machine continues to learn and adapt…

• What about confidence?

• The test must result in the confidence that the

machine will learn and adapt in the right way…

Part 2:

Learning machine testing

Using a robot for dull or dangerous testing

• Mechanical tests use robot-arm

Using a robot for intelligent testing

• Functional tests use machine intelligence to

– create test cases and

– execute test cases

– and (important!!) to judge the results

Using a robot for testing physical objects

Using a robot for intelligent testing

• Non-functional tests use machine intelligence

For example intelligent performance testing

(not just simple transactions but a normal variety)

People and Robots testing together

• Teamwork of person & robot

• Each does what it is best at

– Robot repetitive work

– Human creative work

Machine learning: unsupervised & supervised

Deep learning:

Robot learns at various

layers. Lower layers

are unsupervised.

Highest layer(s) is (are)

supervised.

Machine Intelligence generates test data

• Based on Big Data Analytics of live data

• This is so-called “weak intelligence”

The machine is only intelligent in a limited field

Machine Intelligence generates test cases

• Evolutionary algorithms

• Brute force testing

The machine is able to generate and execute

many many more test cases than a human tester

Conclusion

Testing machine learning

• Testing inputs is very important because the output

is not predictable

• The objective of testing is to gain confidence that

the machine will learn and develop itself in a right

way

Learning machine testing

• Test automation will go to the next level

• Use machine intelligence (e.g. big data analytics)

to create your optimal test data

• Use machine intelligence to create many more test

cases, execute them and analyze results

Conclusion…

As a tester better prepare yourself for:

• Testing of Robots

and

• Testing with Robots

Thanks for attending my talk.

Any questions ???????????

Rik Marselis, Sogeti Nederland

rik.marselis@sogeti.com