22nd Annual SIOPSA Conference –Leading through innovation:

Navigating the new world of work

4 – 6 August 2020The Canvas, Riversands, Fourways

This years themes

• Teams and culture

• Neuroscience of innovation

• Medico-legal innovation

• Wellness and health

• Employee experience and engagement

• Talent attraction, development and mobility

• Innovative leadership

Keynote speakers

People Assessments in Industry (PAI)

Technological Advances in Testing & Associated Practitioner Implications

2020 ACSG Conference

Discussion Points

• About People Assessments in Industry (PAI)

• Current SA legislation and its implications

• ASSA

• Evolution of the World of Work

• Implications on Assessments

• Big Data – LinkedIn and Facebook

• Virtual Reality and Assessments

• The use of Artificial Intelligence in Assessments

• Gamification

• Robotics

• Implication for Assessment Practitioners

• Questions

About People Assessments in Industry (PAI)

PAI background

• Established in 1998 to proactively engage various stakeholders interested

in psychological assessment in South African industry.

• Functions as an interest group of the Society for Industrial and

Organisational Psychology in South Africa (SIOPSA) since 2005.

• Purpose of PAI is to ensure testing exists in the South African context as a

value adding, ethical and fair practice.

PAI acts as an independent body but seeks to build relationships and tocollaborate with such bodies as the:• Health Professions Council of South Africa (HPCSA),• Association of Test Publishers (ATP),• Psychological Society of South Africa (PsySSA)• Assessment Centre Study Group of South Africa (ACSG),• The International Test Commission (ITC),• Business SA, test users and test takers who have similar interests with

regards to assessments in the workplace.

PAI Background

• David Bischof – (Organisational Management Technology - Evalex)• Jaco de Jager (Top Talent Solutions)• Nadene Venter (Independent Consultant)• Dr Paul Vorster (Researcher)• Kevin Distiller (Organisational Management Technology - Odyssey)• Ntswaki Raboroko (Head Talent Acquisition - Nedbank)

PAI Committee

Current SA Legislation and its implications

According to the Health Professions Act (no. 56 of 1974),instruments that measure psychological constructs must be used, interpreted and controlled by psychologists. Furthermore, only individuals registered with the Professional Board of Psychology (HPCSA) may use psychological tests.

Psychological Tests - Chapter 5, point 55 in particular states that -A psychologist shall ensure that tests used have been classified by the board and that the provisions of any applicable legislation, such as the Employment Equity Act, 1998 (Act No. 55 of 1998), have been complied with.

What does the Legislation say

Employers and/or psychologists and psychometrists need to comply with the provisions of the EEA act section 8 (a), (b) and (c) which state that the use of Psychological testing and other similar assessments of an employee are prohibited unless the test or assessment being used-

Has been scientifically shown to be valid and reliable; Can be applied fairly to all employees Is not biased against any employee or group;

The above prohibition holds true for all testing of employees irrespective if it is a Psychological test, or not, being used. All tests measuring a Psychological construct must be registered with the HPCSA and can only be used by a person registered with the HPCSA.

What does the Legislation say

• To classify any device, instrument, questionnaire, apparatus, method, technique or test aimed at theevaluation of emotional, behavioural and cognitive processes or adjustment of personality of individuals orgroups of persons, or for the determination of intellectual abilities, psychopathology, personality make-up,personality functioning, aptitude or interests by the usage and interpretation of questionnaires, testsprojections or other techniques or any apparatus, whether of SA origin or imported, and to report thereonto the Professional Board.

• The Annual publication of a list of psychological tests/psychometric instruments classified by theProfessional Board

• Develop training guidelines/standards related to psychometrics and psychological assessment that caninform and be used in the accreditation of qualifications, universities and internship programmes, whensetting the national Board examinations, and for continuing professional development purposes

• Develop guidelines for ethical practice related to test use and psychological assessment and how to assesswhether a psychological test meets the required standards

• Develop minimum requirements/standards for psychological tests• Classification will entail verifying whether a test was psychological or not. To this end, practitioners and

publishers should submit the full test manual that states the construct(s) tapped by the test, evidence ofpsychometric properties, an indication as to whether the item content was culturally appropriate. No costswould be attached to test classification by the Professional Board.

HPCSA – Revised Mandate of the Psychometrics Committee of the Professional Board for Psychology and its Implications

• Message was that the Psychometrics committee mandate was to classify tests that then goonto list;

• There is no legislative framework for HPCSA to issue certification; no mandate to certify• The reason for training guidelines is to quality assure so that practitioners are trained well

enough to evaluate if can or cannot use a test• The registration of a Psychological test with the HPCSA as published and Gazetted by the

HPCSA is not a certification of the compliance of the EEA or any other evaluative“certification”. The HPCSA is indicating that the test measures a psychological construct andas such is classified as a Psychological test that must be controlled and used by a HPCSAregistered person

• Online and Computer - Based testing - SA guidelines on computerised testing would no longerbe reviewed; HPCSA to develop a position statement instead of guidelines; positionstatement provide practitioners with board stance concerning online assessments;collaboration with all stakeholders on this position statement; position statement to guideprofession; maybe adopt international guidelines

HPCSA presentation by Dr Thandeka Moloi and Justin August at PsySSA 2019 conference

HPCSA presentation by Dr Thandeka Moloi and Justin August at PsySSA 2019 conference

Test classification process entails: test developer submits test to the board You have to be registered Psychologist/Psychometrist to submit test Test manual examined to determine psych constructs classified as psychological or not psychological test publishers no longer obliged to inform board of tests under development or

being adaptedOnce test classified, test material shipped back to publishers due to no space

available to store tests no cost for classification Classification indicates test is limited to use by psychologists - guideline for the

public and practitioners Test classification list to be published twice year on HPCSA website

• An independent non-regulatory external assessments evaluation ‘body’working collaboratively with the support of the Department of Labour and theHPCSA to assist in implementing a robust, best practice and technologyenabled process that could be used to review people assessment instrumentsand tests.

• It is envisaged that this body will look at the broad spectrum of instrumentsthat are used in South Africa and will not be limited to psychological tests only.In cases where an instrument meets the criteria for test classification as apsychological test, the applicants will be informed accordingly aboutsubmitting the test for classification to the HPCSA as per the statutoryrequirements.

ASSA Formation

The core purpose of Assessment Standards South Africa (ASSA) would be to review the quality of tests that are available for use in South Africa. Based on

the experience in other countries the voluntary submission of assessment instruments for objective evaluation and reviews will raise the general

standard and awareness of using quality tests.

Vision and Mission of ASSA

• The ASSA would provide a minimum standards focus in South Africa for all activities inrelation to tests and testing.

• Maintain a website and online platform to provide information for best practice, results oftest reviews and access to information about tests and testing

• Management and dissemination of information on standards relating to tests and testing.This includes information on qualifications in test use, tests reviews, test registration andthe publishing and dissemination of advisory statements, in addition to guidelines andother literature on standards for the construction, use and availability of tests.

• Quick evolution of testing – (AI, Gamification, Big Data) – field is evolving faster thanlegislation can keep pace with; ASSA can assist in staying on top of this andcollaborating locally and internationally to look at ways to maintain quality andadherence to EEA.

ASSA Activities

Evolution of the World of Work

1970s 1980s 2000s1990s Today2010s

Technology

Individuals

Businesses

Public policy

Rate

of

change

The gap between Technology and Public Policy - 2017 Deloitte Global Human Capital Trends

The top 10 skills that will be in demand by all employers by 2020 - World Economic Forum - 2016 report

Position Competence Questionnaires Ability / Cognitive Simulations

10 Cognitive flexibility No Yes Somewhat

9 Negotiation skills Propensity to No Yes

8 Service orientation Yes No Yes

7 Judgment No Yes Yes

6 Emotional intelligence Yes No Somewhat

5 Coordinating Propensity to No Yes

4 People management Propensity to No Yes

3 Creativity Propensity to Somewhat Yes

2 Critical thinking No Yes Yes

1 Complex problem solving No Somewhat Yes

World Economic Forum – Future of Jobs Report 2018

World Economic Forum – Jobs of Tomorrow Mapping Opportunity in the New Economy –January 2020

• Successful professions in future will increasingly require both “digital” and “human” competence and emergingprofessions reflect the continuing importance of human interaction in the new economy.

• There are seven emerging professional clusters and 96 jobs of tomorrow within them that vary in theirindividual rate of growth and in the scale of job opportunities they offer in the aggregate. These clusters are:

Data and AI Engineering and Cloud Computing

People and Culture Product Development

Sales, Marketing and Content Care Economy Green Economy

• The highest-demand skills required in these emerging professional clusters span both technical and cross-functional skills. These in-demand skills can be divided into five distinct skills clusters:

Business Skills Specialized Industry Skills General and Soft Skills Tech Baseline Skills Tech Disruptive Skills

Implications on Assessments

2017 - Robert McHenry - Psychometric Testing: Critical Perspectives – ‘The Future of Psychometric Testing’

• Prediction #1: Smartphones will replace computers for employee assessment• Prediction #2: High-quality psychometric testing services will be sold direct to

consumers• Prediction #3: Advances in neuroscience of personality will reveal which are the

most valid individual differences to measure and how best to measure them• Prediction #4: The digital movement, coupled with the use of big data and new

forms of digital CV will render many of the current applications for high-stakespsychometric testing redundant

• Prediction #5: The basis for employee development will be derived from thedata yielded by wearable devices and not from psychometric tests

• The testing industry - has been in ideational stagnation, change is slow and with minorcosmetic changes, only the numbers in the names of tests substantially change

• Next Generation Assessments are different in design, look, feel, scoring andadministration

• Innovation in technology allow for delivery online through mobile phone and tablets• Completely autonomous scoring and reporting generation is the new kid on the block• Next generation tests – use of artificial intelligence, machine learning, behavioural and

predictive modelling (Computational Modelling and Psychology) gamification, bioscreening, virtual reality, robotics, auto transcribing, language modelling and naturallanguage processing, personality assessment via behaviour-linguistic analysis and socialnetwork data (Facebook; LinkedIn and Twitter).

• Video interviews where AI software is used to analyse the candidate’s facial expression,tone and language to ascertain job fit

• Resume filters scan for keywords and phrases on resumes• Chatbots schedule interviews and algorithms help predict a job candidate’s future success

The next generation of Assessments

The rise of Digital Psychometrics

Big Data - Facebook

Big Data - Facebook

• LinkedIn used as ‘screening tool’ by hiring managers• Provides for Skills Assessments: short, multiple-choice tests that users can take to verify

their knowledge in areas like computer languages, software packages and other work-related skills.

• Research conducted by Julia Levashina, Ph.D., associate professor of management at KentState University, along with her colleague Nicolas Roulin, associate professor ofindustrial/organizational psychology at Saint Mary's University in Canada

• LinkedIn profiles could be used to make inferences about job applicants’ visible skills (suchas leadership, communication and planning), personality traits (such as extraversion) andcognitive ability. However, hiring managers are not recommended to use LinkedIn profiles toassess less visible skills (such as conflict management or adaptability) and personality traits(such as conscientiousness and emotional stability).

• The researchers also demonstrate that hiring recommendations based on LinkedIn profilesare influenced by the job applicant’s LinkedIn profile length, photo and number ofconnections. It was found that other features of LinkedIn profiles (such as listing more skills,collecting endorsements, joining groups or describing volunteering involvements) had anegligible impact on hiring recommendations.

Big Data - LinkedIn

• Virtual and augmented reality (VR and AR) use biometric tracking data - micro-movements ofhead, torso, hands, and eyes - on the job training

• Can diagnose or predict anxiety, depression, schizophrenia, addiction, ADHD, autism spectrumdisorder and more about a person’s cognitive and physical function.

• What if this medically-relevant data is fed into users’ psychometric profiles?• Profiles may start out as relatively harmless, merely predicting when someone might be getting

ready to buy a new car.• However, sprawling psychographic profiles with medical inputs could leave people vulnerable• Anonymizing VR and AR tracking data is nearly impossible because individuals have unique

patterns of movement.• Using gaze, head direction, hand position, height, and other behavioural and biological

characteristics collected in VR headsets, researchers have personally identified users with 8 to12 times better accuracy than chance.

• Dirk Geldenhuys – Use VR to develop resilience in the VUCA world?• Need for stronger regulation

Virtual Reality and Assessments

• Intelligent software has predominantly been used to automate screening • Chatbots used as a part of the screening process• Robotics - Call center in the UK needed to hire representatives to respond to

customers who speak seven different languages; AI-enabled test where candidates were “interviewed” by a robot. The software evaluated their speech for syntax and grammar, and the company made offers based on the results

• Vera - a lifelike Russian robot with impressive conversational skills used to phone and video interview candidates for positions

• Socially capable robots can provide many advantages to diagnostic practice: engage people, guarantee standardized administration and assessor neutrality, perform automatic recording of subject behaviours for further analysis by practitioners.

Robotics and Assessments

Robotics and Assessments

Artificial intelligence is about mimicking

the human cognitive and behavioural

processes to solve more complex

problems faster

Artificial Intelligence

• Robotic process automation: This is achieved by gathering and transferring expert knowledge andthen programming the system with an 'if/then' rule-based approach. Chatbots are a great example.However, this rule-based system is not capable of learning and improving without being givenexplicit instructions. In talent assessment, computer-generated interpretative reports make use ofthis technology.

• Machine learning: Even though a computer cannot think for itself (not yet anyway), statistical toolscan enable a system to model predictions from any given data – and to add to the model to improveits predictions over time. This is used in data analysis to create predictive people analytics as ameans to help employers make better talent decisions.

•

Pattern matching: This AI technique uses a computer to check the sequence of responses todetermine if there is a pattern. It can be used to carry out some 'human' tasks, such as recognizingfaces or identifying emotions.

• Natural language processing: Makes use of text and speech analytics to extract the underlyingmeaning. This can be applied to analyse speech in interview question responses

Artificial Intelligence and Assessments

Artificial Intelligence and Assessments

• Filip Lievens, (Professor from the Department of Personnel Management, Work and Organizational Psychology at Ghent University) - EASE tool (Electronic Assessment Simulation Exercise) -all participants start with one e-mail. They can then obtain further information by organizing meetings, requesting archive documents or sending e-mails to colleagues. The participants need to ask the right people the right questions in the right way. The computer then generates the requested information, and it also functions as an assessor that evaluates and records all actions objectively.

• http://www.fmaconferences.org/Vietnam/Papers/Hrazdil%20et%20al.%20FMA%202019.pdf (2018) study on utilising IBM Watson Personality Insights service to measure executive personalities (big 5) based on CEO’s responses to questions raised by analysts during conference calls.

• Video interviewing – Audio transcribed and analysed for clarity of speech and language proficiency. AI used to analyse the visual elements through emotion tracking software and facial recognition

• Assessment systems designed to integrate constructs from across psychometric instruments to assist with the development of psychometric competency frameworks, the identification of inhibitors and job position fit

Benefits: • Improvement of assessment process flows • Capturing of real-time data as it happens • Equality - a system cannot identify race, gender or age.• Engagement: the candidate finds these types of assessments modern, engaging and novel.• Depth of Information: a tapestry of information collection; not possible by even an

experienced assessor.• If systems are stable then large volumes can be handled• Software algorithms produce accurate and error-free calculations• An AI neurological network can produce a result without any human intervention• AI can analyse vast amounts of data accurately and error-free. This assists with faster decision

making• Elimination of human conscious and unconscious bias BUT an algorithm is only as good as the

data that’s fed into it.

Artificial Intelligence and Assessments

Concerns:• Companies exploit public confusion slap the “AI” label on whatever they are selling ‘ (How to

recognise AI snake oil - Arvind Narayanan https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf)

• Ethical concerns regarding accuracy of areas such as content identification, facial recognition, medical diagnosis from scans, speech to text and ‘deepfakes’.

• Defensibility. How do you defend the outcomes of standardised 'plug-and-play' AI systems which are not aligned with specific job requirements and lead to custom decisions?

• Ethics. There is an ethical question around how much support to take from an AI system for decision making. i.e. allow AI to reject outright or flag to check?

• Do and can organisations provide proof that AI predicts performance? • Data handling. AI excels at analysing massive amounts of data, but the results can be

misinterpreted or even deliberately abused. • Video interviewing – applicability – can you really identify emotions from facial recognition?

Generalisability of facial expressions across racial and cultural groups; If you look anxious are you really anxious?

Artificial Intelligence and Assessments

Gamification

• Instead of developing a full game, gaming elements are layered over an existing program or context in order to amplify users’ motivation to engage within that context (Kapp, 2014)

• Designed to motivate users by providing feedback, recognition, status and the potential for competition among users (Muntean, 2011).

Gamification

Advantages of Gamification

Big Data

Rich data analytics, in a concise

meaningful manner relevant to business

needs and digital transformation.

Psychological Constructs

Founded on strong, well established tried

and tested psychometric principals of

existing assessments.

Generation Z

Strongly appeals to the new generation of

job seekers, millennials / generation z.

Higher Engagement

Provides a unique and positive

recruitment or development experience for

potential candidates.

Mobile Apps

Immersive, rich and fun interactivity

for the candidate, yet still providing

reliable and usable data

Challenges of Gamification

• The time involved in truly customising and validating a game,

• Ongoing investment as the life cycle of the games may be short,

• Some people just don’t enjoy games,

• Games are only fun in low-stake environments,

• Candidates may feel belittled / not taken seriously,

• If poorly designed and irrelevant to the construct being assessed, mechanics introduce error and noise into assessments,

• The game might lack face validity and proving psychometric properties difficult.

• If a game is developed in a Eurocentric environment, gaming mechanics, skin, storyline and challenges may not be culturally relevant to SA candidates.

• Beware high graphics and data requirements in developing environments.

Implications for Practitioners

Implications for Practitioners

• In South Africa psychologists and psychometrists still need to comply with the provisions of the EEA act section 8 (a), (b) and (c)

• The above prohibition holds true for all testing of employees irrespective if it is a Psychological test or not being used.

• All tests measuring a Psychological construct must be ‘classified’ with the HPCSA and can only be used by a registered person with the HPCSA

• ASSA are looking at a Test Criteria Evaluation process and partnership with the HPCSA in this regard

• Local (and global) Legislation and regulations, research and classification/certification is lagging in terms of the next generation of assessments which is already here

• Innovations in technology are allowing for easier delivery of assessments as well autonomous scoring and reporting to improve assessment efficiency and accuracy

• The use of Big Data, Machine Learning, Artificial Intelligence, VR, Robotics and Gamification and are leading the way in the assessment and talent space

• Although proving to be more effective, efficient and accurate in nature, we need to be careful of how these assessments are constructed, where they can be effectively used and what the implications are for the South African environment.

• Research into validity and reliability on digital assessments should be a combination of theory- and data-driven models and more research is required.

• AI’s role should be restricted to providing additional information and enhancing efficiency. AI can deliver useful information, at various stages of the selection process, but should be used to support, and not make the final decision.

• We need to be careful as Algorithms don’t have an ethical compass. They may be based on undesired societal biases. Rubbish in, rubbish out!

• Data-driven models and final decision-making based on assessment results should remain under control of humans - leverage technology and assessments learnings to enable better decisions

• Role of practitioners will probably change – more about job analysis and algorithm design and ensuring ethical and scientific use of technological enabled assessments.

Implications for Practitioners

https://www.psychologytoday.com/za/blog/the-situation-lab/201910/beware-these-marketing-trends-in-psychological-assessment

Ryne A. Sherman, Ph.D., - Chief Science Officer at Hogan Assessment Systems –

‘Many of the new assessment firms use flashy technology and claim new insights into workplace performance. Hiring managers and HR professionals

need to be wary of companies using these common marketing trends.

Only two things matter in psychological assessment: fairness and predicting performance. Companies that emphasize neuroscience, big data, and

gamification may be trying to distract you from the fact that their assessments don’t predict workplace performance’.

Understand what is inside the black box!

Implications for Practitioners

David BischofChairperson: People Assessments

in Industry (PAI)

Director – Research Psychologist &

Psychometrist

OMT | Evalex

Email: David@evalex.comOffice: 073 254 5119

Thank You!