How Artificial Intelligence (AI)

Solves Challenges Inherent in

Traditional Recruiting

Top Articles Seen in Reuters, Fast Company, and Financial Times

PERFORMANCE • RETENTION • DIVERSITY • EFFICIENCY

REUTERSTECHNOLOGYNEWS|TueJun7,2016|9:47amEDT|OliviaOran

Wall Street hopes artificial intelligence software helps

it hire loyal bankers Clinching a job on Wall Street soon may have as much to do with beating an algorithm as nailing the interview. Goldman Sachs Group Inc, Morgan Stanley, Citigroup Inc and UBS Group AG are exploring the use of

artificial intelligence software to judge applicants on traits -such as teamwork, curiosity and grit - that help in the workplace but don't always show up on a resume or come through in an interview. Banks are turning to the hiring software at a time when they are under pressure to cut costs and finding it difficult to lure and retain top talent. Bank executives hope that artificial intelligence will help them avoid the expense of problem hires and turnover, industry sources said. "Up until this point, technology has only allowed you to find the best resume, but now it's a way of truly understanding the people that are applying," said Mark Newman, chief executive of Salt Lake City, Utah-based HireVue, a video-interviewing platform that uses artificial intelligence to screen applicants. Several banks are in the early stages of adding artificial intelligence software to complement in-person interviews and other traditional hiring processes. The banks hope that the technology can help predict which employees will succeed at a given job by creating patterns around large amounts of data that the tests produce. Seattle-based Koru Careers Inc makes one version of the technology, which Citi and other banks are using in pilot programs to sort out applicants. Other banks are experimenting with software created internally. Koru begins by testing a client's employees to identify traits that mark high performance, known as a corporate "fingerprint." Then applicants take the same assessment, and the software identifies which candidates are best suited to that company. The tests can be taken online, at work or via mobile.

"It may be that what it takes to succeed at Morgan Stanley is different than what it takes to succeed at Goldman Sachs," said Koru Chief Executive Kristen Hamilton. Koru charges its employer clients an undisclosed flat fee for the fingerprint and a license fee for the testing that rises with the number of applicants who take it. Applicants also can record a short video in which they talk about their defining qualities and career aspirations. Koru screens the videos for clients, looking not only at what applicants say, but also their delivery style, including body language and pace of speaking. BAD HIRES RAISE COSTS While Wall Street is not the first place the technology has been tried, it is not yet widespread. The banks hope it will help them compete for recent college graduates who are attracted to Silicon Valley firms and hedge funds. They also hope it will help them avoid hiring the wrong person, which can be expensive and can lead to costly mistakes and lost business opportunities, said bank executives and staffing consultants. Capital One Financial Corp estimates the cost of a bad hire can be as much as three times that employee's salary. The goal of hiring software is to avoid human pitfalls, such as overlooking potentially strong candidates who may not seem desirable at first glance, said Matt Doucette, director of global talent acquisition at Monster Worldwide Inc. "The best salesperson usually isn't the one peacocking, it's the mousy person in the corner who is sharp and asks the right questions," Doucette said. "But if that person interviewed at face value, they never would have been hired." Koru says its software decreases the number of bad hires by as much as 60 percent.

THE HUMAN ELEMENT Some human resources experts say artificial intelligence tools and algorithms don't always capture the best people for a given job and could actually perpetuate existing biases. For example, if a company hired mostly white men who were the eldest children and left handed, an algorithm likely would predict such employees were the most successful, said Brian Sommer, a human resources industry analyst. "There may potentially be huge pockets of people who could be even better individuals for a position that end up being excluded because they aren't part of a bigger data set," he said. Citigroup is testing Koru's software on small batches of employees in its corporate and investment bank, a spokesman said. UBS is running an algorithm on digital resumes to identify candidates with preferred traits and is giving those applications priority for interviews, said a person familiar with the bank's hiring process who was not authorized to speak publicly. It also is using the technology to ensure human reviewers don't overlook strong applicants.

Goldman is using software de Laura Noonan, Investment Banking Correspondent veloped internally to mine resumes for attributes that reflect desired qualities, such as teamwork, integrity and judgment, according to people familiar with the matter. It also is exploring personality tests to better understand what traits make the most successful bankers and traders. Applicants take a 20-minute test to measure their thinking style, personal associations and emotional intelligence. Their answers are measured against the broader Goldman employee base. Goldman still is considering whether it will use the data as part of its broader hiring process, the sources said. Anthony Onesto - vice president of human resources at Razorfish who is building a robot to help answer HR-related questions within companies -said it was still early days for hiring software. "We're still early, and ultimately it's computers, technology and humans working together," he said. AbouttheAuthorReporting by Olivia Oran in New York. Follow her on Twitter at @ozoran LEARNMOREABOUTTHESCIENCEBEHINDKORU7TMIMPACTSKILLSDownloadWhitepaper

FASTCOMPANY|THEFUTUREOFWORK|BYSEANCAPTAIN05.18.16|5:58AM

Can Using Artificial

Intelligence Make Hiring Less Biased? Companies promise that algorithms can not only consider more information about applicants, but also be more objective than human recruiters. The resume is on its way out. “[It’s a] data token that boils you down to a data object,” says Pete Kazanjy, founder of TalentBin, a service that uses social media to find job recruits (now part of Monster). That’s especially true for hard numbers, said Google’s SVP of “people operations” Laszlo Bock in a 2013 New York Times interview. “One of the things we’ve seen from all our data crunching is that GPAs are worthless as a criteria for hiring, and test scores are worthless,” said Bock. A growing wave of hiring tech firms are ingesting far more information about candidates—surveys, work samples, social media posts, word choice, even facial expressions. Adding artificial intelligence (AI), they promise to assess work skills as well as personality traits like empathy, grit, and prejudice to provide a richer understanding of who the applicant is and whether they will fit. “This is going to explode in the next few years,” says Kristen Hamilton, CEO of Koru, an AI-based assessment company founded in 2013. “We’ve applied this data-oriented approach to every other aspect of our companies and businesses.” I decided to try a bunch of these AI-driven assessments to see what they might reveal about the science, and about me.

MECHANIZING AN OLD PRACTICE The deep dive into a candidate’s mind isn’t a new idea, says Mark Newman, founder and CEO of HireVue. Founded in 2004, it was one of the pioneers in using AI for hiring. Its specialty is analyzing video interviews for personal attributes including engagement, motivation, and empathy. (Although it also uses written evaluations.) The company analyzes data such as word choice, rate of speech, and even micro expressions (fleeting facial expressions). Like most of the firms I spoke to, HireVue says that it is not yet profitable.

The post-World War II approach to hiring included things like personality tests such as Myers-Briggs as well as the structured behavioral interview: asking every candidate the same questions to objectively compare them. Another classic deep-dive tool is the work-sample test—performing a mock task from the job, like writing code for a software developer or talking to a (fake) irate caller for a customer-service rep. But these interview tactics are performed by busy and potentially biased humans. “Structured interviews are much better and subject to less bias than unstructured interviews,” says Newman. “But many hiring managers still inject personal bias into structured interviews due to human nature.” Scoring work samples takes staff like software engineers away from their real jobs, says Kazanjy. But what if tireless machines could replace

overtaxed humans? “It’s all the science from the last 50 years being empowered by the technology of today,” says Newman. How far can automated hiring managers go, assuming they can do the job at all? Kazanjy says they can at least knock out people who lack the skills to fit the position. Launched in July 2015, Interviewed offers several levels of testing, beginning with basics such as multiple-choice exams on how well someone knows software like Microsoft Excel or Salesforce. Kazanjy believes that software could go even further, such as assessing programmers. “Errors in code samples can be programmatically detected, the same way that spelling and grammatical errors in a written English sample can be programmatically detected,” says Kazanjy. “You couldn’t automate judging between the A+, B+, and B work,” he says, “but maybe you could kick out the C work.”

FINDING RED FLAGS But companies don’t want candidates with stellar skills if they are racist, sexist, or violent. A company called Fama offers to find these problems through automated searches of the web, including news coverage, blogs, and social networks like Facebook, Google+, Instagram, and Twitter. “Forty-three percent of businesses are using social media to screen job candidates,” says Fama’s CEO and founder Ben Mones, quoting a January 2016 study by the Society for Human Resource Management. Companies who vet candidates on their own with social media may be breaking laws, such as the Fair Credit Reporting Act (FCRA), a 1970 U.S. statute (amended several times) that gives consumers the right to challenge the accuracy of public data collected about them and used for decisions like granting employment and credit. Fama follows FCRA’s requirements, says Mones, such as informing candidates that it is collecting information, obtaining their consent, and sharing the results so they have a chance to respond. Fama, founded in January 2015, digs into language and photos that indicate things employers are allowed to consider in hiring (which varies by state): signs of bigotry, violence, profanity, sexual references, alcohol use, and illegal drug use or dealing. It hired dozens of people to read social media posts and figure out how to classify

“It’s all the science from the last 50 years being empowered by the technology of today.”

and rate how offensive they were, using those results to train its natural language processing (NLP) artificial intelligence on what to search for.

I asked Fama to run a report on me. It pulled up several of my articles and flagged salty quotes from protesters I had tweeted while covering Occupy Wall Street, plus my own use of the word damn. “We don’t score a candidate,” says Mones. “We provide simply a way to automate that filter, where you can find those needles in the social media haystack.” “We’re creating these very long histories of behavior over time,” says Pete Kazanjy. “If someone tweeted something racist 3,000 tweets

ago, you wouldn’t find it, but a machine could.” This summer, Fama expects to roll out the ability to also flag positive things about candidates, like posts about volunteer work. GETTING A GOOD FIT Positive traits are an important part of Interviewed’s tests for consumer service jobs, in which candidates field text chats or calls from bots that represent customers. Interviewed’s clients include IBM, Instacart, Lyft, and Upwork, and the company says it’s close to being profitable. Interviewed is beginning to automate the assessment of what cofounder and COO Chris Bakke calls softer skills. It asked hiring managers to look over test transcripts and rate candidates’ qualities like empathy on a 1-5 scale. Interviewed then applied natural language processing and machine learning that looks for patterns in masses of data. “We found that hiring managers and interviewers are four times more likely to perceive and rate a customer support response as empathetic when it contains three or more instances of please, thank you, or some form of an apology, i.e., I’m sorry,'” says Bakke. That data guides what Bakke calls a structured manual review process: A human still makes the call. As Interviewed collects data, its assessments are growing more sophisticated, says Bakke, gradually progressing from testing skills toward judging whether candidates are a “culture fit” for an organization. This moves into territory staked by companies like RoundPegg. Its online CultureDNA Profile starts with what looks like a bunch of word magnets on a fridge. From 36 options, users drag words or phrases like “Fairness,” “Being team oriented,” and “High pay for good performance” into columns of their nine most important and nine least important values. “You are going to excel in an environment that is most similar to you,” says Mark Lucier,

RoundPegg’s director of accounts. Incorporated in 2009, RoundPegg has attracted big-name clients with its method, such as Experian, ExxonMobil, Razorfish, Xerox, and even HireVue.

“If someone tweeted something racist 3,000 tweets ago, you wouldn’t find it, but a machine could.”

RoundPegg works a bit like survey-based dating site OK Cupid: Clients determine what their company culture is by administering the test to current employees. Then applicant assessments show how well personalities match up. Spending about five minutes on the test, I learned that I prefer, like more than 95% of the country, a “Cultivation” company culture, which is described as having “focus on potential and providing opportunities for growth, with less importance placed on rules and controls.” Of the three other cultures, I scored 63 for Collaboration, 62 for Competence (“People are expected to be experts in their field–generalists are not typically appreciated”), and 20 for Command (“Roles are clearly defined and systems and policies are in place to ensure that things are done the same way every time”). That seems right for a freelance writer who recently left an office job.

RoundPegg then helps companies dig deeper with customized interviews to assess their “risk” of not fitting the company’s values. My assessment showed that I may not jibe with RoundPegg’s priority on rewarding team success over individual success. To suss that out, it recommended asking me for an example of how I once handled working in a team success-oriented environment in the past. GETTING THE RIGHT STUFF Koru goes a step further by assessing not only if someone will fit in at the job, but also if they will be good at it. The company started as a training program for college grads, teaching them to develop seven competencies with buzzy names like Grit, Polish, and Impact. Koru expanded to offer a job applicant test that measures these competencies and gathers a bunch of other data. Koru’s focus remains on young workers. In lieu of extensive job experience, companies can evaluate candidates based on Koru’s personal attribute scores. Artificial intelligence backs up the test, says Kristen Hamilton, because it’s based on reverse engineering the attributes of the most successful people in different types of jobs and companies that use Koru, including REI, Zillow, Yelp, Airbnb, Facebook, LinkedIn, Reebok, and McKinsey & Company. “They tell us who performs well at different levels, and we say, okay, let’s look for patterns in this data set,” says Hamilton. She cofounded Koru with Josh Jarrett, who headed up the Next Generation Learning Challenges program for the Bill and Melinda Gates Foundation. Reverse engineering is a mainstay of HireVue’s philosophy as well. “Across our database, we have tens of millions of interview question responses . . . and every response is rich in information we can learn from,” says Newman. “So as you kind of analyze those pieces, and track the outcomes, that’s where you start building these really highly predictive validated models.” Interviewed is moving in the same direction, says Bakke. In contrast to the five-minute RoundPegg test, I spent about 30 minutes answering 82 questions and recording a two-minute video for Koru. Some questions were head-scratcher comparisons. Do I agree more with the statement, “I like to see tangible results of my efforts” or “I’m not afraid to ask questions if I don’t understand

something”? I also had to handle scenarios, like picking which of four options describes how would I handle my team and boss if we weren’t given enough time or resources to complete a big project. Of my top three impact skills, I scored very high (8.3 out of 10) on Polish (effective communication) and Curiosity. I ranked high (6.7) on Grit, described thus: “The ability to stick with it when things get hard. When directions are not explicit, hires can make sense of ambiguous situations.” “The issue that I have as an organizational psychologist is that you can claim to measure all sorts of fancy buzzwords,” says Jay Dorio, director of employee voice and assessment at IBM’s Kenexa Smarter Workforce division. Grit is a great attribute to have, says Dorio, but he calls it “superfluous and funky stuff” that could have an adverse impact—legalese for hiring criteria that appear neutral but could be discriminatory. Kenexa is also cautious about using artificial intelligence—curious for a division of IBM, which keeps promoting new ways to deploy its AI platform, Watson Analytics. Kenexa uses Watson for 4/2/2017 Can Using Artificial Intelligence Make Hiring Less Biased? | Fast Company some of its offerings, such as analyzing employee survey results, says Dorio. But it doesn’t use AI in evaluating employee assessments. Koru’s Hamilton was sure to mention adverse impact. “We have conducted multiple research panel studies that affirm our assessment does not have adverse impact on applicants,” explains Hamilton. A data-driven approach, in fact, is likely to be more objective, she says. “What people do typically in an unsophisticated way in some of these hiring processes is say, we like soccer players because they never give up,” says Hamilton. “So let’s get soccer players.” That’s no joke. Athletic experience has long been a criteria for evaluating candidates to sales jobs, says Pete Kazanjy. It’s believed to indicate high competitiveness, willingness to be coached, and ability to think on their feet. “They may be right, but they’re totally guessing as to whether or not athletes or soccer players or people with a physics class are better employees or better performers,” says Hamilton. “We take the scientific approach and look at any input we want to put in . . . and identify their predictive power.” About the Author Sean Captain is a technology journalist and editor. Follow him on Twitter @seancaptain. KEYNOTEVIDEO:THEHIRINGGAMEHASCHANGEDAspresentedatMicrosoftSummitinMarch2017.EngageNow

Banks are turning to software to widen the limited pool of prestigious US universities such as Barnard College in New York from which they hire talent © AFP

Bank hiring: Wall St turns to machines to find better-behaved bankers

Why a US division of Deutsche Bank is using matchmaking tests to hire graduates FINANCIAL TIMES | SEPTEMBER 7, 2016 | Laura Noonan, Investment Banking Correspondent Noel Volpe owes his marriage to the behavioural profiling used by dating website Eharmony. Now, the affable New York investment banker is using the same technology for a different type of matchmaking: introducing his

employer, Deutsche Bank (http://markets.ft.com/data/equities/tearsheet/summary?s=de:DBK), to perfectly suited graduates. This month, Deutsche’s US corporate finance division has begun screening students for the behaviours identified in its best junior staff, so it can hire high performers from more diverse backgrounds who stay at the bank for longer. With competition always fierce for the best and the brightest, investment banks have been assessing applicants on their educational attainment, extra curricular activities and personality traits for more than a decade. But the beauty of Deutsche’s behavioural profiling approach, according to Mr Volpe, is the way it levels the playing field for a new pool of talent from colleges that banks previously ignored. “You can’t trick the machine,” he says of the 20-minute behavioural test that is carried out online. “We’ve asked juniors to try to trick it and it still catches you”. What the machine aims to tease out is a candidate’s innate behavioural make-up. Fight or flight? Lead or follow? Create or conform? Koru, the company that carries out the profiling, says there are no right or wrong answers, and no politically-correct judgments — only comparisons of candidate responses with those of top performers at the organisation doing the hiring. If those people happen to be difficult loners who never back down, so be it. Kristen Hamilton, chief executive of Koru, says: “We have never had an organisation say I don’t believe it, I don’t buy it.” She describes behavioural profiling as shining a light in the “dark space” that exists beyond measures of cognitive competency and technical skills. Crucially, she points out that the most desirable behavioural profile, or “fingerprint”, is unique to every firm, and even varies within the same industry. Some fingerprints show clear ‘spikes’ in measurements of behavioural traits such as “grit”, “polish”, “teamwork” and “curiosity”. Others show a balance of different traits.

Koru’s work with a range of other clients (http://www. joinkoru.com/employer-partners/) — which have been as varied as McKinsey, Reebok, LinkedIn and Airbnb — has enabled it to build a database of 30,000 applicant fingerprints.

And these 30,000 profiles are helping Deutsche solve a problem that bankers at some Wall Street institutions admit they are still grappling with: how to find talent hidden away in thousands of US colleges. It is a problem that has become more pressing since the financial crisis, which has led many Ivy League graduates to turn their backs on banking (https://ww w.ft.com/content/51e2d8ec-78d8-11e5-933d-efcdc3c11 c89) as a career. For banks still in a state of flux, such as Deutsche — which dropped out of the world’s top three investment banks (https://www.ft.com/content/940 adc00-f7e6-11e5-96db-fc683b5e52db) last year as it restructured itself — attracting talent remains a real challenge.

But even Mr Volpe’s counterparts at the big US investment banks are now intrigued by Deutsche’s approach — and admit to flaws in their traditional reliance on candidates’ grade point averages or college majors. One senior investment banker at a Wall Street giant admits that a test that accurately predicted a candidate’s likelihood of thriving and staying on “would be terrific”.

KEY INDICATORS IN KORU’S BEHAVIOURAL PROFILING

SKILL DESCRIPTION Teamwork Collaborates effectively with colleagues

Ownership Takes initiative in the service of others

Polish Communicates professionally and confidently

Impact Ability to use time efficiently and target

resources

Rigour Holding analytical skills and ability to mine data

Grit Tenacious and resilient in fast-paced environment

Curiosity Capacity to learn quickly, also creative and innovative

Making the grade at Deutsche Bank (https:// www.ft.com/content/be1 45dee-74ed-11e6-bf48b372cdb1043a) FT reporter enters the screening process and finds out if she has what it takes

Using behavioural profilingto widen the candidate pool appeals to banks because academic research consistently finds (http://www.scientificamerican.com/article/how-diversity-makes-u s-smarter/) that more diverse team members improve decision-making. They not only bring new ideas to the table, but also force those from traditional backgrounds to become more open to new ideas. That diversity of thought is particularly sought after in banking where a “herd mentality” was blamed for disastrous decisions in the run-up to the financial crisis. Abbas Elegba, an associate in Deutsche’s New York-based financial sponsors group, is an example of the benefits to the bank of valuing diversity. Of African American descent, he spent his early 20s in the army, and came to Deutsche after the bank visited his MBA school: the University of Virginia (Darden). He believes having staff from diverse backgrounds can also help with staff retention. “When you go to war, it’s not about what you’re getting paid,” he points out. “What you really, really care about is that your leaders care about you and that they’re not going to waste you or your life on something that is fruitless . . . That’s the part that I think [army] veterans add to banks: by really thinking deeply about the people you work with, and the people that are junior to you, and how you treat them . . .[by] leading and not managing.” Deutsche’s previous diversity programmes, and other innovations to empower junior bankers, have resulted in the number of analysts progressing to become associates in its US corporate finance business rise from 11 two years ago to 40 this year. This increase was achieved with a similar intake size.

Deutsche Bank is using behavioural tests to hire bankers resembling the ones it has already. You would have thought it was keen to recruit trainees who are the polar opposite of the current lot lousing up the P&L, says Jonathan Guthrie.

However, the big difference between what Deutsche is doing now, and its approach when people like Mr Elegba were hired, is the more targeted way it can run recruit programmes. Having spoken to his international colleagues, Mr Volpe is optimistic that Deutsche will roll out the Koru behavioural profiling across other parts of its business.

“There must be [a commercial benefit],” he says. Even though he concedes he cannot prove this directly, he fervently believes there is a greater good — for the culture of the firm, and for the satisfaction of those it employs.

“There’s a reason why many managing directors and directors on Wall Street have no idea who their analysts are,” he says. “They’re seen as transitory, two years in and out. I won’t fix all of that. But I’ve already fixed some of it.”

About the Author Laura Noonan, Investment Banking Correspondent Follow her on Twitter @LauraNoonanFT

FROMTRADITIONALRECRUITINGTOPREDICTIVEHIRINGScheduletime withamemberofourcustomerengagementteam

Get Started