The amount of text data we send out in the world is staggering. On average, there are 500 million tweets sent per day, 23 billion text messages, and 306.4 billion emails. Everything we say, every email we send, and every word on our resumes can be used to not only understand the world around us, but as clues about the individual speaking, typing, and writing.

Unfortunately, text data does not fit into the traditional structured format of rows and columns. Text data is messy, unstructured, and not easily analyzed using classical statistical methods. Enter natural language processing, or NLP. NLP is a type of artificial intelligence that uses machine learning to break down, process, and quantify human language. NLP helps us understand the hidden stories within text-based data.

There is no singular method associated with NLP. NLP consists of multiple techniques ranging from using keywords to interpret text or speech to understanding the underlying meaning and context of communication. Because of the varying techniques associated with NLP, in the IO literature, NLP has been used to aid with several business initiatives, such as job analysis and selection, to name just two.

Up to 95% of usable organizational data is unstructured, resulting in an increased drive for using this data to remain competitive. The competition and consistent advancements in computational power, data access, and open-source research initiatives have led to the field of NLP to evolve and grow constantly. At Hogan, we are leveraging this continual growth by using NLP to improve our products and talent analytics solutions.

Hogan and Natural Language Processing

One way we are using NLP is by streamlining the coding process of focus-group notes for personality scale relevance, thereby injecting NLP into our job-analysis strategy to increase the efficiency of the approach and improve the quality of our results. Manually reading and coding focus-group notes is a time-intensive and cognitively draining process. Using NLP, on average, we can decrease the overall time it takes by approximately 6,000% while maintaining predictions that are both consistent and accurate.

Many text-based, data-analytic tasks require similar knowledge about language, such as semantics, structural similarities, and syntax. This knowledge can be shared from one model to another through transfer learning. Transfer learning allows us to quickly take advantage of cutting-edge NLP research without having to spend months and years gathering unneeded data and training similar models from scratch. Transfer learning involves taking a model trained on another dataset for a different task and fine-tuning it on a second dataset for a different task. In other words, we take what the model learned already and adapt it for our purposes. The base model for focus-group note prediction was trained on over 3 billion words. The base model was fine-tuned on a large collection of focus group notes collected across hundreds of organizations where researchers identified which personality scales were relevant based on their expert judgment.

This approach has already shown promising results for correctly identifying the relevance of personality characteristics from focus-group notes. When compared against human-raters (subject matter experts; SMEs), our model was consistent and had an average accuracy score approximately 10% higher than the average accuracy of the SMEs. This indicates that Natural Language Processing is an accurate and efficient method for identifying the critical personality characteristics of job roles from focus groups.

Almost every week I learn about a new psychological assessment company entering the marketplace. Although each company is different, they all tell the same story. First, they tell you that hiring is broken; Personality tests don’t work anymore; Recruiting is out-of-date. Second, they tell you that their company has the answer. Finally, they hit you with the marketing smokescreen: a list of sophisticated-sounding technological advancements designed to confuse you, misguide you, and make you feel like you are missing out. You are not missing out, but you are falling for the common marketing trends used by these new companies. In this article, I expose these trends so that you won’t fall for them.

Trend #1: Neuroscience

Some companies measure how fast you react to flashing objects on a computer screen and say that their assessments are based on neuroscience. Neuroscience is the study of the structure and function of the nervous system. Even though such a broad definition leaves room for debate, the reality is that neuroscience concerns the function of individual neurons and the brain (i.e., a large mass of neurons). So unless the assessment you are taking is directly recording brain activity, it isn’t neuroscience. Pushing the spacebar in response to images on a computer screen isn’t neuroscience. You don’t have to take my word for it. Check the table of contents of this book on neuroscience methods. Here’s another. No mention of measuring reaction times to flashes on a screen. Don’t fall for the neuroscience routine when it’s just measuring reaction time.

Unfortunately, the deception isn’t as innocent as calling a reaction time task neuroscience. Recent scientific studies have shown that reaction time tasks of individual differences are psychometrically useless. First, these tasks are designed to eliminate individual differences. If individuals don’t get different scores on the tasks, how they can possibly predict individual differences in performance? Second, these tasks have poor test-retest reliability. This means that you won’t get the same score each time you complete the tasks. If the scores you get back are random, how can they predict performance? Last, and not surprisingly, these tasks don’t predict real-world outcomes. One recent study showed that self-report measures of personality predicted 20 (out of 30) life outcomes and that reaction time tasks predicted none. Don’t fall for computer-based reaction time tasks that don’t predict anything.

Trend #2: Big Data / Deep Learning

Some companies brag about their stacks of big data and their use of machine learning or artificial intelligence to produce talent insights. However, if you dig deep, you find that most of the data these companies collect is useless; they aren’t even using it. For example, millions of mouse movements, keystrokes, and response times can be measured in a 10-minute assessment. But are they consequential? Do they predict anything? How is moving your mouse five pixels to the left before you respond to a question even relevant to your job as a store clerk? Evidence indicates that these sorts of micro-movements don’t predict anything and aren’t job relevant. Modern assessments might measure millions of things that you do, but only a few of them predict job fit and job performance. Unless the assessment is asking the right questions and measuring the right things, the big data are just another smokescreen.

The second thing you find as you dig deep (and you should be digging deep) into these assessment companies is that the sophisticated statistical methods they tout don’t provide the new insights they promise. Recent advances in deep learning and artificial intelligence have made news; and, these areas are poised to advance human progress. But these techniques are most beneficial for complex problems and huge data sets, not on data sets with a few hundred people and a handful of variables. Don’t fall for grandiose claims about big data and artificial intelligence that aren’t bringing new talent insights.

Trend #3: Gamification

Another marketing trend to watch out for is gamification. Gamification is defined as adding game-like elements such as points, scores, trophies, competition, and entertaining environments to existing assessments. The idea is that if job applicants have more fun taking the assessment, they will be less likely to drop out of the application process. Although the data show that candidates do enjoy game-based assessments, the data also show that gamification doesn’t improve performance predictions. Research indicates that applicants who drop out during the assessment process are unlikely to be your strongest candidates anyway. So you aren’t losing high-quality candidates due to dropout during assessment.

Further, measuring psychologically stable characteristics (e.g., IQ, personality) via games is extremely difficult. Although there is evidence that cognitive ability can be measured via game-based assessments, measuring personality using game-based assessments doesn’t work. In addition, assessments that claim to be game-based often aren’t games at all. In fact, most are just boring psychology laboratory tasks, like the Go, No-Go. Dr. Richard Landers—a global expert on game-based assessments—points out that dressing up boring tasks and adding arbitrary point systems doesn’t make something a game. Don’t fall for games that don’t predict performance.

Trend #4: Profile Matching

Everyone wants to hire high-performing employees. One intuitive way to do that is to hire people who are like your current high performers. Several new companies use a profile-matching approach. First, they assess your high performers. Next, they see what differentiates your high performers from some larger population of people who have taken the assessments. The differences between the two create a high-performer profile. At face value this approach sounds perfect, but it is deeply flawed as the following example demonstrates.

Imagine you are the owner of a professional basketball team. You have three superstars and would like more superstars. A company promises to use their assessments to help you find superstars. First, they measure your three superstars on basketball-relevant skills: speed, height, shooting ability, etc. Next, they compare your players to a large population. Lo and behold, they find out that your superstars are faster, taller, and better shooters than the general population. On this basis, they recommend that you hire players who are fast, tall, and great shooters.

I’m sure you can see the problem here. The assessment company you hired isn’t differentiating between your high performers and your low performers. They are simply telling you what differentiates people, who work in your organization (professional basketball players) from those who work in other organizations (everyone else). Although this profile matching approach used by many companies seems intuitive, it doesn’t work. Only a proper validation study that differentiates high and low performers will give you an accurate profile. Don’t fall for assessments that are only validated on high performers.

Trend #5: Emphasizing Irrelevant Information

The last marketing trend is something that shysters have been doing for a long time: emphasizing features of a product that don’t really matter. New and old assessment companies often emphasize the total number of applicants, time to hire, and the diversity of the hiring class as selling points. The odd thing about emphasizing these is that you don’t need an assessment company to do any of them. A simple lottery will do. That is, if you hire people randomly, you are sure to increase the total number of applicants and the diversity of the hiring class, and likewise you will decrease time to hire. The problem is, when it comes to performance, hiring randomly doesn’t work.

When it comes to performance, the only thing that matters is validity: how well does the assessment predict performance? The reality is that some assessments predict job performance better than others. Rest assured that assessment companies that don’t show or emphasize validity don’t have any. With no validity, they have no choice but to emphasize irrelevant features. The good news is that you don’t have to trade predictive validity for these less relevant features. Well-validated assessments predict job performance and do not discriminate with regard to race, religion, sex, gender, ethnicity, or sexual orientation. As a result, well-validated personality assessments help you build a workforce that is high-performing and diverse. Don’t fall for assessments that emphasize irrelevant information.

Conclusion

Many of the new assessment firms use flashy technology and claim new insights into workplace performance. Hiring managers and HR professionals need to be wary of companies using these common marketing trends. Only two things matter in psychological assessment: fairness and predicting performance. Companies that emphasize neuroscience, big data, and gamification are often trying to distract you from the fact that their assessments don’t predict workplace performance.

*This post was authored by ThreeFish Consulting.

ThreeFish Consulting, the authorized distributor for Hogan Assessments in India, welcomed back Ryan Ross, Managing Partner of Hogan Assessments, for a series of three client events across the country to provide thought leadership around the Science of Personality.

The first stop on the tour was in Pune in the western part of the country. The ThreeFish Breakfast Roundtable on July 30^th at the JW Marriott was well attended by Hogan Users and a wider audience from a range of industries who were interested in learning more about Hogan and the Science of Personality. This was a great “launch” event for Pune as it was the first time ThreeFish-Hogan held a client event in the city. “The growth of ThreeFish into Pune marks an important milestone – while support has existed in the local market for several years, Pune-based businesses are demanding quality assessments, and Threefish is equipped to meet their need” said Ryan. Given the mixture of current and potential clients, Ryan’s talk focused on the link between personality, leadership, and organizational performance. Elaborating on the view, ‘Who you are is how you lead’, Ryan said “it was such a great event, the way the audience engages in such a lively and smart conversation is one of the best things about being in India. Too often participants are quiet or shy, but not in Pune. It was a thrill from a presenter’s standpoint.“ The engaged audience had a lively conversation around how our personality affects the way we think, behave, make decisions, and build relations, ultimately impacting our performance.

After wrapping up in Pune, the team was off to the ‘Garden City of India,’ Bangalore, wheree we hosted the next Breakfast Roundtable on July 31. It was a warm gathering with Hogan users from both corporate and independent practitioners. There was a lot of excitement in the community to participate in a ThreeFish-Hogan gathering given the long gap since the last event. Ryan’s talk, ‘Chasing Leadership Trends – a Recipe for Disaster’, discussed two key macro trends in leadership hiring – ‘short-termism’ which leads to the celebrity CEO and ‘trend-chasing’, where our knowledge on effective leadership is often ignored. Ryan talked about the “multiplier effect” of good and bad leadership across the organization, drawing attention to Laissez Faire leadership. Reflecting on leadership from the lens of the led, Ryan elaborated on what employees actually want from their leaders, which are the four core aspects of effective leadership: Integrity, Inspiration, Competence, and Vision. The talk was a success with the Bangalore audience and was reflected in the lingering conversations over a cup of “chai” and the buzz that followed. “Bangalore is such a smart market, it is an established market, and one in which practitioners are not only implementing current practices, but being bullish about introducing cutting edge ideas,” Ryan said after the event. 

Then, it was off to the premier event, the TechHR Conference 2019 in Gurgaon hosted by People Matters where Hogan-ThreeFish were Gold Sponsors. There was a packed hall on the main stage for Ryan’s session entitled, ‘Bright and Shiny or B.S. – The Role of AI in Talent Acquisition.’ Ryan acknowledged that digital innovations and advances in AI have produced a range of novel talent identification and assessment tools which promise to help organizations improve talent acquisition capabilities. However, he raised the important question of whether these are indeed bright and shiny promises that can propel HR into the digital playing field, or BS promises that are overhyped, over-promising, and perhaps even dangerous to the future of the organization?  Ryan shared his perspective that AI, big data, deep learning all have the potential to help but we must use our ethics and beliefs to guide the future. ‘Bright and Shiny’ can be cool, until it rusts. Ryan’s advice was to choose wisely. The question he posed the audience was “Just because we can, should we?” The immediate discussions among the audience were proof that the question was still very much a subject for debate.

“I have been traveling to India for over a decade,” said Ryan.  “It is such an amazing country with immeasurable potential.  Honestly the amount of skill and talent in the HR/OD space is somewhat intimidating. That said, because India believes in data, it is quite easy to strike up a conversation about assessments and personality – because this is a market that understands assessment has a job to do – that is predict performance.”

The trend toward Big Data shows no signs of slowing down, as businesses, organizations, and governments continue incorporating new technology in the race to collect an almost unfathomable amount of information. But a more critical problem remains – what do you do with all that data? How can you find something useful within?

In this video, Ryne Sherman, Chief Science Officer at Hogan Assessments, discusses how Hogan has embraced Big Data from the very beginning in order to study one of the most complex subjects of all – the human mind.

In this era of Big Data, simply producing or collecting nearly unfathomable amounts of data isn’t enough. The best companies are able to sift through that data to find meaningful trends and, ultimately, specific information that sparks a plan of action.

In the rush to harness that data for job selection, numerous companies are turning to experimental AI and machine learning to discover new forms of data collection and new types of analysis human beings might not be capable of. But not all new methods of data collection are created equal. If set up incorrectly, AI data analysis can go horribly wrong – just ask Amazon.

The Internet giant decided to harness its computing power and expertise to create a job screening program that would scan an applicant’s resume and determine if an applicant is suitable. A person familiar with the effort told Reuters the goal was for the program to receive 100 resumes and spit out the top five.

In order to teach this program how to screen candidates, it was fed resumes submitted tothe company over the last decade. In theory, the program would learn what resume terms lead to successful candidates and which terms lead to rejection. In reality, the program learned to reject female candidates.

The show-stopping side effect was the result of Amazon’s own hiring patterns – most of Amazon’s employees are male. Based on that set of data, the program taught itself that male candidates were preferable. Resumes that included the word “women’s” or the names of all-women colleges were downgraded. Since there was no guarantee the program wouldn’t find other blatantly discriminatory ways to reject candidates, executives pulled the plug.

In short, Amazon’s mistake in this experiment was using biased criterion to judge the resumes, and then give the program free reign. Ryne Sherman, chief science officer at Hogan summed up Amazon’s problem:

“If the criteria are deficient, contaminated, or otherwise systematically biased, big data algorithms will pick up the bias and run with it.”

Today’s supercomputers are immensely powerful and capable of amazing feats. They’re also unfailingly literal. No matter how much power you’re working with, if you set up bad parameters, you’ll get a bad result. Amazon’s mistake was easy to find, but even subtle mistakes made by emerging job screening technology can lead to catastrophic results.

The key takeaway from Amazon’s failure is that big data still needs a human touch. Any type of analysis requires clearly-defined parameters before the supercomputers are even turned on in order to eliminate bias or any other type of noise. Start gathering data without a structure, and your efforts will be wasted.

At Hogan, our assessments were built from the ground up to be free of bias. Even though the database we use to determine scoring and job fit has grown to millions of assessments and has become ever more complex, our results remain valid because of the assessments’ focused structure. In fact, company founders Drs. Joyce and Robert Hogan were inspired to create the company in part out of a desire to eliminate bias from assessments.

Technology has turned HR into a data-driven game. This does not mean intuition is waning, but rather that a larger number of practitioners are likely to experience some shame or guilt if they admit that they are ‘playing it by ear’. The recent rebranding of talent management as ‘people analytics’ has arguably enhanced the status of HR.

The hope here is that HR can empower organisations with robust tech and data to turn the art of people management into a science: an objective, defensible and replicable process with a clear ROI.

That said, there is still room for improvement, as most technological innovations have yet to be rigorously scrutinised or effectively applied. The HR tech world is replete with shiny new objects, including some that warrant a considerable amount of optimism, even among cynics. However, at this stage there is no indication that these toys are more effective than applying well-established scientific principles. This is perhaps clearest in talent identification. Consider these salient examples:

Gamification. Although the application of game-like features to talent identification tools has enhanced user experience, enabling organisations to tap into a much wider candidate pool and turning recruitment upside down (from B2C to C2B), it is hard to get carried away. First, most assessment games look like 1980s arcade games.

If you look at what goes on in the real gaming industry, where it is increasingly hard to see the difference between a game and a 3D movie, gamified assessments seem to belong to a prehistoric era.

Second, most gamified assessments are either good-looking IQ tests or glorified situational judgment tests. However, they are less valid than traditional (non-gamified) tests of the same type. Third, the cost of gamified assessments is much higher than traditional ones, so you end up spending more on less accurate tools.

A more hopeful path may be to mine data from existing gamers – who play real videogames – to assess their job-related potential. There are now more gamers in China than people in the US, and hardcore gamers spend an average of 20 hours a week playing. Imagine a future in which companies send avatars to recruit Grand Theft Auto players as heads of sales because of their kick-ass aggression.

Big Data. There is now so much data available on people that we probably don’t need to gather any more. Internal company data, such as email content or metadata, can be used to monitor people’s performance, engagement, and identify their potential. The opportunities are even bigger online where our digital exhaust is as vast as it is underexploited. If only algorithms could access people’s browser history, Amazon purchases, Spotify playlists, Facebook and Twitter feeds (as in programmatic marketing), we would probably ‘know’ them better than the average manager does. However, there is no evidence this approach can predict work-related behaviours better than established selection methods, which also represent more ethical (and legal) alternatives. There is big a difference between what we could and should know about people.

Digital interviewing. A final area of technological disruption is in interviews, which are still the most widely used selection tool in the world. Today it is possible to interview anyone, and analyse their answers and behaviours, without the need for human intervention. Questions can be asked by avatars, and speech- and video-mining algorithms can translate interviewees’ behaviours into valid predictors of future performance, eliminating unconscious biases while reducing time and cost.

The problem, however, is most interviewers can’t ignore their intuition. It’s like having self-driving cars; people still put their hands on the steering wheels because they have more faith in themselves than in technology.

In short, when measured against intuition HR tech is making the industry smarter. But it has yet to reach the level of rigour of scientifically defensible methods.

This article was originally published by HR Magazine on January 10, 2017 and is authored by Tomas Chamorro-Premuzic.