Almost every week I learn about a new psychological assessment company entering the marketplace. Although each company is different, they all tell the same story. First, they tell you that hiring is broken; Personality tests don’t work anymore; Recruiting is out-of-date. Second, they tell you that their company has the answer. Finally, they hit you with the marketing smokescreen: a list of sophisticated-sounding technological advancements designed to confuse you, misguide you, and make you feel like you are missing out. You are not missing out, but you are falling for the common marketing trends used by these new companies. In this article, I expose these trends so that you won’t fall for them.
Trend #1: Neuroscience
Some companies measure how fast you react to flashing objects on a computer screen and say that their assessments are based on neuroscience. Neuroscience is the study of the structure and function of the nervous system. Even though such a broad definition leaves room for debate, the reality is that neuroscience concerns the function of individual neurons and the brain (i.e., a large mass of neurons). So unless the assessment you are taking is directly recording brain activity, it isn’t neuroscience. Pushing the spacebar in response to images on a computer screen isn’t neuroscience. You don’t have to take my word for it. Check the table of contents of this book on neuroscience methods. Here’s another. No mention of measuring reaction times to flashes on a screen. Don’t fall for the neuroscience routine when it’s just measuring reaction time.
Unfortunately, the deception isn’t as innocent as calling a reaction time task neuroscience. Recent scientific studies have shown that reaction time tasks of individual differences are psychometrically useless. First, these tasks are designed to eliminate individual differences. If individuals don’t get different scores on the tasks, how they can possibly predict individual differences in performance? Second, these tasks have poor test-retest reliability. This means that you won’t get the same score each time you complete the tasks. If the scores you get back are random, how can they predict performance? Last, and not surprisingly, these tasks don’t predict real-world outcomes. One recent study showed that self-report measures of personality predicted 20 (out of 30) life outcomes and that reaction time tasks predicted none. Don’t fall for computer-based reaction time tasks that don’t predict anything.
Trend #2: Big Data / Deep Learning
Some companies brag about their stacks of big data and their use of machine learning or artificial intelligence to produce talent insights. However, if you dig deep, you find that most of the data these companies collect is useless; they aren’t even using it. For example, millions of mouse movements, keystrokes, and response times can be measured in a 10-minute assessment. But are they consequential? Do they predict anything? How is moving your mouse five pixels to the left before you respond to a question even relevant to your job as a store clerk? Evidence indicates that these sorts of micro-movements don’t predict anything and aren’t job relevant. Modern assessments might measure millions of things that you do, but only a few of them predict job fit and job performance. Unless the assessment is asking the right questions and measuring the right things, the big data are just another smokescreen.
The second thing you find as you dig deep (and you should be digging deep) into these assessment companies is that the sophisticated statistical methods they tout don’t provide the new insights they promise. Recent advances in deep learning and artificial intelligence have made news; and, these areas are poised to advance human progress. But these techniques are most beneficial for complex problems and huge data sets, not on data sets with a few hundred people and a handful of variables. Don’t fall for grandiose claims about big data and artificial intelligence that aren’t bringing new talent insights.
Trend #3: Gamification
Another marketing trend to watch out for is gamification. Gamification is defined as adding game-like elements such as points, scores, trophies, competition, and entertaining environments to existing assessments. The idea is that if job applicants have more fun taking the assessment, they will be less likely to drop out of the application process. Although the data show that candidates do enjoy game-based assessments, the data also show that gamification doesn’t improve performance predictions. Research indicates that applicants who drop out during the assessment process are unlikely to be your strongest candidates anyway. So you aren’t losing high-quality candidates due to dropout during assessment.
Further, measuring psychologically stable characteristics (e.g., IQ, personality) via games is extremely difficult. Although there is evidence that cognitive ability can be measured via game-based assessments, measuring personality using game-based assessments doesn’t work. In addition, assessments that claim to be game-based often aren’t games at all. In fact, most are just boring psychology laboratory tasks, like the Go, No-Go. Dr. Richard Landers—a global expert on game-based assessments—points out that dressing up boring tasks and adding arbitrary point systems doesn’t make something a game. Don’t fall for games that don’t predict performance.
Trend #4: Profile Matching
Everyone wants to hire high-performing employees. One intuitive way to do that is to hire people who are like your current high performers. Several new companies use a profile-matching approach. First, they assess your high performers. Next, they see what differentiates your high performers from some larger population of people who have taken the assessments. The differences between the two create a high-performer profile. At face value this approach sounds perfect, but it is deeply flawed as the following example demonstrates.
Imagine you are the owner of a professional basketball team. You have three superstars and would like more superstars. A company promises to use their assessments to help you find superstars. First, they measure your three superstars on basketball-relevant skills: speed, height, shooting ability, etc. Next, they compare your players to a large population. Lo and behold, they find out that your superstars are faster, taller, and better shooters than the general population. On this basis, they recommend that you hire players who are fast, tall, and great shooters.
I’m sure you can see the problem here. The assessment company you hired isn’t differentiating between your high performers and your low performers. They are simply telling you what differentiates people, who work in your organization (professional basketball players) from those who work in other organizations (everyone else). Although this profile matching approach used by many companies seems intuitive, it doesn’t work. Only a proper validation study that differentiates high and low performers will give you an accurate profile. Don’t fall for assessments that are only validated on high performers.
Trend #5: Emphasizing Irrelevant Information
The last marketing trend is something that shysters have been doing for a long time: emphasizing features of a product that don’t really matter. New and old assessment companies often emphasize the total number of applicants, time to hire, and the diversity of the hiring class as selling points. The odd thing about emphasizing these is that you don’t need an assessment company to do any of them. A simple lottery will do. That is, if you hire people randomly, you are sure to increase the total number of applicants and the diversity of the hiring class, and likewise you will decrease time to hire. The problem is, when it comes to performance, hiring randomly doesn’t work.
When it comes to performance, the only thing that matters is validity: how well does the assessment predict performance? The reality is that some assessments predict job performance better than others. Rest assured that assessment companies that don’t show or emphasize validity don’t have any. With no validity, they have no choice but to emphasize irrelevant features. The good news is that you don’t have to trade predictive validity for these less relevant features. Well-validated assessments predict job performance and do not discriminate with regard to race, religion, sex, gender, ethnicity, or sexual orientation. As a result, well-validated personality assessments help you build a workforce that is high-performing and diverse. Don’t fall for assessments that emphasize irrelevant information.
Conclusion
Many of the new assessment firms use flashy technology and claim new insights into workplace performance. Hiring managers and HR professionals need to be wary of companies using these common marketing trends. Only two things matter in psychological assessment: fairness and predicting performance. Companies that emphasize neuroscience, big data, and gamification are often trying to distract you from the fact that their assessments don’t predict workplace performance.
