Measuring personality is complex. How can personality assessment retain what is valuable in traditional methods while leveraging the technology of the future, such as AI and machine learning?

Recently on The Science of Personality, cohosts Ryne Sherman, PhD, chief science officer, and Blake Loepp, PR manager, spoke with Georgi Yankov, PhD, senior research scientist at Development Dimensions International, about the future of personality assessment.

They discussed comparing traditional assessments to new methods, as well as some of the philosophical challenges that new assessment methods may face in the coming months and years.

Let’s dive into the exciting complexity of the personality assessments of the future.

Traditional Assessments Versus New Assessments

The difference between traditional assessments and the newer ones concerns variables and data.

In traditional assessments, the developer designs, pilots, improves, models, and norms an assessment according to a standard process. Hypotheses about the relationships between items and scales are confirmed or disproved. The process can be long, but the tools produced by this method have a history of excellent reliability and validity.

In new assessments, the developer works with large quantities of data and advanced machine learning algorithms. To model the data, the developer must concede some supervision and expertise. With traditional machine learning algorithms, decision trees can provide some insight about which variables are most important for the task at hand (for example, making predictions). With deep learning algorithms, the processes between the input and the output aren’t observable. This is what’s known as the black box problem in data science—we can see the input and the output, but how did we come to these results? Although these powerful tools can integrate with the context the user provides and report instantaneously, they also necessitate new methods that come with their own challenges.

Another way to express the difference is that traditional assessments are theoretically or conceptually driven, while the new assessments are technologically driven. A job analysis might be the first step toward finding the ideal candidate for a particular job using traditional assessment, while an approach using a new assessment might start with data analysis. To find a candidate with a new assessment, the developer might consider what data are available, what data can be collected, what can be measured, and what—among that information—might predict success in the target job.

Why Newer Isn’t Always Better

New assessments that overpromise should be viewed with healthy skepticism. “With assessments about humans, it’s a really serious leap of faith to promise improved overall assessment,” Georgi pointed out. Humans are unpredictable, and we simply cannot explain all their behavior.

New measurement approaches differ from traditional assessments in reliability and validity. Because of the sheer volume of data points, an AI-based measurement system may apply unknown parameters or rely on rules influenced by outdated societal attitudes, demographics, and norms. In other words, the reliability can deteriorate quickly. Traditional assessment offers more control over reliability because developers know the items in an assessment and can retire them if they become obsolete.

AI-powered tools stand the risk of lower reliability—and lower validity as well. They may excel in predictive validity, but personality assessments also rely on construct and content validity. “We can control the inputs and the construct domain in traditional assessments, but in automated ones, it’s difficult,” Georgi explained. “People are not so predictable.”

Some elements of AI-based assessments can be helpful, however. They have the promise of scalability, speed, a lower price point, and removing bias. Assessments that use machines for concrete tasks built on sound research can incrementally improve traditional assessments. “If it is improving what is old in a targeted way to solve a specific user problem, only then can it be better,” he said.

New technologies offer high potential to enhance traditional measurement tools, such as the ability of AI to generate more items and parallel forms based on natural language since humans could control exactly what such models would produce. They could not only produce a job analysis but also personalization for job recommendations, like Netflix for jobs, as Georgi described it.

Personality Assessment and User Experience

Reliable, valid, and in-depth assessments often generate results that are quite complex for the end user to understand. Users don’t know theories of personality and often need a coach to make actionable recommendations based on the results. “They want it boiled down for their busy, everyday lives. They probably have five minutes a day for development,” Georgi said. Reports need to be targeted in how they express, for example, what it means to be high in conscientiousness and low in neuroticism and how such people tend to behave at work and at home.

Validated heuristics that people can use for development are arguably more important than one-dimensional results, such as a high, medium, or low score in extraversion. “People love stories about themselves,” said Georgi. “I encourage everyone who does personality reports to make them a little bit more about everyday life,” he added, noting that machine learning can help with generating nonlinear relationships for reports that people will use and love.

User experience seemingly demands simplicity for quick, applicable comprehension, but personality assessment demands even more detail and nuance. Even the Hogan assessments with their 28 main scales and scores of subscales cannot quite capture the total complexity that is an individual. In moving toward a more holistic picture of personality, we will need artificial intelligence to help us develop the heuristics to better understand ourselves and others.

The Future of Personality Assessment

Considering this discussion, what will be the best approach to measuring personality in the future? An integrated personality measure, Georgi said.

Industrial-organizational psychologists have long thought of personality theory in terms of ingredients, such as cognitive ability, motivations, values, attitudes, and more. However, the founders of the discipline did not intend to dissect people but to functionally organize them to explain how all their characteristics worked together. “The people who we assess and we measure—they are first people, then their personalities,” Georgi said.

The Humpty Dumpty approach with specialists focused on separate traits won’t serve the personality assessment of the future. What will serve instead is a dialectic of induction and deduction, with the induction piece aided by technological tools that help psychologists sift data to study personality in context. “I’d like to see machine learning and AI help us come back to the origins of our field where we wanted to predict behavior,” Georgi said. “We are in the business of prediction, not explanation. We want to serve the person, not just report in general.”

Listen to this conversation in full on episode 63 of The Science of Personality. Never miss an episode by following us anywhere you get podcasts. Cheers, everybody!