Unless you’ve been living under a rock, you’ve likely been hearing about the Mel Gibson audiotapes lately in the news. There’s no need to recount specific quotes in this family-friendly blog, but needless to say that Mel’s language could make a seasoned sailor blush. We’re obviously hearing Mel at a low point. He’s stressed about his finances, some negative publicity, and a recent divorce, not to mention that things clearly aren’t going all that well with his new love interest, who is recording their conversations. So it’s a tough time, and his behavior during this period is unfortunately playing out on a very public stage.

So what does this have to do with assessments and business results? The Mel Gibson thing is a prime example of how critical our behavior is when we’re stressed, under immense pressure, or just fatigued enough to let our guard down. In Hogan parlance, he is “derailing.” His behavior under duress has invited a heap of embarrassment and bad publicity, and his career is in jeopardy. His long-time talent agency has dropped him. The film studios, producers, financers, and other assorted movie-industry folks that have invested in the Mel Gibson brand are watching the value of their investment plummet.

In our world, I see this as a parallel to the growing movement to “focus on your strengths.” Such a perspective emphasizes what we do well, and essentially tells us not to worry about our weaknesses. Intuitively this is an attractive proposition – nobody likes to spend a bunch of time worrying about things we don’t do very well. But the reality is that these moments often occur during crucial periods such as when we have to make a critical decision or during a time of crisis. It is our behavior in these circumstances that can ultimately dictate how well we perform, how we are evaluated, and determine our reputation among others. People are quick to forget how we behaved when everything was going well.

At Hogan we promote the view that strengths are important, but that it is essential to be aware of one’s derailers and to target risks to performance. Indeed, any list of figures who have derailed in spectacular fashion will be populated with immensely talented people. They have ascended to positions of influence and been granted wide latitude and discretion based on impressive bodies of work and dazzling talent. But left unchecked, their derailers eventually led to poor decisions, inexcusable behavior, or other things to initiate their fall from grace.

Tying this back to our current subject matter, what were the first things that you thought of when you saw Mel Gibson at the top of this post? Was it a string of blockbusters? An impressive filmography that has grossed over $2B? An Academy Award for Best Director? Or was it a series of rants being played on the evening news? These brief periods of derailing behavior have dimmed what was otherwise one of Hollywood’s brightest stars.

Jarrett Shalhoop
Senior Consultant
Hogan Assessment Systems

After two pre-dawn trips to retail Mecca, I became the proud owner of the much-coveted iPhone 4. It’s the first smart phone I’ve ever had, and its innovative design, personalized features, and functionality leave me impressed. Now I’m free from my desk and laptop when I want to check e-mails from work, catch up on online banking, or just goof off on Facebook. It would appear that I’ve finally become one of the “cool” people. Or maybe not.

Here’s the problem – I don’t have a case for my new phone. Although a few cases are available for the iPhone 4, they are less effective protection than the one I’d prefer, which won’t be available for anywhere from a few weeks to a couple of months. So while I wait, I’ve relegated my slick new phone to the box it came in, a temporary case to protect it from bumps, scratches, and my two kids, either of whom could strike at any moment. It seems ironic that such an investment is limited to the protection offered by a cardboard box. When my preferred case finally does become available, the condition of my iPhone will determine whether I made a good or bad choice. Either way, it’s a gamble.

Unfortunately, many organizations throughout the world make similar gambles, only with much greater investments and potential losses. Every organization is after the latest “iPhone 4” available to them, which may include new trends in assessment-based selection, High Potential identification, or other such programs. However, after investing considerable resources into selecting and otherwise identifying these individuals, some organizations don’t make further investments to protect their purchases. So instead of investing in assessment-based development programs to coach new hires, provide insights into derailing behaviors, or groom top talent, these organizations gamble on employees developing themselves. When employees take the initiative to develop critical competencies, the gamble works out. But when the employee fails to reach their full potential – or worse, turns over – the gamble fails.

Whether it’s on-the-go access to technology or a better solution for identifying and selecting organizational talent, making significant investments across personal and organizational levels delivers certain advantages. However, if we don’t protect those investments, we risk significant damage or loss.

Come to think of it, maybe I should buy that temporary case after all.

Blaine Gaddis
International Research Manager
Hogan Assessment Systems

Our clients often request a benchmark against which to evaluate their employees on our personality assessments. A common statement is “we want to see how our sales force compares to other sales forces.” Some are even more specific, such as “we want to benchmark our managers against other managers working in large, pharmaceutical companies in East Asia.” Although we always want to be accommodating, we tend to shy away from creating such benchmarks for multiple reasons. The most important reason is that, devoid of the relevant contextual information, benchmarks using personality assessment are not very useful and can lead to false conclusions.

The desire for benchmarks in personality assessment is likely a product of normal business thinking. As organizations, we are always comparing ourselves to our competition to know where we stand. Stock price, annual revenue, number of products sold, customer retention rate and satisfaction ratings, safety violations, and employee turnover rates. These are all things that indicate a clear winner when we are benchmarking ourselves against our competitors. Relative to other organizations, we want higher stock value, revenue, sales, and customer ratings. We want lower safety violations and employee turnover.

Being better than your competition is not as clear when it comes to personality assessment. The general bias when interpreting personality assessment scores is that higher scores are better. This bias stems from our orientation to other standardized testing; we know that higher scores are generally better when evaluating ourselves on tests of intelligence, aptitude, skills, and abilities. The reality is that there are strengths and shortcomings associated with any score on any scale from most decent personality assessments. No one gets off the hook. Although people who score high on our Ambition scale have more natural drive and goal-orientation than the population, they may not recognize when it is time to let others lead and may be perceived as poor team players. For example, let’s assume that an oil company has requested a personality benchmark for their safety inspectors relative to other safety inspectors. We create this benchmark, present it to the oil company, and they see that their safety inspectors are higher on a scale measuring creativity than other safety inspectors. Higher is better, right? Not really. Safety inspectors shouldn’t be creative. Instead, good safety inspectors should be more practical, less creative…following the letter of the law. In this case, lower scores on the creativity scale would be better.

Another bias is that client organizations (and people, in general) often assume that any comparisons we, as experts, make must be relevant and attended to. When ESPN reports on the average number of water bottles kept on the sidelines by World Cup teams, we assume that this has some bearing on the success of the team, or else why would they have reported it? Naturally, the googlesphere will activate with fans searching to find out how many water bottles their team keeps on the sidelines. Companies could compare themselves on the number of staplers they have but what does that matter as a metric of business performance? Further, is it better to have more or less staplers? When it comes down to it, not all comparisons are meaningful. As it applies to personality assessment, not all personality characteristics are related to performance (relevant) in a given job or industry. How important for job performance is it for a custodian to be sensitive to the feelings of others? It is likely irrelevant, but we can still create a benchmark that compares the interpersonal sensitivity of one school’s custodians to all other school custodians. When we provide this benchmark, the client organization is likely to assume it has relevance, or else why would we have reported it?

We feed into these biases when we provide benchmarks without the necessary contextual framework. When an organization sees their employees’ personality data plotted against an industry/subgroup benchmark, they may make inaccurate inferences because a) we always assume higher is better and b) we assume that all reported comparisons are important. We can create meaningful benchmarks with personality assessment but it takes the right kind of data. If we have sufficient data, we are able to indicate what personality scales have the strongest and most consistent relationships with performance in these jobs; hence, relevance. In addition, we can indicate whether higher, lower, or even moderate scores are better. Obviously, the thinner we slice the world (e.g., marketing coordinators at mid-western hotel chains), the less likely it is that we, or anyone, will have sufficient evidence to reliably indicate predictors of performance and create good benchmarks. Further, such specific benchmarks lack relevance and applicability. Truth be told, there probably isn’t much difference in personality between marketing coordinators at mid-western hotel chains and marketing coordinators in general. With all stated caveats, we can help clients benchmark their employees on characteristics that matter for performance, as long as it is a worthwhile endeavor and we have the data to do it. Even then, we should not be creating stand-alone graphs or tables that simply plot group averages against each other. We must tell the story around and behind the graph or table to focus attention appropriately. Only then can a personality benchmark be meaningful, impactful, and actionable.

Because items on personality assessments have no correct or incorrect answers, users must apply norms to interpret scores. Norms exist largely behind the scenes, and aren’t as sexy as other issues. As a result, they receive little attention relative to other issues, and many do not consider their importance. However, norms can determine whether a person gets a job, receives admittance to academic programs, or is deemed clinically “disordered.” These examples show why test takers should care about norms, but why should test users care?

The answer is simple – validity. Put simply, validity concerns the accuracy of assessment-based decisions, or whether the instrument does what it’s designed to do. Because norms inform decisions based on a person’s score, they impact validity. As an example, consider two assessments designed for adult working populations. The only difference between the two instruments is their norms. Norms for one instrument come from an anonymous sample of Internet users. Norms for the other assessment come from working adults sampled based on labor force and demographic estimates. Because both assessments are intended for adult working populations, the validity of decisions based on these assessments depends on how well the scores reflect the intended audience. Here, norms for one assessment give users confidence in making decisions based on an apples-to-apples comparisons; norms for the other do not.

Hogan takes great pride in the quality and continued accuracy of our assessment norms. We continuously monitor our norms, completing incremental updates as necessary. More importantly, we obtain proportionate representation across occupational and demographic categories to make certain that our norms accurately reflect intended audiences. Internationally, we also develop local norms to give our clients and partners confidence that results reflect intended audiences in local populations. Collectively, these efforts ensure that Hogan remains an industry-leader in assessment norming practices.

Blaine Gaddis
International Research Manager
Hogan Assessment Systems