Performace-based personality tests in the assessment and selection of managers

Christer Sandahl, Department of Learning, Informatics, Management and Ethics (LIME), Karolinska Institutet, Stockholm
Eva Birgerson , Sandahl Partners Stockholm AB , Stockholm
Harald Janson, Psykologiska Institusjonen, Oslo Universitet
Anna-Lena Sundlin, Sandahl Partners Gotland AB, Ljugarn
Maria Åkerlund, Sandahl Partners Stockholm AB , Stockholm

Abstract

In today’s rapidly changing work life there is a growing need for managers who can contribute to trust and a belief in the meaning of work. It is argued that performance-based personality tests, in the hands of skilled psychologists, have the potential to identify managers with good judgment, social skills and a will to transform their organizations by a process of long-term sustained development. The prognostic validity of a single test is of less importance than its construct validity and the validity of the decisions that are based on the whole assessment procedure. In a pilot study using such an approach, significant correlations were observed between the results of a market survey and holistic assessments of social skills, judgment and motivation. In order for psychologists to contribute constructively in the selection process of managers they need a clear picture of the context, the goal of the organization and the role. A conceptual tool to describe both role demands and personal strengths and weaknesses is presented. When the assessment was put into a context as described, clients reported that the feed back not only helped them to understand themselves better, but a large proportion of cases also changed their view of their future career.

Key words: personality test, performance-based, projective, assessment, selection, manager.

Accountability and values

When people work together without problems it usually goes unnoticed. A peculiar aspect of human existence is that when mutual relational needs are met, we do not seem to spend much time analysing what made it work so well. Many work groups function well and we tend to attribute this to good social relations or that the demands of the task and the group’s capacity to cope with it were in balance.

When things go wrong and create frustration we are often quick to attribute it to poor leadership. Followership is seldom questioned and maybe rightly so. It is the task of the leader to provide a context for good work to take place. If this fails the leader should be held accountable. This is one example of the rules of a managerial hierarchy. We believe that this is a rule that can also be applied in a post-bureaucratic organisation or when transformational leadership is the ideal. A good leader might question followership, but must then ask him/herself how he or she can improve the situation in order for the employees to be able to be led in a constructive way.

That the manager is held accountable for the output of the work done by the employees on the level below is a simple principle that creates safety and contributes to trust in the company and in the management. The degree of trust in a company and its management is related both to profitability and health (Jaques, 1996). Other factors related to trust are e.g. equitable payment, fairness and justice, knowing that ones work has a meaning for others, to be able to use one’s full potential in the work, to contribute to the company’s development and an understandable organisational structure. If such factors are essential for profitability and the wellbeing of the employees, it becomes clear that management and leadership to a large extent is about values, especially openness and respect.

Personality questionnaires

With this in mind, it is hard to understand the simplistic use of self-report personality questionnaires in the assessment and selection of managers that has lately become frequent in business. How can one expect that a profile of the “Big Five” personality factors or Myers Briggs Type Indicator could be related to success as a manager? Of course a high score on for example Neuroticism might be used as a negative criteria, but how often do you find such a case in a population of successful job seekers? All evidence point to the fact that a manager can be successful irrespective of trait characteristics of this kind. Factors such as extroversion or introversion, intuitive or thinking style etc. include no prognostic information. One exception is tests of integrity and conscientiousness to predict success in job training programs or general work performance. These factors are of less relevance when hiring a manager.

The frequent use of such psychological instruments to measure character traits (usually without any involvement from a professional psychologist) stands in sharp contrast to the lack of empirical support to this practice as it is applied in the selection of managers. People who use these kinds of instruments in assessment and selection of managers have little understanding of the complexities involved.

Precisely the fact that management is such a complex task explains the repeatedly observed connection between job performance and results on tests of general mental aptitude. It has been observed that the higher the complexity of the job, the higher the validity of such tests (Schmidt and Hunter, 1998). In other words it seems to be justified to use intelligence tests but not personality questionnaires in the assessment and selection of managers. Does this mean that personality in a broader sense has no prognostic value at all? Such a conclusion seems to be counterintuitive. It is obvious that there is a connection between for instance values as they are lived and practiced in daily life and personality in a broader (and deeper?) sense.

The primary function of personality assessment in this context

The picture becomes clearer if one takes into account phenomena that are familiar to clinicians and psychotherapists. Managers with paranoid ideation or psychopathic features might contribute to short-term profitability but they will certainly not build trust. On the contrary, people in their surrounding will feel ill at ease and sometimes become sick, competent people will leave the company etc. The same applies to managers with for example strong narcissistic needs, object relations with a low degree of differentiation, primitive defence mechanisms or poor functioning reality testing. From this perspective one might argue that the purpose of personality testing in the assessment of managers is not to find the most suitable person, but to de-select those who with a high degree of certainty would become harmful to the work environment if they had to function in the role as manager.

If one defines that the primary function of personality testing in this context would be to identify negative, not positive, criteria, then it becomes clear that personality questionnaires are of little value. If one responds honestly a questionnaire will reflect the self-image. It is a well-known fact that people, if they are eager to get a certain job, partly unconsciously adjust their response style to what they feel is expected of them. A greater problem with this type of information, however, is that people with problems in personality functioning usually lack self-knowledge. Otherwise they would hardly have applied for a job as a manager. In fact, many want to become a manager to compensate for a deep and hidden insecurity. People with lack of self-insight do not see themselves as other people perceive them. Therefore a self-report measure is of little help in the assessment, except for giving information about self-image.

Performance-based personality tests

This is the reason why performance-based assessment techniques might be more useful in the selection of managers. Tests of general mental aptitude belong to this category. Considering the established validity of these tests one does not have to argue for the use of them. But they will not help to identify persons with personality problems. Assessment centres are also performance based and have proven to have acceptable validity. However, they do correlate highly with intelligence tests and the gain in validity when added to a test battery of general mental aptitude seems to be almost non existent (Schmidt and Hunter, 1998). Since assessment centres are usually run by non-professionals, the question is if problems in personality functioning, other than the obvious cases which would be detected in an interview anyhow, would be noticed by the observers.

The term “performance-based personality test” was first used and defined as follows in an official report from the American Psychological Association (Meyer et al. 1998): “Performance-based personality tests elicit information about behaviour in unstructured settings or information related to implicit dynamics and underlying templates of perception and motivation” (p.16). Performance-based personality tests commonly used in Sweden are for example The Rorschach Comprehensive System (Exner, 2003), Zulliger (Mattlar et al., 1990), The Object Relations Technique (Phillipson, 1955), The Tree test (Bolander, 1977), The psychological interpretation of handwriting (Sandahl and Wallner, 1995), Drawing Completion (Wass and Mattlar, 2000), The Defence Mechanism Test (Kragh, 1985) and The Wheel Questionnaire (Shalit, 1984). A disadvantage connected to the use of the tests mentioned here, is that they make high demands on the user. They require much training (usually several years) in order for a professional to master the interpretation. Even for an experienced psychologist the evaluation of the test results is time consuming.

On the other hand, one obvious advantage is that performance based tests are difficult, if not impossible, to manipulate. They provide rich information about weaknesses as well as strengths and it is usually possible to work within a hypothesis-testing frame.

The prognostic validity of single test varies a lot and seems to be situation specific. A single test, be it handwriting or NEO, can have high prognostic value in one situation and none in another (Hofsommer, 1973; Holt, 1978). This does not mean that the single test has no value, only that the information in the test result must be put into context. This is the work of a professional psychologist, who as part of the procedure integrates all test data in a contextually differentiated fashion.

The feed back to the client (the job seeker) can therefore be enriching and contribute to self-discoveries. In a follow-up study (Eidmark, 1998) of 400 clients at Sandahl Partners (a Swedish consultation firm) 81 % expressed that the feed-back from the psychologist had increased their self-understanding, and as many as 49 % were of the opinion that this two hour conversation had changed their view of their future career.

These tests are often referred to as “projective techniques”, but even in modern applications of a classical “projective” test such as the Rorschach, the focus is on the cognitive-perceptual decision making process. Interpretations of the content (the projections) have shown to have poor validity. Such interpretations sometimes disclose more about the interpreter than about the person being tested. In textbooks on occupational psychology, however, these tests are still referred to as “projective tests”. Within the scientific literature of occupational psychology there is little support for the prognostic value of such tests. Interestingly enough, reliability is seldom questioned, not even for such a controversial method as the psychological interpretation of handwriting (Neter and Ben-Shakhar, 1989).

If one researches the scientific literature within clinical psychology another picture emerges. The problems encountered within this field are different from the ones in occupational psychology in general. There is, however, a striking similarity in identifying personality problems and making recommendations in a selection situation and in making a case formulation and treatment suggestions in a clinical situation. In both cases the test is a tool, such as a stethoscope or a blood test, and cannot be separated from the skill and sophistication of the clinician who draws inferences from it and then communicates it to the client and others involved in the process.

In a recent review by Meyer et al. (2001) of 125 meta-analyses on test validity and 800 samples examining multi method assessment, the authors concluded that “(a) Psychological test validity is strong and compelling, (b) psychological test validity is comparable to medical test validity, (c) distinct assessment methods provide unique sources of information, and (d) clinicians who rely exclusively on interviews are prone to incomplete understanding.” (p.128) They also argue that future research in this area should move beyond the examination of single tests, to focus more on the role of psychologists who use tests as helpful tools. Empirically based theories, models and concepts are also tools in the hands of a psychologist. From this perspective one might add that the construct validity of single tests is of great importance and should also be addressed in future investigations.

In order to be able to interpret performance-based tests, one must have a theory of cognition and personality. It would be to go beyond the aim of this article to expand on this theme. However, theories are good servants but bad masters! Many established psychodynamic theories have little empirical support. Some test methods (e.g. Rorschach and the Defence Mechanism Test) have for example been interpreted within the frame of traditional psychoanalysis. This does not mean that such a method cannot be used within a for example an object-relational or self-psychological theoretical context, but one must be cautious regarding conceptualisations. Although some of these methods are frequently applied within a psychodynamic context, the Rorschach for instance was originally empirically derived and developed as a non-theoretical method, which is how it is presently used within the Comprehensive System (Exner, 2003).

Validation of the assessment system. A pilot study

It would be of interest to evaluate the prognostic value of the whole assessment system and the decisions that were made on the basis of the information gained from interviews and tests. Such an effort was made at Sandahl Partners some years ago and reported at an international conference (Janson and Sandahl, 1996). Psychological assessment was carried out as part of the process of recruiting 130 regional managers for a major Swedish bank. A battery of mental aptitude and performance-based personality tests was group-administered to applicants later seen by psychologists in individual interviews. Twelve psychologists were involved in the work. The interviewing psychologist rated the applicants for social skills, motivation and judgment on nine-point scales. Based on the total information available, applicants were later categorised in a global rating referring to the type of managerial task in the corporation that they might be most suited for. The initial ratings and categorisations were compared to customers’ and employees’ responses to personnel and market surveys (both conducted twice, approximately twelve and 36 months after the appointments) for the two subsequent years. A large proportion of the variance (58-62% in market surveys) in the outcome measures was explained by such background factors as size of the community of the local bank that the manager was responsible for, yearly turnover and the presence of structural difficulties in some local banks due to historical background. When the influence of the background variables were controlled for, the personality factors correlated significantly (except in one case) with the results on the market surveys. (See table 1).

Table 1. Associations between predictors and outcome variables. (Partial correlations controlling for background factors: (a) size of the community of the local bank which the manager was responsible for, (b) yearly turnover and (c) the presence of structural difficulties in some local banks due to historical background.)

Predictor variables	Outcome variables
	Personell surveys		Market surveys
	1993	1995	1993	1995
Intelligence	-.14	-.20a	-.10	.03
Social skills rating	.04	.16	.28**	.34**
Motivation rating	-0.5	.07	.21*	.12
Capacity for Judgment rating	-.14	-.07	.21*	.24**
Global ratingb	-.13	-.06	.25**	.22**

*p<.05   **p<.01 (one-tailed)
^ap<.05 (one-tailed) but in direction contrary to expectations
^b The psychologists made a global rating of suitability based on all test information available.

The correlations were small but significant and there was a consistent pattern, ruling out random significances. Restriction of range was a problem. Only about a third of the applicants were chosen for the job and in most cases because there was an expectation that they would be able to perform well. Most of the cases containing a negative assessment were excluded. In other words, the variation in the psychological scales was restricted to the positive end. Nevertheless we found significant correlations between the personality ratings and the market surveys. This becomes even more interesting in the light of the fact that there was no significant correlation with the global measure of intelligence, which in many studies has been found to be the best overall predictor of success over a range of work tasks (Schmidt and Hunter, 1998). This unexpected result was probably due to the fact that those with low scores had been de-selected early in the process. And it is our experience that low results on measures of intelligence have prognostic value. There is, however, no reason to believe that somebody with high intelligence would function better in a managerial position of this sort as compared with somebody with average intelligence.

It is also of interest that the personality factors did not correlate with the personnel surveys. To a great extent the results of this questionnaire could be interpreted as a popularity measure of the management as a whole, rather than measuring the potential of the employees. In the long run, good management should to some degree result in management popularity. However, in times when great changes, including downsizing, are required (as was the case in this context), managing for the best of the organisation as a whole can result in lower popularity.

In this case, personal characteristics of the managers seemed only to have an effect on the customers’ attitude. This association might be interpreted to indicate that the more effective managers were more successful in influencing their personnel to work with the interest of the customer in mind, (which was considered important in this context) and that the customer had noticed this. The absolute size effect might be considered to be small. However, considering that a large portion of the variance in customers’ attitudes toward the bank depend on external or historical factors, the smaller effect of personality explains an important portion of the variance left to be explained by controllable factors, and may therefore certainly be very important from the organisational point of view. In their meta-analysis Meyer et al. (2001) argue that it is often difficult to appreciate the size of the associations that are studied in psychology or encountered in daily life. This often leads to unrealistically high expectations about effect magnitudes. Compared to results from studies within medicine and other applied sciences, it became clear how challenging it is to achieve correlations that are much greater than .30. They came to the conclusion that when studying complex human behaviour psychologists should be satisfied when they can identify and replicate univariate correlations among independently measured constructs that are of the magnitude observed for e.g. antihistamine effectiveness (r=.11) or criminal history and recidivism (r=.18) etc (p.134).

The results of our study are preliminary and have not been replicated, but we interpret them as a support for the model in which we regard the professional psychologist as the key factor in the multi method assessment process.

Role, goal and context

Another aspect, which must be taken into account when using performance-based tests, is knowledge of organisational context. In order to be able to make a useful assessment, one needs a model of the actual work situation for the manager who is finally going to be selected. This is repeatedly pointed out in literature on occupational psychology, but seldom put into a theoretical perspective. Job analyses are mostly reduced to a list of demand characteristics, -often of divine dimensions! Considering the unrealistic expectations many have when hiring a new manager, it is not surprising that one becomes disappointed when confronted with human limitations. Our conclusion is that part of the task in assessment is to provide a model to describe the job demands in a realistic and useful way.

Group psychology has taught us that people reveal different aspects of their personality in different contexts. Personality is not a matter of either/or. Personality is dynamic, not static. Thus it becomes important to be able to describe the role of being a manager in such a way that one can look for those aspects of personality, which with a high probability will be most important in a specific context at a specific point in time. In order to do that one needs to understand the specific tasks, the organisation as a social system, the overall goal of the organisation and the goal of the department that is going to be managed. The role can be understood as disciplined behaviour, which is a function of the manager’s perception and understanding of the goals and the social context. The task for the psychologist is to assess whether or not a person has the resources available to perform a specified role.

In order to be able to understand the demands of the role we have developed a model in which personality characteristics are related to the social context of being a manager. In order to be able to make a meaningful assessment one has to identify a limited number of factors (four at the most) that are considered crucial in a specific job. Those factors are the ones that will be matched with the job applicant’s personality structure.

The Sandahl Partners Interactive Model

We have chosen three personality dimensions which have a common sense quality to them, namely Social skills, Judgment and Motivation. These dimensions are not in the focus in recent research on personality, but they appear repeatedly in literature on management, leadership and group psychology. They are related to such concepts as relational – or task orientation. Task orientation has one part related to judgment and one to drive. In other words the dimensions are related to emotions and object relations (social skills), cognitive functioning (judgement) and motivational psychology.

As we see it, the social context of being a manager, could be categorised in three main dimensions: (a) running and leading the daily business, (b) leading personnel and (c) leading transformation (change and development) (Bass, 1997). These dimensions create the social foundation of the model, which is best described in a 3 x 3 grid:

• Row A in the grid is an assessment of the skills and competencies required for running and developing services or business. It requires a clear view of service and organizational goals.
• Row B is an assessment of skills and competencies in managing people and working with teams.
• Row C is an assessment of skills and abilities related to managing change and development, ranging from vision and creativity to managing the pace and developmental process (transformational leadership).

By combining the three social dimensions with the three personality dimensions we arrive at the following nine cell grid. (See figure 1.)

For each cell in the grid a score between 1 and 9 can be given in relation to the identified skills and abilities of leadership role for the task, or in relation to an individual. This enables an assessment of the match between the individual and a particular role and/or the particular strengths and weaknesses of any particular person. This system can also be used for assessment of strengths and weaknesses of any particular person when working with personal and organizational development.

Figure 1. The Sandahl Partners interactive model on personality and social context in management.

Personal qualities	1	2	3
Leadership tasks	Social Skills	Judgment	Motivation
A Business activities	1A Ability to initiate, develop, and maintain business contacts and network. E.g. - To be representative and trustworthy - Ability to adapt to different social contexts	2A Ability to make judgements, correctly identify and solve problems. E.g. - Critical abilities and a sober mind - Clarity and structure - Logical/analytical capacity	3A Ability to work toward the goal with energy and efficiency. E.g. - Drive and energy - Goal orientation - Following through
B Personell management	1B Ability to create a team spirit and a good work climate. E.g. - Maintaining and developing relations to employees - Creating dialogue and managing conflict	2B Ability to assess, utilize and develop skills and resources E.g. - To be a good judge of people - Ability to separate the person from the role	3B Ability to motivate and energize others to work toward the goals E.g. - To be a good mentor/coach - Support others to take authority
C Leading trans-formation	1C Ability to create a learning environment E.g. - Inspire and encourage ideas and discussions - Patience - explaining and informing enough for people to catch up	2C Ability to hold a vision and to think in a long time perspective. E.g. - Ability to prioritize and think strategically - Ability to make judgments in unclear and stressful situations	3C Ability to initiate and pursue the course of change and business development E.g. - Curiosity, courage and creativity - Open for change and challenge

Our experience is that working together with the employer to identify the most important factors for each role is very fruitful and contributes to a deeper understanding of the most essential aspects of the role and the goal. The grid gives a good structure for analyzing the gap between the requirements of the role and the personal qualities. It confronts the employer with the limitations of human nature and contributes to realistic expectations of the new manager. As a result of the process, the role is quite often reformulated in order to fit the person. The customers always regard such an outcome as a valuable contribution.

Conclusions

Trust, openness and respect are signs of efficient leadership. Values of this kind can be given lip-service or put into practice. If they are mere words, they are of little value. Many managers in business and public organizations belong to a category of leaders that contribute more to mistrust and paranoia than trust, due to the gap between words and actions. A large group of managers “walk the talk” and of course many fall in between. It is quite clear that these groups of people have different personality in terms of e.g. ego-ideal, identity or self. However, there is no reason to believe that one would be able to differentiate managers who act as they talk on the basis of character traits as measured by common questionnaires.

In today’s rapidly changing organizations there is a growing need for managers who can contribute to mutual trust and a belief in the meaning of work. Performance based tests have the potential to identify managers with a good judgment, social skills and a will to contribute to the transformation of modern organizations into good and healthy work places with a sustainable development. Another advantage is that they provide rich information in the hands of a skilled psychologist. When this information is used wisely it will contribute not only to the selection process, but also both to the development of the person who is tested and to the organization as a whole.

References

Bass, B.M. (1997). Does the transactional-transformational leadership paradigm transcend organizational and national boundaries? American Psychologist, 52,2,130-39.

Bolander, K. (1977). Assessing personality through tree drawings. New York: basic Books.

Eidmark, L. (1998). Psykologisk testning vid rekrytering och urval. En studie över individers upplevelse av att genomföra Personlig ResursAnalys hos Sandahl Partners AB, Psykologexamensarbete, Psykologiska Institutionen, Stockholms Universitet: Stockholm.

Exner, J.E., Jr. (2003). The Rorschach: A Comprehensive System. Basic foundations and principles of interpretations. Vol.1 (4th ed.). New York: Wiley.

Janson, H. & Sandahl, C. (1996). Psychological assessment as part of the recruitment process: Comparison with indicators of subsequent performance of bank managers. (A 36-month follow-up study.) Paper presented at the Midwinter Meeting of the Society for Personality Assessment, Denver, Colorado, March 6-9, 1996.

Jaques, E. (1996). Requisite organization. A total system for effective managerial organization and managerial leadership for the 21st century. Arlington, VA: Cason Hall & Co.

Hofsommer, W. (1973). Untersuchungen zur reliabilität und validität schriftpsychologischer diagnosen. Diss: Rheinischen Friedrich – Wilhelms Universität, Bonn.

Holt, R.R. (1978). Methods in Clinical psychology. Volume 2 Prediction and Research. New York: Plenum Press

Kragh, U. (1985). The Defence Mechanism Test. Manual. Stockholm: Persona.

Mattlar, C-E., Sandahl, C., Lindberg, S., Lehtinen, V., Carlsson, A., Vesala, P. & Mahmood, Z. (1990). Methodological issues associated with the application of the comprehensive system when analysing the Zulliger, and the structural resemblance between the Zulliger and the Rorschach. British Journal of Projective Psychology, 35, 2, 7-27.

Meyer, G.J., Finn, S.E., Eyde, L.D., Kay, G.G., Moreland, K.L. Dies, R.R., Eisman, E.J. Kubiszyn, T.W. & Reed, G.M. (2001). Psychological testing and psychological assessment. A review of evidence and issues. American Psychologist, 56, 2, 128-165.

Meyer, G.J., Finn, S.E., Eyde, L.D., Kay, G.G., Kubiszyn, T.W., Moreland, K.L., Eisman, E.J. & Dies, R.R. (1998). Benefits and costs of psychological assessment in healthcare delivery: Report of the board of professional affairs Psychological Assessment Work Group, Part 1. Washington DC: American Psychological Association.

Neter, E. & Ben-Shakhar, G. (1989). The predictive validity of graphological inferences: A meta-analytic approach. Personality and Individual Differences, 10, 737-745.

Phillipson, H. (1955). The object relations technique. London: Tavistock Publications.

Sandahl, C. & Wallner, T. Does handwriting analysis qualify as a method for personality assessment? British Journal of Projective Psychology, 1995; 40(1): 40-55.

Schmidt, F.L. & Hunter, J.E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin,124, 2, 262-274.

Shalit, B., & Carlstedt, L. (1984). The Perception of Enemy Threat - A Method for Assessing the Coping Potential. National Defence Research Institute Report C 55063-H3, Stockholm.

Wass, T. & Mattlar, C-E. (2000). Warteggs teckningstest. Manual. Stockholm: Psykologiförlaget.