Skip to main content Skip to main navigation menu Skip to site footer
Projects and researches
Published: 2022-06-10

Face Validity in the Context of Personality Assessment: An Experimental Approach

University of Bucharest
face validity visceral credibility feedback perceived accuracy perceived usefulness

Abstract

In the online environment, there are free psychological questionnaires available for anyone to complete, but test-taker perceptions have not been studied for these tools. Starting from the distinction between cognitive and visceral credibility and from the model of applicants’ reactions in the selection process, this paper has the objective of evaluating whether manipulating the feedback format can predict perceptions related to face validity, accuracy and usefulness. The secondary objective was to test the predictor role of face validity in relation to perceived accuracy, respectively to perceived usefulness. To achieve these objectives, the participants have been randomly assigned in two groups, depending on the feedback format of the IPIP questionnaire. The results indicate that feedback format does not predict any of the three variables, but that face validity significantly predicts the perceptions related to accuracy and those related to usefulness. These results suggest the fact that respondents do not perceive differently the face validity, the accuracy and the usefulness of a test depending on the visual aspect of feedback. Face validity proves, however, to be a relevant predictor even outside of personnel selection.

References

  1. Anderson, N., & Witvliet, C. (2008). Fairness reactions to personnel selection methods: An international comparison between the Netherlands, the United States, France, Spain, Portugal, and Singapore. International Journal of Selection and Assessment, 16(1), 1-13.
  2. Anglim, J., Horwood, S., Smillie, L. D., Marrero, R. J., & Wood, J. K. (2020). Predicting psychological and subjective well-being from personality: A meta-analysis. Psychological Bulletin, 146(4), 279–323.
  3. Barańczuk, U. (2019a). The five factor model of personality and sense of coherence: A meta-analysis. Journal of Health Psychology, 1-14.
  4. Barańczuk, U. (2019b). The Five Factor Model of personality and social support: A meta-analysis. Journal of Research in Personality, 81, 38-46.
  5. Baylor, A. L. (2011). The design of motivational agents and avatars. Educational Technology Research and Development, 59(2), 291-300.
  6. Berberick, S. N., & McAllister, M. P. (2016). Online Quizzes as Viral, Consumption-Based Identities. International Journal of Communication, 10, 3423–3441.
  7. Bogg, T., & Roberts, B. W. (2004). Conscientiousness and health-related behaviors: a meta-analysis of the leading behavioral contributors to mortality. Psychological Bulletin, 130(6), 887-919.
  8. Buecker, S., Maes, M., Denissen, J. J., & Luhmann, M. (2020). Loneliness and the Big Five Personality Traits: A Meta‐analysis. European Journal of Personality, 34(1), 8-28.
  9. Chan, D., Scmitt, N., Jennings, D., Clause, C. S., & Delbridge, K. (1998). Applicant Perceptions of Test Fairness: Integrating Justice and Self-Serving Bias Perspectives. International Journal of Selection and Assessment, 6(4), 232–239.
  10. Choi, W., & Stvilia, B. (2015). Web credibility assessment: Conceptualization, operationalization, variability, and models. Journal of the Association for Information Science and Technology, 66(12), 2399-2414.
  11. Colineau, N., & Paris, C. (2010). Talking about your health to strangers: understanding the use of online social networks by patients. New Review of Hypermedia and Multimedia, 16(1-2), 141-160.
  12. Costa, P. T., Jr., & McCrae, R. R. (2014). The NEO Inventories. In R. P. Archer & S. R. Smith (Eds.), Personality assessment (p. 229–260). Routledge/Taylor & Francis Group.
  13. Curry, K. T., & Hanson, W. E. (2010). National Survey of Psychologists’ Test Feedback Training, Supervision, and Practice: A Mixed Methods Study. Journal of Personality Assessment, 92(4), 327–336.
  14. Dineen, B. R., Ling, J., Ash, S. R., & DelVecchio, D. (2007). Aesthetic properties and message customization: Navigating the dark side of web recruitment. Journal of Applied Psychology, 92(2), 356-372.
  15. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.
  16. Fayard, J. V., Roberts, B. W., Robins, R. W., & Watson, D. (2012). Uncovering the affective core of conscientiousness: The role of self‐conscious emotions. Journal of Personality, 80(1), 1-32.
  17. Finn, S. E., & Tonsager, M. E. (1997). Information-gathering and therapeutic models of assessment: Complementary paradigms. Psychological Assessment, 9(4), 374-385.
  18. Fogg, B. J., Soohoo, C., Danielson, D. R., Marable, L., Stanford, J., & Tauber, E. R. (2003). How do users evaluate the credibility of Web sites? Proceedings of the 2003 Conference on Designing for User Experiences - DUX ’03, 1-15.
  19. Furnham, A., & Varian, C. (1988). Predicting and accepting personality test scores. Personality and Individual Differences, 9(4), 735-748.
  20. Giles, D. C., & Newbold, J. (2011). Self-and other-diagnosis in user-led mental health online communities. Qualitative Health Research, 21(3), 419-428.
  21. Gilliland, S. W., Groth, M., Baker, R. C., Dew, A. F., Polly, L. M., & Langdon, J. C. (2001). Improving Applicants’ Reactions To Rejection Letters: An Application of Fairness Theory. Personnel Psychology, 54(3), 669-703.
  22. Goldberg, L. R. (1992). The development of markers for the Big-Five factor structure. Psychological Assessment, 4(1), 26-42.
  23. Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. C. (2006). The International Personality Item Pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84-96.
  24. Grand, J. A., Ryan, A. M., Schmitt, N., & Hmurovic, J. (2010). How far does stereotype threat reach? The potential detriment of face validity in cognitive ability testing. Human Performance, 24(1), 1-28.
  25. Hall, J. A., Schlegel, K., Castro, V. L., & Back, M. (2019). What laypeople think the Big Five trait labels mean. Journal of Research in Personality, 78, 268-285.
  26. Harari, M. B., Reaves, A. C., Beane, D. A., Laginess, A. J., & Viswesvaran, C. (2018). Personality and expatriate adjustment: A meta‐analysis. Journal of Occupational and Organizational Psychology, 91(3), 486-517.
  27. Harland, L. K. (2003). Using personality tests in leadership development: Test format effects and the mitigating impact of explanations and feedback. Human Resource Development Quarterly, 14(3), 285-301.
  28. Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112.
  29. Hausknecht, J. P., Day, D. V., & Thomas, S. C. (2004). Applicant reactions to selection procedures: An updated model and meta‐analysis. Personnel Psychology, 57(3), 639-683.
  30. Holden, R. B. (2010). Face validity. In I. B. Weiner & W. E. Craighead (Eds.), The Corsini Encyclopedia of Psychology (4th ed., pp. 637-638). Hoboken, NJ: Wiley.
  31. Holtz, B. C., Ployhart, R. E., & Dominguez, A. (2005). Testing the rules of justice: The effects of frame‐of‐reference and pre‐test validity information on personality test responses and test perceptions. International Journal of Selection and Assessment, 13(1), 75-86.
  32. Holtz, B. C., Ployhart, R. E., & Dominguez, A. (2005). Testing the rules of justice: The effects of frame‐of‐reference and pre‐test validity information on personality test responses and test perceptions. International Journal of Selection and Assessment, 13(1), 75-86.
  33. Holzwarth, M., Janiszewski, C., & Neumann, M. M. (2006). The influence of avatars on online consumer shopping behavior. Journal of Marketing, 70(4), 19-36.
  34. Iliescu, D., Popa, M., & Dimache, R. (2015). Adaptarea românească a Setului International de Itemi de Personalitate: IPIP-Ro [The Romanian adaptation of the International Personality Item Pool: IPIP-Ro]. Psihologia Resurselor Umane, 13(1), 83-112.
  35. International Personality Item Pool: A Scientific Collaboratory for the Development of Advanced Measures of Personality Traits and Other Individual Differences (http://ipip.ori.org/). Internet Web Site.
  36. Jin, S. A. A., & Bolebruch, J. (2009). Avatar-based advertising in Second Life: The role of presence and attractiveness of virtual spokespersons. Journal of Interactive Advertising, 10(1), 51-60.
  37. Kern, M. L., & Friedman, H. S. (2008). Do conscientious individuals live longer? A quantitative review. Health Psychology, 27(5), 505-512.
  38. Malouff, J. M., Thorsteinsson, E. B., Schutte, N. S., Bhullar, N., & Rooke, S. E. (2010). The five-factor model of personality and relationship satisfaction of intimate partners: A meta-analysis. Journal of Research in Personality, 44(1), 124-127.
  39. McCrae, R. R., & Costa, P. T. (2003). Personality In Adulthood: A Five-Factor Theory Perspective (Second Edition). Hoboken: Guilford Publications.
  40. McCrae, R. R., & John, O. P. (1992). An Introduction to the Five‐Factor Model and Its Applications. Journal of Personality, 60(2), 175-215.
  41. McGloin, R., Nowak, K. L., & Watt, J. (2014). Avatars and Expectations: Influencing Perceptions of Trustworthiness in an Online Consumer Setting. PsychNology Journal, 1(1-2), 7-28.
  42. McGrew, S., Breakstone, J., Ortega, T., Smith, M., & Wineburg, S. (2018). Can Students Evaluate Online Sources? Learning from Assessments of Civic Online Reasoning. Theory & Research in Social Education, 46(2), 165-193.
  43. Mull, I., Wyss, J., Moon, E., & Lee, S.-E. (2015). An exploratory study of using 3D avatars as online salespeople. Journal of Fashion Marketing and Management: An International Journal, 19(2), 154–168.
  44. Mutén, E. (1991). Self-reports, spouse ratings, and psychophysiological assessment in a behavioral medicine program: An application of the five-factor model. Journal of Personality Assessment, 57(3), 449-464.
  45. Myers, I., & McCaulley, M. (1985). Manual: A Guide to the Development and Use of the Myers-Briggs Type Indicator. Palo Alto, CA: Consulting Psychologists.
  46. NERIS (n.d.). 16 Personalities. Retrieved from: https://www.16personalities.com/
  47. Nevo, B. (1985). Face Validity Revisited. Journal of Educational Measurement, 22(4), 287–293.
  48. Nevo, B., & Sfez, J. (1985). Examinees’ Feedback Questionnaires. Assessment and Evaluation in Higher Education, 10(3), 236-248.
  49. Nowak, K. L., & Rauh, C. (2005). The influence of the avatar on online perceptions of anthropomorphism, androgyny, credibility, homophily, and attraction. Journal of Computer-Mediated Communication, 11(1), 153-178.
  50. Nowak, K. L., Hamilton, M. A., & Hammond, C. C. (2009). The effect of image features on judgments of homophily, credibility, and intention to use as avatars in future interactions. Media Psychology, 12(1), 50-76.
  51. Oostrom, J. K., Born, M. P., Serlie, A. W., & Van Der Molen, H. T. (2010). Effects of individual differences on the perceived job relatedness of a cognitive ability test and a multimedia situational judgment test. International Journal of Selection and Assessment, 18(4), 394-406.
  52. Park, A., Conway, M., & Chen, A. T. (2018). Examining thematic similarity, difference, and membership in three online mental health communities from Reddit: a text mining and visualization approach. Computers in Human Behavior, 78, 98-112.
  53. Prinzie, P., Stams, G. J. J., Deković, M., Reijntjes, A. H., & Belsky, J. (2009). The relations between parents’ Big Five personality factors and parenting: A meta-analytic review. Journal of Personality and Social Psychology, 97(2), 351-362.
  54. Quinn, S. M. (2017). Feeding our Identities:BuzzFeed Quizzes as a Tool for Personal Identification in the Social Digital Age (Unpublished master's dissertation). Roger Williams University, Bristol, Rhode Island.
  55. Reich, Z. (2011). Source credibility and journalism: Between visceral and discretional judgment. Journalism Practice, 5(1), 51-67.
  56. Reinecke, K., Yeh, T., Miratrix, L., Mardiko, R., Zhao, Y., Liu, J., & Gajos, K. Z. (2013). Predicting users’ first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13, 2049-2058.
  57. Rieh, S. Y., & Danielson, D. R. (2007). Credibility: A multidisciplinary framework. Annual Review of Information Science and Technology, 41(1), 307-364.
  58. Robins, D., & Holmes, J. (2008). Aesthetics and credibility in web site design. Information Processing & Management, 44(1), 386-399.
  59. Robins, D., Holmes, J., & Stansbury, M. (2009). Consumer health information on the Web: The relationship of visual design and perceptions of credibility. Journal of the American Society for Information Science and Technology, 61(1), 13–29.
  60. Ryan, A. M., & Huth, M. (2008). Not much more than platitudes? A critical look at the utility of applicant reactions research. Human Resource Management Review, 18(3), 119-132.
  61. Secolsky, C. (1987). On the direct measurement of face validity: A comment on Nevo. Journal of Educational Measurement, 24(1), 82-83.
  62. Smith, S. R., Wiggins, C. M., & Gorske, T. T. (2007). A survey of psychological assessment feedback practices. Assessment, 14(3), 310-319.
  63. Smither, J. W., Reilly, R. R., Millsap, R. E., Pearlman, K., & Stoffey, R. W. (1993). Applicant reactions to selection procedures. Personnel Psychology, 46(1), 49-76.
  64. Steiner, D. D., & Gilliland, S. W. (1996). Fairness reactions to personnel selection techniques in France and the United States. Journal of Applied Psychology, 81(2), 134-141.
  65. Stoet, G. (2010). PsyToolkit - A software package for programming psychological experiments using Linux. Behavior Research Methods, 42(4), 1096-1104.
  66. Stoet, G. (2017). PsyToolkit: A novel web-based method for running online questionnaires and reaction-time experiments. Teaching of Psychology, 44(1), 24-31.
  67. Tuch, A. N., Bargas-Avila, J. A., & Opwis, K. (2010). Symmetry and aesthetics in website design: It’s a man’s business. Computers in Human Behavior, 26(6), 1831–1837.
  68. Vogel, D. L., Wester, S. R., Wei, M., & Boysen, G. A. (2005). The Role of Outcome Expectations and Attitudes on Decisions to Seek Professional Help. Journal of Counseling Psychology, 52(4), 459–470.
  69. Wilmot, M. P., Wanberg, C. R., Kammeyer-Mueller, J. D., & Ones, D. S. (2019). Extraversion advantages at work: A quantitative review and synthesis of the meta-analytic evidence. Journal of Applied Psychology, 104(12), 1447–1470.

How to Cite

Neagu, A. (2022). Face Validity in the Context of Personality Assessment: An Experimental Approach . Studia Doctoralia, 13(1), 69–81. https://doi.org/10.47040/sdpsych.v13i1.144