Drawing from the experiences of the authors in developing, conducting, and evaluating sport psychology interventions, several considerations are highlighted and recommendations offered for effective psychometric assessment. Using the Test of Performance Strategies (TOPS; Thomas, Murphy, & Hardy, 1999) as a working example, opportunities for bias to undermine a measure’s validity and reliability are discussed with reference to a respondent’s four cognitive processes: (a) comprehension, (b) retrieval, (c) decision-making, and (d) response generation. Further threats to an instrument’s psychometric properties are highlighted in the form of demand characteristics athletes perceive in the environment. With these concerns in mind, several recommendations are made relating to the process of questionnaire administration and how possible compromises to the psychometric soundness of measures used in applied interventions can be minimized.
|Journal||The Sport Psychologist|
|Publication status||Published - 2012|
Bibliographical noteReference text: Anderson, A.G., Miles, A., Mahoney, C., & Robinson, P. (2002). Evaluating the effectiveness
of applied sport psychology practice: Making the case for the case study approach. The Sport Psychologist, 16, 432–453.
Baumeister, R.F. (2010). The self. In R.F. Baumeister & E.J. Finkel (Eds.), Advanced social psychology: The state of science (pp. 139–175). New York, NY: Oxford University Press.
Beckmann, J., & Kellmann, M. (2003). Procedures and principles of sport psychological assessment. The Sport Psychologist, 17, 338–350.
Brener, N.D., Billy, J.O.G., & Grady, W.R. (2003). Assessment of factors affecting the validity of self-reported health-risk behavior among adolescents: Evidence from the scientific literature. The Journal of Adolescent Health, 33, 436–457.
Brewer, B.W., & Shillinglaw, R. (1992). Evaluation of a psychological skills training workshop for male intercollegiate lacrosse players. The Sport Psychologist, 6, 139–147.
Choi, B.C.K., & Pak, A.W.P. (2005). A catalogue of biases in questionnaires. Preventing Chronic Disease, 2, 1–13.
Cohen, A.B., Tenenbaum, G., & English, R.W. (2006). Emotions and golf performance: An IZOF-based applied sport psychology case study. Behavior Modification, 30, 259–280.
Conroy, D.E., Kaye, M.P., & Schantz, L.H. (2008). Quantitative research methodology. In T.S. Horn (Ed.), Advances in sport psychology (3rd ed., pp. 15–30). Leeds: Human Kinetics.
Dietrich, H., & Ehrlenspiel, F. (2010). Cognitive interviewing: A qualitative tool for improving questionnaires in sport science. Measurement in Physical Education and Exercise Science, 14, 51–60.
Duda, J.L., Cumming, J., & Balaguer, I. (2005). Enhancing athletes’ self regulation, task involvement, and self determination via psychological skills training. In D. Hackfort, J.L. Duda, & R. Lidor (Eds.), Handbook of applied sport psychology research (pp. 143–165). Morgantown, WV: Fitness Information Technology.
Filby, W.C.D., Maynard, I.W., & Graydon, J.K. (1999). The effect of multiple-goal strategies on performance outcomes in training and competition. Journal of Applied Sport Psychology, 11, 230–246.
Fletcher, D., & Hanton, S. (2001). The relationship between psychological skills usage and competitive anxiety responses. Psychology of Sport and Exercise, 2, 89–101.
Gravetter, F.J., & Forzano, L.B. (2008). Research methods for the behavioral sciences (Vol. 3, 3rd ed.). Belmont, CA: Wadsworth Cengage Learning.
Hagger, M.S., & Chatzisarantis, N.L.D. (2009). Assumptions in research in sport and exercise psychology. Psychology of Sport and Exercise, 10, 511–519.
Hardy, J. (2006). Speaking clearly: A critical review of the self-talk literature. Psychology of Sport and Exercise, 7, 81–97.
Hardy, L., Roberts, R., Thomas, P.R., & Murphy, S.M. (2010). Test of performance strategies (TOPS): Instrument refinement using confirmatory factor analysis. Psychology of Sport and Exercise, 11, 27–35.
Holland, M.J.G., Sharp, L., Woodcock, C., Cumming, J., & Duda, J.L. (2010a, October). Evaluating MST research procedures: Observations, reflections, and progressions. In J. L. Duda (Discussant) An action research approach to mental skills training: Experience, evaluation and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI.
Holland, M.J.G., Sharp, L., Woodcock, C., Cumming, J., & Duda, J.L. (2010b). Validity and reliability of the Behavioral Regulation in Sport Questionnaire (BRSQ) with youth athletes. Journal of Sport & Exercise Psychology, 32, s175–s176.
Howorka, K., Pumprla, J., Schlusche, C., Wagner-Nosiska, D., Schabmann, A., & Bradley, C. (2000). Dealing with ceiling baseline treatment satisfaction level in patients with diabetes under flexible, functional insulin treatment: Assessment of improvements in treatment satisfaction with a new insulin analogue. Quality of Life Research, 9, 915–930.
Huang, C., Liao, H., & Chang, S. (1998). Social desirability and the Clinical Self-Report Inventory: methodological reconsideration. Journal of Clinical Psychology, 54, 517–528.
Jobe, J.B. (2000). Cognitive processes in self-report. In A. Stone, C.A. Bachrach, J.S. Turkkan, J.B. Jobe, H.S. Kurtzman, & V.S. Cain (Eds.), The science of self-report: Implications for research and practice (pp. 25–28). Mahwah, NJ: Lawrence Erlbaum Associates.
Kirschenbaum, D.S., Owens, D., & O’Connor, E.A. (1998). Smart golf: Preliminary evaluation of a simple, yet comprehensive, approach to improving and scoring the mental game. The Sport Psychologist, 12, 271–282.
Lonsdale, C., Hodge, K., & Rose, E.A. (2008). The Behavioural Regulation in Sport Questionnaire (BRSQ): Instrument development and initial validity evidence. Journal of Sport & Exercise Psychology, 30, 323–355.
Markus, H. (1977). Self-schemata and processing information about the self. Journal of Personality and Social Psychology, 35, 63–78.
McCarthy, P.J., Jones, M.V., Harwood, C.G., & Olivier, S. (2010). What do athletes implicitly understand about psychological skills? Journal of Clinical Sports Psychology, 4, 158–172.
Morsbach, S.K., & Prinz, R.J. (2006). Understanding and improving the validity of self-report of parenting. Clinical Child and Family Psychology Review, 9, 1–21.
Podsakoff, P.M., MacKenzie, S.B., Lee, J.Y., & Podsakoff, N.P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Sport Psychology, 89, 879–903.
Ravizza, K. (2006). Increasing awareness for sport performance. In J.M. Williams (Ed.), Applied sport psychology: Personal growth to peak performance (5th ed., pp. 228–239). New York: McGraw-Hill.
Raedeke, T.D., & Smith, A.L. (2001). Development and preliminary validation of an athlete burnout measure. Journal of Sport & Exercise Psychology, 23, 281–306.
Schober, M.F., Conrad, F.G., & Fricker, S.S. (2004). Misunderstanding standardized language in research interviews. Applied Cognitive Psychology, 18, 169–188.
Schutz, R.W. (1998). Assessing the stability of psychological traits and measures. In J.L. Duda (Ed.), Advances in sport and exercise psychology measurement (pp. 393–408). Morgantown, WV: Fitness Information Technology.
Sharp, L., Woodcock, C., Holland, M.J.G., Duda, J.L., & Cumming, J. (2010a, October). Youth athletes responses to a mental skills training program. In J. L. Duda (Discussant), An action research approach to mental skills training: Experience, evaluation, and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI.
Sharp, L., Woodcock, C., Holland, M.J.G., Duda, J.L., & Cumming, J. (2010b). Validation of the Athlete Burnout Questionnaire with youth athletes. Journal of Sport & Exercise Psychology, 32, s218–s219.
Tenenbaum, G., Kamata, A., & Hayashi, K. (2007). Measurement in sport and exercise psychology: A new outlook on selected issues of reliability and validity. In G. Tenenbaum & R.C. Eklund (Eds.), Handbook of Sport Psychology (3rd ed., pp. 757–773). Hoboken, NJ: John Wiley & Sons.
Thomas, P.R., Murphy, S.M., & Hardy, L. (1999). Test of performance strategies: Development and preliminary validation of a comprehensive measure of athletes’ psychological skills. Journal of Sports Sciences, 17, 697–711.
Vealey, R.S. (1988). Future directions in psychological skills training. The Sport Psychologist, 2, 318–336.
Vealey, R.S., & Garner-Holman, M. (1998). Applied sport psychology: Measurement issues.In J.L. Duda (Ed.), Advances in sport and exercise psychology measurement (pp. 433–446). Morgantown, WV: Fitness Information Technology.
Willis, G.B. (2005). Cognitive interviewing: A tool for improving questionnaire design. London, UK: Sage.
Woodcock, C., Holland, M.J.G., Sharp, L., Duda, J.L., & Cumming, J. (2010a). Psychological skill use in adolescents: Exploring the structural and temporal validity of the TOPS. Journal of Sport & Exercise Psychology, 32, s233.
Woodcock, C., Holland, M.J.G., Sharp, L., Duda, J.L., & Cumming, J. (2010b, October). Origins and evolution: Coming full circle with mental skills training. In J. L. Duda (Discussant), An action research approach to mental skills training: Experience, evaluation, and evolution. Symposium conducted at the meeting of the Association for Applied Sport Psychology, Providence, RI.
Young, B.W., & Starkes, J.L. (2006). Measuring outcomes of swimmers’ non-regulation during practice: Relationships between self-report, coaches’ judgments, and video observation. International Journal of Sports Science & Coaching, 1, 131–148.