Data from a clinical study presented an opportunity to examine the psychometric properties of the Leeds Multiple Sclerosis Quality of Life scale (LMSQoL), which has undergone limited psychometric evaluation. LMSQoL and Multiple Sclerosis Quality of Life-54 (MSQoL-S4) data were collected from 90 people with multiple sclerosis (MS) living in the community. Standard psychometric methods to examine data quality, scaling assumptions, scale to sample targeting, reliability, validity, and responsiveness were employed. The LMSQoL satisfied criteria for data quality (no missing data), scaling assumptions (item-total correlations: 0.24-0.56), reliability (Cronbach alpha: 0.71), and demonstrated responsiveness (effect size: 0.34). Correlations between the LMSQoL and MSQoL-54 physical (range: -0.02 to -0.50) and emotional subscales (range: -0.38 to -0.65) were similar; the magnitude and pattern was not consistent with predictions based on the construct purported to be measured by the LMSQoL. The LMSQoL satisfied many psychometric criteria in this small study, however it was difficult to interpret the validity data. From this, two fundamental measurement issues are highlighted. Firstly, current methods of examining rating scales provide only circumstantial evidence of validity, secondly, health-rating scales should be developed on the basis of clear conceptual definitions.