Using a standard Measurement Scale in a Survey within a fresh Population

Many years ago, a researcher came to me with a simple scale of over all well being. The first question asked how well over all they felt and the rest asked about performing specific tasks such as climbing stairs. The aim was to measure overall well being. She was investigating the response of a specific group of patients who had leg injuries from motorcycle accidents. The scale did not seem to behave as expected. Upon investigation I found the first term was routinely scored as doing fine while the rest could have quite negative responses. This was because the group heard the general question as having the words "except for you leg" in it, even though the words were not there.

If the culture of the population you are investigating is different from the one that the Measurement Scale was developed on then you need to check you can apply it.  This is particularly true if you are translating the survey and that even if you back translate to check equivalence. No matter how careful your translation is if the topic you are dealing with has been conceptualised differently then you will have trouble with the scale.

There are a number of things you can do to check it. Firstly you can simply run a Cronbach's alpha, and if this fails then you know you are in trouble. If you want to be more thorough you can carry out a Confirmatory Factor Analysis (CFA), but what you should not do is just assume that ideas from one setting can translate unproblematically in another.

I came across another one this week.  The person was looking at the way disability was viewed in this country and in a Middle Eastern Country. Both the CFA, Cronbach's alpha and exploratory factor analysis suggested problems. This was found after she had conducted two large surveys and from what I can see there is also an intervention in one survey. Plus she has sampled from several subpopulations such as health care professionals and parents of disabled children.

In some ways she is fortunate, the analysis conclusively demonstrates that disability is constructed differently in the two cultures. However it is not enough to say they differ we also have to say how they differ and probably why the differ. The only clue to this is to go back to each item and compare it across the cultures. So not the fancy techniques but a huge lot of fairly simple statistics. Then you try and make sense of it!