This is a blog in two halves, the first is totally practical and will tell you somethings you must note if you are going to compare codings. The second is theoretical and I want to sound a warning over the use of a particular statistics use in Qualitative analysis for comparing codings by different raters.
Right the first part is probably best told as a cautionary tale. A student turned up this week wanting to compare his coding with that of another student. The problem was that when he imported it into NVivo 9 it saw the documents in the two files as different documents. Thus making it very difficult to comparing coding schemes. It took us a while to find out why it was doing it and it was in the end because he had made minor edits to the files after the copy was taken for his friend to code on. That meant he had to go away and transfer all the coding from his friend onto his data files before he could do the comparison. There are ways to avoid this.
Firstly if you have coded your data and are now going to take a clean copy for a friend to continue coding on I suggest you also take a backup copy of your set with which to compare it with, just in case you get itchy fingers.
The second is a tip for getting around this in NVivo and making sure NOBODY edits the files including yourself and that is to print them to pdf before you import and only import the pdf. To do this you would need to download a free print to pdf utility such as
CutePDF and use it to produce the pdfs. When these are imported into NVivo you cannot edit them. So there should be no problem with anyone editing them.
However now my concern, he went onto say he was going to calculate a Cohen's Kappa and was surprised when I knew what it was. I know it because it has come up regularly in my statistics degree and I have calculated it a fair number of times. There has nearly always been debate about its applicability and when I was a junior statistician it was usual to report percentage agreement as well. The thing is I have always used it for agreement on the categorical data where I have clearly defined units, such as children grading by two teachers. Or whether two radiologist classified tumours in the same way from xrays. Child and xray are clearly defined coding units and the grades are normally exclusive.
Now the big problem I have is that there is no nicely defined coding item and it quite often happens that the same bit of text is coded to two different codes. Think about it you might want to code the short sentence "John wore a red dress to the party" to both cross dressing and the use of red as a colour. This is perfectly sensible qualitative coding. One simply might look at present and absence of coding on a sentence but then there is a problem of coding with the un-coded units, easy perhaps to deal with if the coding unit is defined as a sentence but if you have "red dress" coded to red and "John wore a red dress" codes, then what is coded in both instances is less than a sentence. Equally there may be times when the code only covers several sentences. If coding unit implies that it is coded to that then how do we know how many code units there are that neither rater has found? I don't have a clue?
So I checked the literature and came across the paper "
Content Analysis Research: An examination with Directives for Improving Research Reliability and Objectivity" by Richard H Kolbe and Melissa S Burnett which says:
"However, the use of kappa is difficult in content analysis because a key value, the number ofchance agreements for any particular category, is generally unknown."
which is basically what I am getting at, the underlying fluidity of the coded unit means I would have grave cautions about using Cohen's Kappa. You can read them to see what alternatives there were already, and I suspect more have come about since. I think I probably need to do some serious literature searching on this when I have the time so I can give accurate advice.
