Comments for Slowly, but Surely:- Hi Sara, The formulas used in the OKC are listed in the description and the references at the bottom of the OKC page. The java program that the OKC is based on is password protected. Only I have the password to the java program. I haven't checked the calibration lately, but it was extensively tested by the programmer and I by hand-calculating to compare results between the formulas and the OKC. Specifically, I used known kappa values from a data set from the formula's source, among other tests, and compared that to OKC results. The results were accurate at the time. To the best of my knowledge, nothing has been changed in the original program. You could run a few tests comparing the OKC results to hand-calculated results from the formulas used. Write again if you need clarification. I appreciate your attention to detail. Take care, Justus

Comments for Slowly, but Surely:- Hi Isabelle, Thanks for the great resource! This looks as if it will really be a benefit to the research community of Kappa users. I look forward to reading it. Justus

Comments for Slowly, but Surely:- Hi Justus I've been having problems using my University's preferred statistics package to calculate values using Kappa statistic and no one there knew how to use the package for that purpose. I came across your online Kappa calculator and was able to obtain some values. What I would like to know is how did you formulate this calculator? Is it regularly calibrated and can anyone change it so that the results it provides aren't accurate? I would like to use the results I obtained but need to be assured of the accuracy of the calculator. Many thanks

Comments for Slowly, but Surely:- Hi Christian, Thank you for your comment. An article by Altaye et al. (2001). A general goodness-of-fit approach for inference procedures concerning the kappa statistic. Statist Med, 20, 2479-2488 has solved my problem. I recommend this paper to anyone who wants to estimate the sample size in a reliability study that involves 3 categories, and would like to compare the numbers needed when only two judges make the ratings vs. when there are 3-7 raters. I have downloaded the paper you suggested - a good resource to have for future reference! Isabelle

Comments for You can always change this later.:- I haven't see such a good comment spam for years. Thanks Andrés. :)

Comments for You can always change this later.:- Pearls, that was one of the most brilliant spam comments I have ever witnessed. It is 100% relevant. Should I praise randomness or the flying spaghetti monster?

Comments for You can always change this later.:- Oh the glorious PhD. I can't wait till I actually have one of those one day.

Comments for Slowly, but Surely:- @Isabelle Maybe this is helpful: Bonett DG. Sample size requirements for estimating intraclass correlations with desired precision. Statist Med 2002; 21:1331–1335.

Comments for Slowly, but Surely:- Hi Justus, Thank you for your prompt reply. I have sent you an email with the additional info. Best regards, Isabelle

Comments for Slowly, but Surely:- A few more questions: Do you want to make an inference from a subset of cases, which you will have both judges rate, to all cases that you have data for? Or Do you want to make an inference from all cases you have data for to the universe of possible cases? Or Do you want to make an inference to all judges from the judges you have chosen? Justus

Comments for Slowly, but Surely:- Hi Isabelle, I'm not sure what you mean by "sample estimate" in the sentence: "estimating sample size for significance tests for kappa, but nothing on what would be considered appropriate if kappa were to be simply used as the sample estimate." Please explain. Could you send me the papers you referred to by e-mail and I'll try to see what you mean. I'm guessing that you could use those formulas or tables there if you are using Cohen's kappa to estimate the inter-rater reliability of your scale. You might also want to check out generalizability theory. That might be another method of answering your question, if you want to think of raters as random factors.

Comments for Slowly, but Surely:- Dear Justus, I have come across your website with the online kappa calculator, which seems like a fantastic resource. However, unlike the other users of your blog, I am not yet at the stage of data analysis, my question is much more preliminary. My research project investigates risk factors for suicide attempts. As part of it, I have developed a rating system to determine whether or not a given incident of self harming behaviour constitutes a suicide attempt. Accordingly, an event can be classed as a suicide attempt, not a suicide attempt, or undetermined. Despite an intensive literature search, I have not been able to work out what would be an adequate sample size of the vignettes to be rated and the number of judges for calculating kappa. The papers I have read on the sample size requirements (e.g., Cantor, 1996; Flack et al., 1988; Sim & Wright, 2005) provide formulae or tables for estimating sample size for significance tests for kappa, but nothing on what would be...

Comments for You can always change this later.:- Some interesting predictions about EdTech year 2007, made by Roman: http://cs.joensuu.fi/~rbednari/blog/index.php/a-look-at-the-2007-edtech Can you imagine what could be ahead, let's say at 2011, or even in the second half of 2010?

Comments for You can always change this later.:- [...] feed. You can leave a response, or trackback from your own site. Leave a Reply. Name (required) ...You can always change this. Blog Archive Meet me at the ...Meet me at the CS dept. store. After reading this article about how cool it is even for ... one of [...]

Comments for You can always change this later.:- [...] [...]