Can Researchers Agree on Levels of Resilience?

Abstract

Resilience has been challenging to define, though it is intuitively apparent in many ways as the capacity to recover from adversity. We set out to determine if we could agree on resilience ratings based on our assessments of collected life story interviews. After reviewing our life stories, we settled on five resilience ratings based on a water/swimming metaphor. At pattern 1, the person sinks to the bottom and remains there. At pattern 2, the person is not sitting at the bottom but has yet to reach the surface. At pattern 3, the person is treading water. They are working hard to stay aoat but not making signicant progress in altering their overall situation. At pattern 4, they’re swimming toward shore, toward a more favorable environment where they can thrive. At pattern 5, they've managed to climb out of the water and change their circumstances. Their resilience allows them to overcome challenges and seek better surroundings. We reviewed the life story interviews of thirty-five participants with an eye to these categories. We also used ChatGPT to determine if it could perform better than us. For 35 participants with five ratings and three raters, we had a percent overall agreement of 84.00% with a free-marginal kappa of 0.80 (95% CI for free-marginal kappa; 0.66 to 0.94). The fixed-marginal kappa was 0.78, with a 95% CI of 0.62 to 0.93. ChatGPT had a percent overall agreement of 71.33% with a free-marginal kappa of 0.65 (95% CI for free-marginal kappa was 0.50 to 0.79); fixed-marginal kappa of 0.61 (95% CI for fixed-marginal kappa of 0.51 to 0.71). Cohen suggested the Kappa result be interpreted as follows: values ≤ 0 as indicating no agreement and 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41– 0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement (Cohen, 1960). All results were statistically significant at p < 0.01. The self-administered Ohio State Resilience Scale, reflecting the participants own assessment of their level of resilience, did not correlate with our ratings, suggesting that people’s assessments of their level of resilience do not match those of clinically trained observers.

Warning: This paper includes discussion of categories of adverse childhood experience, including child sexual assault, as well as accounts of adult violence, domestic violence, abuse, sexual assault, neglect, and household dysfunction.