How do our services benefit individuals?

Emma Smith explains why we need to take into account the clinical significance of the findings from service evaluations

Child on a swingOne thing I love about my job as an evaluator at the NSPCC is letting practitioners know when a new service we’ve developed to support children and families has been successful. There’s nothing better than being able to tell the practitioners who have worked so hard delivering the programme, that the findings of the evaluation are positive and that the programme makes a real difference to children’s lives. Usually everyone is delighted to hear about the success.

So I was really surprised when, a few years ago, a group of practitioners were sceptical about the positive evidence I showed them for their service. At first I wasn’t sure why the team didn’t agree that the evidence suggested the programme was a success, but a while later I carried out an analysis looking at the clinical significance of our findings, which revealed their concerns were justified. What I learnt will influence all my future evaluations.

The complexity of positive findings

I was evaluating Domestic Abuse Recovering Together (DART), our service to support the recovery of mothers and children who have experienced domestic abuse. It seemed at first glance to be a valuable, well-designed service, so I was pleased when early evaluation results were positive. We found several statistically significant improvements in the families’ well-being after they completed DART and the children from DART showed greater improvements than a comparison group who hadn’t taken part in the programme.

In the early sessions I ran to share the evaluation findings, the practitioners who had delivered DART agreed with the results - they said they had observed these improvements in the families. However, the final group of practitioners’ responses surprised me. They seemed cynical about the positive findings and felt that although some families had shown improvements, they were not fully recovered and would still need support after DART. They seemed concerned that the evaluation had not identified these families.

A year later, when we had sufficient data, I carried out different analyses to compare the clinical significance of the results. This revealed some truth in the practitioners’ concerns.

The importance of looking at clinical significance

Statistical significance focuses on the average change in a group as a whole at the end of the programme they are taking part in, whereas clinical significance concentrates on changes that are meaningful for each individual. For example, a child with behavioural difficulties might find their behaviour improves to the extent that they no longer require specialist support after finishing the programme.

Some researchers have warned against focusing purely on statistical significance when evaluating a service. This can lead to too much weight being placed on results that are clinically unremarkable.

The children who took part in DART completed a Strengths and Difficulties Questionnaire (SDQ) at the beginning and end of the programme. By looking at the results of these questionnaires, we can assess the level of a child’s needs based on their emotional needs, peer interactions and behaviours. We identify whether they have ‘high needs’, ‘some needs’ or ‘low needs’. If a child had moved from having ‘high needs’ or ‘some needs’ to a lower level of need by the end of DART, we considered this change to be clinically significant.

The analyses I carried out showed that just over half the children moved to a lower need category of need after taking part in DART. So the improvements were both clinically and statistically significant. However, I was surprised to see that just over half of the children identified as ‘high needs’ before DART began still had this level of need after DART finished. This highlighted the practitioners’ concern: despite improvements after DART, some families still needed support.

This picture was reflected in interviews we conducted with families who had taken part in the programme: although they were overwhelmingly positive about DART and felt it had helped them, some mothers still struggled with their child’s behaviour and felt anxious about their support from DART ending.

Using a multi-agency approach

I tried to make sense of these findings by speaking to a DART service manager. She told me families who have experienced domestic abuse often face a series of challenges, some of which are not specifically addressed by DART. For instance, some of the families taking part in the programme were at risk of homelessness, very isolated and facing extreme financial hardship. She explained that we would continue to support families on a one-to-one basis after DART finished, if they were still considered to be in need. She highlighted the importance of our relationships with services such as Women’s Aid, housing organisations and Domestic Abuse Safety Units to make sure families can access any extra support they need, both during and after DART.  

How this has changed my practice

I am grateful to the practitioners who questioned the original findings of my evaluation. The concerns they raised were valid, and showed their dedication to the families they work with.

This experience reminded me that I need to continue to question evaluation findings. Even if results seem to be overwhelmingly positive, it is important to find out more. We need to look at how a service affects all the children and families who take part, as well as just looking at the success stories.

Was this page helpful?

If so, let us know how. Or you can let us know how we can improve our Impact and Evidence hub.

Get in touch

More from impact and evidence

Domestic Abuse, Recovering Together (DART™)

Helping children and mothers strengthen their relationship after domestic abuse.
DART - Domestic Abuse Recovering Together service

Learning from domestic abuse survivors

NSPCC evaluator Emma Smith reflects on her experiences and challenges interviewing domestic abuse survivors.
Read more

Improving services using negative findings

Dr Alice Haynes discusses the implications and importance of identifying a service that isn't working.
Read more

Using evaluation tools and standardised measures in assessment

A practitioner's perspective on whether evaluation measures help practitioners in their assessment of a child
Read more

Support for professionals

Follow @NSPCCpro

Follow us on Twitter and keep up-to-date with all the latest news in child protection.

Follow @NSPCCpro on Twitter

Library catalogue

We hold the UK's largest collection of child protection resources and the only UK database specialising in published material on child protection, child abuse and child neglect.

Search the library

New in the Library

A free weekly email listing all of the new child protection publications added to our library collection.

Sign up to New in the Library

Helping you keep children safe

Read our guide for professionals on what we do and the ways we can work with you to protect children and prevent abuse and neglect.

Read our guide (PDF)

Impact and evidence

Find out how we evaluate and research the impact we’re making in protecting children, get tips and tools for researchers and access resources.
Impact and evidence

Impact and evidence insights

Each week we’ll be posting insights from professionals about evaluation methods, issues and experiences in child abuse services and prevention. 
Read our blogs

Sharing knowledge to keep children safe

Read our guide to the NSPCC Knowledge and Information Service to find out how we can help you with child protection queries, support your research, and help you learn and develop.

Read our guide (PDF)