Using propensity score matching in evaluation

Denise Coster used an "experimental design" to evaluate the Coping with Crying film

Mother and baby in the gardenIn the health sector, activity, treatment or medicine must have a body of research evidence behind it, with the randomised control trial (RCT) being the gold standard for evidencing the impact of any intervention.

The NSPCC wants to know if changes that occur for parents, families or children can be attributed to the work we do, are due to other things happening, or are simply down to chance.   

In the social care sector, proving impact is difficult: there are many factors, unclear boundaries and overlapping roles.

When I came to evaluate our Coping with Crying service, these issues were clear, but I identified a method to help solve the issue of attribution.

Evaluating the Coping with Crying service

There are no standardised ways of looking at a parent’s reaction to a screaming baby at 3am in the morning.

But I wondered how I could measure if the film we show to parents, helping them prepare for parenting, cope with crying and prevent non-accidental head injuries, actually made a difference. 

How could I prove that anything parents reported was as a result of seeing the film, rather than receiving parenting information from somewhere else? Was there a way to find out if having a chat with a mum at the local children’s centre was just as effective as watching our film, for example?

In an "ideal" evaluation, with willing participants and plentiful resources, I would have recruited parents to an RCT. I would have relied on the good will and professionalism of healthcare staff to randomly show the film to some parents and not to others, under carefully controlled conditions. We would then be able to look at differences between the 2 groups.

But this is not an ideal world.

Implementing an RCT might be setting us up to fail. The nature of having a baby means that, understandably, many parents aren't willing to be recruited to a study at such a busy and momentous time in their lives.

And I’d spoken to enough midwives (and been involved in enough RCTs) to know how much work this would entail. They were already struggling to show the film to willing parents in a small window of opportunity between mothers giving birth and parents taking the baby home. How could I also expect them to recruit parents to a “research study”?

The difficulties in trying to set up an RCT, given the resource we had and the ethical and practical difficulties, felt insurmountable.

An "experimental design"

So much hard work from colleagues and professionals, inside and outside the NSPCC, had gone into producing and showing the film that I wanted to say more than just “parents tell us the film helped them”.

I really wanted to be able to say that, when compared with parents who hadn’t watched the film, parents thought and did something different which helped them cope with their babies and react appropriately when their babies were crying.

At this point I hit upon the idea of using comparative surveys and propensity score matching.

The idea was to survey parents who had seen the Coping with Crying film and compare their responses with a sample of parents who hadn’t seen the film.

Parents would be asked about their knowledge of the dangers of crying, what was normal in relation to babies crying and how they behaved and felt when their babies cried.

This was a good start in thinking about the impact.

But even if parents who had seen the film answered the survey in a particular way, this wouldn’t prove that this was because they had watched the film. There might be something else different about these parents that affected their responses. For example, parents who agreed to watch the film might tend to be first time parents and react differently to crying than second time parents. 

Propensity score matching

This is where propensity score matching comes in.

Parents were asked about a range of other factors which might affect the way they reacted to the film.  

In order to ensure that differences between the parents were not causing any differences in responses to the survey, the groups were matched on these factors so that they were equivalent.

Now, when the data was compared, any differential impacts on parents could be attributed to the film rather than, for example, being due to whether or not they had a baby who cried a lot. 

Solving the issue of attribution

Although this method is not without its limitations (you can’t always match for everything) it's a practical and sensible way forward in trying to solve the issue of attribution.

This method certainly allowed us to talk with some authority about how the Coping with Crying film actually did help parents.

You can read the full report here.

Like this blog?

Let us know which blog you've read, what you think, share information you have on the topic or seek advice. 

Get in touch

Learn more

Impact and evidence insights

Each week we’ll be posting insights from professionals about evaluation methods, issues and experiences in child abuse services and prevention. 
Read our blogs

Impact and evidence

Find out how we evaluate and research the impact we’re making in protecting children, get tips and tools for researchers and access resources.
Impact and evidence

Coping with Crying

Helping parents to keep calm and soothe their crying baby.
Coping with Crying service