Carrying out evaluations with families facing adversity

Professor Jane Barlow shares what we learnt from carrying out a randomised control trial with families where parents misuse drugs or alcohol

SandpitIn my previous blog I explained how we carried out a randomised control trial (RCT) of our Parents Under Pressure™ service, to find out what works to support families where parents have drug or alcohol problems.

RCTs are now recognised as being the ‘gold standard’ for evaluating how effective a treatment programme is. But despite this, they aren’t often used in social care settings. In this blog I’ll be discussing why this is, outlining some of the problems we experienced during the RCT and sharing tips for how these challenges can be overcome.


Why use a RCT and what does it involve?

During a RCT, participants are allocated at random to different treatment paths. In this case, parents either took part in Parents Under Pressure™ or received standard treatment for their drug or alcohol problems. The participants are monitored to see how their behaviour changes throughout the programme (and in this case, up to 6 months afterwards). Because they are allocated randomly to each group, we can be confident that any differences we notice between the 2 groups after the programme finishes are due to the intervention they received. Any other factors that might explain differences between the 2 groups, such as parents’ socioeconomic status, level of education or ethnic background will be equally distributed between the 2 groups.

What are the problems with conducting an RCT in children’s social care services?

Although RCTs are now recognised as being the ‘gold standard’, they are still not often conducted in social care settings for a range of reasons. One of the most prominent is to do with perceptions about the ethics involved in randomly assigning participants to receive either the treatment we are testing or standard care.

There is often a belief among service providers and people who refer families to services such as Parents Under Pressure™ that these services are superior to standard care because they involve innovative and more in depth methods of working with families. Many people feel that all eligible families should receive the service, and that it’s unethical to randomly deny some families access to it.

However, RCTs are conducted because we don’t yet know how well a programme works. So we don’t know for certain whether the families who are selected to be in the ‘treatment as normal’ group will be worse off than the others. In today’s world of limited resources in the social care sector, many services use some form of selection criteria to make sure that the people taking part are likely to get maximum benefit. RCTs use one of the most equal forms of rationing - randomisation - because everyone has an equal chance of receiving the service.

So one of the first things we had to overcome as part of the RCT for Parents Under Pressure™ was that some of our practitioners were uncomfortable with the idea of randomly selecting who was going to take part in the programme. The practitioners expressed feelings of anger and anxiety about this, and we needed to help them to understand that this approach to evaluation isn’t unethical.

Many of the service providers for Parents Under Pressure™ hadn’t taken part in an RCT before so practitioners were understandably concerned about what was involved. In addition we needed to make some changes to the way we selected participants in the initial stages of the RCT, and this didn’t help with these difficulties. The learning for us as evaluators was that we should pilot all our methods before introducing them to the practitioners involved in the trial.

What challenges did we experience when conducting the RCT?

One of the real challenges of conducting this RCT was that we had to be structured and consistent in collecting the data during the period of time that the families were taking part in Parents Under Pressure™ and up to 6 months after they had completed the programme. These families were experiencing a lot of challenges and we had to be sensitive to the unpredictability that parental substance misuse in particular can bring to a family’s life.

Our researchers needed to visit participants outside of the programme sessions to collect data from them. We made sure we gave plenty of notice to parents about our visits, reminded them that we were coming and encouraged them to continue taking part by giving them a £10 gift voucher every time we collected data from them. However, our evaluation team still experienced some difficulties. For example our researchers could arrange a time and place to meet a participant and send a text on the day to confirm details, but when they arrived (often having travelled a long way) the person wasn’t there.

We had to understand things from the participants’ point of view. If they stopped responding to phone calls and texts, we realised that there could be many possible reasons. It could be due to changing their number (perhaps because they were trying to avoid debt collectors); not being able to afford the mobile tariff; or just not wanting to engage with the researcher. If they didn’t want to answer our questions, it could be because they were going through a particularly tough time and didn’t want to talk about it, for example if their children had been taken into care. Our researchers had to find the balance between being persistent so they could get the data we needed for the RCT, and not making the participants feel uncomfortable.

Overcoming challenges

One of the Parents Under Pressure™ researchers shared the following advice for collecting evaluation data from reluctant parents:

“At our first meeting, before the participant had consented to take part in the evaluation, I made it clear their participation was voluntary but said I would do my very best to get hold of them when I needed to collect data. I explained this was because it was important for the study to include everyone who had consented to take part. I apologised in advance for my persistence but I explained I was passionate about the study and the data collection.”

“I took the approach that if I was going to catch any of the more elusive participants, it would be early in the morning, before they left the house.”

“If I’d arranged to visit a participant at home but there was no answer when I knocked on the door, I would call and text them again. If there was still no reply, I often waited in my car. I then took the opportunity to speak to the participant when they came home, or to go up to the house if the participant had answered the door to someone else.”

“Sometimes participants said they weren’t in the mood today and didn’t want to see me. If this happened I tried to sensitively explain about the importance of them completing the questionnaire and apologised for my persistence but wondered if we could continue with the data collection, and I usually managed to get them to agree.”

“I was always sensitive and tried hard to show understanding and empathy. I tried to make it clear to the participants that they were providing me with a valuable insight into their lives, and how privileged I was that they were willing to see me.”

Data collection was an interesting and challenging task. Our researchers were careful to never upset any of the participants or force them to answer questions, but had to be very persistent. Having a non-threatening, non-judgmental approach enabled the participants to complete the research questionnaire, even when the situation was extremely difficult for them.

Conclusion

Illustration megaphoneDespite these difficulties, 100 families consented to take part in the RCT, and nearly 3 quarters of these participants provided data at the 6-month follow-up. This enabled us to get a much better understanding of how Parents Under Pressure™ works to improve outcomes for families where parents misuse substances - and which families it works best with.

Although RCTs are a vital part of evaluating the effectiveness of a programme like Parents Under Pressure™, it’s also important to get the views of the professionals who provide the treatment and the people who receive it.

So in addition to carrying out this RCT, we also conducted interviews with parents and practitioners. We wanted to find out what they liked and disliked about Parents Under Pressure™ and if there were any difficulties with its delivery. This study design is known as mixed-methods because it involves the collection of both quantitative and qualitative data. We’ll be sharing what we learnt in a future blog.

Was this page helpful?

If so, let us know how. Or you can let us know how we can improve our Impact and Evidence hub.

Get in touch

More from impact and evidence

Parents Under Pressure™

Improving parenting in families with drug or alcohol issues.
Parents Under Pressure™ service

Parents Under Pressure™ - Evidence, impact and evaluation

How the impact of the NSPCC Parents Under Pressure service is evaluated.
Parents Under Pressure - Evidence, impact and evaluation

Evaluation of Parents Under Pressure™

Evaluation of a service delivered by the NSPCC to support families where a parent has alcohol or substance misuse problems.
Find out more

What can we do to improve the lives of families affected by parental substance misuse?

Overview of the Parents Under Pressure™ programme, which supports families where parents have a drug or alcohol problem.
Find out more

Using randomised control trials in social care

Richard Cotmore offers advice about using randomised control trials (RCTs) to evaluate the impact of support services for children and families.
Find out more

What are the best tools to measure outcomes for children and families?

Rachel Margolis, Emma Smith and Mike Williams discuss the best tools for measuring the impact of social work interventions to support children and families.
Find out more

Impact and evidence insights

Each week we’ll be posting insights from professionals about evaluation methods, issues and experiences in child abuse services and prevention. 
Read our blogs

Support for professionals

Sign up to CASPAR

Subscribe to CASPAR, our current awareness service for child protection practice, policy and research.
Sign up to CASPAR

Information Service

Our Information Service provides quick and easy access to the latest child protection research, policy and practice. 
Find out more about our Information Service

Follow @NSPCCpro

Follow us on Twitter and keep up-to-date with all the latest news in child protection.

Follow @NSPCCpro on Twitter

Library catalogue

We hold the UK's largest collection of child protection resources and the only UK database specialising in published material on child protection, child abuse and child neglect.

Search the library

Sign up to New in the Library

Subscribe to receive weekly alerts on new additions to the library collection.
Sign up

Helping you keep children safe

Read our guide for professionals on what we do and the ways we can work with you to protect children and prevent abuse and neglect.

Read our guide (PDF)

Impact and evidence

Find out how we evaluate and research the impact we’re making in protecting children, get tips and tools for researchers and access resources.
Impact and evidence

Training and consultancy

Grow your child protection knowledge and skills with CPD certified courses delivered by our experts nationwide and online.
Get expert training