Understanding biased baseline data

In the first of a two-part post, Mike Williams discusses the NSPCC’s experience of biased baseline data

We’ve spent the last 5 years conducting outcome studies on interventions designed to support children and families. Testing whether interventions have had a positive effect on our service users has been central to our work.

To demonstrate effect, we need to know what life is like for service users when they start an intervention, so we can see if things are better for them at the end.

What is baseline data?

Baseline data are measures collected at the start of an intervention that can be used for comparison with data from during or after an intervention..

At the NSPCC, we ask practitioners, not evaluators, to administer the measure. On occasion, practitioners use the measure data to inform their practice.

Baseline data varies depending on which elements of the service user’s life are measured and who provides the data.

We use a form that is shown to be a valid and reliable indicator of the thing it tries to measure.

Collecting honest and reliable data

It's crucial that the service user fills the report in honestly so it’s a reliable measure of how things actually are.

If the service user underestimates problems, the evaluation has less chance of capturing genuine improvements brought about by the intervention. If there's an overestimation, the evaluation could appear to detect an improvement that isn’t actually there.

We've found that even when self-report and parent report measures have been tested for validity and reliability, the data does not always appear as a reliable representation. Unreliable self-report and parent report data can be a real issue in service evaluation.

Underestimating and overestimating examples

These NSPCC examples show baseline data that appears to underestimate service users’ problems:

    • Fathers attending Caring Dads, our service for abusive fathers, reported positive parenting behaviour above the population average. We would have expected their scores to be below average
    • Parents attending the SafeCare programme, which focused on children at risk of neglect, also rated their parenting skills highly. Again, we would have expected below average scores
    • Young people attending Turn the Page, a service for young people displaying harmful sexual behaviour, appeared to under-report symptoms of trauma.

Other baseline data appears to overestimate service users’ problems:

    • Parents whose children attended Letting the Future In, our service for children who have experienced sexual abuse, reported higher levels of traumatic symptoms in their child than the child reported.

Why can baseline data be inaccurate?

In our experience, when practitioners administer measures, service users may present themselves in a positive light because the results can have consequences.

For example, fathers using our Caring Dads service may be ashamed of past behaviour and worried that social workers would restrict access to their children if they were honest.

Some children displaying harmful sexual behaviour are obliged to attend the Turn the Page service as part of their Youth Offending Service order. A desire to convince the practitioner that there are no problems may explain the low levels of trauma reported.

Conversely, parents may report higher levels of trauma in their children to communicate their desperation for support.

The wording of a form plays a major role in skewing baseline data, despite previous testing.

We used an adapted form of the Mother Child Neglect Scale to get a sense of the quality of parenting in a service focused on neglect, but we felt the questions were worded in such a way that parents could be put off admitting neglectful behaviour.

For example, the questions asked parents if they would take action if a problem arose: this sends a strong message about what parents should be doing, encouraging positive bias. It also assumes parents are aware of problems; a big issue with neglect is the lack of understanding of good parenting.

Disassociation and post traumatic symptoms can often stop service users from recognising their own behaviour and feelings.

Moving forward

Our experiences show that self and parent report data can be problematic - but does this mean we should give up on our attempts to collect self-report and parent report data?

Not at all – we need to be honest about the problems we face with parent and self-report data and persevere in our attempts to minimise bias.

Part two…

Read the second part of my blog to find out my 8 ideas on how to manage bias in baseline data.

This Impact and evidence insights blog was written with input from NSPCC experts: Gill Churchill, Emma Belton, Nicola McConnell, Vicki Jackson, Paul Whalley and Richard Cotmore.

Like this blog?

Let us know which blog you've read, what you think, share information you have on the topic or seek advice. 

Get in touch

Learn more

Impact and evidence insights

Each week we’ll be posting insights from professionals about evaluation methods, issues and experiences in child abuse services and prevention. 
Read our blogs

Impact and evidence

Find out how we evaluate and research the impact we’re making in protecting children, get tips and tools for researchers and access resources.
Impact and evidence

How we evaluate our services

We choose the best approach for the question we're trying to answer - whether we're learning about something very innovative or something that's well developed.
Read more