Real time evaluation
April 2020
Just a month ago the Covid-19 pandemic had barely begun to affect Cordis Bright’s work and that of our clients. In the second week in March members of our team facilitated a workshop in London with 50 health and social care professionals and service users; travelled to Wales to discuss a new project on childhood obesity; delivered a training session on peer research in Blackpool and continued to commute from our homes to the London office. Less than two weeks later the country was in lockdown, most of us were working from our homes and our NHS, council and charity clients found themselves at the forefront of responding to the biggest global health emergency in the last hundred years. So much has changed so quickly.
With our colleagues in health and social care organisations working flat out to save lives and keep staff and service users safe, it might have been reasonable to assume that ‘non-essential’ activities like evaluation should be put on the back burner until the crisis eases. There will be lessons to learn, but all that can be done retrospectively at some point in the future. That was certainly our first thought, as we began to speak to project managers and steering groups about postponing evaluation activities and revising project timetables. Our expectation was that no-one would have the time to think about evaluation now.
But as we began to think about a new approach with the local team in Bury, Greater Manchester (GM), where we’re leading a longitudinal evaluation of the local care approach in seven areas as part of the GM devolution arrangements, it occurred to all of us that there could be so much to learn now about how the system works as it responds to this challenge. The present crisis presents an opportunity to do this learning in real time, while events are fresh in people’s minds, and to feed the learning back into the local system’s evolving response, rather than waiting until the immediate emergency is over before thinking with hindsight about what went well and what could have been done better.
Instead of pausing our activities we’re now thinking about how we can adapt local evaluations to answer our clients’ immediate questions using an approach that draws on ‘real time evaluation’ (RTE). RTE is usually used in humanitarian or other disaster response contexts; it involves rapid cycles of data collection, analysis and reporting to inform decision making. It often involves researchers working with those involved in a crisis response, capturing and understanding what they are doing to respond to the crisis, and then rapidly analysing and synthesising the evidence and feeding it back into the system to support better decision-making.
For example, lessons to be drawn now from local integrated health and care systems’ response to Covid-19 might include:
- The effectiveness of governance and decision-making processes employed in a time of crisis.
- How services relate to local and regional emergency command structures.
- The work of the intermediate tier to get people out of hospital as quickly as possible.
- What changes have occurred to the way in which care at home is delivered by the intermediate tier and integrated neighbourhood teams.
- How residents and staff are protected from infection in all settings, not just hospitals.
- How vulnerable people are being identified and supported in neighbourhoods.
- How health and social care staff in local community teams are working together in this new environment.
- The impact of any disparities in resources available for health and social care.
- The experience and impact of technological solutions and remote access to services.
- The contribution of the voluntary sector and how to make best use of resources.
- How to harness the efforts of local people who come forward to volunteer.
- The impact of changed ways of working on people who use services.
In practice, the RTE approach we are developing has the following features:
- It will focus on process, rather than on impact and sustainability, and its purpose would be immediate lesson learning. The imperative would be to generate findings quickly and cost effectively so that systems can respond to them.
- Our team will discuss and agree with a group of key stakeholders the areas to be explored and the questions to be answered. This will probably be limited to three or four key questions.
- Our approach will be rapid, flexible and responsive. We aim to carry out field work over a short time period, probably two to three weeks, and feed back findings immediately.
- Fieldwork will involve mainly qualitative methods such as interviews, group discussions and observation of meetings. These can all happen via web-based conferencing and telephone. We can also use video-diary and case study techniques. We will seek to talk to system leaders, operational managers, staff and beneficiaries. If any documents or monitoring data are available we may be able to use these to triangulate findings from the qualitative work.
- We will not plan to produce an additional evaluation report unless this is specifically requested. The primary focus will be on the recommendations made (and decisions acted on) whilst the evaluation is in progress. The findings might be fed into a final retrospective evaluation at a later point.
- We are acutely aware that the people we want to talk to are busy with urgent priorities. Our primary objective as evaluators is about being useful at this time and to do this whilst minimising the burden on participants. Ways we might do this include: (a) joining scheduled meetings as an observer; (b) keeping one-to-one interviews brief and to the point, with limits on time and the number of questions to be covered; (c) short video-diary approaches; (d) designing short surveys as an alternative to being interviewed, for those who would rather participate in this way; (e) involving people who are not currently able to be at work because they are having to self-isolate (although not if they are actually unwell).
As scientists put their expertise to work to find effective treatments and a vaccine to protect against Covid-19, social science also has a role to play in helping those responding to the crisis now to learn vital lessons for the future. If you are interested in finding our more about our work using real time evaluation please contact Jane Harris, Senior Consultant (janeharris@cordisbright.co.uk) or Dr Stephen Boxford, Head of Research (stephenboxford@cordisbright.co.uk).
