Measuring Outcomes in Youth Justice Programmes: A Review of Literature and Practice Evidence

By John Reddy and John Reddy, Sean Redmond

This Research Evidence into Policy, Programmes, and Practice (REPPP) study examined outcome measurement in youth justice programmes, youth work, and human services. Outcomes for young people are the effects or contribution to effects for young people that can reasonably be attributed to their participation in a programme. The research was commissioned by the Department of Justice to support improved measurement in Garda Youth Diversion Projects (GYDPs). Messages from Literature and Practice Reports The evidence presented indicates that timely information from practice helps to strengthen programmes, improve standards, and provide accountability. Service providers use information collected in their work with young people to measure the impacts of programmes. Programmes collect data about a young person’s circumstances, demographics and ethnicity, offence history and likelihood of reoffending, referral and placement information, and their interaction with other services. This data informs case management and intervention planning and service-use evaluations such as the number and costs of programmes delivered, and any gaps in service. To date, there has been a tendency to assess programmes using ‘hard’ programme input and output data (e.g. programme completion numbers, young people’s participation in education/training, school attendance, and rates of offending behaviour) at the expense of harder to measure positive or negative changes in behaviour.  Evidence of change in a young person’s social and emotional capabilities (soft outcomes) is increasingly regarded as intrinsic in efforts to effectively evaluate outcomes for young people. Programmes that gather soft data typically do so by embedding observation and recording processes into practice routines. When aligned to policy and programme objectives, data reflective of practice with young people can assist service providers to contextualise the ‘hard’ data produced by standardised measurement instruments. Data processes that included soft data were suggested as providing programmes with enhanced capacity to evaluate a young person’s engagement in the programme, their development, and changes in their behaviours and attitudes. Integrating soft information can help service providers to identify factors that may have shaped a young person’s life: identifying the part that a young person played in the changes observed, a practitioner’s role in achieving change, and how project activities may have contributed. The following table presents findings from a rapid (realist) review of outcome measurement literature and practice reports:  Outcomes for young people in programmes: 7 step measurement checklist 1. Measure outcomes for young people in programmes: To maintain and improve the quality of a programme and demonstrate its impact and value To ensure accountability and transparency in the delivery of public services To record what young people describe as important to them and barriers they face in achieving a good life • To improve efficiencies, realign resources, maintain standards, and strengthen practices 2. Things to consider when measuring outcomes for young people in programmes: Performance-led data alone rarely produces assessments that reflect a programme’s true value Developing young people’s social and emotional capabilities is associated with positive life outcomes Understanding how participants experience programmes provides a basis for better decision-making Evidencing improvements in personal development can be difficult due to the many influences impacting on young people’s lives 3. What can help the measurement of outcomes for young people in programmes? A logic model identifying outcomes can focus programme delivery and measurement practices Research and practice collaboration on data and monitoring processes A mix of measures and/or the development of new data processes to suit the task Active data leadership, specialised data skills, and support and technical assistance 4. Factors influencing outcome measurement: Integrating quantitative and qualitative data is associated with comprehensive assessments Qualitative data improves understanding of the factors contributing to outcomes Data quality and accuracy is linked to the quality of relationships established between a practitioner and a young person, their families, and other services Data practices can provide opportunities for young people to contribute to identifying outcomes and working towards these goals 5. It is important that the tools used to measure outcomes: Are relevant to the programme, local contexts, and culturally appropriate Produce quality data that is timely and comparable across groups and programme types Are comprehensible to practitioners and those completing them Are sensitive to change, reliable, consistent, and repeatable Produce useful practice and policy information 6. Challenges in measuring outcomes: Measurement can be a lengthy process, from design to collection to analysis and reporting Tools may not be designed to meet programme needs, be costly, untested, and difficult to adapt Tools may be difficult for young people to complete and may not differentiate between aspects of youth development 7. Things to consider when analysing data from practice: Evidence of a young person’s progress can be observed, interpreted, and documented Data collection and analysis processes should be documented for transparency and credibility Focus on a particular outcome and identify from the data if an anticipated change has occurred Time, resources, sample size, practitioner bias, and research expertise all impact the quality of  Messages from Practice Outcomes for young people in programmes should align with youth justice policies to reduce offending and improve attitudes and behaviours. GYDPs collate significant volumes of information from young people using routine administrative and assessment procedures. This data is predominantly quantitative (input/output) and details participation in a project, education, health, safety, and risk of offending/re-offending. However, service providers have advocated for greater use and reporting of supplementary data collected through observational processes implemented by practitioners. They suggested that integrating ‘soft’ data into existing outcome measurement processes would be a welcome and useful addition to efforts to evaluate outcomes for young people and to demonstrate the value of their work. This research aimed to establish a robust knowledge-base of outcome measurement from literature and practice for practical application by GYDPs. One additional but critical dimension was the challenge to bring scientific evidence of soft outcome measurement to bear on realworld constraints. This is compounded by the complexities of diverse administrative systems within the overall GYDP structure. Of the 105 Projects now operating nationwide, many are national youth organisations providing multiple services and operating well-developed information technology (IT), while others are more local and operate with less IT resources. In acknowledging organisational diversity in GYDPs, the study established a common minimum threshold for applying the scientific evidence of soft outcome monitoring in practice. To this degree, the report has been necessarily pragmatic. The report provides three data options that balance substantive progress in outcome-based recording practices with the need to ensure implementation with the minimum of disruption and impact on frontline work. REPPP recommends developing and embedding a non-invasive routine observation and recording process into GYDP practice to assess a young person’s engagement in the programme, their development, and changes in their behaviours and attitudes. A time-efficient evaluation template could record information from practice based on the expected outcomes of the Garda Youth Diversion Programme to address behaviour and offending problems and to facilitate personal development. When combined with existing data processes, this data could yield a more nuanced understanding of the outcomes for young people in GYDPs and inform judgements about the impacts of interventions.

Limerick: University of Limerick, 2022. 59p.