Evaluation Summary: Final Results – Stop Now and Plan (SNAP®)
Table of contents
Introduction and Program Description
Stop Now And Plan (SNAP®) is a community-based program for children ages 6 to 12 who have come into contact, or are at risk of coming into contact, with the criminal justice system, and/or who display early signs of anti-social or aggressive behaviour.
The program uses a cognitive-behavioural, multi-component approach to decrease the risks of children engaging in future delinquent behaviour. The SNAP® model is based on a comprehensive framework for effectively teaching children with serious behavioural problems, emotional regulation, self-control and problem-solving skills.
The core program components include the children's and parent's groups. The SNAP® Boys and SNAP® Girls offer 12-week gender-specific groups that teach emotion regulation, self-control and problem-solving skills. The concurrent SNAP® Parent Group teaches parents effective child management strategies. Other program components include individual counselling/mentoring, family counselling, academic tutoring, youth leadership and a gender-specific component called “Girls Growing Up Healthy”. These are recommended based on a continuing assessment of the child's risk and need levels.
Although there is evidence regarding the effectiveness of SNAP® in Canadian and United States contexts (within accredited mental health centres and community based settings), a further evaluation was conducted to assess the impact the program in a variety of other community-based organizations across Canada (i.e., youth justice, mentoring, and Aboriginal reserves).
This summary provides an overview of the multi-site impact evaluation of SNAP® that was funded by the National Crime Prevention Strategy (NCPS). The multisite impact evaluation assessed the efficacy of this program in three unique communities (Toronto, Edmonton and Cree Nation – Quebec), contributing to the collective body of knowledge of what works in crime prevention.
The NCPS contributed approximately $10 million to fund nine SNAP® programs across Canada (including process evaluation costs). An independent firm was contracted to conduct the multi-site impact evaluation of SNAP®. The study, valued at $875,000, started in 2010 and ended in 2014.
Target Group
Children who are admitted to the SNAP® program must have had some former police contact, or had demonstrated aggressive/oppositional behaviour at clinical levelsFootnote2.
Across the three sites, children were referred to the program largely by schools and other social/human services organizations. Table 1 below provides a summary of risk levels of participants across the three sites, as measured by the Early Assessment of Risk List for boys (EARL-20B) or girls (EARL-21G) instruments. Out of a total of 375 children participating in the evaluation, 247 children had completed these instruments. This data shows that approximately three-quarters of the children entering the program were of moderate to high risk. It should be noted that the Cree sample has a greater percentage of high risk than the other two sites.
Overall | Edmonton | Toronto | Cree | |||||
---|---|---|---|---|---|---|---|---|
Risk Level of Child |
N = |
% |
N = |
% |
N = |
% |
N = |
% |
High |
68 |
27.5 |
14 |
16.3 |
24 |
22.9 |
30 |
53.6 |
Moderate |
117 |
47.4 |
33 |
38.4 |
62 |
59.1 |
22 |
39.3 |
Low |
62 |
25.1 |
39 |
45.3 |
19 |
18.1 |
4 |
7.1 |
Total |
247 |
100% |
86 |
100% |
105 |
100% |
56 |
100% |
Evaluation Objectives
The objectives of the evaluation were as follows:
- Incorporate process evaluation information and assess the extent to which the project is being implemented as intended;
- Determine whether the intended outcomes were achieved and identify any unintended outcomes;
- Provide a descriptive cost analysis for each project and determine feasibility in order to conduct a cost effectiveness analysis;
- Identify lessons learned and recommendations; and
- Assess the extent to which each project has been adapted to meet the needs of the youth and the community.
The evaluation assessed the efficacy of SNAP® in three project locations: Toronto, Edmonton and a Cree Nation community in Quebec. Key evaluation questions were:
- Are there any between-site similarities and differences found in relation to the outcomes of interest?
- To what extent might these similarities and differences mediate outcomes of interest?
- What factors contribute to any differences and similarities between the sites with respect to location, target group, time, culture, etc.?
- What are the longitudinal changes with respect to the experimental group only?
- What differences and similarities can be identified between the experimental and comparison group sites in respect to the outcomes of interest?
- To what extent can outcomes generated between the three sites under examination be generalized for different target groups and settings?
Evaluation Methodology
Across the three sites examined over four years, 375 children were admitted and began services in the 12-week core SNAP® group(s), with 254 completing 8 sessions or more, resulting in a 68% completion rate for group attendance. Two standardized instrumentsFootnote1 were used to measure program effect, one completed by parents (CBCL; Child Behavior Checklist) and the other by a child's elementary school teacher (TRF; Teacher Report Form). The CBCL and the TRF were administered at pre-program, post-program and at 6, 12, and 24 months post program. EARL was used to assess risk and was administered at pre and post program only.
Program staff administered some of the instruments, the evaluators administered others, and the program developers (CDI) participated in the fidelity audits. CDI created the initial fidelity tool that was adapted for multivariate analysis.
Longitudinal analyses were performed in all three sites comparing pre and post treatment using the previously specified standardized instruments. In addition, a delayed treatment “comparison group” was created in the Toronto and Edmonton sites to compare changes between SNAP® participants and children on a three month waiting list who were not receiving the SNAP® program or comparable services.
For children allocated to a wait list in Edmonton and Toronto, measures were administered at intake, and again at the conclusion of their waiting period. While a longer waiting period was desirable, it was found to be difficult to maintain families requiring an intervention on a waiting list for more than three months. Therefore, at the conclusion of a 3-month stay on the waitlist, children were assigned to a SNAP® treatment group. This compromise means that the evaluator is unable to compare long-term differences between the SNAP® children and the Delayed Treatment Group (DTG) group at the 6-month and 12-month follow-up intervals. Comparison results at more than one post follow-up point increases confidence when determining differences.
All staff at the SNAP® sites confirmed that the children did not receive any SNAP®-related treatment during the DTG period or other comparable services during their waiting period. Using a series of t tests, the SNAP® and DTG group clients were compared at baseline with no statistically significant difference seen. When the SNAP® and comparison group are deemed to be similar, this increases confidence that the results can be attributed to the program and not to the predisposition of the groups.
Insights derived from qualitative methods (key informant interviews with parents and teachers of children completing the SNAP® program) were triangulated with quantitative results to obtain a more comprehensive understanding of why theses results were occurring.
Outcomes Measured
With respect to children, targeted outcomes include decreases in externalizing behaviours (identified as a combination of rule-breaking and aggression on the CBCL and TRF), conduct problems, oppositional defiance, attention problems as well as co-morbid internalizing behaviours (anxiety, withdrawn, depression, and somatic complaints), and increases in pro-social behaviours (e.g., competency and adaptive skills). For parents, outcomes measured include improved child management strategies, reduced family risk factors, and improved relationships with their children.
To obtain results beyond parent and teacher report, the research division encouraged the use of police contact and school suspension data. Data could not be obtained from police in any of the three sites could not be collected due to the age of the children. School suspension data was not standardized at the schools located near the evaluation sites creating a challenge to better understanding how the children's behavioural changes impacted outcomes in the school setting.
Fidelity Assessment
At each replication site, the evaluator in collaboration with CDI conducted a fidelity assessment to identify the extent to which their respective programs were implemented as per standard for SNAP® implementation and to determine what impact program fidelity has on outcomes of interest. A quantitative fidelity assessment instrument and scoring model was developed with scores assigned to each site across several elements including implementation, dosage, quality and participant responsiveness.
Statistical Analysis
brentTo compare the treatment and comparison groups on behaviour problems, a series of analysis of covariance (ANCOVAs) were conducted controlling for the initial level of behaviour problems. To examine the effect of the SNAP® program and assess differences between pre- and post-program at 3, 6 and 12 month follow-up periodsFootnote3. Mixed model approaches (multilevel model for change) were conducted on each targeted outcome variable to examine the overall effect of the SNAP® program in reducing children's antisocial behaviour or augmenting their social competencies. The advantage of using mixed model statistical procedures over the traditional repeated measures analysis of variance (ANOVA) is that the evaluators can use all available information and can provide projections where there is missing data.
With respect to data from the fidelity assessment process, for each site, ANOVA was conducted to examine the difference in the intensity of program services between three risk groups of children as defined by the EARL instrument (low, moderate and high risk). The Spearman correlation was used to examine the relationship whether children with higher risk levels were receiving the appropriate amount of programming after the 12-week core components were complete.
Evaluation Limitations
There were some threats to validity that limited the evaluator's ability to attribute measured changes to the SNAP® program. A delayed treatment group (DTG) comparisonFootnote4 approach was used at the Edmonton and Toronto site because a more robust design (e.g., random control trial) was not feasible due to the priority focus of the funding agreement to service all of the at risk children eligible for the program. While this is a priority, this limits the ability to randomize children into a group that will not receive the program. Moreover, the Edmonton and Toronto DTG groups were small, at 19 and 18 children respectively, resulting in limited statistical power. As well, the waiting period was only three months, prohibiting the evaluators from comparing differences between treatment and comparison group children at longer follow-up intervalsFootnote5.
Only data acquired from parents and teachers were used, limiting the evaluator's ability to triangulate other sources of data such as school suspension records or police data.
Finally, the SNAP® sites evaluated were NCPS funded and selected community-based programs operating in a non-clinical environment. All of these sites were operating different programs prior to being funded by the NCPS. In reviewing the results in this summary it should be noted that the sites had a relatively short period (six months) to learn and implement the SNAP® model. Scientific literature regarding implementation indicates that it takes approximately two to four years to effectively implement an evidence-based model program (Fixen et al., 2009).
Key Findings
Program Participation
Table 2 presents the participation rates across the three sites for the impact evaluation.
Variable | Edmonton | Toronto | Cree Nation |
---|---|---|---|
# Referrals | 147 | 202 | 150 |
# Children Enrolled | 104 | 121 | 150 |
# Children/8 or more of the 12 SNAP® Group sessions | 96 | 109 | 49 |
Attrition Rate | 7.7% | 11.0% | 69.3% |
Average Duration in Program (Weeks) | 61.3 | 55.3 | 6.3 |
Mean Age (years) | 9.1 | 8.4 | 8.8 |
Male/Female (%) | 70/30 | 75/25 | 63/37 |
Primary Source of Referrals | Schools (41%) | Schools (64%) | Schools (86%) |
Average Group Sessions/Child | 10.4 | 10.3 | 6.8 |
Average Group Sessions/Parent | 10.2 | 9.3 | 4.2 |
Program Fidelity
The maximum possible score of the fidelity framework was 102, with Edmonton scoring 90 (very high fidelity), Toronto 83.5 (high) and Cree Nation 54.5 (low). These results indicate that Edmonton and Toronto sites have implemented the program as planned and reached the appropriate target group, whereas the Cree Nation site experienced a number of fidelity challenges related to quality of implementation, child/parent participation and matching children's risk with the appropriate treatment dosage.
With respect to risk and the total number and total hours of other sessions attended, for Edmonton and Toronto there was a statistically positive relationship for these variables and level of risk, providing evidence that children with higher levels of risk received a greater dosage of services relative to lower risk children. The Cree Nation site showed no statistically significant positive correlations suggesting that children at higher risk did not receive a greater dosage of services compared to lower risk children.
Results from the Longitudinal Analysis
The longitudinal analyses assessed changes on key variables at three follow-up points after the program was completed. These results do not include differences between the SNAP® program participants and those that did not receive the program.
The parent measure (CBCL) contained sixteen (16) individual variables including aggression, conduct disorder, rule-breaking, and attention. Table 3, below, illustrates outcomes summarized across the three sites on the basis of the repeated measures design, treatment children only and in respect to the CBCL measure. Results are categorized either as Favourable (F), Unfavourable (U) or Not Significant (NS). Comparing this data from pre to post treatment in Edmonton, Toronto and Cree Nation, the program produced statistically significant and clinically important improvements in the variables respectively. However, the teacher measure did not show any significant improvementFootnote6.
Scale/Item | Edmonton Pre (N = 80) to 24 M post (N = 39) |
Toronto Pre (N = 84) to 12 M post (N = 65) |
Cree Nation Pre (N = 67) to post (N = 50) |
---|---|---|---|
Problem Scales | |||
Externalizing problem | F | F | F |
Rule-breaking | F | F | F |
Aggression | F | F | F |
Attention | F | N | F |
Internalizing problems | F | F | F |
Anxious | F | F | F |
Withdrawn | F | NS | NS |
Somatic | F | F | F |
Total problems | F | F | F |
DSM affective disorder | F | F | F |
DSM anxiety disorder | F | F | F |
DSM somatic problems | F | NS | F |
DSM ADHP problems | F | NS | F |
DSM ODD | F | F | F |
DSM conduct disorder | F | F | F |
Competency Scales | |||
Total competency | F | N | N |
Note: All favorable changes (F) were statistically significant at the p<0.05 level |
Although the above results are promising, it is important to note that program attribution cannot be confirmed with longitudinal results. An attempt was made to increase confidence in the results above by triangulating the quantitative data with some qualitative data (interviews with parents and teachers).
Interviews with parents in Edmonton and Toronto indicated that there were treatment gains in the area of externalizing behaviours and to a lesser extent in internalizing behaviours, parent/child communications and relationships, child sociability and social competence. Moreover, interviews with Cree parents suggest solid treatment gains in children with regards to externalizing behaviour (especially rule breaking, aggression and defiance), parent/child communication and quality of relationship, and overall child sociability. Several parents noted the positive impact SNAP® made on their roster of parenting skills resulting in an increase in parenting confidence. This is summarized below either as Favourable (F), Unfavourable (U) or Not Significant (NS).
Item | Edmonton Parents (N = 19) Teachers (N = 18) |
Toronto Parents (N = 20) Teachers (N = 15) |
Cree Nation Parents (N = 15) Teachers (N = 9) |
---|---|---|---|
Qualitative FindingsFootnote7 | |||
Externalizing behaviours | F | F | F |
Internalizing behaviours | F | F | N |
Child sociability/social competence | F | F | F |
Parent/child/family functioning | F | F | F |
School behaviour | F | N | N |
N: No change noted from pre to post program F: Favorable change U: Unfavorable change |
This longitudinal analysis combined with the qualitative findings contribute to evidence that the SNAP® program contributes to favorable changes in key externalizing and internalizing behaviours that when altered can reduce future involvement in the criminal justice system. Interviews with teachers in Edmonton, Toronto, and the Cree Nation indicated, to a much lesser extent, improvements in child behaviour and sociability within the classroom settingFootnote8.
Results from Comparison Group Analyses
Using a waitlist comparison group allows us to determine if the program changes identified in the longitudinal analysis in Table 3, is a result of the SNAP® intervention. When evaluating changes in children for example, their maturation over time can lead to favorable change even when they do not participate in an intervention. This is why it is important to compare program participant results with children who do not receive the intervention.
Results from both the Toronto and Edmonton sites did not show any significant differences in outcomes between the treatment group and the comparison group. However, this may have been partially due to the relatively small sample size and the potential for reduced statistical power. The results could also be due to the limited time frame participants remained on the waiting listFootnote9. A comparison group was not achieved in the Cree Nation site.
Scale/Item | Edmonton SNAP® Group N= 85 Comparison Group N=19 |
Toronto SNAP® Group N=90 Comparison Group N=18 |
---|---|---|
Problem Scales | ||
Externalizing problem | NS | NS |
Rule-breaking | NS | NS |
Aggression | NS | NS |
Attention | NS | NS |
Internalizing problems | NS | NS |
Anxious | NS | NS |
Withdrawn | NS | NS |
Somatic | NS | NS |
Total problems | NS | NS |
DSM affective disorder | NS | NS |
DSM anxiety disorder | NS | NS |
DSM somatic problems | NS | NS |
DSM ADHP problems | NS | NS |
DSM ODD | NS | NS |
DSM conduct disorder | NS | NS |
Competency Scales | NS | NS |
NS: No statistically significant change between pre and post measures F: Favorable change U: Unfavorable change |
||
Note: These results are based on a pre and post test follow-up with program participants and a group who did not receive the program over a 3 to 4 month period. |
Economic Analysis
While cost per participant was computed for each of the three sites as shown in Table 4 below, no cost-effectiveness analysis (CEA) was possible at the conclusion of the program since there were no statistically significant effect sizes for any scale related to the comparison group analysis in Edmonton or Toronto. Regarding cost per participant in the three sites under examination, these were higher than standard costs identified by CDI. These higher costs may be a result of “start-up” sites fees where significant time and expense had been invested at the outset to recruit and train staff, establish initiatives, build awareness and trust in their communities, stimulate referrals of eligible children and ensure staff were well equipped for group delivery. By their nature, these developmental activities raise the cost per participant in the short term therefore, the cost economies and process efficiencies at CDI that keep costs per participant low, are not evident for a community initiating a SNAP® program and operating it for a short, fixed period of time.
Site | Cost/graduate | Cost/participant |
---|---|---|
Edmonton | $10,451.65 | $9,647.68 |
Toronto | $16,811.94 | $15,144.64 |
Cree Nation | $35,303.78 | $16,633.51 |
CDI Guidance | -- | $6,735.00Footnote11 |
Lessons Learned and Recommendations
There are a number of lessons learned as a result of conducting the National Multi-Site Evaluation of the Stop Now and Plan Program. The following lessons learned should be considered for future programs being implemented by non-governmental organizations that are considering implementing or evaluating a comprehensive evidence-based program for children under the age of 12.
Adequate Time for Program Development
The SNAP® program is resource intensive in terms of intake, case management, evaluation, data management, client outreach and other processes necessary to ensure high fidelity and concordance with CDI's licensing agreement, especially for community-based groups without clinical management expertise. Program managers must set reasonable expectations of these various tasks, ensuring ample time for program development and acclimatization to the SNAP® model.
Recruitment
Program managers should have close relationships with police and the school board. These institutions will help with appropriate referrals and will ensure the necessary information about police contact and school suspensions are provided and included for potential analysis.
Access to Clinical Staff
SNAP® projects that are not being implemented within a clinical setting may require enhanced training to increase the likelihood of producing expected behavioural change typically demonstrated by the SNAP® model. Program staff without a clinical background had difficulty assessing need for more programming beyond the 12 week core component.
Translation of Materials
Evidence-based programs still need to be adapted to meet cultural needs, geographical location, gender and other unique factors that may contribute to changes in program implementation. In particular, the Cree children and their parents required the materials to be in their language to be able to respond and engage effectively.
Evaluation Methods
Although a quasi-experimental evaluation design with a comparison group or equivalent design is always useful for determining the efficacy of an intervention, when working with programs with small sample sizes, it is important to incorporate strong qualitative methods. It is recommended that realist approach methods that consider the relationship between context, mechanisms and outcomes be explored and used to complement traditional evaluation designs.
Instruments to Measure Outcomes
It is important that programs attempting to implement the SNAP® program recognize that administration and data entry of the required instruments are time-intensive. For program managers and evaluators that wish to add additional measures, it is important to assess whether these outcomes will have added value.
If evaluators are able to work with a large enough sample, attempts to explore the feasibility of reducing CDI' required standardized instruments to a manageable number of questions is recommended. Various statistical tests such as factor analysis can be used to reduce the number of questions in an instrument while still maintaining construct validity. Efforts to make the instruments more manageable will enhance program engagement, reduce attrition and will allow for the testing of other complementary outcomes that may be introduced in each new evaluation.
Honoraria
Non-financial and financial incentives should be used to encourage participants, parents and teachers to participate in both the pre and post test follow up sessions. In addition to using financial incentives, evaluators and program administrators are encouraged to develop creative non-financial incentives for participants that will add value to their learning and engagement in the measurement process.
Standardized Protocols for Teacher Data
Children will often have more than one teacher during the program and evaluation cycle. To ensure consistency between sites, a standard protocol should be implemented to ensure the results have similar meaning. For example, a teacher who observed a child for three hours may report unfavorably as a result of not having the appropriate time to observe changes over a reasonable time period. Lack of standardization in this area can contribute to inaccurate reporting.
Site Specific Lessons Learned and Recommendations
Cree Site
The following lessons were learned from the evaluation at the Cree site:
- Aboriginal youth who reside in rural communities may have more difficulty completing the required number of cognitive-behaviour therapy sessions due to distance between the home and the program location;
- Aboriginal youth may have more difficulty understanding the content if their first language (Cree) and cultural norms are not adapted in their curriculum; and
- The parents of Aboriginal youth at risk who reside in rural communities have a higher rate of concern related to stigmas and therefore their involvement in parental groups may be significantly reduced.
These three key variables in the Cree site may have affected the achievement of typical outcomes. These confounding variables were not present in the other two sites.
References
Astwood Strategy Corporation; Technical Authority and Contract Manager: Donna Smith-Moncrieffe, CSCCB, Research Division.
Fixsen, D. L., Blasé, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19, 531-540.
Smith-Moncrieffe, D. & Léonard, L. (2014). The Feasibility of the Realist Approach with Canadian Evaluations. Presentation at the Canadian Evaluation Society Conference, Ottawa Convention Centre, June 17, 2014.
Footnotes
- 1
developed by the Child Development Institute (CDI): http://www.childdevelop.ca/about-us.
- 2
Clinical levels are assessed by standardized instruments. A T score above 70 indicates the child’s externalizing and internalizing behaviours puts them at risk of future criminal justice contact.
- 3
Follow-up data was achieved up to 24 months for the Edmonton site.
- 4
Due to challenges with sample size at the Cree site, keeping children on a waiting list for use in the delayed treatment group design was not feasible.
- 5
Conducting a randomized control experiment includes the need to randomly assign children to the experimental and control group to ensure that each group is equal on a number of characteristics. Conducting this type of study is not likely using the funding vehicle of a contribution agreement. The contribution agreement is funding a vehicle used to ensure youth receive the intervention—this arrangement can compromise the need to randomly assign youth to the program and a control group for research purposes.
- 6
All the statistically significant effect sizes generated were low ranging from 0.06 to 0.25.
- 7
For the final annual reports for each site, a total of 54 parents were interviewed using a standard interview guide (Edmonton, N = 19; Toronto, N = 20; Cree Nation, N = 15), along with 42 teachers (Edmonton, N = 18; Toronto, N = 15; Cree Nation, N = 9). For each item listed, all parent/teacher commentary was analyzed to determine program effect. Where there was clear qualitative evidence of program effect for each item, the site was allocated Favorable Change (F). Where qualitative findings demonstrated little if any change, sites were allocated “N” for No Change. No site reported Unfavorable Change (U).
- 8
Further information about the qualitative findings can be reviewed in the final report.
- 9
Ideally, the evaluation plan had the participants on the waitlist for six months, however, the program did not have enough participants for the program which eventually resulted in the need to shorten the waitlist period.
- 10
SNAP® Program Costs are typically calculated by the degree of service delivery for Low-High ($1,729), Moderate-High ($4,166) or High-High ($8, 503) risk levels. Included are indirect costs (e.g., case coordination, program supplies, intake services) estimated at 50 %. Farrington, D. P. & Koegl, C. J. (2014). Monetary benefits and costs of the Stop Now And Plan program for boys aged 6-11, based on the prevention of later offending. Journal of Quantitative Criminology, DOI: 10.1007/s10940-014-9240-7.
- 11
Based on a high risk boy assessed by the EARL instrument.
- Date modified: