NEW DIRECTIONS EVALUATION

Executive Summary: Resident Outcomes and Program Environment

Methodology

The evaluation followed a “pre-post” design in which the intervention residents served as their own controls. After admission, evaluation staff administered the baseline resident survey consisting of measures of symptoms, medication compliance, substance use, quality of life, and functioning.  At 6 and 12 months after their baseline resident survey, residents completed the same interview minus items that were required only once, such as demographics. 

The evaluation team concluded baseline enrollment into the evaluation as of April 30, 2005 with 147 participants enrolled in the study. Six-month follow-up was completed as of December 31, 2005, with 108 of a potential 146 six-month follow-up interviews (1 resident passed away before the evaluation team was able to locate and interview him). This represents a six-month follow-up rate of 74.0%. As of June 30, 2006, the evaluation team finished twelve-month follow-up for the evaluation, completing 102 of a potential 145 twelve-month follow-up interviews (2 residents passed away before the evaluation team was able to locate and interview them). This represents a twelve-month follow-up rate of 70.3%. While the evaluation study of the New Directions program has been completed, New Directions received a no-cost extension of the grant through December 31, 2006 and continues to enroll and follow-up with residents with the GPRA interview, as required by SAMHSA. Therefore, while enrollment and follow-up rates for the resident outcome survey are final, GPRA enrollment and follow-up is still ongoing.

Client services (i.e., number of resident service contacts) were tracked on a Service Tracking Form by service delivery staff.  Also staff tracked employment information for all residents who obtained employment after enrollment. All data was entered into the evaluation database.  To assess the New Directions program context, residents and staff completed a program survey six months after the evaluation began and six months before it ended.  Two series of resident and staff focus groups were also conducted at these time-points.

Data Analysis and Feedback

Resident Interviews

Demographics

Upon entering New Directions North (NDN), the average resident is 46.7 years of age with slightly more than a high school education. Somewhat more residents are African American/Black race (43.5%) than Caucasian/White (35.4%), and just under one-fifth (17.7%) identify as Latino/Hispanic.

Housing

At the twelve month assessment, most residents were no longer homeless. Regarding residents’ living situations immediately prior to entering NDN, just under half (40.8%) of the residents report having been living in a shelter or on the street.  The remaining 59.2% of respondents were living in a variety of conditions ranging from hospitals, jails, or other institutions to private residences.  At twelve months post-baseline only 8.8% of respondents are in living situations that would classify them as homeless.  However, about half of respondents are still being housed in some form of residential treatment (53.9%). Nevertheless, 35.3% of respondents report living in an apartment or house at twelve months post-baseline.

Employment

At the twelve month assessment, about one third of residents had obtained a job at some point during the study. While none of the respondents was employed at the time of the baseline interview, 24.5% of follow-up respondents were employed at the time of the twelve-month follow-up.  In addition, 36.7% (54 participants) of all evaluation participants (n=147) worked at a job for pay at some point after enrolling in NDN, with the number employed at permanent jobs somewhat lower. Further, 5.9% of follow-up respondents report being enrolled in school or job training.  Average monthly income from all sources was about the same at baseline as at twelve-month follow-up.

Mental Health 

Though they still are not as symptom-free as the general population, respondents report significant improvements in their mental health symptoms and functioning at the twelve-month follow-up as compared to baseline.  Specific improvements at follow-up include a significant 11.7 average point drop in overall symptoms of psychological distress as measured by the Colorado Symptom Index (CSI), a significant decrease in average mental health severity on the Addiction Severity Index (ASI) Psychiatric Composite score, a significant 9.1 point increase in emotional functioning as measured by the Global Assessment of Functioning (GAF), and a significant 19.5 point improvement on the Medical Outcomes Study (MOS) Mental Health Scale.  In addition, respondents report 5.2 fewer days (out of 30) of psychological disturbance and improved medication adherence at follow-up, with 74.7% of respondents reporting that they take most or all of their prescribed psychiatric medication, up from 50% at baseline.

Health

Residents improved their health over time, but are in poorer overall health than the general population, with higher reports of pain, and lower health and functioning scores.  Respondents reported significant improvements in their pain and general health scores at twelve-month follow-up as compared to baseline, with improvements of 6.4 points and 7.2 points respectively.

Social

Although most of the improvements in social measures do not rise to the level of statistical significance, there were some areas of significant improvements.  Respondents report a significant improvement of 11.9 points in overall social functioning and also report significantly more people that they felt they could turn in times of need.

Substance Abuse

Respondents’ reported alcohol and drug consumption decreased significantly at the twelve-month follow-up, as compared with the baseline.  On the ASI measures of substance use, the overall average use rates include an alcohol composite score of 0.1 (as compared with 0.3 at baseline) out of 1.00 and a drug composite score of 0.0 (as compared with 0.1) out of 1.00.  Both alcohol consumption rates and intoxication rates decreased from baseline to twelve-months, by an average of 7.8 days and 5.6 days respectively, both statistically significant decreases. Further, while the overall use of all illicit substances also decreased significantly from baseline to twelve-months, there were also statistically significant decreases in use of every individual drug; opiates (2.5 day decrease), cocaine/crack (6.3 day decrease), depressants (1.6 day decrease), marijuana (2.5 day decrease), and amphetamines (1.7 day decrease)

Program Services

In general, the improvement in respondents’ mental health, as measured by the GAF and by the MOS Mental Health Scale, has a strong positive correlation with several of the types of services received at New Directions North. Respondents who had better scores on the GAF and MOS Mental Health at twelve months as compared to baseline received more mental health services, chemical dependency services, and aftercare services. Respondents’ scores on the GAF were also positively correlated with their receipt of social recreation services, and respondents’ scores on the MOS were also positively correlated with receipt of family services and vocational services. In addition, respondents who received more aftercare services had fewer days that they were bothered by psychiatric problems at follow-up as compared to baseline. In terms of substance use, these outcomes were overall not correlated with receipt of services; however, respondents who received more family services used less alcohol at twelve months as compared to baseline.
                       

Respondents’ physical health and functioning was also related to their receipt of services. Respondents who received more social recreation and vocational services had better MOS General Health scores at twelve months as compared to baseline, and those who received more aftercare services had better physical functioning scores. Interestingly, physical functioning scores were negatively correlated with receipt of supportive services, meaning that those who received more supportive services had lower physical functioning scores at twelve months as compared to baseline.
                       

Finally, respondents’ scores on social measures were also related to their receipt of services. Respondents who received more chemical dependency services, social recreation services, and vocational services had more people in their social networks at twelve months as compared to baseline. As with physical functioning scores, receipt of supportive services had a negative relationship with social functioning, with respondents who received more supportive services scoring worse on social functioning at twelve months as compared to baseline.

Program Environment: Resident and Staff Surveys

Results from the resident and staff surveys indicate that residents and staff had a number of differing and similar views about the program climate at NDN at both time points, and that some of these views changed over time. 

Time 1
When considering staff and resident differences at time 1, staff tended to believe that the program was significantly more open, more empowering, and more helpful than did residents. In contrast to staff, residents tended to find that the program provided significantly less assistance with paid employment supportive services, less post-program housing support and assistance with obtaining public benefits, and less emphasis on medication management. Finally, residents tended to feel more negatively than did staff about the level of program confidentiality and staff commitment to the program, while feeling much more confident about resident commitment to and responsibility for the program.
Residents and staff also demonstrated agreement regarding several program elements at Time 1, with both feeling that program personnel cared about their clients and that the program environment was generally friendly, open to family involvement, and placed a high value on the importance of work for clients. Staff and residents agreed that that there were good levels of staff-resident and resident-resident respect, that the program goals were clear to residents, and that program rules were generally reasonably enforced. Finally, both staff and residents agreed that residents understood, implemented, and participated in program processes in a physically and emotionally safe environment in general, and that, while having a fairly limited amount of choice about participating in the treatment process, they still participated fairly actively.

Time 2
Compared to residents, staff at Time 2 still tended to believe that the program was significantly more open and more empowering. In addition, at Time 2 there were significant differences between the groups’ perceptions of the program’s friendliness, with staff feeling much more positively than residents. Furthermore, residents felt significantly less positive about the program’s friendliness at Time 2, as compared to Time1. Compared to staff, residents still tended to find that the program provided less post-program housing support and assistance with obtaining public benefits. As at Time 1, staff still felt significantly more positively than residents about the program’s emphasis on medication management. At Time 2 staff also felt more positively than residents about the program’s emphasis on family involvement. As at Time 1, residents tended to feel more negatively than did staff members about staff commitment to the program; while feeling much more confident about resident commitment to and responsibility for the program. However, unlike at Time 1, residents felt less positively than staff about residents’ participation and trust in the program, and also felt much more negatively than staff about respect among the residents and the enforcement of program rules. Residents’ views about resident respect and rule enforcement were significantly more negative at Time 2, as compared to Time 1.
In terms of areas of agreement at Time 2, both staff and residents felt that program staff cared about their clients, treated them with respect, maintained their confidentiality, placed a high value on the importance of work, and provided assistance with paid employment supportive services. Residents and staff agreed that the program was very helpful to the residents and that program goals were clear, though residents played a limited role in the treatment process. Finally, both staff and residents agreed that residents understood and implemented the program processes in a physically and emotionally safe environment.

Program Environment: Round 1 Focus Groups

1. Residents and staff were fundamentally appreciative of NDN, but raised several concerns and discussed components of the program that are need of improvement.
2. Staff expressed that clarification of the relationship between 257 and 116 is necessary, and resource (e.g., food) management between the two is in need of improvement.
3. Food at 257 is in many cases unacceptable, according to both staff and residents.
4. Staff indicated that staff relationships are changing for the better (though better communication is desired) in large part due to less micromanagement.
5. According to staff, the more relaxed environment has taken a toll on residents, who are making excuses for not participating fully in the program and who are “working” or “playing” the staff.
6. Residents described a harsh, “boot-camp” environment in which they are not well treated and in which their mental illnesses are not treated with sensitivity. This included being subjected to “attack therapy” and feeling forced to participate in activities even when they did not want to participate due to their illnesses. Some staff stated that “attack therapy” was justified.
7. According to residents, staff maltreatment was due, in large part, to a lack of staff professionalism and mental health/dual diagnosis training.
8. Residents depend on each other for recovery, and they do not feel that the program follows the 12-step philosophy except in the aspect that they are helping each other as brothers in recovery. Residents expressed that there are too many in-house 12-step meetings, in that they know each other well and are therefore not exposed to new perspectives on recovery.
9. Staff and residents expressed that additional personnel are needed; this personnel would ideally consist of a nurse, on-site therapists/psychologists, and case managers.
10. Residents were fairly unanimous in their depiction of a lack of psychological support overall; several had not seen a psychologist or a psychiatrist, and several had seen psychiatrists for medication purposes only.
11. Residents expressed significant concerns about their financial situations and requested clarification of the ways in which their money was being allocated.
12. Residents expressed that vocational rehabilitation options were limited and not always appealing.

Program Environment: Round 2 Focus Groups in Comparison to Round 1

1. While residents and staff expressed issues between 257 and 116, few expressed a strong need for clarification or adjustment of the relationship.
2. There were no complaints about the food; if anything, participants complained that they were gaining weight because the food was so good.
3. While communication among service coordinators and case managers seems vastly improved, communication between program staff and clinicians seems to pose a challenge.
4. The environment was no longer described as “boot camp” by the residents; in fact, when prompted with this finding from round 1, residents definitively disagreed and generally expressed that the structure was acceptable but that, for some, more clarity of procedures and directions was desired.
5. No residents described being subjected to attack therapy or other subjugating treatment.
6. There was a qualitative shift from residents focusing on how they were being (mis)treated as dually diagnosed individuals to how they felt the program should be run as a dual diagnosis program.
7. Staff training continues to be an issue, particularly with regard to the question of whether the recovery-oriented staff should learn/know more about mental health. Some staff described their on-the-job training and growing awareness with exceptional self-criticism and introspection.
8. Residents continue to depend on each other, but felt that their relationships were being undermined by a logging system whereby they were required to report a certain number of their peers’ infractions; the logging system was the main theme raised in the round 2 resident focus groups that was not raised in round 1.
9. Staff and residents called for additional personnel (in particular, another clinician) but not with the sense of urgency apparent in round 1. Several noted that the program now has the full compliment of case managers and sufficient service coordinators.
10. A lack of psychological/clinical support for residents remained a concern.
11. Several staff expressed that residents should have more opportunities to educate themselves about their illnesses and medications.
12. Several residents expressed concern about how their money was being spent, though this concern was not as widespread as in round 1.
13. No residents expressed that they observed residents being pushed out of the program early; if anything, residents and staff alike expressed appreciation that residents were given the time to complete their treatment course.
14. Finally, there were fewer concerns about vocational options, with more residents describing more options and fewer residents expressing that there were a limited number of options.

Recommendations

Here we revisit the initial recommendations and actions taken, and we then provide a new set of recommendations based on the second round of findings.

Final Recommendations

1. Engage in the evidence base on how to structure an integrated dual diagnosis program. NDN could benefit from learning more about the state of the art in integrating substance abuse and mental health treatment (see, e.g., Brunette & Mueser, 2006).  For example, SAMHSA has published a toolkit called, “Co-Occurring Disorders: Integrated Dual Disorders Treatment” that has a great deal of information about the evidence base for integrating the two treatments and specific strategies and self-assessment tools for its implementation. (http://mentalhealth.samhsa.gov/cmhs/communitysupport/toolkits/cooccurring/)

1. Continue to enhance the vocational rehabilitation aspect of the program. While qualitative satisfaction with this aspect seemed greatly improved, the rate of employment is still low. Consider encouraging staff to be more engaged with residents’ vocational achievements. It is expected that the newly acquired SAMHSA funds will greatly enhance NDN’s capacity in this area.

1. Make efforts to strengthen the communication between the clinicians and the program staff. Consider developing a clear decision-making policy such that residents know when decisions are final and cannot be changed at any level. Meeting structures should be put in place that guarantee regular communication.

1. Revisit the logging system and structure it in a way that does not undermine the peer support among residents.

1. Continue to look for ways to provide more clinical support. Consider providing more Dual Recovery Anonymous (DRA meetings). A listing of existing meetings in Southern California is available: http://www.draonline.org/meetings_dra/usa/california_south.html. DRA offers a free guide for starting a group: http://www.draonline.org/copyright/startpac.shtml.

1. Continue to provide trainings for program staff on mental illness and medications in order to increase the sensitivity of the staff toward the residents. The US Psychiatric Rehabilitation Association now offers low cost, remote access trainings in various topics relevant to the residents at New Directions. See the current listing at: http://www.uspra.org/i4a/pages/index.cfm?pageid=3286.

1. Provide more opportunities for residents to learn about mental illness and medications. Psychoeducation is thought to be an essential component of illness management (Mueser et al. 2002). Consider re-instituting an incentive program for residents to engage in education about their illnesses. Consider providing education/training during the first phase when residents cannot be involved in vocational rehabilitation. This could ameliorate the frustration that residents expressed about not being able to pursue vocations until the second phase.

1. Continue outcome and services data collection for quality improvement purposes. Over the next several years, this will involve collaborating with MIRECC staff in the upcoming SAMHSA evaluation.