Evaluation Plan

Projects involving community-based collaboration can be difficult to evaluate due to several features including complexity across multiple services and levels (both horizontal and vertical complexity); broad range of outcomes; flexible and evolving interventions, contextual issues affecting the community; and the absence of a comparison or control group (Kubisch et al, 1995). A recommended evaluation approach will be used (Fishman et al., 2000) that minimizes these difficulties as well as focuses on measuring outcomes directly resulting from project activities. Recommended approaches include articulating a theory of change while developing a logic model; using the theory of change to evaluate the collaboration; and conducting a time-series analysis in order to observe outcomes over time, before the start of the intervention, and following it (Kubisch et al., 1995). Finally, an experienced and knowledgeable program evaluator (Irvin) with a history of conducting evaluations in social service and educational settings will be planning and conducting the evaluation.

Both outputs and outcomes of the FEAT model will be analyzed in the evaluation process. Outputs include those countable events or features resulting immediately (direct outputs) and intermediately from Project FEAT, such as improved policies, new procedures, number of collaborators, number of referrals to early intervention and drug treatment services. Outcomes, including the measurable effects of outputs (i.e. efficacy), will be evaluated such as increased knowledge of CAPTA requirements, improved knowledge of issues in prenatal drug exposure.

Outputs and outcomes can be categorized into three major categories: 1) service integration, 2) child development, and 3) caregiver/family functioning. (Note: In 2001, the Department of Health and Human Services Administration for Children and Families implemented the Child and Family Services Review (CFSR) process to increase states' accountability. Outcomes indicated with "*" are in direct alignment with outcomes reviewed through this process. The alignment of these outcomes and information gathered during the course of the proposed project will assist local and state Child Welfare systems with future reviews.)

The following is an outline of projected outcomes. These will be modified and refined through model planning, development, and implementation.

Service Integration Outcomes

• All stakeholders (staff, agencies, evaluators, etc.) will work together to develop a basic theory of change, or reason why activities may lead to outcomes.
• Logic model is refined.
• Steps in model are identified.
• Expected connections between program design and outcomes are identified.
• State and local agency representatives are satisfied with working group progress.
• Collaboration is improved, as measured by on-going development of a community "collaborative framework" (Fishman et al., 2000).
• Clear policies and procedures are developed at state and local levels resulting in
• Increased numbers of identified children from the medical community
• Increased timeliness of identification, assessment, referral and treatment.

Child Development Outcomes

• *Children receive adequate services to meet their educational needs.
• Periodic developmental screening
• Access to needed developmental services
• * Children receive adequate services to meet their physical and mental health needs.
• Children receive periodic well child checkups and access to needed medical/health services
• Children receive periodic screening and access to needed mental health services
• *Children are protected from abuse and neglect.
• Increased numbers of children exposed to prenatal substance abuse are identified from medical and other providers.
• Decreased numbers of repeated maltreatment are reported from Child Welfare.
• Children participating in Project FEAT model will experience improved outcomes including increased safety, reduced incidence of repeat maltreatment, increased access to needed medical, educational and mental health services.
• Prenatally substance exposed children are not repeatedly maltreated.
• Children are screened and referrals made as needed to early intervention.
• Children's medical needs are assessed at in-take, and services provided as necessary.
• Children receive well child checks as scheduled.
• Children's mental health needs are assessed and referrals made as necessary.
• Children and families participating in the FEAT model are safely maintained in their homes whenever possible and appropriate, resulting in:
• Services are provided to family to protect children in home and prevent removal.
• Services provided to families reduce risk of harm to children.
• Prenatally substance exposed children have permanency and stability in their living situations.
• Services provided to families reduce length of involvement with child welfare/protective services.
• *The continuity of family relationships and connections is preserved for children.
• Relationship of child in care with parents is enhanced due to provided services.
• Families receiving FEAT home visitor have a decreased number of removals of children from their care.
• *Families have enhanced capacity to provide for their children's needs.
• Parents feel that they contributed to their case planning (goal setting).
• Families receiving FEAT home services have an increased participation rate in targeted services.
• Services provided to families result in increased responsivity of parent (s) to child's needs.
• Needs of diverse children and families are served by project activities.

Caregiver/Family Functioning Outcomes

• *Families have enhanced capacity to provide for their children's needs
• Families are involved in development of service plan.
• Diverse families receive and participate in culturally-appropriate needed services.
• Families experience enhanced parenting ability
• Families provide enhanced quality of home environment
• Parents experience decreased depression
• Parents remain clean and sober

* Outcomes that align directly with outcomes reviewed during the Child and Family Service Review (CFSR) process

Outcome Measures

Measures fall into three categories: service integration, child development, caregiver/family functioning and safety. Measures for collaboration and service integration will include satisfaction surveys and an on-going measurement of discrete levels of "collaborative functions" (Fishman, et al., 2000). Proposed child development measures include the Ages & Stages Questionnaires and Child Welfare data base and file reviews. Measures for caregiver/family functioning include the Parenting Stress Index, the Home Screening Questionnaire, the NCAST, Center for Epidemiological Studies Depression Scale (C-ESD) and Child Welfare database and file reviews. In addition, staff will review participant Child Welfare files to collect child and family demographic data such as child ethnicity, caregiver education and caregiver income.

Service Integration Measures

All individuals who participate in FEAT activities (e.g., state and local working group participants, community partners, caregivers) will be asked to complete evaluation measures including Likert scales measuring satisfaction with FEAT objectives. These measures will be developed during the planning phase.

The FEAT logic model will be continuously reviewed and refined to document the theory of change. A well specified theory will reflect activities that were implemented as planned and outcomes that followed the activities, with desired results and no obvious pervasive contextual shift that might have impacted outcomes.

Child Development Measures

Ages and Stages Questionnaires (ASQ) . The ASQ (Bricker, & Squires, 1995, 1999; Bricker, Squires, & Mounts, 1999) is a parent-completed developmental screening questionnaire, designed to screen for developmental problems for young children from birth to 5 years. The validity and reliability of the ASQ have been studied with favorable results. The ASQ is recommended for clinical use for early identification of developmental delays ( American Academy of Pediatrics Committee on Children with Disabilities, 2001; Filipek et al., 2000) and used widely in programs nationally, state-wide and locally in Lane County for developmental screening and monitoring. The Ages and Stages Questionnaires: Social-Emotional (ASQ:SE) will also be used, measuring problem behaviors in social, emotional, and behavioral areas. Many protective service programs have adopted these tools to measure potential developmental delays in infants and toddlers. The ASQ is available in English, Spanish, Korean, and French.

Child Welfare Data Base and File Review

Outcome specific data that will be retrieved from files includes timeframes for investigation, assessment and placement of infants after initial referral information has been gathered. Reports of incidence of repeat maltreatment, number of placements for child and documentation of receipt of needed services for child's educational, physical and mental health needs will be gathered. Information about the numbers of children born prenatally exposed to substances who are referred to and enter the system will be gathered.
Caregiver/Family Functioning

Nursing Child Assessment Satellite Training (NCAST). The Feeding and Teaching Scale of the NCAST (Barnard, 1976 ) will be used for rating parent-child interactions and for guiding intervention activities. The Feeding Scale, containing observable communication and interaction behaviors will be used with infants up to one year of age. The Teaching Scale will be used with infants from birth to three years to observe and rate caregiver-child interactions during a novel situation in order to assess a dyad's strengths and areas needing improvement. The 73 items of the Teaching Scale are organized into six subscales related to parent and child behaviors, with headings similar to those in the Feeding Scale. The NCAST is the most widely used research tool for measuring and intervening with parent-infant interactions and is considered to be the "gold standard" for the birth to three year age range. It is a valid and reliable measure for observing, rating, and intervening with interactional behaviors (Kelly & Barnard, 2000).

Home Screening Questionnaire (HSQ). The Home Screening Questionnaire (Coons et al., 1981) adopted from the Home Observation and Measurement of the Environment (HOME) (Caldwell & Bradley, 1984) will be used as a measure of the quality of cognitive stimulation and emotional support provided to the child by the family. The HOME has been widely used in research studies and has adequate reliability and validity. The HSQ is a brief parent-completed instrument tapping similar constructs with straightforward data collection techniques, taking parents 15-20 minutes to complete. The Birth to 3 year form will be used.

Parenting Stress Index/Short Form (PSI/SF) (Abidin, 1990) will be used as a standardized, validated measure of parent stress levels. It is hypothesized that access to a family advocate who provides support and increases accessibility to community resources will reduce family stress levels (Olds & Kitzman, 1993). Studies on the PSI/SF have found the measure to be a valid and reliable scale (Abidin, 1990).

Center for Epidemiological Studies Depression Scale (CES-D; Radloff, 1977) will be used to assess mothers' level of depression. This is a 20-item scale with a 1-4 Likert response scale, asking about frequency within the past week of symptoms associated with depression. Validity and reliability have been demonstrated (Radloff, 1977). Maternal depression will be measured as part of emotional functioning. Recognition of depression and referral to community resources may reduce stress and improve parent-child interactions and child functioning (Sameroff & Fiese, 2000; Shonkoff & Phillips, 2000).

Child Welfare Case Record Review. Information regarding a caregiver's involvement with case/service planning, timeliness of implementation, participation and completion of required services will be gathered from case reviews.

Data Analyses

A time series approach will be used to measure outcomes on an on-going basis. Service integration and child and family outcomes will be measured prior to implementation and on going as the model is implemented. The logic model will guide how specified FEAT activities should lead to short, medium, and long term outcomes. The theory of change should begin to explain why the results should be attributed to the intervention, or to suggest alternative explanations. The outcomes will then be fitted to a line or curve to estimate the impact of the intervention. A counterfactual strategy will be adopted-comparing outcomes that would have occurred in the absence of collaboration with outcomes that occurred based on FEAT implementation.

Data for the FEAT Project will be derived from multiple sources using multiple measures. Collaborator and services provider report, parent/caregiver report, developmental and social emotional measures, and child welfare case review will be included. Data management procedures similar to those used in previous Early Intervention Program projects will be employed to help ensure the integrity of results. A management plan will be created for each instrument and procedures including coding, administration, data set cleaning, data entry, verification, and locked storage of originals being specified.

Replication

During Year 3, replication of Project FEAT will occur in Jackson County, Oregon. Jackson County was selected for replication because of the interest of county human services and medical personnel (see letters of support, Appendix A), and because some model features are already in place such as a perinatal working group (known as the Perinatal Task Force) and a part time Family Advocate. During Year 3, follow-up of the Lane County model will continue, with the Project Liaison working approximately 75% FTE in Jackson County and 25% in Lane County , continuing with collaboration meetings and data collection. The majority of time will be focused on model replication and refinement in Jackson County . (As CAPTA systems are put in place in Lane County , less time should be required for liaison activities.) The Family Advocate position will be funded at .50 FTE in Jackson County , to supplement the .50 FTE position already in place. This additional time will enable Jackson County to fully implement the proposed FEAT model. A half time Family Advocate will continue in Lane County , with additional funds coming from community and foundation monies as needed.

Documentation of the replication will be carefully evaluated, with consideration of local context, cultural/ethnic context, service integration issues (Leutz, 1999), county agency needs, individuals and systems involved in the process, and additional replication issues (Miller, Townsend et al., 2001). A logic model and theory of change will be developed and evaluated by the Perinatal Work Group during the replication effort. Replication data will assist in developing a FEAT Replication Plan for wider dissemination.

Dissemination Plan

A dissemination plan will be written in collaboration with participating agencies and providers during Year 1 and modified as needed during Years 2 and 3. Model development products and results will be made available to interested agencies and professional organizations including child welfare, public health, human services, hospitals, pediatric and early childhood organizations. A dissemination web site will be prepared by UO project staff and maintained by the University Center on Developmental Disabilities (UCEDD) after the completion of the project. Activities related to CAPTA provisions and Oregon and national implementation is a priority area for the UCEDD and additional assistance of dissemination will be offered as appropriate. The project directors have a long history of publishing manuscripts and preparing presentations for a national audience and they will continue such activities with this project. On-line and journal articles will be prepared and submitted. While presentations will be on-going throughout the project, final outcome presentations will be presented at professional meetings. The Project FEAT model will be clearly documented so that replication is possible at additional sites. Finally, project final reports will be prepared and disseminated.

Factors that might accelerate or decelerate the work

Potential obstacles to effective interagency collaboration exist. The uniqueness of agency structures and systems, selective communication, perceived differences in status and power, and conflicting professional and organizational priorities are all potential obstacles to the success of collaboration (Robinson et. al, 2003). However, this project proposes to include a number of essential elements of successful collaboration, including a recognition of the need for collaboration; clear, concrete goals; effective informal relationships; clear policies and procedures; training; clear benefits to all parties; shared responsibility; and a clear mechanism for coordination (Robinson, et. al. 2003).

There is clearly a shared recognition of the need for collaboration on this issue, illustrated by the strong letters of support received for this project's effort from state and local health, medical, drug and alcohol, education and social service agencies. At state and local levels, agency representatives have been meeting on this and related issues, and have developed excellent working relationships. The commitment of agencies and perceived urgency for CAPTA policies may accelerate the planned activities. It is clear that Lane County's identification rate of babies born prenatally exposed to substances is lower than expected, based on state statistics, and agency providers are anxious to improve procedures. Some counties in Oregon have agreements between Child Welfare and local hospitals regarding referrals, but no policies or procedures are consistent statewide (DHS-CAF mtg. minutes, 2005).

According to the DHS Safety System Improvement Plan (DHS website, 2005), Oregon 's practice of "localizing" policy and procedures results in inconsistent application of statewide safety intervention models. Historically, local community partners and advisory groups have had strong influences in setting priorities which has resulted in community-defined models that are inconsistent with safety requirements. DHS has stated the need for increased statewide consistency for clarifying policy, establishing procedures and requiring on-going monitoring. The FEAT model can assist DHS in their identified goal, providing a focus on infants born prenatally exposed to illegal substances.

Plan for continuing FEAT model beyond federal funding

Features of the FEAT model are already in place in Lane and Jackson counties, including state and local groups and on-going communication among agencies regarding CAPTA provisions. The continued funding of the Family Advocate position will be vital to continuing the model. Several of the participating agencies including the Relief Nursery and DHS have suggested a willingness to contribute funds for a family advocate position, depending upon the success of the project. Finally, private foundations supporting children and families in Oregon will be approached for funding as necessary.

Reason for taking the proposed approach

The Family Early Access to Treatment (FEAT) model will increase the knowledge base for how best to identify and serve infants born affected by illegal substances. The model builds on projects funded by state CAPTA monies, and adds new elements that will assist the state in providing guidance to counties. The focus of the Oregon CAPTA five-year plan for 2000-2004 was to explore ways to protect children, strengthen families, and counteract the destructive effects that alcohol and drug abuse have on families. Project FEAT will employ a family-centered, ecological approach guided by a clear logic model for evaluating effects.

(3) Organizational Profiles

Key Organizations, cooperating entities, consultants, key individuals

Following is a description of key organizations involved in Project FEAT, including the University of Oregon 's Center for Excellence in Disabilities (UCEDD), the Child Development and Rehabilitation Center (CDRC)/Oregon Health Sciences University (OHSU), state and local collaborating agencies and the Lane County Relief Nursery. A description of key project personnel follows.

The Center on Human Development, University Center for Excellence in Disabilities (UCEDD), emphasizes the areas of infants and children at risk, children having disabilities, and children and youth demonstrating anti-social and violent behavior patterns. The UCEDD is lifespan-inclusive as well as strongly person and family centered and fully accessible. The UCEDD provides programmatic support of the interdisciplinary training of professionals, the provision of direct services and supports, and the development of model programs and best practices for serving individuals with disabilities and their families. The UCEDD is able to provide support to this project in the form of facilities, equipment, and expertise and is fully accessible, providing services daily to individuals with various disabilities and their families.

The University of Oregon's Early Intervention Program (EIP) is housed in the Center on Human Development and College of Education . The EIP is directed by FEAT Co-Principal Investigator, Dr. Jane Squires. The EIP offers graduate programs in early intervention, interdisciplinary training, program development, service delivery, provision of supports, and applied research on behalf of young children having developmental disabilities and related risk conditions (e.g., child abuse and neglect). The EIP focuses on children in the 0-5 age and their families and exemplifies best practices in the field of early childhood intervention. The Early Intervention/Special Education area of the University of Oregon consistently ranks among the top four departments among U.S. research universities in the annual survey of university programs by U.S. News & World Report. The College of Education is currently ranked fifth among public universities in the nation (U.S. News & World Report, 2004) and is a member of the American Association of Universities consisting of the top 40 research universities in this country. Staff in the Early Intervention area have worked closely with local and state Child Welfare and Early Intervention systems to implement recent CAPTA amendments that require cooperation between these two agencies. In the past year, a Memorandum of Agreement was developed between The Oregon Department of Human Services and The Oregon Department of Education to outline agency roles and responsibilities to meet this CAPTA amendment. Lane County , with representatives from Child Welfare, Early Intervention, Public Health and Lane County Safety Net created a local system that the EI program is helping to evaluate. Over the next year, EI personnel will assist agency representatives in disseminating findings at state and national meetings. Project FEAT builds on this existing relationship, at both state and local levels, to provide an additional level of focus and attention to the babies with founded allegations who additionally have been identified as prenatally exposed to drugs. EI personnel have well established relationships at both state and local levels with Child Welfare, Public Health and Early Intervention systems which should help facilitate collaborations.

The Child Development and Rehabilitation Center (CDRC) of the Oregon Health Sciences University is a statewide agency that serves children with special health care needs. CDRC is co-directed by Robert Nickel, M.D., co-principal investigator of the proposed Project FEAT. CDRC's mission is to improve the health, development and well-being of children and youth with special health needs and their families. CDRC includes the CaCOON program, which provides public health nursing services to every Oregon County to support identified families who have children with special health care needs, including prenatally exposed infants, and infants with fetal alcohol syndrome. CDRC provides leadership in policy development and advocacy at the state level and works closely with the Oregon Office of Family Health and Oregon 's Maternal and Child Health agency to build capacity through training programs, sponsorship of multidisciplinary clinics and community planning activities.

State Collaborating Organizations The Oregon Department of Human Services includes Oregon 's Department of Health Services and Child, Adult and Families (CAF). Within CAF lies Child Protection Services , a key player in CAPTA legislation, and within Health Services lies the Office of Mental Health and Addiction Services , the Office of Multicultural Health , the Office of Family Health Services as well as other departments that could provide guidance and resources for the state level CAF Working Group which is committed to working on development of policies and procedures for FEAT objectives. A letter of support from Sharon Bolen, CAPTA grants coordinator and member of the CAF working group, is provided in Appendix A. In addition, the State Department of Education, which administers Early Intervention and Early Childhood Special Education services will be a collaborator. As the working group evolves, additional input will be sought from identified agencies.

County Collaborating Organizations . Lane County Child Welfare/Child Protective Services is a key member of the local Perinatal Working Group and is the agency responsible for administering self-sufficiency and child-protective programs in Lane County . Local hospitals, including Sacred Heart , the largest hospital, and the only hospital with NICU facilities, will be collaborators. Lane County Health and Human Services , which administers drug and alcohol treatment services and mental health services, will be active collaborators. Lane County Early Childhood Cares , which provides assessment and intervention services for children, birth to age five, with identified developmental or social emotional delays and their families. The Lane County Relief Nursery/Healthy Start, is a local non-profit agency whose mission is to prevent the cycle of child abuse and neglect by early intervention that focuses on building successful and resilient children, strengthening parents and preserving families" and will provide a base agency for the proposed FEAT Family Advocate. In 2003 the Relief Nursery received national recognition through the Office of Child Abuse and Neglect as a "Program with Noteworthy Aspects". The Relief Nursery, recognized by the Office of Child Abuse and Neglect as exemplary, provides a myriad of culturally and linguistically appropriate services, including a therapeutic early childhood program, home visiting services, case management, respite care, and recovery support.

Project Co-Director (Jane Squires, FTE .05). Dr. Squires is an Associate Professor and Director of the Early Intervention Program (EIP), College of Education . She directs a graduate training program, as well as outreach and research projects related to early identification and referral of infants with developmental disabilities. Dr. Squires is a nationally recognized authority on developmental screening and assessment and has had extensive experience in developing early intervention programs for young children with disabilities and their families. She co-authored the Ages & Stages Questionnaires and recently the Ages & Stages Questionnaires: Social-Emotional , which targets screening of infants and preschool children with mental health concerns. She is Associate Director of the UCEDD and a member of the Oregon Disabilities Council and will donate additional time to this project as part of her UCEDD activities. As project director, Dr. Squires will be responsible for coordination of all research activities and will monitor progress toward project objectives, supervise activities of the research assistants and manage project meetings and work with the project's statistical consultant to conduct analyses and interpret findings, develop project reports and other written materials (e.g., journal articles) and attend the required research meetings in Washington, D.C. Her vita and those of other project staff can be found in Appendix B.

University Co-Project Director (Dr. Robert Nickel, FTE .05 contribution). Dr. Robert Nickel is a developmental pediatrician, professor of Pediatrics at Oregon Health and Sciences University and Director of Oregon Center for Child and Youth with Special Health Needs at OHSU. Dr. Nickel has a long-standing interest in the early identification of children with developmental and behavioral programs and working with primary care offices to identify cost effective strategies for Early Identification, and is co-author of the ASQ, and is principal author and editor of the Physicians Guide to the Care of Children with Disabilities and Chronic Conditions (Nickel & Desch, 2000), authoring chapters on Children with Special Health Needs and Prenatally-Drug Exposed Children. Dr. Nickel is a member (representing CDRC) of the State Interagency Coordinating Council on EI and Chair of the Oregon Pediatric Society's Committee on Children with Disabilities. He will provide the expertise necessary to effectively work with the medical community at state and local levels.

Project Evaluator (Deanne Unruh , FTE .20). Dr. Unruh's research area includes transition services for adolescents with disabilities. She specifically focuses on community-based services for high-risk youth with disabilities who have been incarcerated or are on probation. Dr. Unruh specializes in evaluation methods using a participatory approach through both qualitative and quantitative methods.

Project Research Coordinator/Project Liaison (Elizabeth Twombly, .50 FTE). Elizabeth Twombly is a Senior Research Assistant at the University of Oregon 's EIP. Ms. Twombly has been the "liaison" between state and local working groups as part of a model demonstration project aimed at early identifications of social emotional delays, and is evaluating county pilot project's effort in implementing CAPTA requirements. Her previous experience with state and local agencies and the relationships that have been developed should provide an excellent base to begin the collaboration for the FEAT project. Ms. Twombly's areas of research include infant mental health, developmental and social emotional screening, and support and training of relationship-based home visitors through video feedback.

FEAT Family Advocate (TBA, .60 during Year One -1.0 FTE during Years 2 and 3). The FEAT family advocate will participate in Year 1 planning activities including attending project, state, and local workgroup meetings. During Year 1, the Family Advocate will participate in baseline data collection and during Years 2 and 3 provide home visiting and advocacy services to referred FEAT families. The family advocate position has been described previously.

Data Management/Analysis (Rob Hoselton, .25 FTE). Mr. Rob Hoselton has a bachelor's degree in computer science with specializations in web development and data design and analysis. He is a research assistant in the EIP, setting up and maintaining data entry and analysis systems for two local research projects and a web-based national standardization study.

FEAT Child Welfare Link (TBA, .10 FTE) will attend local Perinatal Task Force meetings and be responsible for communicating between the Lane County Child Welfare Agency and the working group. The Child Welfare Link will assist University personnel in accessing data systems and case files needed for evaluation.