There is general agreement among the Kellogg Project Group and the Evaluation Committee that the Kellogg evaluation will be relatively broad, but that the depth of the analysis will depend upon the centrality of an initiative or program to the Kellogg Grant. As the Project and its various components evolve, the breadth and depth of evaluation will also change. However, it is valuable at this point to delineate the factors we will consider in determining the breadth and depth of evaluation.
The starting point for determining the scope of the Kellogg Evaluation is the set of five issues listed by the Kellogg Foundation in their grant to Trinity College. With numbering added for reference:
The evaluation should include but not be limited to periodic assessments of:
1. Various components of the Neighborhood Initiative/Learning Corridor,
2. Trinity’s urban academic program, and
3. Overall record in building a viable and sustainable extended community of learning.
Trinity will also evaluate:
4. The pace, direction, and impact of change in the neighborhood and
5. The pace, direction, and impact of change within the College
“Findings from these evaluations will be made available to the W.K. Kellogg Foundation and to other educational institutions with an interest in this or similar projects In addition, Trinity faculty and staff will present findings at conferences and visit other institutions to offer insights on the project.”
The grant does not indicate for issue 1 which “components of the Neighborhood Initiative/Learning Corridor” are to be included in the evaluation. To keep the evaluation task manageable, the Evaluation Committee has elected to focus on those that are either (a) funded in part by the Kellogg grant or (b) can be expected to be affected by components funded by the grant. The large “smart neighborhood” initiative in Kellogg, for instance, would seem to related to issues 1, 3, and 4. Trinity’s grant proposal also set forth a wide range of anticipated outcomes that we expected our efforts at building an extended community of learning might accomplish. We have extracted them in the document, “Major Research Questions” (February 1998) that is available on the Kellogg Evaluation web site. Because these anticipated outcomes are specified in the grant, they seemingly would also come under the general purview of the evaluation related to issues 1 through 3, and we must establish a scope that includes them.
For the purposes of defining scope in terms of breadth and depth, we can distinguish three levels of evaluation:
1. Description of history, organization, activities, participants, and accomplishments.
2. Measurement of movement toward objectives using both quantitative and qualitative indicators. Includes both obtaining baseline information and periodic measurements.
3. An analysis of both process and outcomes that is oriented toward understanding the factors behind successes and problems with the goals of providing timely feedback to help guide “midcourse corrections” and of guiding the Kellogg Foundation and others interested in implementing similar programs.
A level 3 evaluation, of course, would also include level 1 and 2 evaluations. By themselves, level 1 and 2 evaluations provide information that is useful for building a detailed process and outcome analysis.
At the core of the Project are those initiatives, programs, and structures funded in whole or in part by the Kellogg grant. Following the theory model of the Project presented in our “Evaluation Plan,” we can work out from this core and distinguish several different subject areas within the broad scope of the five issues posed by the Kellogg Foundation.
A. Initiatives and Positions Funded by the Grant. These core aspects of the Project are the ones for which the Evaluation Committee must provide the detailed type of level 3 evaluation that can provide relevant information for those designing and implementing an initiative and that can provide detailed guidance to the Foundation and others interested in establishing similar programs. This evaluation of necessity will include level 1 and level 2 evaluations of events and outcomes within the College and within the community.
B. Programs and Activities Originating in or Advanced by a Kellogg Initiative. Examples are the various neighborhood planning and action groups whose organization and work were advanced by the Kellogg-funded Urban Forums and various programs fostered by the catalytic activities of the Professor of Comparative Urban Studies. These also should receive at least a level 1 evaluation. Where feasible, a level 2 measurement of progress toward objectives would also be valuable for addressing Kellogg’s issues, especially if the measurement can be through available or routinely gathered information. The more specific questions of how they were affected by the Kellogg initiatives would be part of the level 3 evaluation of the initiative, although the move from level 1 to level 3 might well be largely through document review and interviews with some of the key players.
C. Learning Corridor. The organization and program development of the components of the learning corridor and the coordination of the components with each other, aspects of Trinity, and individuals and organizations in the community are all somewhat related to the Kellogg evaluation and to Kellogg’s issues 1, 3, and 4. The grant expects that the catalytic leadership provided by the Project (Chief Education Officer, Program Development Coordinator, and Community Partnership Coordinator) will facilitate and advance these organizational, programmatic, and coordination aspects of the Learning Corridor. Therefore, a chronicle and description of them over time are an important part of the indicators for evaluating this aspect of the Kellogg Project. Thus, they should receive a level 1 and level 2 evaluation and a level 3 evaluation of those aspects financed by Kellogg.
D. Related Academic and Non-Academic Programs at Trinity. Examples are the academic internship programs, community service programs, Trinity Center for Neighborhoods, and other aspects of Trinity’s urban academic program. In addition to being covered by issue 2, these related programs are part of the organizational context for the academic components of the Kellogg grant. In general, we believe these should receive a level 1 evaluation. This is also the minimum level needed to respond to Kellogg’s issues 2 through 5. To provide answers to questions about the pace, direction, and impact of change (issues 4 and 5), they should also be covered by a level 2 evaluation. For instance, some of the internship programs; community service activities, including those with the Boys and Girls Club; and TCN’s projects are expected to have positive effects on students, faculty, and staff, on residents and organizations in the neighborhood, and on the institutionalizing of links between the College and the community. In would be valuable to have level 2 information about all of these effects.
E. Indirect “Ripple” Effects in the Neighborhood. A major goal of Trinity’s Neighborhood Initiative and the Kellogg Project is to improve the lives and well-being of residents, businesses, and organizations in the neighborhood, as well as to make the neighborhood more cohesive and self-sustaining. Kellogg’s interest in progress toward this goal underlies its issue 4. As one works through the anticipated direct and indirect effects of many aspects of the Project and of the Neighborhood Initiative, one can readily make the argument that there will be many “ripple” effects working toward this goal. There would be very little point in trying to allocate causal effects between Kellogg and the Neighborhood Initiative, indeed, many would probably end up being allocated to interaction effects. However, Trinity and the Kellogg Project without question need to describe and track “the pace, direction, and impact of change in the neighborhood.” Doing so requires identifying some key indicators, establishing baseline values for them (at this point retrospectively), and tracking changes over time. Some indicators will be quantitative aggregate statistics, but others will be qualitative chronicles of changes in the neighborhood. Thus, the Kellogg evaluation committee and others interested in the transformation of the neighborhood should begin planning and coordinating efforts for level 1 and 2 evaluations of the anticipated ripple effects. The evaluation committee feels strongly this aspect of the evaluation must be done in a cooperative and coordinated manner.
Related Programs with an Evaluation Requirement. Some other programs related to the urban academic program or Trinity’s efforts in the community may have funding sources that require evaluations similar to those required by Kellogg. For instance, CLI has received funding with such an evalution requirement. Programs that must conduct an evaluation can fit any where in the ripples out from the core of the Kellogg Project. To help meet such evaluation requirements, the Kellogg evaluation committee will be happy to provide advice and coordinate evaluation activities so that we can build a cumulative evaluation picture rather than duplicate efforts.
Guiding Principles for the Evaluation
The Kellogg evaluation committee is firmly committed to making its evaluation efforts as meaningful and useful as possible to those responsible for the implementation of the Kellogg Project, Trinity’s urban academic program, and the Neighborhood Initiative. We are also concerned to make the evaluation process as unobtrusive and with as little burden as possible. As we set forth in our evaluation plan, some of our guiding principles are
· Involve those who are planning and implementing the initiatives and, where feasible, the intended beneficiaries, users, and partners in the activities;
· Be appropriate for the initiatives’ organization, activities, and objectives;
· Provide useful information about the achievement of the initiatives’ long-term and short-term objectives;
· Provide useful information about the effectiveness of the organization and activities of the initiatives;
· Provide timely information for improvements and mid-course corrections and for optimal resource allocation;
· Keep the burden and intrusiveness of the evaluation process as low as possible while still meeting the other criteria.