|
After the extensive deliberations
just described, the Committee believes that the framework outlined
in Chapter IV represents the best design available for conducting
a national analysis of the impact of Head Start. At the same time,
the Committee believes that several key next steps are crucial to
translating this design into a credible and feasible research effort
on a topic of great national importance. As a result, the members
of the Committee call on the Secretary-and, where indicated, the
broader research community and the Head Start community-to commit
to the following next steps for implementation.
Provide strong leadership and support for the impact
evaluation.
The Department, in conjunction with the Head Start
community, must provide strong leadership and a clear message that
the impact evaluation is important. Such leadership is important
for gaining broad support for the research design among local Head
Start programs and enabling programs to understand that their participation
in the impact study efforts is critical for shaping the future of
the program.
Ensure partnership between researchers and the
Head Start community through a full dialogue with the Head Start
community, and ensure involvement of Head Start programs from the
earliest phases of the design.
The Committee believes that Head Start programs need
to be partners throughout all phases of the design and implementation
of the impact study or studies, so that the research is not done
"on" the programs, but rather with the programs' full engagement
and support.
The Committee believes a full partnership is important
because the experience of Committee members, whether in evaluating
Head Start or other public programs, suggests that early, complete,
and honest sharing of information is essential to successful implementation
of a rigorous research design. A true partnership will help to build
the necessary trust and collaboration among program staff, families,
and researchers. Failing to engage Head Start programs and families
in this way would risk compromising the impact research as well
as making it harder for programs and researchers to collaborate
on the rest of the research agenda.
Therefore, the Committee urges the Department and
the research community to reach out to the Head Start community
to share the recommendations of this Committee and to hear the voices
of Head Start programs and families. At the same time, the Committee
urges the leadership of the Head Start community to continue to
work actively to engage programs and to communicate the importance
of well-designed evaluation research to achieving the outcomes programs
are seeking for children and families.
Collect the range of information and consult the
range of experts needed to make the remaining design decisions,
determine feasibility, and fine tune these recommendations.
Throughout this report, the Committee has identified
information that is not currently available and that needs to be
gathered and assessed in order to make the next level of detailed
design decisions. While members of the Committee believe that their
recommendations represent the best design given the available information,
they are mindful of the limits of that information and believe that
the design will need to be fine-tuned, and possibly even changed
in more substantial ways, as the missing information is gathered,
whether during a formal feasibility phase or as part of the detailed
design, planning, or early implementation phases of the study or
studies.
Among the questions that Committee members raised
for further consideration and investigation were the following:
-
How many unserved children are there and where
are they located? How concentrated are they in particular communities,
particular regions of the country, and among particular populations
of children? Why are these families not enrolled in Head Start?
Are they enrolled in other programs?
-
What sample size of programs and children is required
to achieve adequate statistical power (i.e., the ability to
determine impacts in various domains)?
-
What incentives would programs and families need
in order to willingly participate in the impact research? Which
of these incentives would be most consistent with the design?
-
What is the best way to ensure informed consent,
so that families understand their involvement in the impact
study or studies and what it means for their ability to access
Head Start services for their child?
-
What culturally appropriate measures are available
and what measures still need to be developed in order to appropriately
capture the development and school readiness of children who
speak English as a second language?
-
What experience is there with subject assignment
and tracking, data collection, and other design considerations
of the magnitude needed for this research effort? Is a field
test needed and if so, when and how should it be done?
Consider costs and time frames in moving from this
design framework to an actual detailed design.
The Committee notes that the Department will need
to consider cost and time frames as it refines the design for the
impact study or studies. The Committee urges the Department to set
priorities within the design in a way that promotes as much knowledge
as possible for the resources invested (for example, the Committee
has given explicit guidance on the resource allocation across experimental
and quasi-experimental components of the design), and it urges the
Department to set priorities across the research agenda as a whole
in a way that ensures a rich and active research agenda beyond the
impact research itself. That is, because the Congress appropriates
a limited amount of money each year that can be used for research
within Head Start, if the impact design is too expensive, it would
compromise the ability to complete other ongoing Head Start research
priorities. The Head Start Amendments of 1998 authorized "not more
than $5,000,000 for each of fiscal years 1999 through 2003 to carry
out impact studies under Section 649(g)" (Head Start Amendments
of 1998). Thus, the Department ultimately will have to weigh recommendations
made by the Committee and decide how best to allocate funds among
the priorities.
Similarly, as noted in Recommendation 11 in Chapter
IV, with respect to time frame, the Committee notes that none of
the options it considered, including the recommended design framework
in Chapter IV, would meet the Congressional time frame of producing
a final report on the impact study or studies by September 30, 2003.
This is because the Committee recommends, consistent with the statutory
requirement, that at a minimum data on children should be collected
at three points in time (Head Start, kindergarten, and first grade),
which would delay completion of data analysis and a final report
until approximately the year 2006. The Committee urges the Department
to make priority choices so that the deadline does not slip beyond
this date, and the Committee also urges the Department to make the
fullest possible use of valuable information on outcomes that will
be available sooner from other ongoing and new research efforts
such as the FACES, ECLS-K, and ECLS-B and to present this information
in the forms and at the times that are most useful to policymakers.
Commit to taking stock at key points in time.
No matter how effectively the Department carries out
the information collection tasks identified above, the Committee
believes that in an effort of this magnitude, much new evidence
about the strengths and weaknesses of the design will come to light
during the research itself. Therefore, the Committee believes that
the Secretary must commit to re-examining the Head Start impact
research effort at key points in time to take stock of what is being
learned and determine whether changes need to be made to the study
design. In addition, the Secretary should consider the implications
of other research findings from the early childhood field and factor
these into any reconceptualization of the Head Start impact research
effort. Consistent with the Congressional charge to the Committee,
the members of the Advisory Committee would like to indicate their
willingness to assist in this ongoing fine-tuning of the design.
Conclusion
In conclusion, the Committee offers the information,
ideas, and recommendations of this report to leaders in the Head
Start and research communities and to public officials who share
the crucial responsibility for shaping the future of America's programs
for young children and their families. The Committee believes that
the proposed design framework represents the best currently available
approach to responding to the important research questions raised
by the Congress, based on the present forms and functioning of Head
Start programs.
We believe that this proposal for a rigorous, credible,
and feasible evaluation of the impact of Head Start on the school
readiness of low-income children across the country will contribute
to the nation's ability to achieve its goals of providing high quality
education and enhancing opportunities for all children. We have
sought to design a framework for a national analysis of the impact
of Head Start that-in conjunction with the rich and active research
agenda now being implemented in Head Start and other early childhood
programs-will assist policymakers and the Congress in ensuring that
the goals of Head Start are fully accomplished and will help early
childhood professionals, in Head Start and other programs, to learn
more about how to improve their efforts to enhance results for children. |
