|
Head Start provides comprehensive early child development
services to low-income children, their families, and communities.
With more than 18 million children served since the program began
in 1965, a federal appropriation of $4.7 billion in fiscal year
1999, and increased attention to outcomes and accountability for
federal resources, the program has been challenged to demonstrate
its effectiveness through rigorous research designed by nationally
renowned experts. Specifically, the Head Start Amendments of 1998
(P.L. 105-285) directed the Secretary of Health and Human Services
to establish an expert panel on Head Start research and evaluation,
charged with offering recommendations for a study or studies that
provide a national analysis of the impact of Head Start, advising
the Secretary on the progress of the research, and commenting
on the study reports. The Advisory Committee on Head Start Research
and Evaluation was organized to meet this charge.
The Committee met three times between April and
July 1999 to fulfill the first part of the Committee's charge.
This report to the Secretary summarizes the deliberations of the
Committee and sets forth a framework for evaluating the impact
of Head Start.
Recommendations
The Committee concludes that a study or set of
studies of the impact of Head Start must address two main questions.
First, as specified in the statute, the study or studies must
answer the question of impact: what difference does Head Start
make in the development (and, in particular, the multiple domains
of school readiness) of the nation's low-income children? Second,
and consistent with the legislation, the Committee believes that
a successful study or studies must address the question of how
impact varies in certain key situations: under what circumstances
does Head Start work best and for which children?
The Committee saw its charge as developing a research
design that is capable of answering these questions and that meets
two key criteria:
-
An acceptable design must be scientifically
valid and widely credible. It must provide evidence
that is scientifically convincing and persuasive to a variety
of audiences, such as the Congress, the research community,
program staff, and parents.
-
An acceptable design must be feasible.
It must be capable of being implemented in the real world
by researchers working in close partnership with Head Start
programs.
Much of the Committee's deliberations focused on
the potential tension between these two criteria. In the end,
after a rich and lively debate, the Committee set forth a framework
for impact research in Head Start that we believe is both credible
and feasible. The key elements of this framework are as follows:
-
The Committee believes that the research design
should include random assignment of children and families
to Head Start and non-Head Start groups at a diverse group
of sites located across the country. The Committee spent a
considerable portion of its deliberations discussing the feasibility,
ethics, and credibility of random assignment designs and concluded
that random assignment of children within the framework
described here represents the best approach that the Committee
can identify to answering the two central research questions
and meeting the two key criteria. Committee members believe
that random assignment will not be easy to implement but is
nevertheless important.
-
To ensure that random assignment is feasible
and to ensure that families are not unfairly denied Head Start
(an ethical concern to many members of the Committee), sites
where Head Start saturates the community (i.e., where there
are not enough unserved children to permit random assignment
of a sufficient number of children to an unserved control
group) would be excluded from the random assignment portion
of the study or studies.
-
Every effort should be made to ensure that the
sites selected are representative of Head Start sites nationally.
Diversity should be sought on key criteria (e.g., region of
the country and poverty level of the community). Sites should
reflect the range of Head Start quality across the country.
Sites would be provided appropriate incentives and supports
to facilitate their involvement in the study or studies. The
small number of sites that are out of compliance with Head
Start standards or extremely new to the program would be excluded.
-
To answer the research questions rigorously
and credibly, the Committee believes that the study or studies
must measure quality in the Head Start sites and in the child
care, prekindergarten, and other settings experienced by control
group children. More specifically, the Committee believes
that the study or studies must collect the same or closely
comparable information on the Head Start children and control
group children across all the areas of measurement, to the
extent feasible. These recommendations are particularly important
to help address the concerns raised by some members of the
Committee that some Head Start programs (particularly the
best) are likely to have influenced other child care and prekindergarten
programs available to low-income children, so that the environments
of control group children have been influenced (or, in research
terms, "contaminated") by the Head Start treatment.
-
Outcome measurement in the study or studies
should focus on the multiple domains important for school
readiness of children1
and on parental practices that contribute to school readiness.
The Committee has specific recommendations regarding the domains
of school readiness on which to focus, the nature of the measures
that should be used, and the need to improve measurement for
children for whom English is a second language.
-
The Committee identified several strategies
for selecting sites. Each strategy has advantages and disadvantages,
which should be fully assessed and reviewed by the Department
of Health and Human Services (the Department) during development
of the detailed research design.
-
The Committee believes more consideration needs
to be given to the option of using quasi-experimental or other
embedded studies to supplement the information from the randomized
impact study or studies. Some members believe quasi-experimental
studies could yield useful information about Head Start, but
others question the validity of these studies. All members
agree that the amount of money spent on quasi-experiments
should be small relative to the amount spent on a randomized
study or studies. This option should be more fully developed
and reviewed by the Department during development of the detailed
research design.
-
The Committee believes that the Department should
consider carefully, in consultation with the Head Start community,
what incentives for parents and for programs would be most
helpful to secure participation in the study, consistent with
the research methodology. The Committee strongly encourages
the use of an appropriate range of incentives that are offered
to Head Start programs and families as well as control group
programs and families.
-
Immediate and continuing efforts should be made by the Department
to promote the use of research and the findings from the impact
study or studies to improve the effectiveness of Head Start
programs for the benefit of children and families.
-
The Committee believes that it is critical to draw on information
from the existing Head Start research agenda to complement
the information gained from the impact study or studies. In
addition, the Committee believes that the research design
proposed here should be part of a rich and active overall
research agenda, not a substitute for it.
-
Finally, the Committee notes that none of the
design options it considered, including the recommended design
framework presented here, would meet the congressional time
frame of a report by September 2003. Because the statute (and
the Committee) endorses follow-up of children through at least
the end of first grade, all of the options considered would
lead to a final report in approximately the year 2006. As
a result, the Committee urges the Department to make the fullest
possible use of valuable information on outcomes that will
be available sooner from other ongoing and new research efforts-such
as the Head Start Family and Child Experiences Survey and
the birth and kindergarten cohorts of the Early Childhood
Longitudinal Survey-and to present this information in the
forms and at the times that are most useful to policymakers.
Major Issues and Challenges
The Committee developed this framework for rigorous
Head Start impact research after extensive deliberations that
focused on two broad areas.
Challenges Related to Credibly and Accurately
Assessing Impact
Because most Committee members agreed that the most
rigorous methodological approach to answering questions about
impact is to compare children and families who are randomly assigned
to Head Start with children and families who are assigned to a
control group that does not receive Head Start, the Committee
spent a great deal of its time discussing the credibility, feasibility,
and ethics of random assignment in the Head Start context. Most
Committee members believed ethical issues were diminished once
programs with waiting lists or unserved children were considered
as the basis for a random assignment sample. Among the key issues
considered by the Committee in these deliberations were the following:
-
Is it ethical, and if so, under what circumstances,
to assign children to a control group that receives no Head
Start services?
-
Is it feasible to maintain adequate participation
in the research by families assigned to a control group that
receives no Head Start services? Is it feasible to recruit
parents who are fully informed that participation in Head
Start will be determined by lottery? How do parents see their
choices and what incentives for participation in the research
might be helpful, ethical, and not in conflict with the research
approach?
-
Are there special ethical and feasibility issues
with regard to children identified as particularly high-risk,
who are typically given first priority for enrollment in Head
Start programs?
-
Is it feasible to expect that Head Start program
staff will be willing to implement the random assignment approach?
Are they willing to keep control group children out of the
program?
-
How does the dramatic expansion of other child
care and early childhood alternatives, including state preschool
programs for low-income children, affect the feasibility and
credibility of a randomized study design?
-
How does the potential effect of Head Start
on broader community child care services affect the credibility
of the random assignment design? That is, if Head Start programs
affect the quality of the services received by control group
children, how serious a problem will that be to the research
design?
-
Are there alternatives to a randomized study
design that would be sufficiently credible in answering the
key research questions?
During the course of these deliberations, the Committee
benefited from the experiences of a set of Head Start feasibility
studies of random assignment conducted by researchers in the Head
Start Quality Research Centers in partnership with local programs.
The Committee also drew on existing research and data about Head
Start, child care, and other early childhood programs, as well
as the experiences of its members in state-of-the-art research
and evaluation across a wide variety of policy areas.
The Committee's resolution of these issues appears
in Chapter V-Rationale for the Recommendations: Addressing Key
Challenges.
Challenges Related to Generalizing Findings to
the National Head Start Program
The Committee focused extensively on several issues
raised by the congressional charge to provide a national analysis
of the impact of Head Start in the most rigorous manner possible.
Among the key issues considered as part of these deliberations
were the following:
-
What is known about the capacity of sites to successfully
comply with the demanding task of random assignment, and about
the number and distribution of sites that are at saturation
with respect to the percentage of eligible children currently
being served by Head Start or comparable programs?
-
What are the advantages and disadvantages of a
strategy that involves a nationally representative, stratified
random selection of sites?
-
What are the advantages and disadvantages of a
more purposive strategy that seeks to target a group of sites
selected for diversity on key variables and chosen to maximize
the likelihood of successful implementation of the research
design?
-
Are there alternative, quasi-experimental strategies
that might yield useful knowledge at sites without the capacity
to carry out random assignment?
Next Steps
Based on extensive discussion of the information
available to answer these and other key questions, the Committee
believes that the framework outlined in this report represents
the best strategy for evaluating the impact of Head Start on children.
At the same time, the Committee believes that several key next
steps are critical to translating this strategy into a credible,
powerful, and feasible study or set of studies. In particular,
we urge the Secretary, the research community, and the Head Start
community to commit to the following next steps:
-
Demonstrate clear leadership and commitment
to the rigorous evaluation of the Head Start program, at all
levels of the Department and the Head Start community;
-
Ensure true partnership between researchers
and the Head Start community and involve the Head Start community
from the earliest phases of the design;
-
Conduct an initial feasibility study or set
of activities to collect additional information that is essential
to the detailed planning and refinement of the design; and
-
Pay close attention to the ongoing implementation
of the research, including ensuring several opportunities
to review the design and modify it where appropriate.
In conclusion, the Committee hopes that a rigorous,
credible, and feasible evaluation of the impact of Head Start
on the school readiness of low-income children across the country
will contribute to the nation's ability to achieve its goals of
providing high quality care and education and enhancing opportunities
for all children. We have sought to design a framework that-in
conjunction with the rich and active research agenda currently
underway in Head Start and other early childhood programs-will
assist policymakers and the Congress to ensure that the goals
of the Head Start program are fully accomplished and will help
early childhood professionals, in Head Start and other programs,
to learn more about how to improve their efforts to enhance results
for children.
|
