Integrated Evaluation Results and
Meta-Evaluation
2007-2008 Grant Year
Third Party
External Meta-Evaluation Team:
Gary Phye, Ph.D.,
Mary Herring,
Ph.D.,
Don Yarbrough,
Ph.D.,
December 1, 2008
Table of Contents
ILTC
Observations, Conclusions, and Recommendations……………………………………………………..3
ILTC
Executive Summary............................................................................................................. 6
Projects Description…………………………………………………………………………………………………..…6
Model Projects Meta-Analysis.......................................................................................... 7
Primary Project Outcomes.......................................................................................................... 8
Impact on Student Engagement....................................................................................... 8
Disciplinary Problems....................................................................................................... 9
Writing, Analysis, and Research....................................................................................... 10
Student Centered Classrooms........................................................................................... 11
Parental Involvement........................................................................................................ 11
Vendor/Business Relationships......................................................................................... 12
Student Achievement........................................................................................................ 13
Unanticipated
Outcomes/Recommendations.......................................................................... 14
Appendix:
Individual Project Reports…. Available
online at www.iowaltc.org
ILTC
Observations, Conclusions, and Recommendations
One hundred-eighty three applications were received over 3
years, 18 of which were funded by the legislature. A total of $1,361,256 has been awarded to
school districts from this initiative.
Competing school districts submitted 61 concept paper
applications in this second cycle.
The number of ideas submitted indicated the existence of need
in the field of education that was greater than the Commission could fill. This may be a reflection of how difficult it
is for districts to keep up with technology, especially as it relates to
infrastructure, wireless access, and other basics. In terms of promising
practice, the Commission is forced to eliminate a lot of deserving ideas due to
funding constraints.
Our report shows that most grants awarded are having a
positive impact on kids due to careful planning on the part of school leaders,
but the evaluation of this process is on-going, since grant round three is
still in progress.
Commission members have observed promising practices with
regard to positive and innovative integration of technology into instruction.
There is agreement that the grant process is worthwhile, captures promising and
innovative practices, and includes a large range of stakeholders who support
the programs. This support, it is perceived, leads to student and staff
engagement, as well as community and parent enthusiasm and investment in the
educational process. Community engagement, in particular, appears to be
impacted by participation in the pilot projects, bolstered by the requirement
for stakeholder role in the implementation of and/or the funding match for the
project, as well as expectation for ongoing support of stakeholders.
In each round of grant awards, there was far more demand for
access to grant funding, especially among small districts than the given
appropriation could accommodate.
Parents and community members are proud of the
accomplishments of their schools. There is agreement that these accomplishments
need to be communicated to Legislators, Iowa Association of School Boards
(IASB), Iowa Technology and Education Connection (ITEC), and School
Administrators of Iowa (SAI).
Life-Long
Learning
Education is moving toward a 24-7 model and promotion of
life-long learning. ILTC members are encouraged by the promising practices and
innovation observed during on-site visits to participating schools. High levels
of student, staff, and parent engagement were evident. Teachers were observed
integrating technology into classroom instruction. Additionally, support for
multiple student learning styles is being addressed through the use of
technology.
All
students would benefit from greater access to technology because technology
integration is a teaching and learning strategy that works for kids. Commission
members witnessed firsthand the high levels of student engagement. In most
cases, the data reported regarding decreases in discipline referrals may be an
indicator of student engagement. Technology can support a diverse set of
learning styles. Students impacted by these pilot projects are not only engaged
in learning, but are immersed in exploring a greater variety of sources of
information.
In planning for participation in these pilots, each district
selected the type of project that fit its needs; therefore, each participating
district has experienced different successes. For that reason, it would be
difficult to identify one single project that exemplifies best practice. At
this point, the Commission is not in a position to say any of the instructional
practices being utilized by the second round of grantees should be replicated
by all districts.
Sustainability of efforts is another difficulty facing
technology grantees. Technology efforts that are funded for a one-year period
will be difficult to sustain. To maintain their efforts with technology
integration, schools may be forced to seek alternate means of funding.
Additionally, it is clear that staff turnover/continuity has an impact on the
success of such projects. Without the sustained commitment of all staff,
success is unlikely. Sustainability is often impacted by teacher training
support. Technology resources must be accompanied with teacher training to
ensure that technology is integrated into existing teaching practices.
Professional
Development
It
is generally agreed that instructional practice is key to the success of
technology integration efforts. This equates to the need for professional
development, a necessary component for ensuring that teachers are comfortable
with technology, and know how to successfully integrate technology into
existing effective teaching practices.
Recommendations
Based upon these conclusions and observations, the Commission
makes the following recommendations:
·
Continue the study of grant funding in round three, using the lessons
learned in rounds one and two.
·
Ensure grantees are reporting consistently on mandated data points and
understand legislative intent.
·
Resume funding technology grants to support schools, especially small
rural districts.
·
Consider additional funding for this grant program, given that the
demand for this initiative has consistently out-paced available resources.
·
Continue to promote grant awardees that emphasize innovation regarding
the use of technology as a tool to enhance instruction and learning.
This
is the second cohort of schools that has been funded by the Iowa Learning
Technology Commission (ILTC). This is viewed as a 3 year initiative that has
funded a total of 18 schools across
Six
projects make up the second cohort of school districts funded during the 2007 –
2008 academic year: 1)
Cedar Rapids Community School District installed
activboards, student response systems and projectors in every seventh grade
math classroom with the intent of increasing student achievement in math. Each
building also received student response system devices to be used with the
activboards for formative and summative assessment. The district was also
implementing a new math curriculum and professional development was centered on
the use of acquired technology to support the new curriculum, create
instructional resources for teachers and increase student learning.
Southeast Polk focused on providing their high school
students with the opportunity to perform authentic research using cutting-edge
tools for data collection in both the classroom and in the field. Equipment purchased included laptops,
multi-media tools, cameras, digital microscopes and probes. This high school science technology infusion
project was designed to provide 21st Century tools to enhance the
science curriculum and the field station.
Williamsburg Community Schools focused on grades seven through twelve in an
effort to improve the reading skills for middle school and high school
students. This project was integrated with a Study Skills course that all
struggling readers in grades seven through twelve were required to take with
the goal of improving reading comprehension. Funding was used to initiate the
use of the
A
common theme that runs throughout all of these projects is awareness by
applicants of the changing landscape in
The quality of the second cohort projects is quite high. The
level of technology use by teachers and students is impressive. This is a significant improvement over the
first year in terms of both accountability and the integration of funded
projects into existing curriculum efforts. This is in part due to the efforts
of the ITLC committee to provide recipients of funding the opportunity to
become aware of “lessons learned” from the first cohort of funded schools.
The spring ITLC conference where previous participants and future participants
are in attendance, is a critical factor in the improvement of both
accountability and quality of individual project efforts.
The extended comments that follow are organized around the
same basic themes that provided the story line for the evaluation of cohort one
schools (2006-2007 academic year). This organizational theme was maintained in
order to provide the opportunity for the analysis of change promoted by ILTC
funding across the three-year funding period.
The external evaluation team again provided participating schools with
oversight and structure in the development and collection of data (both
qualitative and quantitative). Appendix
A is a copy of the framework provided to each grant recipient. This Framework
served as the basis for the organization of both the ITLC midterm and final
accountability and evaluation report.
Appendix B contains all of the second cohort project final
reports. This provides readers with the
opportunity to access detailed information about individual projects. A perusal
of the reports provides insight into the varied quality of reporting. This is not inconsistent with the quality of
reporting observed by the first cohort of schools. In an attempt to address this problem, a
proactive systemic change view is also included in this executive summary.
The Anita
project
issued laptops to all 9-12 students in August 2008, reflecting a project goal
of equalizing the technology playing field by emphasizing the use of laptops
for at-risk and special needs students. By the end of the project year, 3
classrooms were totally immersed in using Moodle. Seventy percent of classes
were using technology in daily lessons with coverage of 98-100 users in core
courses (except in math).
The
The Cedar
Rapids Project has an extensive data collection system in place. Student survey data,
classroom observations focusing on “time on task” and classroom interviews of
students all indicate a high level of student engagement in the math learning
process. Furthermore, students’ attitudes toward math are positive and student
self-evaluation is good. Teacher comments about student engagement are
extremely positive. Thus, the use of technology in seventh grade math classes
appears to increase both student motivation and engagement.
The Okoboji
Project has
addressed student engagement in three ways. First, during the project year,
there was a decrease in unexcused absences compared to the prior academic year.
Second, referrals for discipline problems decreased from 74 the prior academic
year to 31 during the project year.
Third, there was an increasing usage of laptops across classes as the
project year progressed.
Southeast
Polk has
provided data on student engagement with technology surveys, anecdotal comments
and tardy data. Direct comments by both students and participating teachers
were extremely positive on this count. The “hands on” data collection
experiences were well received by students and teachers. The perception was
that student engagement had improved in their classes. The tardiness data
provides evidence of engagement as a positive motivational factor for
attendance. Tardy referral data for the project year demonstrated a decrease
from a high of 30.55% during the Fall semester to 25.47% in the Spring
semester.
Anita did not address this issue
in their project.
Okoboji also collected disciplinary data gathered from
the district’s student information system (JMC). These data are reported above in the context
of student engagement.
Southeast
Polk did
track discipline problems. Across all high school classes, the total number of
discipline referrals increased during the project year. In contrast, the
percent of school wide discipline referrals written by teachers declined from
26.96% to 15.27%. When students are engaged in relevant activities, there are
less discipline problems.
Anita has done a good job of
integrating professional development and technology by focusing on the infusion
of reading, writing, and performance activities into the curriculum in addition
to oversight by the school improvement team.
The approach taken by Anita reflects a good example of how technology
and professional development provide the basis for curriculum change that
foreshadows a recommended approach to changes anticipated by the implementation
of the Iowa Core Curriculum.
Okoboji has reported extensively on
the frequency of technology (computers and software) for writing, analysis and
research (see Okoboji final report in Appendix B). The district sees this as a
primary means by which the high school curriculum is moving toward a more
student centered classrooms. Unfortunately, the data provided have not been
disaggregated or analyzed to address specific evaluation questions. This is as
much as anything a simple fact of life that teachers are not trained in the rudiments
of statistical analysis. The software and data entry activities are implemented
with fidelity by the use of data requires someone with
statistical analysis skills and many small school districts do not have such
expertise on the staff. Frequency of use
was graphed across the school year and trend lines
suggest that frequency of technology use at Okoboji high school is increasing.
Southeast
Polk reports
the use of laptops and digital accessories to gather, analyze and report data
during student experimentation. Laptops were also used to write science reports
on such topics as genetics and other topics. The technology was used for
writing, analysis and research activities by all students in the core
curriculum science curriculum. The infusion of these activities into the
science curriculum involved thirteen teachers.
All of the
reporting projects (Anita,
Anita has used Moodle as a
student, teacher, and parent portal in order to enhance the communication
process among building stakeholders. In addition to enhance student and teacher
communication, Moodle was also provided student parent and teacher discussion
and e-mail groups. The Moodle portal also provided enhanced communication about
school activities, projects, etc. to both parents and the community at large.
Parental involvement was also required at the outset of the laptop initiative
before the issuance of the laptops to students.
Okoboji did not monitor parental
involvement. However, it is anticipated that the Moodle portal will be used
next year as a means of increasing parent involvement in the educational
process.
Southeast
Polk made a
concerted effort to inform parents of the ILTC grant initiative. In February of
2008, participating teachers demonstrated the new technology and how it was
being used during the parent-teacher conferences. In April 2008, the high
school hosted an open house to showcase the ILTC technology and students made
presentations informing attendees about how the technology was being used in
their projects and assignments. This open house was well attended and two state
school board members were in attendance.
Anita has entered into a long-term
agreement with Apple Inc. in order to provide sustainability for the project.
Vendor relationships have been positive. Anita has made extensive efforts to
involve the community by showcasing student technology projects.
Okoboji purchased their equipment
through Hewlett-Packard Direct using the WSCA pricing. While the relationship
started out in a strained manner, the relationship improved dramatically
throughout the funding period. Aaron Bennis, the HP K-12 Iowa representative
has proven to be a valuable resource. This relationship has flourished and will
continue.
Southeast
Polk. Hewlett-Packard provided significant price cuts on the first order of
laptop computers. Apple Corp. provided a free multimedia bundle which
included: A pro-scope, digital camera,
camcorder, and keyboard when Mac laptops were purchased. Vernier provided free
shipping and a 10% discount on all equipment purchased. Metro Waste continues
to support the SEP/MWA field station.
This included property and funds for the upkeep and maintenance of the
field station.
The impact of
project activities on student achievement is almost impossible to determine
with and certainty because a lack of a comparison or control group. In
Anita did report the administration
of ITBS tests although ITBS is not administered at grades 9-12. Student achievement was not directly
addressed except under the general topic of assessment. Assessment activities included individual educational plans, lesson rubrics, walk
through surveys and self assessment by students.
Okoboji did report ITED summary
statistics for freshmen, sophomores and juniors from the 2006-2007 academic
year and the 2007-2008 academic year. However, since
Southeast
Polk reports
student achievement for the project year using district adequate yearly
progress data. The data collection efforts and the resulting presentation were
well done and primarily reflected performance by all students in all classes on
the ITED. However, data were not disaggregated by course so that data would
reflect only those students in the science classes who would receive exposure
to the ITLC project.
Anita advises projects to purchase
the latest and best laptops fully loaded with needed software. This will be a
benefit in the long run for projects initiating one-to-one laptop projects.
Don’t skimp, as it will cause more troubles than it is worth in savings over
time. We also learned to not charge an initial user fee, but do charge an
insurance fee to cover damages to the student’s computers.
Ankeny reports a critical need for
advanced planning, the purchase of technology as soon as possible in order for
IT department to get involved, provide assistance and have equipment setup long
before school starts in the fall. Also, you must have a curriculum element in
place before school starts that focuses on teaching students the proper use of
the software and hardware. This cannot be done informally, on the fly, nor can
you expect all students to be technology savvy.
Okoboji’s recommendation to others is
to make sure to thoroughly test all hardware and software and if possible put
demos in the hands of those that will be using it prior to ordering. Again
differentiated staff training opportunities is recommended. In this case,
suggestions pertained to group size as well as ability level of teachers.
Interestingly, a barrier to the use of Moodle was the difficulty at the high
school level in scheduling staff development time to train teachers on its many
uses.
Southeast
Polk
reports both positive lessons learned and recommendations. For such a large project (school size),
involve as many teachers and administrators as possible. Expecting two teachers
to take primary responsibility for implementing a project of this scope can be
extremely optimistic. The technology department needs to be extremely involved.
Other delays involved vendor delays in shipments, and compatibility with
existing technology (e g bandwidth availability). Due to the scope of the
Southeast Polk project, the principal investigators recommend an earlier
dispersal of grant money to give schools plenty of time to purchase equipment,
get it installed and train teachers prior to the beginning of the academic
year.
Williamsburg
reports
that student and teacher use of the mobile lab has met their expectations, with
interesting class projects showing a gradual shift to more student-centered
classroom activities and more use of the computers for research, collaboration,
and student publications and presentations. This will continue to grow, as
teachers find new ways to use the mobile lab in their classes. Moving to block
scheduling allows longer class periods which works well with scheduling the
mobile lab. However, managing the mobile lab is challenging: check out and
keeping the laptops all running smoothly requires extra time and effort on the
part of media center staff.