ILTC Final Evaluation Report, July 25,
2008
Williamsburg Community Schools
Executive Summary
The Williamsburg
Community School District was granted ILTC funds to improve the reading skills
of the 130 Williamsburg Community School District students with the lowest
reading achievement (<46th percentile ITBS/ITED) in grades 7-12 through use
of the Academy of Reading program (reading intervention software) and mobile
computing. This project is integrated with the Study Skills course (formerly
Reading Comprehension) that all struggling readers in grades 7-12 are required
to take, with the goal of raising reading test scores and improving reading
comprehension. These gains will be measured by performance on ITBS and MAP
tests, class performance in Study Skills, and on the Jamestown Reader
Assessments. Our expectation is that student motivation and attitudes toward
school will improve along with improved reading ability, as measured by
attendance and behavior referrals, as well as student and parent survey data.
We have seen some improvement on standardized test scores since
starting to use the
Description of the Project
Williamsburg
Community School District was granted ILTC funds to improve the reading skills
of the 130 Williamsburg Community School District students with the lowest
reading achievement (<46th percentile ITBS/ITED) in grades 7-12, out of a
total of 558 full-time students. We will increase the percentage of proficient
readers and raise reading performance two grade levels through use of a mobile
wireless computer lab, allowing for one-to-one computing, and the innovative
There were 16
sections of Study Skills this past year, with an average of 7 students in each
section. Students are enrolled in the Study Skills course if their ITBS/ITED
Reading score is less than or equal to 45% proficient. Students who raise their
scores above this level may be transferred out of the Study Skills course at
semester. Students who test below this level may be transferred in to the Study
Skills course at semester as well, and start using AoR mid-year. Teachers from English,
Social Studies, Math, and Special Education departments teach this course the
last period of the day. The course does not meet when school is dismissed early
for staff development or for weather, and students missed several days this
year. The Study Skills course uses Jamestown Readers and assessments,
vocabulary building, and individualized reading in the student’s lexile range
in addition to the AoR software to achieve the goal of raising student test
scores in reading and improving student reading skills. Students who do not
finish the AoR program in one year will finish the following year, as long as
their ITBS scores remain below the proficient level. Because the ITBS tests are
given in November, senior students who raise their score above the proficient
level are allowed to transfer out of the Study Skills course and they do not
finish the AoR program. The course is supervised by the Asst. Principal, with
curricular assistance from two of the participating teachers.
Since the grant
was funded in Fall 2006, the district has experienced several personnel
changes. There is a new superintendent, a new Jr/Sr High Principal, a new Jr/Sr
High Asst. Principal, new Study Skills Coordinator, and a new Curriculum
Director. This change, while delaying some of the items in the Action Plan, has
provided a new vision and excitement for the district and for the Jr/Sr High in
particular. The Principal, Asst. Principal, and Curriculum Director have all
attended AoR training, and take an active role in assisting Study Skills
teachers as they use the AoR program. There has been no direct involvement of
college or university staff or community partners as yet. However, we expect
that reading improvements on the part of our students will result in more of
them attending college and/or becoming productive members of the local
workforce.
The equipment and
software purchased from the ILTC grant is supervised by the Technology Director
with day-to-day management of the laptop computers by the
Most of the focus
so far has been on the AoR software program: learning to use it with the
students, and managing use of the 30-station mobile lab and software for 137
students. To increase the possibilities for creative use of the mobile lab, we
purchased MacBooks for the laptops instead of the HP laptops we had originally
planned to purchase.
The decision to
purchase MacBook computers instead of the HP laptops originally planned is
primarily due to aggressive pricing and support by Apple, and by personnel
changes at HP. The HP representative originally involved in the planning was
promoted, and her replacement was not as helpful nor as good at finding
aggressive pricing for us. Because there had been no commitment made on HP’s
part – no special discounts or other involvement – we were not obligated to
purchase HP machines. Apple sent trial equipment and a support engineer here to
work with the Technology Director to look at the MacBooks in our network
environment, and to make sure that the MacBooks would work well for our project
and for our school in general. The Education Technology Partners (the
The most
significant barrier to clear success for the project has been scheduling
students to use the AoR program as much as the software recommends. A decision
was made last fall to start all Study Skills students in the AoR program at the
same time, which limited access to the program to one session per week. This
slowed down student progress through the program, and made it more difficult to
see the program’s effect on student reading improvement. Administrators are
currently looking at changing the way AoR is scheduled in Study Skills classes
for next year, to provide students increased weekly access to the program.
Student engagement
We expect to see
improved attendance, student disposition toward reading instruction and greater
on-task behavior as a result of this program. We expect that attendance and
attitudes toward school will improve for at-risk students as they begin to see
success in the reading comp program and to develop more confidence in their
academic abilities. These student engagement outcomes will be monitored in
three ways: 1) attendance records; 2) pre- and post-AoR intervention surveys of
student attitudes toward reading instruction; and 3) periodic observation of
student on-task behavior using an observational checklist and time sampling
procedure. Of the 12 students sampled for the interim report, 7 students showed
50% fewer absences than their totals for the 0607 school year. Because of
administrative personnel change, the pre-AoR intervention survey was not
completed as scheduled. A post-AoR intervention survey was completed in May,
including students who had completed the program and students who had been in
the program during the school year but did not complete the program by the end
of the school year.
Disciplinary problems
We anticipated a
25% reduction in office referrals by targeted students. We will compare the
number of referrals in September to the January and May data using our Student
Information System to provide the necessary discipline data. We also expect to
see fewer discipline referrals for at-risk students as they become more engaged
in their own academic progress and as they start to see gains from the reading
intervention program. During the 0607 school year, there were 30 behavior
referrals in the Study Skills classes, with 27 of those referrals occurring in
the first semester, all but 2 for non-compliance or not reading. In 0708, there
have been 44 referrals for Study Skills students in the fall, and 24 in the
spring.
There were 533
absences for Study Skills students in the fall (absence defined as missing more
than one period, for any reason other than school field trips), and 641
absences in the spring. We will continue to monitor attendance and behavior
referrals to look for improvement.
Use of computers and software for writing,
analysis, and research
We expect
students to report increased frequency of using computers for writing, analysis
and research. In August of 2008, we will administer a survey asking students to
compare their current use of computers to past use, using a Likert scale to
assess changes. Teachers have had one professional development session
introducing them to the Macintosh laptops and the applications that are part of
the Macintosh suite. This was new for some of the teachers but was review for
several who are already familiar with Macs, their interface and multimedia
application suite. Professional development sessions are planned for the 0809
year, and will include small group work sessions on the Macintosh interface and
multimedia applications. The Macintosh cart is reserved each day for the Study
Skills classes (5th period only), and is available for use by other teachers
the rest of the day. The special education teachers use laptops from this cart
throughout the day to give their students extra time in the
Here are some
examples of such use:
“We used them quite a bit during nutrition lessons. The
government has a great website www.mypyramid.gov. This website allows
students to calculate their daily needs for calories, vitamins, etc. They
can log their food intake and it lets them know if they are coming close to
meeting recommendations. The macs were a great tool for this as I could
instruct in the classroom using the projector and my own computer to help them
navigate their way through.”
“Students in a Conservation Ecology class put together a short
video/movie clip showing the consequences and problems associated with
overpopulation in countries such as
Movement toward student-centered
classrooms
To monitor movement toward
student-centered classrooms we will use subjective measures such as looking at
student book journals (which include summaries, personal vocabulary selection,
and character reviews), observation of class discussions (based upon readings
and vocabulary work), independent work with students on topics of special need,
and measuring progress on those topics on a weekly basis.
Parental involvement
A school
publication sent in August 2007 explained the AoR program to parents and
community members. We intend to increase parent involvement by including
information on the reading intervention software program in all orientation
sessions. Study Skills progress is made part of parent-teacher conferences, and
invitations to the project open hour were sent home to all parents of Study
Skills students. Parents will be surveyed in August 2008 to assess interest, satisfaction,
and observations, with regular annual surveys in coming years.
Increased student achievement
We will use
ITBS/ITED data to determine if we have achieved our intended outcomes of: 1) increasing
the average percentile from 28.02% to 56% for struggling readers on the ITBS
Reading test, 2) 45% of the targeted students reading in the proficient range,
and 3) a 2-grade improvement in reading level as measured by the AoR
assessment, weekly in-class measures (vocabulary quizzes and Jamestown reading
assessment), and the MAP test administered in Fall and Spring of the school
year.
We see these
improvements in the ITBS Reading Comprehension tests taken in November: 39% of
the targeted students tested proficient, and the average score for the targeted
group was 42.07%. Of the students with IEPs (Individualized Education Program),
19 improved their scores over the previous year, with an average improvement of
13.6 points. One student improved his score by 46 points. Of the non-IEP
students, 36 improved their score, 1 was unchanged, and 21 lowered their score.
Of those students who improved their score, the average increase was 19.9
points.
By the time of
the interim report, we had only 2 students complete their AoR program. Of these
students, one showed a 2-grade improvement between the pre-test and posttest,
and the other student showed no improvement. However, this latter student went
from an average of 66% on
On the
Jamestown Reader
Growth (measured comprehension and fluency)
7th
grade – 13 of 15 students showed growth between first test and last test
8th grade
– 14 of 21
9th
grade – 9 of 21
10th
grade – 11 of 22
11th
grade – 7 of 17
12th
grade – 6 of 10
MAP test scores (difference between fall 2007 and spring 2008 scores)
7th grade - 13 students took both tests; 8 of 13
improved, 7 of those have IEPs
Fall
average score 201; spring average score 205
8th grade – 19 students took both tests; 14 of 19
improved, 5 of those have IEPs
Fall
average score 206; spring average score 214
9th grade – 21 students took both tests; 15 of 21
improved, 4 of those have IEPs
Fall
average score 215; spring average score 220
10th grade – 22 students took both tests; 19 of 22
improved, 8 of those have IEPs
Fall
average score 216; spring average score 222
11th grade – 18 students took both tests; 14 of 18
improved, 5 of those have IEPs
Fall
average score 220; spring average score 226
12th grade – seniors did not take spring MAP tests, 6
seniors have IEPs
Fall
average score 221
29 students with
IEPs took both tests; of those all but 6 improved their scores between fall and
spring.
We had most Study
Skills students complete the AoR
post-test in May, and these were the results:
Out of 135
students who started the program:
9 left the
district without completing the posttest
32 did not
complete the posttest
16 decreased
their level between pre and posttest
30 saw no change
48 increased
their level.
Conclusion
This project has
been a valuable addition to the efforts to improve reading comprehension at
Williamsburg Jr/Sr High. The
“it [AOR] has been a wonderful tool both for instruction and
progress monitoring. The weekly progress is quantifiable and the scope and
sequence is correlated closely with the Descartes material from MAPS.
“I think that overall the idea of the program is great. The kids
truly need help with these skills. One concern that I have with the
program would be the time requirement - requiring the students to get
consistent times for each trial is very hard to achieve. I have seen some
of my students have more than 30 trials with few/no errors, but their time was
inconsistent.”
“The only problem my students and I have with the program is the
visual match skills that go according to time. My students earn 100% but
are not allowed to continue because their timing was not fast enough. They
start to get frustrated at this point. Otherwise, my students tell me they
enjoy it.”
My other concern is that it’s really hard to make them understand
that there’s a point to the program. I’ve explained that it helps automaticity,
and they’ll read better for it, but because it seems out of context (imitating
sounds and recognizing letters) they don’t see how it’s helping them.”
“Pros I’ve seen: 1) It does reinforce
necessary skills. 2) I already have a student in the stories, and he’s
progressing well. He’s also progressing well through his books and
Jamestowns, so it appears not to be hurting. 3) The way the kids can see their
rewards immediately is good. 4) The fact that most activities are short and we
can modify if necessary.
Cons I’ve seen: 1) The students are
frustrated that the sound-matching takes so long. 2) It takes a while before they
understand they’re not being played down to with this program, it’s about
automaticity. 3) It does feel like it’s geared for younger kids. 4) This
program needs to be done in a small group setting, so space is an issue.
(Since bringing my kids in my room, they have done much better with focus than
the library…bad attitudes are also harder to perpetuate in smaller settings).
It’s very early yet, but those are my
observations thus far. My students are presently working very hard, so it
does keep their attention when there’s no one to distract them.”
“By far the number one frustration is pacing segment. Students are angry that they get it correct
but if slightly mess up on pacing they have to do the whole thing over.”
“Not using them frequently enough to see a gain.”
Student and
teacher use of the mobile lab has met our expectations, with interesting class
projects showing a gradual shift to more student-centered classroom activities,
and more use of the computers for research, collaboration, and student-created
publications and presentations. This will continue to grow, as teachers find
new ways to utilize the mobile lab in their classes. Block scheduling allows
longer class periods, which work well with scheduling the mobile lab. However,
managing the mobile lab is challenging: managing checkout and keeping the
laptops all running smoothly requires extra time and effort on the part of
media center staff.
The district’s
administrative changes have complicated implementation of the AoR project, and
have delayed some of the items in the action plan. There were simply too many
things going on at the start of the school year to follow the action plan as
designed for the first year of the AoR implementation. We have learned a lot
from the project so far, and expect to make some changes for next year’s
implementation:
The project is
one that could be duplicated in many other schools. Assuming our standardized
test scores show reading gains, we want to expand the AoR program to our upper
elementary building (grades 2-6). Teachers have commented that junior high
students have had a more positive approach to the program, while high school
students feel that the approach taken by the program is geared toward younger
students. If we intervene earlier in student reading problems, we should see fewer
struggling readers by the time they reach high school. Strong program
leadership is critical to ensure teachers buy in to program goals and methods,
to provide the mechanism for student access to the program, and to provide
adequate support, collaboration time, and professional development for teachers
using the program. Although we don’t have the clearly positive results we had
hoped for after the first year, we expect to see some positive growth in all
areas of measurement in the coming years.
7th
grade – 13 of 15 students showed growth between first test and last test
8th
grade – 14 of 21
9th
grade – 9 of 21
10th
grade – 11 of 22
11th
grade – 7 of 17
12th
grade – 6 of 10
MAP test scores (difference between fall 2007 and spring
2008 scores)
7th
grade - 13 students took both tests; 8 of 13 improved
8th
grade – 19 students took both tests; 14 of 19 improved
9th
grade – 21 students took both tests; 15 of 21 improved
10th
grade – 22 students took both tests; 19 of 22 improved
11th
grade – 18 students took both tests; 14 of 18 improved
12th
grade – seniors did not take spring MAP tests
AoR Posttest
Out of 135
students who started the program:
9 left the
district without completing the posttest
32 did not
complete the posttest
16 decreased
their level between pre and posttest
30 saw no change
48 increased
their level
ITBS
135 students
32 had lower
Reading Comprehension scores
1 had no change
54 had higher
Reading Comprehension scores
48 had no comparison
test (took PLAN test previous year or were not
Disciplinary Referrals for students in Study Skills class
Fall 44
Spring 24
Attendance
533 absences fall
(more than one period, does not include field trips)
641 absences spring