Integrated Evaluation Results and Meta-Evaluation
2006-2007 Grant Year
Third Party External Meta-Evaluation Team:
Gary Phye, Ph.D.,
Mary Herring, Ph.D.,
Don Yarbrough, Ph.D.,
November 19, 2007
Table of Contents
|
Executive Summary................................................................................................... |
5 |
|
|
Introduction to this work and Report |
|
|
|
|
Purposes of this Work and this Report................................................................... |
7 |
|
|
Background and Perspectives Bought by the Meta-Evaluation Team....................... |
7 |
|
|
Strengths and Limitations of this Work.................................................................... |
8 |
|
|
|
|
|
Descriptions of the 2006-2007 Projects |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
9 |
|
|
|
10 |
|
|
North Cedar (Middle School)................................................................................ |
11 |
|
|
|
12 |
|
|
|
14 |
|
|
Sioux Central (Middle and High School)................................................................. |
14 |
|
|
|
|
|
Outcomes on Specific Indicators |
|
|
|
The Impact on Student Engagement |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
15 |
|
|
|
16 |
|
|
North Cedar (Middle School)................................................................................ |
17 |
|
|
|
17 |
|
|
|
17 |
|
|
Sioux Central (Middle and High School)................................................................. |
17 |
|
|
|
|
|
The Impact on Student Achievement |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
18 |
|
|
|
18 |
|
|
North Cedar (Middle School)................................................................................ |
18 |
|
|
|
19 |
|
|
|
19 |
|
|
Sioux Central (Middle and High School)................................................................. |
19 |
|
|
|
|
|
Demonstration of Successful District-to-Vendor Relationships |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
20 |
|
|
|
20 |
|
|
North Cedar (Middle School)................................................................................ |
20 |
|
|
|
21 |
|
|
|
21 |
|
|
Sioux Central (Middle and High School)................................................................. |
21 |
|
|
|
|
|
Development of Individual Education Plans for Students |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
22 |
|
|
|
22 |
|
|
North Cedar (Middle School)................................................................................ |
22 |
|
|
|
22 |
|
|
|
22 |
|
|
Sioux Central (Middle and High School)................................................................. |
22 |
|
|
|
|
|
Effective Integration of Technology and Teacher Training |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
22 |
|
|
|
22 |
|
|
North Cedar (Middle School)................................................................................ |
23 |
|
|
|
24 |
|
|
|
24 |
|
|
Sioux Central (Middle and High School)................................................................. |
24 |
|
|
|
|
|
Curriculum Development to Establish Successful Research-Based
Instructional Methods |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
24 |
|
|
|
25 |
|
|
North Cedar (Middle School)................................................................................ |
25 |
|
|
|
25 |
|
|
|
25 |
|
|
Sioux Central (Middle and High School)................................................................. |
25 |
|
|
|
|
|
Other Outcomes for Schools |
|
|
|
Use of Computers and Software for Writing, Analysis, Research, and
Communication |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
26 |
|
|
|
26 |
|
|
North Cedar (Middle School)................................................................................ |
26 |
|
|
|
26 |
|
|
|
26 |
|
|
Sioux Central (Middle and High School)................................................................. |
27 |
|
|
|
|
|
Movement Toward Student Centered Classrooms |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
27 |
|
|
|
28 |
|
|
North Cedar (Middle School)................................................................................ |
28 |
|
|
|
28 |
|
|
|
28 |
|
|
Sioux Central (Middle and High School)................................................................. |
29 |
|
|
|
|
|
Disciplinary Issues |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
29 |
|
|
|
29 |
|
|
North Cedar (Middle School)................................................................................ |
29 |
|
|
|
30 |
|
|
|
30 |
|
|
Sioux Central (Middle and High School)................................................................. |
30 |
|
|
|
|
|
Other Outcomes Important to the Projects |
|
|
|
|
|
30 |
|
|
|
|
|
Other Outcomes for Educators, Administrators, Parents, and Community
or Organizations |
|
|
|
|
Clay Central-Everly (High School)......................................................................... |
30 |
|
|
|
30 |
|
|
North Cedar (Middle School)................................................................................ |
31 |
|
|
|
31 |
|
|
|
31 |
|
|
Sioux Central (Middle and High School)................................................................. |
32 |
|
|
|
|
|
Lessons Learned by Individual Projects |
|
|
|
|
North Cedar (Middle School)................................................................................ |
32 |
|
|
|
32 |
|
|
|
33 |
|
|
Sioux Central (Middle and High School)................................................................. |
33 |
Executive Summary
Six very different projects were funded and implemented during the 2006-2007 school year. These projects varied not only in their goals but in the implementation of the proposed innovations. The varied nature of these projects makes it impossible to provide a single summary evaluation statement about the successes observed. This is in part due to the fact that one is really dealing with six case studies rather than six variations on a single theme. Consequently, each project stands on its own merits and individual project strengths and weaknesses are identified using a basic organizational theme reflecting the mandatory project goals and outcomes that were identified in the request for proposals (RFP- 2005).
A
dependable finding across all six projects was an interest in promoting the
development of student achievement through the integration of technology and
pedagogy. Some projects were more successful than others in this regard. If one
were to apply a three category evaluation rubric with the category labels of
outstanding, good, and needs improvement, the following evaluation rating would
be appropriate. Outstanding projects were observed at
A reading of the following report will provide suggestions and insights for the ILTC commission, ITLC sponsors, and third party evaluators. First, it is difficult to demonstrate change in a school environment in a single year. This is particularly the case when the outcome is an improvement in student achievement. Realistically, when the innovation requires teacher change, system modification, and/or technology infusion as a precursor of student change, any achievement growth is typically observed between the second and third years. Second, the qualitative case study approach is appropriate for these evaluation efforts. However, one cannot infer cause and affect relationships between educational innovations or interventions and student change. Without inclusion of appropriate comparison groups as a part of the evaluation design, such inferences are not possible. Third, we are still working on cultural change involving communication among various stockholders in education. There is a question of the fidelity of communication between grantees and evaluators. This is viewed as an issue that third party evaluators must be willing to deal with from the beginning of the grantee/evaluator relationship.
This latter point has precipitated a change in the procedures used by the evaluation team. When meeting for the first time with the second cohort of grantees, the evaluation team provided each project with an outline of the evaluation report that would be due at the end of the project year. At that time the type of quantitative data to be collected was verified. This is consistent with the purpose of the action plan that was required of everyone responding to the RFP.
Overall the first year projects were a success. As with all projects, some were superior to others. Two primary themes were recurring. Greater success is likely when ILTC grants can be integrated with existing school initiatives. Second, strong leadership at the building level is necessary. While a cadre of quality teachers is critical to the successful implementation of innovative practices, it is the building administration that must take responsibility for the oversight of the action plan and the evaluation activities.
Introduction
Purposes of this Work and this Report
The Iowa Learning Technology Commission grant recipients were each asked to provide internal evaluation information about their funded project accomplishments during 2006-2007. Each grantee’s evaluation report was expected to address individual education plans for students; students’ engagement and achievement; successful district-to-vendor relationships; research-based curricula and instruction; and the effective integration of technology and teacher training. The external meta-evaluation team (Phye, Herring, & Yarbrough) traveled to each site for at least two site visits, provided on-going evaluation and research consultation, reviewed grantees’ evaluation plans and interim reports, suggested ways to address and document their project goals, and provided review and assistance where feasible and desirable. This third-party meta-evaluators’ report provides the following:
1. Documentation and summaries of the evidence presented in the six individual evaluation reports about the impact of the ILTC funding.
2. An evaluation of the quality of the evidence from these six reports and what can accurately be concluded from them.
3. Suggestions for improvements in the ILTC grant program and in the evaluation guidelines and reports for the coming year (2007-2008).
Backgrounds and Perspectives Brought by the
Meta-Evaluators
The three meta-evaluators bring numerous years experience
with theoretical and applied educational research and are all experienced,
practicing program/project evaluators.
All three are tenured faculty at Board of Regents universities and are
intimately familiar with K-12 and higher education in the State of
Mary Herring is interim head of the Curriculum and Instruction
Department at
Dr. Herring
provided consultation and review of the evaluations for the Clay Central-Everly
Project: Standards Based, Student Centered, Technology Initiative (SST) and the
North Cedar CSD’s project. Other than
her work as a 3rd party reviewing these districts’ studies, Dr.
Herring had no other direct relationship with these projects and no conflicts
of interest.
Gary Phye is director of the Psychology in Education
Research Lab (PERL) at
Dr. Phye provides consultation and review of the evaluations
for Sioux Central’s project, Project INNOvATE and the
Don Yarbrough is Director of the University of Iowa Center
for Evaluation and Assessment (CEA) and an Associate Professor of Educational
Measurement and Statistics, with additional appointments in Educational
Psychology, Educational Policy and Leadership Studies, and the Institute for
Clinical and Translational Science (
Dr. Yarbrough provided consultation and review of the
evaluations for
Strengths and Limitations of this Work
Strength 1. High degree of expertise and dedication. This is important work and was taken seriously by the administrative bodies at the legislature and the Department of Education.
Strength 2. The utilization of a request for proposal (RFP) format provided a more equitable and valid procedure for the selection of projects than an earmark approach to funding. This approach provides the basis for project accountability as determined by the applicants when writing their respective grant proposals to address RFP requirements and recommendations.
Limitation 1. A one year time frame did not allow for more than preliminary work. The impact of project initiatives is difficult to observe in less than a three year period. This is basically due to the fact that organizational structures and systems are slow to respond to changes in operational conditions. This is particularly the case when intended outcome measures are changes in human behavior.
Limitation 2. There was uneven application of effort and interdependence with previous and contemporaneous funding and programs or projects. The less successful grantees in the first cohort of schools were those that did not integrate the new initiative with existing initiatives but tried to develop a new stand alone initiative.
Limitation 3. Grantees felt that the level of funding was limited given the array of intended goals and outcomes.
Limitation 4. Lack of attention to and evaluation of program evaluation theories. In some respects, the RFP is too broad. It may be feasible to consider sharpening the focus of the RFP in terms of goals and outcomes to “better fit” the requirements of current program evaluation models.
Limitation 5. Lack of alignment between grant assessment and evaluation section and program evaluation criteria. A process must be in place for data collection of the mandatory outcomes, this is an accountability requirement. However, from a research design perspective, being able to infer any causal relationship between these mandatory outcomes and the outcomes identified in the respective evaluation sections of a grant proposal is not possible in a case study approach (single case of project). The best one can do as an external evaluator is to provide evidence that appropriate processes were in place as a requirement of meeting RFP conditions.
Descriptions of the 2006-2007 Projects
Clay Central-Everly
(High School)
This project equipped high school students with Tablet PCs. It also included software intended to improve reading performances by delivering a standard-based curriculum. The intent of the quality student-based curriculum was to achieve three district goals:
1.
To increase high school reading proficiency.
2.
To deliver a standards-based curriculum.
3.
To ensure all students graduate with key 21st
Century technology skills.
High school staff members received support for continuous professional development with Tablet PCs. Implementing this one-to-one SST Initiative at the high school was designed to bring this technology directly to the students. Targeted professional development focused on standards-based, technology-rich lessons and assessments with an emphasis on reading. Scientifically based research strategies in reading–such as think-, read-, and talk-alouds, QAR, and graphic organizers—were incorporated in all curricular areas. Grant and district funds provided the continuous professional development, hardware, and software necessary for implementation.
The Clay Central-Everly final report offered some reflections on the use of computers and software applications by the classroom teachers: Mathematics: Class Server a big plus! Both staff members reflected that their teaching techniques changed drastically. The learning curve for both teachers on Class Server would like to be enhanced next year!
Art Education – Class Server used exclusively for this discipline- also Adobe Suite using specifically for our new Graphic Design course. Cross curriculum items were addressed much better with our technology program. All assignments and tests were taken on the computer. Research and resources available through the computer for this discipline were another huge plus!
The district determined the need for the SST Initiative after examining relevant indicators. For example, according to ITED scores, 23% of the Clay Central-Everly 11th grade population is not proficient in reading and 50% fall between the 40th and 75th percentile. Furthermore, 32% of students who took the ACT scored below 20. These are serious problems, because students who cannot read classroom materials are less motivated by textbooks, are less inclined to read and may hesitate to complete assignments that reflect their deficits in reading. Students who can access texts on the computer and can use technology to demonstrate their proficiency in multiple formats will, in theory, be more motivated, challenged, and engaged. The most important process goals over the course of the grant year was to achieve these changes through the development of new staff and student skills as well as through changes in instructional strategies and curricula.
In Davenport, PLTW was implemented at Central, North and West
High Schools (making Davenport the first PLTW school system in Iowa) during
2006-2007 with the goal of transforming existing technology education to
prepare students for high skill, high demand occupations in technical fields. Funding from the ILTC was especially helpful
by providing support for software, technology and teacher professional
development, all requirements for being certified as a PLTW school. Major milestones for the 2006-2007 academic
year included teacher and counselor professional development, enrolling more
than 250 students in the program, and installing computer labs and Cognitive
Tutor Algebraic Principles (CTAP) software to teach Algebra to 225 students. The
PLTW is a very hands-on, project-based
curriculum. In the first course, Introduction to Engineering Design (IED), the
students do the majority of their work on a computer using the 3-D modeling software,
Inventor, which is part of the
North Cedar (Middle
School)
North Cedar’s project centers on the transition of sixth
grade students entering into the
Based on these findings, a leadership committee was formed to determine ways in which North Cedar could better serve its students. The committee investigated several instructional strategies to improve student performance on the Iowa Test of Basic Skills (ITBS) and the Measure of Annual Progress (MAPS).
The committee sought to improve students’ test scores by incorporating two key strategies:
1. Using a one-to-one computer initiative to support students in their daily academic setting and to create greater interest in their work.
2. Changing how instruction would be delivered from teacher focused to project-based learning.
To use the computers effectively, the leadership team determined that project-based learning would allow for greater student engagement in the learning process, while keeping the teachers involved in the instructional process as facilitators.
The project-based learning concept was a new instructional strategy for the teachers. In the past, teacher led discussion or lecture were the primary means of instruction in core classes. The daily schedule was changed to create a period when all teachers would be available to meet with students. The committee developed assigned rooms for the students to be placed into according to test results. For example, if a student had difficulty with math on the ITBS or MAPS test, the student was placed with a math teacher. The committee wanted students to have teachers available to them in the area of most need. If a student struggled in two areas, that student alternated the teachers from day to day which gave him or her opportunities to meet with both teachers.
The last component of the ITLC grant initiative was to develop a connection with local cable television providers. The school formed a partnership with the providers to have DSL placed in the students’ homes at a reduced cost to the homeowners.
Sixty-eight 6th grade students participated in the year-long project. Four of the students entered the program during the middle of the school year or left prior to the completion of the school year. A 7th grade student with special needs participated in the pilot program, but in a limited number of classes. Parents and students were in-serviced prior to the start of the school year. They were informed of the appropriate use of the computer and how to log onto Class Server, the system or platform chosen by the school to present the information to the students.
These goals were accomplished in the high school science department using a team of four teachers working with approximately 650 students. The teachers, who were all familiar with constructivism and techniques to increase knowledge scaffolding, engaged students in a cycle of learning that encouraged them to connect learning to practical experiences. Such learning required that students become engaged in the topic, explore through problem solving, explain their results to others, elaborate on their initial conclusions and evaluate their own thinking as well as that of others. These skills are important in real life and are skills that can be used on the job. They go beyond and can not be acquired through traditional lectures, science demonstration experiments, or preparation for traditional classroom tests alone.
Most of the ILTC funding went to vendors to purchase classroom laptops, digital document cameras, science probes, ceiling-mounted projectors, DVD players, flex cam microscopes and Sympodium software. These items were used to increase engagement in daily lessons and increase the frequency and quality of interactions with science professionals.
Significant partnerships needed to be developed to make the
project work.
Education and business partnerships were established with
members of the
Training and curriculum planning were completed during the summer of 2006 while the equipment was being installed. Teachers prepared for integration opportunities that would match the skills needed by the business partners with the problem solving activities they were developing. This allowed them to be ready with their first activities during the fall of 2006.
Probeware training took place during the summer of 2006 and helped teachers prepare lab activities to facilitate students thinking “like scientists.” Teachers were also trained to use the display equipment. Additional (and potentially useful) training in some of the advanced applications was not available, given time limitations.
Implementation began in the fall of 2006. To start the year, teachers and their
industrial partners developed realistic problem-solving scenarios. In this way,
students were exposed to the way business and industry use scientific concepts
in practice. Additional topics emerged
throughout the year that allowed for the use of business, medical, and college
partners. These electronic and personal
interactions led to further blogging and emailing among students and teachers to
expand upon students’ initial knowledge.
Even students who were not comfortable interacting with others face to
face were willing to offer their views on the web or through e-mail. This extended dialogue has also increased the
students’ depth of understanding of the topics covered. The combination of increased rigor and
relevance fed nicely into
New modes of communication and display technologies made it possible to construct problem-solving solutions, lab reports, PowerPoint slides, podcasts, profcasts, and a variety of other presentations with greater flexibility and expertise than ever before.
Sidney Community Schools is a rural district with 385
students in preschool through 12, located in
Prior to the award of the ILTC grant, the district’s access to technology for student and staff use was limited to one lab at the elementary school and two labs at the secondary school. Additionally both buildings had at least one computer in each classroom, where access to the Internet was limited.
The
Sioux Central (Middle
and High School)
This grant was written during the 2005-2006 school year. During the summer of 2006, the former elementary principal/curriculum director who wrote the grant resigned from his position. The district also had a change in superintendent positions. The superintendent under whom the grant was written also resigned in July 2006. So, it was not until fall 2006 that the new elementary principal and new superintendent were brought on board for this project. Therefore, the initial implementation efforts for this project were slow.
The original grant application was for students in grades 4-12. The intent was to implement Project INNOvATE in all 4-12 science classrooms. As implementation of this project began in fall 2006, the new leadership realized that full implementation would not be possible in all grades 4-12. The school narrowed its focus and implemented Earth Science in grade 8, Physical Science in grade 9, and Biology in grade 10.
The biggest problem with this project was the lack of funds in the grant for additional training on the usage of the probeware and laptop computers. The grant provided one day of training on September 18, 2007 with the use of the probeware. No additional funds were available for training throughout the year. Therefore, teachers were left on their own to experiment with the probeware implementation as well as how to utilize inquiry learning with laptop computers.
Outcomes on Specific Indicators
For each project, this report evaluates the evidence provided and reports evidenced based information (any data collection, including qualitative and quantitative) related to the specific dimensions of quality listed in the Request for Proposals. Where the individual grantee evaluation report includes only opinion-based conclusions or did not address a dimension of quality, we indicate that no evidence was provided in that grantee’s final report.
The Impact on Student Engagement
Clay Central-Everly
(High School)
Visits by the external evaluator to the school found students using the computers in the classroom for classroom projects and note taking. The district’s final report provided an overview of activities within the different departments.
The Business department used laptops daily for the grade nine required Computer Applications class. Frequent laptop used was observed for Accounting I and II; Intro to Business; and Business Law. The Industrial Technology Department used laptops and desktop computers for Diverse Tech. a computer aided drafting program. A CAD program was incorporated into the Woodworking classes at a more basic level with students drafting projects using the CAD program.
Math, Science and Social Science had frequent laptop use throughout the year and textbook for these classes were loaded on student laptops. The Art Design class, Music Theory and Music Theatre classes were also laptop based. Student consistently used their laptops 80% to 90% of the time with most assessments for the high school curriculum being done on the laptop.
Student online surveys were not done consistently and only one survey was reported completed, as opposed to the originally proposed three surveys. It was not indicated when this survey was completed. This survey reflected the installment of the laptop computer program and reactions to the new program by parents and students. The majority (97%) of the parents and students were supportive of the program and 3% were not, stating that the computer program was only another diversion from the normal school curriculum.
The final report offered that walk-throughs planned to gather data on this indicator were sporadic and that little data was recorded. No data was reported on student assignment completion as was indicated in the original grant proposal.
During 2006-2007, 269 students participated in PLTW. During informal observations and interviews
at
Student engagement was measured by the Coordinator of Curriculum & Instruction responsible for implementing Project Lead the Way, Betty Griffin. She utilized the Instructional Practices Inventory model. Six levels of engagement on the part of the teacher and the student were recorded. This was done over the course of the school year as a result of 21 visits to PLTW classrooms and then merged onto one summary document. (See attachment: Project Lead The Way – Instructional Practices Inventory 2006-07) The optimum rating of a 6 for active learning/active teaching occurred 56% of the time. Teacher led conversation, a 5 rating, occurred 5% of the time. This is not surprising with such a computer-driven, project-based curriculum. Teacher led instruction, a 4 rating, occurred 29% of the time. This was especially prevalent at the beginning of class when teachers delivered PowerPoint presentations which are part of the PLTW curriculum and set the foundational knowledge necessary for students to do their work. Only 10% of the time was described as seatwork/teacher engaged work, a 3 rating. This tended to be time spent giving instructions for a test or for the students’ portfolios. There was no evidence that the teachers were disengaged when students were doing seatwork, a 2 rating, or that there was total disengagement of the students and teachers, a 1 rating.
Students also wrote papers on engineering-related occupations and met practicing engineers in the local community, providing additional opportunities for motivation and engagement in the subject both in and outside of the classroom. Discipline problems were so rare that there was no need to track them in this group of students, according to the evaluation report.
North Cedar (Middle
School)
Teachers reported the time their students were engaged on the computer compared to the traditional assignment process and how much of the time was centered on the project-based learning as compared to teacher-directed instruction, using their lesson plans as a reference (see Table 1).
Table 1. Percentage of Student Computer Use and Project-Based Learning Use by Subject.
|
Subject Area |
% of Student Computer Use |
% of Project-Based Learning Use |
|
English and |
95% |
100% |
|
Math |
50% |
10% |
|
Science |
90% |
90% |
|
Social Studies |
80% |
80% |
The
The 5E’s were also used as a means of measuring the opportunities for students to engage in problem-solving experiences that mirror how scientists research in their own field. The tracking of teacher lesson plans indicated a student-centered approach that focused on posing research questions for investigation. This approach involved problem-solving activities that used the new technology and equipment to engage in experimentation or communication about experimental problem-solving and ethics issues.
Sidney reported that as a result of the ILTC project, they believed that student engagement had increased because more teachers were incorporating group projects into their curriculum activities No quantitative was provided to support this belief. The increase was evidenced by their weekly lesson plans and by regularly scheduled administrative walk-throughs. Administrators reported that it was “not uncommon to see students working in pairs in various locations in the building, … actively engaged in writing, rehearsing, recording or revising a podcast on the laptop computer.” The number of projects is recorded in the check-out log in the appendix of this report, as well as by the number of podcasts published on the school website. End-of-year reflections by staff also indicated that students are more actively engaged with the use of the Apple laptop computers.
Sioux Central (Middle
and High School)
The goal was to use the number of failures to do assignments in the Earth Science (8th grade) class as a measure of increased student engagement. The number of Failure to Do Assignments (FDA’s) in the 8th grade science class decreased from 35 FDA’s in 1st quarter to 18 FDA’s in 3rd quarter. The 4th quarter saw an increase to 27 FDA’s. Comparing the science data according to semesters, there was a decrease from 67 FDA’s in the first semester to 45 FDA’s in the second semester. The ACE Science points (inappropriate classroom behaviors) decreased from 28 in the first semester to 19 in the second semester.
The Impact on Student Achievement
Clay Central-Everly
(High School)
No data were provided or analyzed by the district demonstrating impact on student achievement. Anecdotal observations indicated, “We used the SRI tests for an assessment format in reading and then adjusted our instruction in each class as indicated by these assessments. We gave the SRI test format; used our ITED reading pieces and used teacher reflections on their own assessments to determine adjusted teaching methods for our students.”
Data provided in the evaluation report suggested that many students
who took the CT Algebra Principles performed better than students with other
pre-algebra preparation classes in the Fall 2006 Algebra 1 classes. However, this needs further study and may be a
factor of who selected the CTAP. In addition, approximately 30 students scored
high enough on the end-of-course college assessment and their in-class
assessments to pay for and receive credit at the college level at
North Cedar (Middle
School)
The scores in Table 2 indicate the average growth of the students, using the National Grade Equivalency scale. In order to make any inferences about the relationship between implementation of the new program and changes in achievement based on scores it would be necessary to compare the average growth in grade equivalency of the 6th grade class which benefited from the program to the average growth in grade equivalency of 6th graders who did not, perhaps starting with 6th graders from the year before.
Table 2. Average Grade Equivalency Growth by ITBS Subject Category
|
ITBS category |
Average growth, measured as grade
equivalency |
||
|
6th Grade (n=2013) |
7th Grade (n=2012) |
8th Grade (n=2011) |
|
|
|
.8 |
.5 |
.7 |
|
Reading Comprehension |
.7 |
.4 |
.6 |
|
Math Total |
.4 |
.9 |
.9 |
|
Science Total |
.7 |
1.3 |
.6 |
|
Social Studies |
-.2 |
1.8 |
.6 |
|
Core Total |
.4 |
.6 |
.7 |
Although North Cedar sixth graders showed growth in reading,
math, and science, the table shows a decrease in social studies. The social
studies results may be attributable to several different factors. The change from teacher-directed strategies
to a project-based learning structure allowed students to gather a wider range
of information than what was presented on the testing materials. Concentrating on themes rather than textbook
chapters, the social studies teacher created a class that was less aligned with
the content of the ITBS social studies test.
Finally, an evaluation of the teacher’s performance this year showed
that he engaged students at a much greater level with many different activities
than he had in the previous years. For
example, he made a large map of the continent of
proficiency in problem solving, measured by locally established problem-solving tasks, 89.6% of Pella high school students achieved an average score of 70% or higher. Data on the ACT scores will be collected in future years.
The district continues to compile data from standardized tests, including the ITBS and ITED, but at this point reports that the data are inconclusive with regard to the grant’s impact.
Sioux Central (Middle
and High School)
Sioux Central reported that 84% of their 8th grade students in 2006-2007were proficient in science, as compared to 82% of the 7th grade students the year before. 74% of Sioux Central students were proficient while in 9th grade in 2006-2007, as compared to 80% of the 8th grade students the previous year. 90% of Sioux Central students were proficient in science while in the 10th grade in 2006-2007, as compared to 72% of the 9th graders the previous year. In order to assess the impact of the new technology, it would be interesting to compare these changes in proficiency to the changes in proficiency among the same grades from the period 2004-2005 to 2005-2006, but that has not been done.
Demonstration of Successful District-to-Vendor
Relationships
Clay Central/Everly
(High School)
The district did not contact with Knowledge Network Solutions (KNS) to provide assessment of current technology usage or 8 days of Tablet PC integration into the classroom training, 3 days of curriculum development training, on-going support via e-mail, online conferences, and on-site visits. No reason was given for this decision.
It was reported that they “received support services from
Karen Appleton, AEA 8 consultant concerning the use of resources for both staff
and students” Data as to effectiveness of this relationship was not provided.
The district did report on Microsoft’s Class Server classroom management system
use, “Most students when off school time used our “Class Server” frequently and
about a 60% usage was observed for this activity.” The district also “abandoned
the
Getting materials from the vendors on time was a big challenge because PLTW has become so popular the suppliers could not fill the vendors’ orders. In some instances we cancelled the order and paid a little more to get what we needed in time. There is too much PLTW curriculum so pacing was a challenge the first time the teachers taught the classes. The teachers kept good notes on what they did and met in June to discuss how to adjust the curriculum to create a timeline for next year.
It is anticipated that vendor relationships will improve even more as the PLTW curriculum and technology becomes more clear and standardized in the local high schools.
North Cedar (Middle
School)
North Cedar reported that one vendor in particular had promised that the instructional portion of a classroom could be filmed and then played on a local access channel during the evening hours. Unfortunately the cable system started installing a fiber optic system at an inopportune time. During the changeover process the school was unable to transmit on their station. To compensate for this, classes were filmed and disks provided to students who had been absent upon their return to school. According to their report, it wasn’t the ideal solution, but it did fulfill the purpose of the grant.
Other vendors that were influential in allowing the project to run smoothly included: Microsoft, Intel, Gateway, F & B Communications, Knowledge Network Solutions, Classserver, Broad Education and the Clarence Telephone and Cable Company. The Grant Wood Area Education Agency provided numerous individuals in support of the project. The members from Grant Wood included Keith Stamp (Area Administrator), Vicki Bone (Special Education Consultant), Diane Peters (Reading Strategist), Mike Macklin (School Improvement Consultant) and Jon Nietupski (Grant Writer).
According to the
While
Sioux Central (Middle
and High School)
Sioux Central did not report measuring the success of vendor relationships with vendors this year. They are planning to develop a partnership for 2007-2008 with Gateway, where the portable wireless lab and server were purchased. A second vendor is PASCO Science, which provided the software/probeware for the project. One teacher also utilized material as part of prior training from the National Aeronautics and Space Administration (NASA).
Development of Individual Education
Plans for Students
Clay Central/Everly
(High School)
The Clay Central/Everly report did not address individual education plans.
According to the
North Cedar (Middle
School)
The North Central report did not address individual
education plans.
The
Sioux Central (Middle
and High School)
Sioux Central reported that IEPs were not appropriate for
this project.
Effective Integration of Technology and
Teacher Training
Clay Central/Everly
(High School)
Clay Central/Everly reported that project members thought that they tried to implement far too much new computer technology in the first year. Despite frustrations and complications, however, they worked extremely hard to make sure that the system worked and to help one another learn the system, with definite benefits for the whole high school. They reported that all staff were engaged and on task on each and every staff development day and also trained every other Wednesday evening to learn the program.
PLTW is very computer and software intensive, requiring
professional development for the in-service teachers and counselors. As part of the requirement to participate in
the program,
North Cedar (Middle
School)
To institute the change in the delivery of instruction from teacher driven to teacher facilitated, the leadership team received training from Knowledge Network Solutions in developing and utilizing project-based learning in the classroom. Three units using the project-based learning format were developed by the committee during in-service opportunities afforded to the committee by the grant from ITLC. The three units of study that the leadership team developed were Ethics, Diversity, and Change.
The professional development was led in two different pieces. The first portion of the professional development was in the area of using the Microsoft platforms to support data acquisition and student presentations. The teachers were led through a process to use Power Point in making classroom presentations more powerful for student learning. Microsoft based Excel spreadsheets, and data base to support instruction in the classroom. KNS was the resource responsible for presenting the information to the teachers. Kelly Dietrich, KNS employee, presented the material to the staff and had the staff start developing materials that they would be able to use in their classrooms during the school year. Ms. Dietrich then led the staff in developing the project-based learning system that was used by the staff. Kelly led the committee in looking back at our past curricular area lesson plans and to see if there were three or four common themes, that ran though all of the teachers areas. After identifying the common themes the group started to develop the first unit.
The staff continued to work on the unit several times during the summer and staff in-service days prior to the students starting school. During the school year the team met every third day during “teaming” time and after the end of the school day working on the units. During the school year, staff was provided three days of in-service to help develop the units. Substitute teachers were hired to replace the teachers in their classrooms to support the cultivating of curriculum.
As part of the project, we incorporated a co-teaching format in the social studies and science areas. MaDonna Gretten, Level II & Level III teacher co-taught with Todd Hoefler in social studies and with Laura Cady in science. The balance that was achieved by these teachers in the classroom was amazing to see. Much of the success can be attributed to the personalities of the teachers and to the special education teacher’s willingness to work extremely hard before and after school in developing the lesson plans with each individual teacher. If you were to attend a class, you could not pick out who was supposed to be the lead teacher and who was entering the room as the special education instructor.
Ms. Gretten also carried this project back to her room. She had a much greater level of success with her Level II students and their participation in the project than did our Level I instructor. In many other school systems this is a non-typical situation. This may attributed to several factors: 1. Ms. Gretten participated in the initial training of the teachers. 2. Ms. Gretten communicated frequently with parents on the phone about areas of concern with the students and the students’ computer difficulties. 3. Ms. Gretten spent numerous hours of her own time preparing the lessons for the students, so that her students would not be frustrated with some of the more advanced computer issues.
The
Sioux Central (Middle
and High School)
Sioux Central reported that Rod Haenke from Instructional Designs, Inc. was hired in the summer of 2006 to provide two days of professional development training on using science inquiry with technology. One day of training was provided in September 2007 on how to use the probeware.
Curriculum Development to Establish
Successful Research-Based Instructional Methods
Clay Central/Everly
(High School)
The Clay Central/Everly report addressed this question through teacher comments. One language arts teacher reported that “research was more enhanced.” Science teachers noted that “research topics and information available online was enhanced by the program.”
PLTW is a nationally developed, well-researched
intervention. Research on PLTW
effectiveness, how to recruit students typically under-represented in
engineering and technical occupations, and how to make the curriculum and
instruction more effective is taking place nationally and at
North Cedar (Middle
School)
The teachers were engaged with the project-based learning for a core of the instruction. Prior to the one-to-one computer initiative, the math instructor used lecture-guided practice with a limited amount of manipulatives used during the instructional process. By becoming involved in the project, he has greatly increased the use of manipulatives in his classroom. North Cedar faculty reported that students participated in research activities from 35-45% of the time within the English, Social Studies, Math and Science classrooms (see Table 3).
|
Table 3. Percentage of time students participate in research activities by subject. |
|
|
Subject |
Research |
|
English |
35% |
|
Social Studies |
50% |
|
Math |
45% |
|
Science |
45% |
During the site visit, the
Not reported.
Sioux Central (Middle
and High School)
During the year, teachers began to revise their lesson plans
to include inquiry learning into the curriculum. Teachers also began the expansion
of the use of scientific equipment and labs as part of the instruction in
inquiry learning.
Use of Computers and Software for
Writing, Analysis, Research, and Communication
Clay Central/Everly
(High School)
Although Clay Central/Everly’s grant application stated that the district would use “Teacher logs and Student product portfolios” no data was analyzed. Examples of student portfolios were submitted.
Use of computers and software for writing, analysis, and
research is integral to the PLTW intervention.
North Cedar (Middle
School)
North Cedar reported, in the Table 4 below, the amount of time that the core teachers spent in their classroom doing activities that involved writing, analysis, research, or non-computerized activities.
|
Table 4. Percentage of time core teacher spent doing activities in their classroom by subject |
||||
|
Subject |
Writing |
Analysis |
Research |
Non-computer |
|
English |
40% |
20% |
35% |
5% |
|
Social Studies |
10% |
20% |
50% |
20% |
|
Math |
2% |
3% |
45% |
50% |
|
Science |
15% |
30% |
45% |
10% |
According to the
1. Use of scientific probeware for investigation
2. Gathering and graphically displaying problem-solving results for elaboration to peers
3. Posting of results on Moodle web sites
4. Feedback on results through blogging with teachers, peers, and professionals
5. Podcasting using iTunes and Profcast
6. Virtual problem solving
7. Multimedia presentations
The use of computers and software has increased as evidenced
by a number of indicators provided in the
Sioux Central (Middle
and High School)
In the Sioux Central report, increased use of computers was measured by the collection of computer log
data from the science teachers. The data show an increase of computer usage in
the Earth Science class from 7 times during the first semester to 25 times in
the second semester. Computer usage decreased in the Physical Science class
from 14 times in the first semester to 7 times in the second semester. Computer usage in the Biology class increased
from 11 in the first semester to 32 in the second semester. The overall computer usage (grades 8, 9, 10)
increased from 37 in the first semester to 64 in the 2nd semester.
Movement Toward Student Centered
Classrooms
Clay Central-Everly
(High School)
The district reports use of the following formats in the classroom for the 2006-2007 school year as evidence of student centered classrooms:
1.
Several research data was collected for all of
the classes
2.
Class
Server was consistently used by both parents and students
3.
Student
was required in several classes at all grade levels to develop slide shows and
PowerPoint presentations. These were done not only for classes, but
also other community activities and extra-curricular activates the students
were involved in.
4.
We
had one evening with the parents and used students to demonstrate the use of
our new software and how teacher/student communication was used. Parents also were involved in the learning of
the wireless system so they could also view student work, school announcements,
etc.
5.
Portfolios
were developed for each student with most of the emphasis based on the production of assignments and storage
of school work used.
Not Reported.
North Cedar (Middle
School)
North Cedar reported that teachers were engaged with the project-based learning for a core of the instruction. The three teachers who taught science, social studies and English established high expectations for using project-based learning in their classrooms on a daily basis. The math instructor was involved in project-based learning but on a much smaller scale than the other teachers. The numbers reported in section 4.2.6 indicate this. Prior to the one-to-one computer initiative, the math instructor used lecture-guided practice with a limited amount of manipulatives used during the instructional process. When he became involved in the project, he greatly increased the use of manipulatives in his classroom.
The enjoyment teachers derived from the experience and the degree to which they felt they were making a difference were not measured with any specific instruments. However, four out of the five main teachers involved in the project stated that it was a positive experience for them and want to continue to use the computers in the future. Only the math teacher is hesitant in using the computers for the 2007-2008 school year. The teachers expressed that it could be “fun” to be teaching and that it was incredible to see the amount of information students could obtain in each topic area. The teachers expressed that their students were very engaged in the projects, especially the election debates, polls, campaign and the voting.
Teachers have developed real-world problem-solving scenarios
for each unit of study in the high school science curriculum. Students are offered choice in activities
through multiple scenarios and independent research tasks. The tasks have specific criteria to inform
evaluation but are open-ended to allow students to tailor them toward specific
areas of interest.
Explicit instructional plans submitted by each teacher throughout the year indicated that teachers were beginning to shift the focus of their classrooms from “teacher-centered” to “student-centered”. Examples of this appeared in instructional lesson plans collected by the building administrator. An increase in the number of plans that include individual research and writing, group work on podcasting, and peer assistance was documented in explicit instructional plans as well as in weekly lesson plans.
Sioux Central (Middle
and High School)
The Sioux Central report indicated that the move to more student-centered classrooms was measured by the use of inquiry-based lessons at the start of the 2006-2007 school year compared to the end of the year. The number of estimated inquiry-based lessons remained about the same from first semester to second semester. Sioux Central concluded that additional efforts will need to be made in 2007-2008 to do a better job of collecting lesson plans to get an accurate count.
Disciplinary Issues
Clay Central/Everly
(High School)
Student disciplinary issues were tracked by a comparison of referrals between the 2005-2006 school year and the 2006-2007 school year. There was a dramatic decrease in the numbers, from 204 in 2005-2006 to 87 in 2006-2007. The district felt that the use of computers contributed to this decrease.
North Cedar (Middle
School)
The disciplinary problem data was based on the TATs for behavior issued by the teachers. The 6th grade teachers issued 139 TATs compared to 192 for the 7th grade students and 282 for the 8th grade students. According to the North Cedar report, the comparison of these numbers to the number of TATs generated by 6th graders before the implementation of the program was valuable in helping to determine the effect of the program on discipline. For the 6th grade students over half of the disciplines issued occurred in a single classroom. A group of students in an exploratory classroom misbehaved frequently in the second quarter, reflected in the reporting. The staff members believed that the students were better behaved when they had their computers and were engaged in an activity. The only problem that the team experienced as a whole occurred when students were instant messaging each other against their teachers’ instructions. Even 7th and 8th grade teachers who had the 6th grade students in their 6th hour guided studies classes noted how much more engaged the 6th grade students were compared to the other students.
Sioux Central (Middle
and High School)
See discussion under student engagement.
Other Outcomes Important to the Projects
According to
Other Outcomes for Educators,
Administrators, Parents and Community or Organizations
Clay Central-Everly
(High School)
Did not report.
This is a tremendous program that can serve as a model “program of study” for education in general with its emphasis on staff development for the teachers before they teach the class plus on-going teacher staff development, counselor training, rigorous and relevant curriculum linked to industry standards, end-of-course assessment, and college credit.
The 3rd party evaluator reported that this was a mature program with widespread and solid district and external support. In addition to ITLC funding, the PLTW intervention received extensive external and internal resource allocations. He noted especially how effective the project management was and how community support, steering committee work, and local administration worked together effectively.
North Cedar (Middle School)
The data for the parental involvement is derived from a
parent survey taken at the end of the school year. Of the possible 64 sets of parents, only 31
parents responded to the on-line survey. Fifty-five percent of the responding parents
indicated that they logged on to their child’s Class server account on a daily
basis. Eighty-seven percent of the
parents indicated that they logged on to their child’s Class server account on
a weekly basis and one hundred percent logged at least once during the
month. Another question, which was relevant
for student achievement, was the extent to which parents felt involved in their
child’s educational pursuits. Every parent responded that they have assisted
their child on research or homework assignments.
Parental involvement was not a focus of the project. However,
Parental involvement in the use of technology has increased
over the past year. A technology survey
was completed by 66 parents in the district.
From these surveys, it was determined that 68.2% of them receive e-mail notices
about school activities and announcements.
35.5% of the parents who responded indicated that they access school
communication using technology on at least a weekly basis. When asked if the
Comments from parents on the survey include the following to support positive outcomes from the grant:
The number of “hits” on the school’s parent access page was collected. Data indicate an increase of 25% in the number of hits on the parent access page from November 7, 2006 to May 20, 2007.
Sioux Central (Middle
and High School)
Parental involvement data were collected using PowerSchool, a school wide database system that monitors student attendance, grades and student demographics. The middle school data show the total access hits by parents to the PowerSchool system increased from 1182 in the first semester to 1322 in the second semester. The high school data shows the total accesses by parents to the PowerSchool system remained essentially the same from the first semester (4659) to the second semester (4624).
Lessons Learned by Individual Projects
North Cedar (Middle
School)
In retrospect, we should have been less aggressive at trying to change so many areas at once. We went from teacher directed instruction to project-based instruction, changed the daily schedule to add a guided study hall for all students, and went to a computer initiative for the 6th grade students. It would have been much easier to manage one or two of these initiatives than all three simultaneously.
We are hoping to bring the project forward to the seventh graders in several classrooms. The social studies teacher and reading teacher would like to be able to use class server in their classrooms next year. This means, that a third of the seventh graders’ classes would be on-line and using computers during their daily lessons. We can do this plan if we allow the seventh grade students to check out computers similar to what we anticipate doing for the sixth grade students.
We have plans to expand the use of the technology in the classroom. This will include the videoconferencing features of our equipment, professional critique of projects, expanded problem solving scenarios, and a variety of experimental design additions.
Our plans also include the model we have established with Pella Corporation as a result of our partnership with them on this project. It is built in five levels:
Level 1: Awareness activities on what an engineer does that articulate with our curriculum in grades 3-12.
Level 2: Engagement
activities at Pella High, Pella Corporation, and the
Level 3: Mentoring for students who enter engineering
college by Pella Corporation and the
Level 4: Possible early acceptance into the
internship program at Pella Corporation.
It would also include reverse mentoring of
Level 5: Potential hiring by Pella Corporation
We would like to repeat this model in various other science
careers. Many of these may need to use
technology to complete, as it would not be possible to bring them to
As we designed the professional development with Apple professionals this year, we grouped the staff according to grade level, K-6 and 7-12. While this was effective in that some of the staff who was more proficient with technology could assist others, we have decided that next year is will be more beneficial to provide “ability grouped” instruction. Those staff members who are proficient will be grouped together to move at a more accelerated pace through the training. Those who are still at the frustration level will be able to move at a less intense rate. Following the training from Apple, we will again rely on those more proficient teachers to become mentors as the teachers work in study teams to implement and integrate technology.
We have discussed the sustainability of the work from this grant and believe we have in place several items for that purpose. First of all, the AEA will provide two days of professional development in this district on the use of iMovie we have again contracted with Apple for technology training from their facilitators. We have two dates in August and two in November contracted with them. Secondly, we have made arrangements to update the elementary computer lab with Apple computers that will have more capabilities for this project. Thirdly, we have entered into a leasing agreement with Apple to allow us to purchase twenty-four additional iBooks to expand the number of mobile labs in the district to six. Lastly, we will continue to provide staff with the time to collaborate together in study teams with a weekly late start devoted to professional development.
Sioux Central (Middle
and High School)
It is the intention of Sioux Central to continue to implement this project within the next school year. We will look at the data (lesson plans, computer use logs, ITBS/ITED, etc.) at the end of the second and third year to see if trends change and were impacted by the use of the material. Since we have the equipment and have begun to implement the probeware, we will continue to do our own in-house professional development and continue to collect data