Case Study 2: Creating Lifelong Readers
Principal and Frustrated Reader

The principal stood in the doorway of the classroom and watched the teacher interact with his young charges. As each of the third graders took his or her turn reading a passage in the book, Mrs. Dupree thought about the unevenness in their performances. While some of them got through their selection with only a few bumps, a number of the students stumbled over too many of the words and seemed totally disinterested in the story they were striving to find meaning in.

North Brook School is a rural K-6 school with self-contained classrooms and approximately 600 students, 97% of whom are Caucasian. With 44% of the student population receiving either free or reduced lunch, the school is also considered economically disadvantaged. Over the years, North Brook has received many national and state awards and has been designated a model pre-service education school.

What was the problem faced by the school district?

While North Brook had many strengths, there was cause for concern. The school’s reading scores had fallen below the state and national average for two consecutive years, and many of the students identified reading as an assigned and unexciting task.

North Brook

What could be done?

During the 2000-2001 school year, the school steering committee targeted the development of students as lifelong readers as a priority goal for the school. Their belief was that setting this goal would also result in the improvement in students’ reading comprehension and fluency. The staff felt there was not enough emphasis being placed on motivating students to embrace recreational reading, either at school or in their homes. Struggling students were frustrated by the level of the basal reading series, and they seemed unable to cope with available library books. The challenge was to find a program that would not only boost student achievement and comprehension in reading but would at the same time support the development and enthusiasm of reading for enjoyment. 

Goal

How did they go about it?

In order to achieve the desired goal, the principal, Mrs. Dupree, advocated piloting a curriculum- based assessment tool called The Pro Reader in grades 3-6 (312 students). Before starting the program, students’ reading levels are first measured using software-based assessments. After the individual reading level is determined, students receive a book list with ability-appropriate titles from which to choose and with point values assigned to each. Students read the book at their own pace and take a computerized quiz that is immediately graded, providing instant feedback. The program provides continuous monitoring of student progress, allowing instant teacher intervention if needed. The Pro Reader Program sets a goal point value based on the students’ initial reading level on the pre-assessment.

How does the program work?

Computers and Books

The theoretical premise of the program is that the more books a student reads, the more their comprehension and recreational reading will improve. Point values are used as student incentives with rewards for meeting goals determined by the school. They also found the computer-based assessment to be rewarding and motivational, since 21st century students are accustomed to using computers on a daily basis. The program requires teachers to set aside 60 minutes daily for recreational reading and focused activities to increase comprehension. Students are asked to spend at least 30 minutes daily at home engaged in recreational reading as well.

In order to get a clear picture of how the program worked and the factors that affected implementation, the program planners developed a logic model to portray these elements. This helped them to engage in discussions about what they would need to do to ensure success—by considering all the elements. It also helped them to think about the questions they wanted to answer by conducting an evaluation of the program.

For detailed information about developing a logic model, go to Evaluation 101: Prepare and to Lab 1: Logic Models.

How would they know if the project was successful?
 

The main question for North Brook School was:  Can the program increase achievement and motivation for reading? It was time to find out. Administrators, parents, and teachers at North Brook were eager to determine if the Pro Reader Program was effective in meeting the expected outcomes and goals (increased student achievement and interest in reading).

For more information on developing a plan, go to Evaluation 101: Design Plan.

Pro Reader

What did they want/need to know?

Mrs. Dupree asked a group of teachers representing the grade levels involved to work as a team to consider carefully how to evaluate the program—to know if it was achieving their goals.

In order to determine if the goals are met, the team will use summative evaluation techniques upon completion of the pilot program, i.e. standardized assessments, the Pro Reader Post Test, and end of the year basal reading test. Formative evaluative techniques will also be used throughout the duration of the program.

Additional questions that may need to be addressed in order to measure the success of the program included:

  1. Do students realize a long term gain in comprehension, or do they simply memorize facts from the book and then forget?
  2. Does participation in the Pro Reader Program equate to increased student achievement on standardized reading tests?
  3. Will long term recreational reading carry over into subsequent grades, or will students only read to receive points and incentives?
  4. Are other content areas positively impacted by the Pro Reader Program, i.e. increased science and social studies assessment scores?
  5. Do all students benefit equally from participation in this program?
  6. Will parental involvement in promoting recreational reading increase through this program?
  7. Are computer based assessments motivating and less intimidating to students than traditional paper and pencil?
  8. Do the test questions accurately measure higher levels of comprehension or just the literal level?

For more information on formulating evaluation questions, go to Evaluation 101: Design Plan - Framing Questions and to Lab 2: Framing Questions.

What data would they need to collect?
 
The school decided to utilize the existing school improvement team to follow the progress of the students by monitoring test scores and analyzing behavioral or attitudinal changes in relation to reading.  A mixed approach to data collection will be utilized and involves both qualitative and quantitative methods. 
In order to establish effective evaluation methods, the team decided to first examine the data collection methods provided by the reading program software, including: diagnostic reports, proficiency charts that track reading skills, as well as formative feedback for both teacher and student. The team also wanted to utilize data to assess the long-term effects of the reading program.  Data collection techniques that would be employed consisted of tracking student achievement beyond participation in the Pro Reader Program, administering parent and student surveys and interviews to measure increases in recreational reading, using teacher observations to note changes in student performance, and the analysis of Pro Reader test questions to determine if appropriate levels of reading comprehension are being assessed.
Interview and Surveys

For more information on ways to collect data, go to Lab 5: Evaluation Methods and to Lab 6: Data Collection.

How would they analyze the data?

In order to analyze and correctly interpret the data, the team employed both qualitative and quantitative techniques. The assessments and surveys would be analyzed by quantitative methods, in which test scores would be tracked and compared to another school in the district (i.e. third grade at North Brook will be compared to third grade at a school with similar demographics). The interviews and observations, on the other hand, would need to be qualitatively analyzed.  Upon completion of the pilot program, all staff met to begin the first stages of data analysis.   

According to the pre- and post- test data, students reading comprehension scores increased 30% over the year-long implementation of The Pro Reader Program. Data collected from parent surveys and interviews revealed that student interest in recreational reading also increased and book checkouts from the school library indicated a 35% increase in book loans.

For more information on ways to analyze data, go to Lab 7: Data Analysis.

Increase in reading

What did they find out, who did they need to report to, and what decisions could be made?

Following data analysis, decisions need to be made regarding the effectiveness of the Pro Reader software program in increasing reading comprehension and the joy of reading. By looking carefully at all collected data, the team will be able to decide if their goal has been met. 

Close examination of the parent-completed surveys revealed that an increase in recreational reading was observed among 65% of the students. Also, pre- and post-test comparisons showed that students’ reading comprehension scores increased significantly over the duration of the pilot program. Was the improvement significant enough to outweigh the cost of the Pro Reader Software, which averages around $5 per student? Did it justify the significant amount of time required to implement the program by teachers, students, and parents?

Based on student interviews, the use of The Pro Reader Program had positive impacts on student motivation and interest in recreational reading. Students viewed the computerized assessments as non-threatening and data revealed that test scores from computer-based tests were 25% higher than those tests administered with pencil and paper.

Teachers were very supportive of the program, noting that the collected data helped drive their instruction and that the immediate feedback from the computer-based assessments allowed them to quickly catch problems students were encountering in reading comprehension and provide supplemental materials to help counteract those concerns.

Parents were also encouraged by the program.  Not only did their children’s reading comprehension scores increase, but parents were also becoming more involved in the education process. 

For more information on making decisions, go to Lab 8: Taking Action.