Research Into Practice

2006

Methodological Challenges of Study Participation and Attrition:
Pacific Region Case Studies

By
PREL Staff

The design and conduct of field-based research studies present numerous methodological challenges that must be addressed in the study design in order to ensure validity and reliability of the study conclusions. This report addresses the challenges of recruitment, retention, and attrition of study participants.

It is commonly accepted that the process developed for the selection criteria for study participants is critical in the study design. It is not uncommon that issues related to study participation take up less time as the study proceeds, as this process is often thought of as only pertaining to the beginning phase of the research. This beginning phase is important, as the initial selection of participants has a domino effect that has an impact on participant retention and attrition, and thus the generalizability of the study outcomes. For those reasons, researchers need to give careful consideration to issues of retention and attrition at the outset of the design and throughout the duration of the study. In a carefully conducted study, issues of participation, retention, and attrition remain visible throughout the study’s progress.

Once study participants are selected, informational meetings held, memoranda of understanding signed, and such, the task of retaining the participants must be managed. This presents its own set of challenges for the duration of the study. Long-term studies especially may require creative strategies, trust, and commitment. Competing demands experienced by school districts from states, parents, and communities may complicate the ability of the school to remain loyal to the requirements of a study as originally agreed. These demands present researchers with the challenge of implementing strategies at the beginning of the study in order to minimize attrition rates. When participants are compelled to withdraw from a study, researchers are challenged to manage the attrition in a way that minimizes impact on the integrity of the study conclusions.

The case studies described in this report present these particular methodological challenges as experienced during the Pacific Communities with High-performance In Literacy Development (Pacific CHILD) research project undertaken by Pacific Resources for Education and Learning’s (PREL’s) Regional Educational Laboratory (REL). The case studies are presented in an attempt to deepen the understanding of methodological challenges facing educational research and development efforts, and their implications for research in field settings.

THE PACIFIC REGION
The region served by PREL’s REL program includes 10 U.S. affiliates in the Pacific region whose political statuses range from statehood to free association. In addition to economic and political diversity, the Pacific entities are characterized by a multiplicity of cultures and languages. At least nine different Pacific cultures are prominent in the region. The languages of instruction in the region’s schools include English, as well as a dozen or more Pacific languages. In the region outside of Hawai‘i, there are approximately 410,183 students attending 446 schools, including private institutions. Hawai‘i’s public school enrollment is 187,395 students in 241 public schools and 37,078 students in 132 private schools. In developing the Pacific CHILD research design, a conscious effort was made to acknowledge the cultural characteristics of the region PREL serves, including the importance of community and collectivistic value systems, while maintaining the highest methodological standards for social science research.

PURPOSE AND DESIGN OF THE STUDY
PREL initiated the Pacific CHILD research project in response to an overwhelming need to improve teachers’ knowledge and instructional practices in early reading. Educators in the Pacific region perceive three problems as major issues:

Problem 1: Significant numbers of students are not reading independently in either English or their home language by the end of 3rd grade.

Problem 2: Low-performing schools in the region have difficulty improving student learning in part because they lack adequate or appropriate assessment systems and accountability processes.

Problem 3: A significant percentage of teachers do not have the necessary content knowledge or pedagogical skills to create high-performing learning communities.

For the purposes of this study, one overarching question provided the focus and direction: What are the components of an effective professional development model for early reading improvement? This question guided the inquiry. Within this question, several sub-questions elicited more information to identify the components of the model.

The research design measured correlations between Pacific CHILD professional development delivered to teachers and teacher knowledge and instruction, as well as student achievement in reading. This information provided support in identifying components of a model that are effective and worthy of experimental study. As the individual components of the model were correlated with increases in teacher knowledge and change in instruction, the effectiveness of the components became evident. The research design did not begin with a hypothesis to be proved or disproved, but rather with a flexible plan to investigate phenomena. With this in mind, the research intends to fill a gap in existing knowledge particular to the Pacific region.

PREL researchers developed a research-based set of criteria for the purpose of selecting one school site in each of PREL’s 10 service entities. Among the criteria was support from the entity’s chief state school officer, the school leader, and the school teaching staff. Prior to site selections, meetings were held at each entity outlining the requirements and criteria of the project. These meetings were held in an attempt to increase likelihood of participant retention throughout the duration of the study. The following factors are key to retention: community, staff, and leadership willingness to participate; capacity for using the intervention; provision of needed data; and sustaining ongoing involvement over the duration of the project.

Ten elementary schools in the Pacific designated as research sites were the main foci of data collection. These sites are known as Co-Development Partner (CDP) schools. Eight of the 10 schools use a different Pacific languages for instruction. The research sites targeted grades K–2 or grades 1–3 (depending on whether the school offered kindergarten) and were judged by the Pacific CHILD research team to have teacher and administrative characteristics conducive to more in-depth observation and systematic data collection.

Criteria for CDP Schools:

1. The school receives approval of chief state school officer.
2. . A member of the Pacific Curriculum and Instruction Council (PCIC) recommends the school.
3. The school leader (principal) supports and buys into the program.
4. The majority of school teaching staff and parents support and buy into the program.
5. School staff and parents understand the extent of the commitment to participate and a majority agree that the school should be involved.
6. School data show low achievement for students in the area of reading.
7. The school will designate a single point of contact at the school to facilitate communication and coordination of activities.
8. The school is willing to participate in both the reading initiative and the linked assessment and accountability interventions.
9. In the event that another site-based initiative is underway at the school, the REL initiative fits into the overall school plan.
10. In the event that another PREL program is working with the site, the REL initiative can be coordinated with the other PREL program to avoid teacher burnout or overload.
11. The school and staff commit to participation for at least 3 years and possibly all 5 years of the contract.
12. The school will provide access to student data in reading achievement and other areas, as related to becoming a high-performing learning community.
13. Where necessary, in collaboration with PREL, schools will facilitate the administration of reading assessments in Pacific languages and/or in English.
14. Teachers are willing to participate in ongoing weekly, biweekly, or monthly staff development activities.
15. Teachers and others as appropriate are willing to allow in-class interventions, which may include partner teaching, classroom observations, demonstrations, and lesson modeling.
16. School staff will allow the PREL research team to conduct activities related to the documentation, implementation, and outcomes of the improvement effort at the school.

An important characteristic of the Pacific CHILD professional development was the inclusion of a reading specialist who was from the local community and fluent in the local language. The site-based reading specialist delivered professional development at the CDP school on a regular basis. In addition to these initial qualities, people who had some knowledge in early reading and classroom experience were sought. Teacher preparation is inconsistent throughout the Pacific region. Understanding that the majority of educators in this region possess, at most, an associate’s degree, finding staff with specific content knowledge in early reading proved challenging. Thus, the level of early reading content knowledge of staff members who were hired ranged from none to quite extensive knowledge. Also of importance to this equation was the necessity of having these specialists skillfully communicate early reading content knowledge to teachers. In other words, our reading specialists also needed to know how to provide professional development in early reading. While the professional development delivered to each school was consistent in content, the staff developers’ ability to effectively deliver the material and the teachers’ ability to make the professional development their own differed considerably. In addition to the reading specialists, REL staff included site-based assessment specialists who were available to assist and support the assessment work at various sites. During the course of the Pacific CHILD research project, 5 of the 10 schools retained their status as CDP schools. The remaining five schools were dropped from the research project, but were monitored at different intervals for the purpose of providing insight into factors that hinder evidence-based research in low-performing schools. The entity schools in Hawai‘i, Chuuk, Yap, the Commonwealth of the Northern Mariana Islands (CNMI), and Guam are subjects for this brief. The following case studies describe the circumstances under which each school was no longer identified as an intensive research site.

KALIHI ELEMENTARY SCHOOL, HAWAI‘I
Kalihi Elementary School was identified as a CDP school in June 2001. It participated in the Pacific CHILD project from August 2001 to March 2003.

There were several reasons Kalihi Elementary School became a CDP school site. In selecting a site, special consideration was given to Hawai‘i schools that had already been working with PREL. In addition, Kalihi Elementary School had been placed on “Needs Improvement” status in Hawai‘i, which matched the REL school selection criteria as a low-performing school. Unlike many of the other CDP schools chosen to participate in the Pacific CHILD project, the Hawai‘i State Superintendent allowed Honolulu District Superintendents to be involved with selecting a site. The District Superintendent agreed that Kalihi Elementary School would be a viable site, pending agreement of the school Principal and staff . Subsequently, informational meetings were held with REL staff and Kalihi Elementary School faculty. Pamphlets regarding the project were distributed to better inform the teachers when it came to making a faculty-wide decision permitting the REL to work with the school. The Kalihi Elementary School faculty met to vote on school participation. Once the school faculty agreed to become a CDP school, a formal letter was sent out by the REL to Kalihi Elementary School, as well as to the District Superintendent and State Superintendent, notifying all of the ensuing partnership.

Many of the early meetings with Kalihi Elementary School were spent discussing issues related to school improvement. The REL helped the Kalihi Elementary School faculty work on creating Standards Implementation Designs (SIDs), which, at the time, were a state mandated task. As work on school profiles and plans progressed, the school realized the need to collect assessment data to inform the SID work, as well as for planning instructional next steps. At the same time, the REL had been working on creating some new assessments and adapting other formative assessments. Kalihi Elementary School, as it happened, became the site where many of the new REL assessments were piloted, as well as some of the adapted formative assessments that were used to collect data. In addition, there were many other required assessments administered at the school at that time. At first, REL staff administered most of the assessments to K–2 students. While teachers were interested in the information gleaned from these assessments, they appeared disinterested in administering them. Consequently, at spring assessment time, REL staff administered most of the assessments, even though they had trained Kalihi teachers. This defeated the purpose of teachers administering and using the assessments to inform instructional next steps with their students.

There were several occurrences that led to the eventual withdrawal of Kalihi Elementary School. With the help of REL staff, the school applied for and was awarded the Hawai‘i Reading First grant. While this was an accomplishment for the school, there were several implications. First, in addition to the assessments that the REL required for research purposes, new assessment demands came from the grant. Second, the grant required mandatory training sessions, which conflicted with some of the on-site weekly professional development work of the Pacific CHILD project. From the point of view of both Kalihi and PREL, there was little time left on the school calendar to work with the teachers. Third, there were mounting concerns that the REL would not be able to conduct research work alongside the Reading First grant and still have fidelity of implementation. While the REL and Kalihi staff tried to align the assessments with other project expectations, this clearly took its toll on the staff. In addition, the faculty felt that not all teachers were benefiting, as REL staff provided reading information to only the lower grades. The lower grade teachers, however, felt burdened by multiple requests from the district and state, as well as the REL. While there was no formal meeting to sever the partnership between the REL and Kalihi, key faculty were dissatisfied and made requests to the school principal to “drop out” of the project. There was a final meeting between the REL director, the associate director, the school principal, and other key staff to end the services provided by the REL. A letter written by the REL director and the school principal followed, informing all concerned of the decision to officially terminate.

Since then, Kalihi Elementary School has undergone staff changes, including hiring a new school principal. Many teachers have left either to go to other schools or pursue new challenges, or for personal reasons. Kalihi’s status under the No Child Left Behind Act of 2001 (NCLB) is Corrective Action Year 1, which means the school has not met Adequate Yearly Progress for 4 consecutive years.

SAPUK ELEMENTARY SCHOOL, CHUUK, FEDERATED STATES OF MICRONESIA (FSM)
In 2001, the chief state school officer, with the approval of the school principal, selected Sapuk Elementary School in Chuuk as a CDP school. A team from the PREL Honolulu office, Department of Education (DOE) staff, and the site-based reading specialist visited Sapuk to announce the selection of Sapuk to the parents and community leaders and to share information about the research project.

Sapuk’s student enrollment is 257, with 7 teachers in the target grades of K–2. Sapuk Elementary School is one of the lowest performing schools in Chuuk. The school has two old buildings covered with marks and writing both inside and outside. At the time of our first visit, the classrooms were totally empty, with no tables or chairs for either teachers or students. There was no electricity and no telephone.

On the walls, there was no indication that the room was a classroom—there was nothing related to students’ or teachers’ work and no school schedules or reading charts in the room. There were no signs of writing or reading activities in any of the classrooms. The students were lying on the bare floor to write.

During the first year of involvement, parents did not participate in any activities with the school. The site-based reading specialist spent much of the first year conversing with the teachers and principal regarding expectations for improvement, providing early reading strategy demonstrations for teachers at the school, and sharing this with the community. The teachers eventually became hopeful that change was possible.

Improvement started with the most basic needs. At the teachers’ request, students brought brooms from home, which was the entry for community and parent involvement. But problems persisted at Sapuk. The principal was absent more often than not. The lack of interest exhibited by the principal had an impact on teacher morale and teacher attendance. Community obligations, such as funerals and church-related meetings, also had an impact on teacher attendance. In addition, all of the children walked to school and some lived a significant distance away. The school lacked instructional materials, including reading materials in the vernacular. Because the community had not been involved with the startup of the school, there was no communication system between the school and parents. Issues with teacher and student attendance made data collection inconsistent and unreliable.

These issues resulted in the decision to drop Sapuk as a CDP school, but to continue working with the teachers and community on ways to improve the conditions for learning at the school. The site-based reading specialist continued working intensively with teachers, parents, and community leaders.

One of the first solutions for the lack of reading materials was to assist teachers in writing stories on flip charts, which were then posted on the walls. Later, teachers began writing local stories to make books. Early on, community members walking down the road were stopped and invited to the school. Discussions began about the problem of school supplies and ways in which the community could assist. That chance encounter on a dirt road began the parents’ involvement at the school. In the following weeks, parents brought pandanus leaves to weave local balls with which the children could play. The news was passed to other women, and soon parents started coming to the school.

Meetings were held frequently and gradually student attendance increased. The commitment of the reading specialist encouraged both the teachers and the parents. Meetings were held among the village leaders and teachers. As a result of those meetings, community members gathered and built benches and small tables for the students. Parents began volunteering in the classrooms, which, in turn, encouraged teachers and had an impact on student and teacher attendance.

The lack of commitment on the part of the principal continued, but the impact was lessened. Five of the teachers at the school made a commitment to come to school every day in spite of the school leadership. One of the teachers took on the role of informal leader. The improvement at Sapuk came from community involvement and the belief that change is possible.

NORTH FANIF AND GAANELAY ELEMENTARY SCHOOLS, YAP, FSM

North Fanif Elementary School
North Fanif Elementary School was identified as a CDP school by the Yap DOE in January 2002. The individual hired as the site-based reading specialist was a previous Yap DOE employee who had worked with the Head Start program, but had little experience working directly in a classroom or teaching reading. After identification of North Fanif as the CDP School, the site-based reading specialist held meetings with the school’s teachers and principal. Although they had not been brought into the participation conversation earlier, they were interested in participating and welcomed the opportunity for training and on-site assistance. Professional development began immediately for the reading specialist, the teachers, and the principal.

North Fanif is a small rural school with young and inexperienced teachers, many of whom are only high school graduates. High teacher turnover and low school enrollment at North Fanif plagued the project from the outset. At the end of the 2002–2003 school year, the decision was made to pull the project out of the school.

Gaanelay Elementary School
In November 2003, Gaanelay Elementary School was selected to replace North Fanif. Gaanelay’s student enrollment was 136, with 6 teachers in the target grades of 1–3. However, since the project was already underway, it was decided that Gaanelay would be a non-research site. As a non-research site, school staff would receive ongoing assistance and assessments would be piloted, but student achievement data and professional development observation data would not be collected.

In April 2004, typhoon Sudal hit Yap. The island was heavily damaged, including the school. In the following months, life on the island revolved around repairing businesses and homes. Gaanelay Elementary School was closed for 7 months until November 2004. During that time, the site-based reading specialist suffered a heart attack and was unable to return to work until January 2005. In addition, teacher turnover occurred, with veteran teachers being replaced by young, inexperienced high school graduates. In spite of the continuing challenges on Yap, assistance has continued for the teachers and principal.

DANDAN ELEMENTARY SCHOOL, SAIPAN, CNMI
Although the Pacific CHILD project was aimed at working with low-performing schools, the CNMI Public School System (PSS) leadership asked that we take a different approach in selecting the school. At the time, two new elementary schools had just opened on Saipan, and both had inexperienced principals. Dandan Elementary was one of the two schools. The school was located in a new homestead area and enrolled students who had been attending schools in different villages. Thus, the school did not have any baseline data that designated it as a low-performing school. The decision was made for Dandan Elementary School to be involved in the Pacific CHILD project in 2001.

Dandan Elementary School was selected because it was a smaller school and had an experienced vice principal to support the new principal. Dandan Village was established as a result of the CNMI government opening up the area for homestead lots. The area continues to grow, as young families move in and their children attend the village school.

When Dandan Elementary School became part of the Pacific CHILD project, its student enrollment was 445, with 10 teachers in the target grades of K–3. Initially, support from a reading specialist was to come from Guam, but Hawai‘i-based staff helped until a site-based reading specialist was hired in July 2002.

A school improvement profile was the first project tackled, along with the administration of the REL assessments. The Dandan teachers took on the responsibility of administering all of the assessments for their students. Training was provided by REL staff on the purpose and procedures in administering each assessment, as well as sharing the results in graph form. In March 2003, the CNMI PSS decided to allow each school to choose its own reading program to go along with NCLB. Direct Instruction was selected for Dandan as the mandatory reading program. At this point, it was decided that Dandan could not continue as a Pacific CHILD CDP school because of the conflicting programs. The Commissioner of Education communicated to PREL that she was unaware that the selection of the reading program would result in Dandan Elementary being dropped as a research site. She had understood the Pacific CHILD project as supplementing the ongoing instruction and offering support to the teachers in areas in which they or their students may be weak. However, as the project is a research study, the conflict in programs meant there would be “contamination” of data, so PREL decided Dandan could not continue as a research site.

Despite not being a research site, the teachers and administrators of Dandan continued to administer some REL assessments that were not part of the Reading First assessment battery and professional development for teachers.

MACHANANAO ELEMENTARY SCHOOL, YIGO, GUAM
Machananao Elementary School participated in the Pacific CHILD project for 2 years. Guam’s Superintendent of Education was invited to select one elementary school for the project based on the set of criteria established at the beginning of the research design. The Guam DOE team selected Machananao Elementary School as the CDP school pending school staff agreement. Conversations with the principal and teacher leadership team began in May 2001. Additional presentations about the project were held at the Pacific Educational Conference in Guam in July 2001. The superintendent signed a memorandum of understanding in October 2001.

Machananao Elementary School is a K–5 school located in Yigo near Anderson Air Force Base on the north end of the island of Guam. The school was built in 1999. The school began the project with 375 students and 21 teachers. There were 271 students and 14 teachers in the target grades of K–3. The language of instruction was English for all students.

Assessment administration began in Machananao in the fall of 2001. After the pretest was given, teachers received professional development in the teaching of each pre-reading skill area. The site-based reading specialist worked with Machananao teachers on a daily basis. The professional development included small group instruction and modeling in the classroom.

At the end of the 2002–2003 academic year, the superintendent resigned. During the search for a new superintendent, the Guam Board of Education adopted Direct Instruction as the only reading program that would be permitted in Guam elementary schools. As a result, the Pacific CHILD project at Machananao Elementary School was terminated in the summer of 2003. No work continued at the school.

LESSONS LEARNED
Understanding the reasons a school drops out of a program provides insight for revising future studies to reduce attrition and increase the likelihood for generalizability of the research conclusions. Tracking issues of participation, retention, and attrition provides opportunities for the modification or redesign of studies in order to garner greater commitment and improved implementation of treatment interventions. Thoughtfully studied, attrition can be highly informative.

Competing demands on schools for improved student achievement had an impact on the research participation. At three of the original research sites, required statewide initiatives that conflicted with the REL initiative were implemented at the schools. When resolution to the issues could not be found, schools were dropped from the intensive research program. In a couple of cases, chief state school officers designated schools as research participants prior to ensuring staff support for the project. The differing perceptions and resentment that resulted from the required participation presented obstacles to communication and buy in for professional development activities. After the first few years, the schools were dropped from the project.

Other factors that influence the retention and attrition of study participants are: (a) funding source requirements and/or restrictions that can result in loss of control of the original project; (b) staff in low-performing schools who may be motivated to participate in a study because of availability of technical assistance, with research requirements a secondary consideration; (c) impact of school leadership; (d) communication among research staff, and between research staff and school staff; (e) impact of the school community; and (f) accounting for cultural norms and expectations versus research expectations.

Case studies, such as those described in this report, offer insight into the methodological challenges of recruitment, retention, and attrition of study participants that can inform future educational field-based research and development efforts.

Copyright 2005 by Pacific Resources for Education and Learning.