Research Into Practice

2005

Challenges to the Planning and Evaluation of Educational Programs in the Former U.S. Trust Territories of Micronesia

By
Dr. Mary B. Church
Program Evaluation Specialist
PREL's PRELStar Program

The purpose of this paper is to discuss challenges associated with the planning and evaluation of educational programs in less developed nations, including the former U.S. Trust Territories of Micronesia. I begin with a discussion of the geographic, linguistic, and economic diversity of Micronesia and its associated educational contexts. This is followed by a description of some major difficulties inherent to educational planning and evaluation that occur, organized in relation to a model of the planning and evaluation process. Recommendations to successfully meet the challenges associated with each stage of the planning and evaluation model are also presented.

CONTEXT

The U.S.-affiliated region of Micronesia includes five political entities: the territory of Guam, the Commonwealth of the Northern Mariana Islands (CNMI), the Republic of the Marshall Islands, the Republic of Palau, and the Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap) (see Figure 1). Except for Guam, which has been a U.S. territory since 1898, each nation is a former U.S. protectorate member of the United Nations Trust Territory of the Pacific Islands, established consequent to World War II.

FIGURE 1
Map of the Pacific Region

These nations are culturally, geographically, linguistically, economically, and educationally diverse. Geographi-cally, the region is spread over 3 million square miles of ocean containing 2,100 islands, which, together, comprise 923 square miles of land. Approximately 30 indigenous languages exist as primary languages in the region, with the second language of English used as the official language for government and commerce. With the exception of the CNMI, the economies of the former U.S. Trust Territories are largely underdeveloped, with substantial portions of the population engaged in subsistence labor. Public administration and education tend to be the largest industries in the region, accounting for up to one-half of the labor force, with subsistence agriculture and fishing accounting for a large portion of non-wage earnings. Per capita gross domestic product figures range from a low of US $1,680 (Marshall Islands) to a high of US $10,500 (CNMI) (Central Intelligence Agency, 1996).

Each of the former U.S. Trust Territories is currently assuming independent responsibility for the political, economic, and social services previously administered by the U.S. The organization that I work for—Pacific Resources for Education and Learning (PREL)—is a not-for-profit organization that assists in promoting the educational services of the region through collaboration in staff development, curriculum development, student assessment, applied research, and program planning and evaluation.

EDUCATIONAL PROGRAM PLANNING AND EVALUATION

A story described by Benjamin Franklin (Brislin & Segall, 1975) sets the stage for a discussion of international educational programming:

At the treaty of Lancaster, in Pennsylvania, anno 1744, between the government of Virginia and the Six Nations, the commissioners from Virginia acquainted the Indians, by a speech, that there was at Williamsburg a college with a fund for educating Indian youth and that, if the chiefs of the Six Nations would send down half a dozen of their sons to that college, the government would take care that they be well provided for and instructed in all the learning of the white people. The Indian’s spokesperson replied: “We are convinced that you mean to do us good by your proposal, and we thank you heartily. But, you, who are wise, must know that different nations have different conceptions of things, and you will not therefore take it amiss if our ideas of this kind of education happen not to be the same with yours. We have had some experiences of it. Several of our young people were formerly brought up at the colleges of the Northern provinces. They were instructed in all your sciences. But, when they came back to us, they were bad runners; ignorant of every means of living in the woods; unable to bear either cold or hunger; knew neither how to build a cabin, take a deer, nor kill an enemy; spoke our language imperfectly, were therefore neither fit for hunters, warriors, nor counsellors; they were totally good for nothing. We are, however, not the less obligated by your kind offer, though we decline accepting it, and, to show our grateful sense of it, if the gentlemen of Virginia will send us a dozen of their sons, we will take care of their education, instruct them in all we know, and make men of them. (p. 57)

Because Micronesia was a protectorate of the U.S., its system of education has been largely influenced by the U.S. government. Soon after World War II, the U.S. government provided assistance to encourage Micronesian communities to develop primary schools in major settlement areas, which were to be supported by and responsible to the local culture. In the 1960s the federal investment in the region increased enormously, and the previously autonomous educational system was transformed to resemble a U.S. curriculum, with universal education, academic orientation, and the use of English as a language of instruction—all provided largely by American teachers. This “Americanization” of the educational system removed control from local authority and disengaged schools from the community. According to Conklin (1984), “Micronesians now came to think of education as a responsibility of the distant American authorities, neither structured to their needs nor requiring their cooperation” (p. 10).

With the dissolution of the Trust Territory agreements in the last few decades and the corresponding political independence of the Micronesian nations, control of the educational system has now reverted back to local rule. The vast majority of teachers and administrators are now Micronesians. While this autonomy has moderated the Americanization of the schools, the current public school curriculum remains largely based on the California school system, with little adaptation to the regional context.

Relative to the U.S., the educational achievement of students in the region is characterized by low levels of attainment and performance. While education to grade 8 is compulsory, financial constraints in most entities allow only approximately one-half of students to continue to high school, and only about one-half of these students to gain a diploma.

Education remains one of the largest industries in Micronesia. Program planning and evaluation is centered on large federal grants provided through the U.S. Department of Education, including the Freely-Associated States Educational Grant Program, Special Education, Pacific Vocational Education Improvement Program, and School-to-Work.

PROGRAM PLANNING AND EVALUATION MODEL

Factors that inhibit the planning and evaluation of educational programs in the Micronesian nations served by PREL are presented within the context of a model of the nine major stages of program development and evaluation (see Figure 2).

In the first stage, the awareness of a problem, issue, or concern occurs. This is followed by the conceptualization and selection of potential alternative solutions to the problem. The objectives that define the realization of the solution are formulated and detailed, and the program is then planned to meet these specified objectives. Based upon the program plan, an evaluation is designed to monitor program implementation and effectiveness. The evaluation is then conducted and the findings reported. Finally, the results of the evaluation are used in continuous feedback to inform further iterations of the planning and evaluation cycle.

In this model, evaluation is seen as inextricably connected to program planning. Each stage in the process of program planning and evaluation is viewed as a link in the chain of effective program planning, evaluation, and improvement. Stages in the cycle insufficiently concluded negatively impact the potential of each subsequent stage in the process.

CHALLENGES TO PROGRAM PLANNING AND EVALUATION

This section describes impediments to program planning and evaluation in the American-affiliated Pacific by reference to each of the model’s stages. Recommendations to avert each of these potential obstacles are then presented.

Define Problem

One of the most important determinants in defining problems is cultural beliefs and values. Cultural norms determine what is defined as a problem, and what is, as a consequence, worthy of the investment of limited human and material resources. If something is defined as a problem by an outside culture, but not by those within, it is unlikely that the intrinsic motivation necessary to initiate and sustain ameliorative efforts will exist. Of course, extrinsic motivators can be applied and adopted by the cultural group, such as power, prestige, and money, but once these features are removed, so goes the motivation to change.

In much of the American affiliates, local and traditional village leadership is separate from that existing in the government, which carries out the interests of U.S.-based and funded education. For this reason, the community-at-large is often a stranger to the educational system and little parental and other community-based involvement is present. Cultural patterns include a strong sense of community, and community values tend to be preeminent. Many parents do not recognize the need or value of the Western educational system for their children and some discourage their children from attending school if other more traditional and/or immediate needs for their time exist (e.g., agricultural assistance, funerals). In fact, the educational systems that have been transplanted to this region have in themselves been seen as problematic, resulting in family separation, a disturbance of traditional authority, and increased urbanization (Conklin, 1984).

The values existing among the cultures of the Pacific region are also characterized by strong interpersonal relationships and styles of interacting that avoid confrontation. These communication styles derive from cooperative values and result in the avoidance of disagreement. As Conklin (1984) states, “putting oneself forward, directly challenging or demeaning another, and disturbances of consensus are generally negatively valued” in Micronesian cultures (p. 19). Respect for and a deference to authority are also characteristic of the cultures of this region, making the admission of problems difficult, since doing so may be perceived as criticism of an authority.

The cultures of the Pacific also tend toward a different sense of time than that experienced in the West. The urgency associated with daily living occurring in much of the industrialized world is not common here. The director of one of the departments of education in the region described this slower pace as resulting from a “breadfruit culture” in which abundant life resources (e.g., food, water) and a warm climate are conducive to leisure.

A final impediment to educational program planning in the U.S.-affiliated Pacific is the presence of few highly trained local personnel. For example, in one of the entities only a single indigenous PhD resides.

Recommendations. Recommendations for improving this primary and all-important stage in the planning and evaluation process include involving and training host country people in the process of problem definition. Circumstances viewed as problematic within the existing community should then be prioritized and selected for further program planning and evaluation. A respect for and accommodation of cultural differences in interaction styles is also recommended.

Define Solution

Once an educational problem is considered to exist, one moves to the second part of the process of program planning and evaluation: solution definition.

Again, cultural characteristics influence this process. One of the detriments to defining a solution to a problem is, again, the norms for interpersonal interaction existent in cultures of the Pacific. Because communication styles tend to foster cooperation and deference to leaders, rather than conflict, individuals may not be prone to proffer solutions to problems, though they may recognize them.

Insufficient program planning is also a detriment to generating solutions to problems. If not enough time or human resource is devoted to this process, it is unlikely that a sufficient solution will be discovered. If no problem is recognized, no solutions will be generated.

Again, the absence of trained local personnel makes the process of problem solution difficult. Expertise is essential to solving many of the major and multifaceted problems that exist in the educational sector.

Recommendations. To engender solutions sufficient to solve or lessen existing educational problems, host country people need to be involved. After all, these persons are the service providers and beneficiaries of the major educational programs involved, responsible for engineering, implementing, evaluating, and, ultimately, sustaining them. Those having the greatest expertise in the field of interest should be convened for a sufficient period of time to allow adequate levels of consideration and solution generation.

Define Objectives

The development of measurable objectives for a program is the next step in the program planning and evaluation process. Again, an understanding of cultural features is critical to the thorough accomplishment of this stage. Standards and widely accepted measures of effectiveness are not always relevant to developing nations. Outside researchers who lack cultural awareness regarding what is important to program stakeholders will likely fail to provide measures and resulting findings of interest and use to others. If management concerns are missed in the development of specific program objectives, little interest and use by this group in the evaluation process and outcome will result (Bamberger, 1989; Binnendijk, 1989).

Other common problems associated with this stage include the formulation of research questions. First, program goals are often too vague to be operationalized. Insufficient clarity of objectives often results in insufficient clarity of program planning, and nonspecific goals are less likely to be realized. Outcome targets that are overly optimistic, and therefore unrealizable, but to which the program is held accountable, are also problematic. There is also the omnipresent “outputs as goals” problem, in which program activities and service provisions are considered objectives, in place of measurable program outcomes.

Since objectives are by definition measurable, it is necessary to decide at this stage how such outcomes will be observed. A common problem associated with the observation of program objectives is the costs incurred by data collection. Many of the intended impacts of educational programs require routine and large-scale data collection efforts, uncommon in many regions of the Pacific. For this reason, more immediate and smaller-scale educational outcomes may have to be considered for purposes of evaluating program effects.

Recommendations. To ensure that program objectives are relevant to local stakeholders, it is imperative to involve host country people in their selection and definition. To this end, evaluation teams should include local consultants able to contribute the perspective of the larger community of program beneficiaries, providers, and decision makers.

As Binnendijk (1989) notes, an evaluation’s focus should be directed at intended users, including management, to meet their specific informational questions and needs. Program objectives should therefore be relevant to others, in addition to the agency performing the evaluation. This is accomplished by first defining who program stakeholders are, and then by identifying their needs. These multiple needs, once discovered, will then need to be prioritized. In addition, provision should be made for changing program objectives over time.

Plan Program

The fourth stage in the planning and evaluation cycle involves the process of developing step-by-step program activity plans. These activities should be directly related to the objectives considered in the preceding stage. Again, this process can be impeded by cultural aspects, including nonconfrontational styles of interaction. As previously noted, this style of interaction, while supporting and maintaining interpersonal cooperation and harmony, may result in deficits in program planning when valuable opinions of group members are not expressed and, therefore, not considered by the larger group of planners.

In addition, when no sense of urgency in problem solution is experienced, plans to carry out complex activities may tend to be tentative or otherwise not motivated by time constraints.

A further problem with educational program planning in less developed countries in the Pacific region involves the acceptance of educational systems, programs, and innovations exported from the United States. Often these educational programs have been imported without regard to the countries’ capacities to absorb, modify, or sustain them. This is, in part, due to a lack of expertise on the part of program recipients necessary to the development of their own programs, and partly to financial considerations when program funding is based on implementation integrity. As a consequence of this ready acceptance of prepackaged educational programs, long-term planning has not been a common feature of educational programming. When program planning does occur, it is often not comprehensive or carried out.

Recommendations. To remediate these impediments to program planning, it is recommended that a greater number of host country persons gain expertise specific to educational administration and curriculum development. In the short term, others in the educational community should be provided with training on basic program planning and implementation strategies. Together, these persons could lead an effort to develop programs that are relevant and responsive to the educational needs of the society in which they are found. At the very least, host country people should be involved at every level of the program planning and decision-making process.

Design Evaluation

The next phase in program planning and evaluation is evaluation design. One major factor contributing to ineffective evaluation design involves the use of culturally biased research tools. Some research methodologies are more relevant to the cultural characteristics of a site than others. For example, survey instruments with unknown and, therefore, ambiguous response formats may result in inaccurate responses from participants. Surveys themselves may be inappropriate modes of question asking within a particular cultural context. For example, in Micronesia direct question asking is not always appropriate. Verbal exchanges of question asking and answering are often done indirectly, making the development of standardized question asking and response interpretations ambiguous and perhaps impossible. It is also critical that the language in which questions are drafted and presented be in the primary language of respondents.

Survey responses may be biased if the respondent does not understand or have confidence in the expressed use of their answers. A positive response bias may occur when respondents have a desire to please the interviewer or seek to avoid giving a negative evaluation because of loyalty to the organization referenced by the survey.

In addition, some research methodologies may not be possible, as is often the case for the preeminent method for causal analysis in Western cultures—experimental design. An overemphasis on quantification, including the use of sophisticated research methodologies and statistical analyses, is problematic. First, the application of experimental or quasi-experimental designs in the field is often impractical, and the findings from such research are nongeneralizable and of limited interest to program managers. Sophisticated research designs imposed from the outside are often viewed as being irrelevant to program personnel. In addition, such designs require a reliance on the specialized skills of external evaluators, and may impede the continuation of evaluation efforts. A reliance on quantification may also result in the neglect of other research methods that could provide effective data gathering and important information.

Because data archives are often incomplete, out of date, unreliable, or unavailable, quantitative data must be measured directly. In the field of education, one of the most relevant measures involves academic performance. Unfortunately, many of the performance tests used have questionable validity and reliability, and an absence of norms.

In planning for data collection, evaluators should also consider impediments associated with sampling. Sampling frames are often unavailable, making the selection of representative samples difficult. In addition, the cost of data collection restricts sampling, especially where large-scale survey or interview methods are necessary or most appropriate. Time and financial constraints disallow casting a wide net, especially to outer islands within the region.

A further obstacle to proficient evaluation planning is the negative attitude that program managers often have regarding evaluation. Evaluation is seen as an auditing or surveillance function, rather than as an activity meant to provide information useful to program personnel. For this reason, the interest and cooperation of program managers in the evaluation process may be limited.

Recommendations. To overcome some of the difficulties inherent to designing appropriate evaluations in international settings, it is recommended that a variety of research strategies and information resources be used (Binnendijk, 1989). Examples of these techniques include review of documentation, key informant interviews, observation, group interviews (Van Sant, 1989), small sample surveys, proxy indicators (Binnendijk, 1989), and beneficiary assessments (Salmen, 1989). In addition, intrusive or inappropriate methods of data collection should be avoided by soliciting the advice of local field personnel and program beneficiaries (Lawrence, 1989). To reduce problems associated with inaccurate or incomplete translation of questions and responses, back translation techniques should be applied.

To maximize program personnel interest and cooperation in the evaluation process, evaluation teams consisting of host country experts and personnel should collaborate in the design of the evaluation. Host country involvement in evaluation planning is essential to understanding contextual and policy factors, as well as the local language, and enhances institutional capacity to conduct future evaluations (Binnendijk, 1989).

Implement Program

The implementation of programs can be impeded by a number of factors. First, foreign experts insensitive to cultural considerations may execute programs in culturally inappropriate or irrelevant ways.

Once programs are initiated, poor management can result in program ineffectiveness or discontinuance. A failure to instill or maintain mechanisms for accountability can also lead to poor program implementation and/or insufficient program delivery. In addition, the failure to hire or develop local exper-tise for program delivery may promote program discontinuity.

Recommendations. Host country persons should be involved in decisions and activities related to the implementation of programs to promote the likelihood of their success. Training should be provided to foreign experts to increase their awareness of the cultural context in which implementation occurs and to local personnel to improve program effectiveness. In addition, oversight should be provided to ensure that program implementation, management, and delivery are conducted as intended.

Conduct Evaluation

Once the program is planned and implemented, evaluation activities are conducted. Hindrances to evaluation include the language in which the evaluation is made. Even when a common language is used among evaluation personnel, data collection is often made by others. If checks are not made to ensure that program personnel are collecting the data as intended, data interpretation and analysis may be ambiguous.

There is a common misperception of evaluation as being simply a monitoring function, in which the review of project activities, expenditures, and schedules are overseen for purposes of accountability. This has often resulted in the belief among local personnel that evaluation is a policing activity rather than one that offers information to be applied in improving program effectiveness and efficiency. This negative image of evaluation is intensified by evaluator requests for the time and resources of program staff.

Finally, everything takes longer than expected in the planning and evaluation cycle in less developed countries, as compared to the U.S. Many complications in data collection should be expected. Because the pace of work in Micronesia is less hurried than in the U.S., additional time for every aspect of the process of evaluation should be planned for. If it is not, the evaluation may be hastily completed, incomplete, and/or too expensive.

Recommendations. To improve the likelihood of evaluation success, the following recommendations are offered. First, local personnel should be involved in the evaluation. Preferably, these local experts would be part of an evaluation team established to gather information. A greater participation of local personnel in the process of conducting the evaluation should promote positive attitudes toward the purpose and usefulness of evaluation activities, and discourage the perception of evaluation as audit (Lawrence, 1989). The data collectors thus assembled should be overseen to ensure the integrity of data collection, including the language of use in the process of data gathering. These activities should serve to strengthen the capacity of host country personnel to independently carry out future evaluations.

The information gathered during the course of the evaluation should include program implementation, so that the intended delivery of each stage in the process of service coordination and provision can be confirmed. Finally, additional time to conduct the evaluation should be built in to cost estimates to minimize the potential for insufficient evaluation due to funding or time shortages.

Report Findings

Once the evaluation is completed, the analysis and reporting of findings are carried out. Avoiding hindrances to this stage in the program planning and evaluation cycle is vital to the acceptance and use of findings. Common impediments to successful reporting are, again, related to the cultural and political features of the context in which programs are established. First, an understanding of the styles of interpersonal interaction is important. If those reporting on evaluation findings do not understand appropriate means of relaying potentially critical and/or sensitive information, the information thus provided may be rejected or ignored. In addition, if foreign experts are the only ones involved in the provision of program feedback, it is much easier for the host country to blame a cultural incongruity for the lack of positive evaluation findings.

Recommendations. To reconcile these potential obstacles to effective reporting, persons from the host country should be involved in the reporting process, both in written and oral presentations. This will assist the appropriate and effective communication of findings.

Use Findings

The final stage in the planning and evaluation process is also of critical importance. The utilization of findings is essential to the process of program efficiency and improvement. If such findings are not applied to the betterment of program servicing, then the investments made throughout the process of program planning and evaluation are largely futile.

Many of the obstacles to the use of evaluation findings are common to the previous stages. Again, the interaction styles of those responsible for program decision making need to be considered. It may be too much to encourage and expect program administrators to use findings critical of segments of the program in an effective or expedient manner. This is especially true when individuals responsible for ineffective program provision are valued members of the society, either through ascribed familial, village, or other high-status positions. In addition, if program personnel are assigned program responsibilities because of their status in relation to program decision makers, it is unlikely that the motivation for change, on the part of either group, will be present, nor the results useful. Further, the lower sense of urgency and time constraints associated with many cultures of the Pacific make the expeditious use of information less likely.

The misuse of findings is possible when members of the community critical of the program are known to program personnel or other members of the society. As mentioned earlier, one of the most common features of interpersonal interaction in the Pacific is deference to authority and cooperation. Persons who criticize authority are therefore open to censure from the community if the public expression of their critiques is discovered.

Finally, evaluation results may not be used because of a lack of interest in them. As previously mentioned, evaluations are often conducted due to a mandate that requires it as a contingency for the receipt of federal funds. Because the motivation to conduct evaluation is not derived from an intrinsic interest in program planning and improvement, but as a result of the need to satisfy donors, the entire process of evaluation is viewed as a mechanism for accountability. As a result, the process is viewed as providing oversight rather than insight, and as necessary rather than desirable.

Recommendations. To mitigate the potential for ineffective use of findings, the following recommendations are offered. First, personnel from the host country should be a central source of information regarding what changes in program structure and personnel are realizable. Those aspects of the program that are both vital and amenable to change should be targeted in efforts to use the findings. As Binnendijk (1989) notes, the use of host country evaluation teams promotes the development of local evaluation understanding and capacity, maximizes the benefits of their contextual expertise, and furthers the use of evaluation findings and recommendations.

The misuse of findings is also a potential problem, especially when the anonimity of those providing information critical of the program cannot be ensured. Safeguards to the misuse of information should be provided to the highest extent possible, to protect the identity of individuals who could be harmed for their participation. Only essential identifying information should be provided in reporting.

To increase the intrinsic interest in evaluation findings among program planners and decision makers, it is advisable to communicate the validity and reliability of the data gathered. Once this is accomplished, the potential use of the information should be presented in sufficient detail to provide step-by-step direction.

As with most social programs, change is incremental and often at the perimeter (Shadish, Cook, & Leviton, 1995), if it comes at all. So it is with educational programming in Micronesia. Because the pace of change in much of the developing world tends to be less rapid than in the U.S., the wait for systematic program planning, and the acceptance and application of evaluation findings, is likely to be longer. Recognition of this makes our task more challenging, frustrating, and sometimes fraught with existential dilemma (e.g., “What am I doing with my life?”). Still, if done correctly, what we do is important because it hastens to some extent the process of social improvement. It is hoped that the lessons described herein serve to inform and expedite this process.

REFERENCES

Bamberger, M. (1989). The monitoring and evaluation of public sector programs in Asia: Why are development programs monitored but not evaluated? Evaluation Review, 13(3), 223–242.

Binnendijk, A. L. (1989). Donor agency experience with the monitoring and evaluation of development projects. Evaluation Review, 13(3), 206–222.

Brislin, R. W., & Segall, M. H. (1975). Cross-cultural research: The role of culture in understanding human behavior. New York: Learning Resources.

Central Intelligence Agency. (1996). The world factbook. Washington, DC: U.S. Government Printing Office.

Conklin, N. F. (1984). Culture and education in Micronesia. Unpublished manuscript.

Lawrence, J. E. S. (1989). Engaging recipients in development evaluation: The “stakeholder approach.” Evaluation Review, 13(3), 243–256.

Salmen, L. F. (1989). Beneficiary assessment: Improving the design and implementation of development projects. Evaluation Review, 13(3), 273–291.

Shadish, W. R., Cook, T. D., & Leviton, L. C. (1995). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.

Van Sant, J. (1989). Qualitative analysis in development evaluations. Evaluation Review, 13(3), 257–272.

Article will also appear in Volume 10 of The Micronesian Educator. Copyright 2005 by Pacific Resources for Education and Learning