2014/4, стр. 428 - 441

USING THE RESULTS OF A NATIONAL ASSESSMENT OF EDUCATIONAL ACHIEVEMENT

Резюме:

Ключови думи:

Chapter 4

Translating Assessment Findings Into Policy And Action

Although the primary purpose of a system of national assessment is to describe students’ learning, its role is not limited to description. To justify the effort and expenditure involved, the information that an assessment provides about the achievements of students, their strengths and weaknesses, and how they are distributed in the population (for example, by gender or location) should be useful in informing policy and making decisions (related, for example, to resource allocation). A national assessment is also envisaged as providing information to curriculum developers, textbook writers, teacher trainers, and the public. In this view, an assessment should provide more than information; following dissemination, it should become a lever of reform.

This chapter outlines five topics that are relevant when considering how assessment information might be translated into policy and action: (a) institutional capacity to absorb and use information, (b) trustworthiness and relevance of the information provided by the assessment, (c) procedures to identify appropriate policy or action following an assessment, (d) determination of a systemwide or targeted intervention, and (e) complexity of policy formation and decision making. The focus is primarily on institutional arrangements in the education system. Later chapters address more specifi c uses of a national assessment for policy and educational management (chapter 5) and for teaching (chapter 6).

Institutional Capacity to Absorb and Use Information

A number of conditions relating to institutional capacity should be met if optimal use is to be made of the findings of a national assessment (Kellaghan and Greaney Тhe text started in issue 2/2014 of “Strategies for Policy in Science and Education” 2004; Postlethwaite 1987; Vegas and Petrow 2008). First, political will, openness to new information, and readiness to consider change or reform on the part of policy makers are particularly important. Second, policy and decision makers in the ministry of education should have the capacity (knowledgeand skills) to interpret and use information from a national assessment. Considerable investment may be required to develop this capacity, particularly in countries in which national assessment activity is new. Third, a national assessment should not be seen as an isolated activity. Rather, it should be integrated into existing structures, policymaking and decision-making processes, and channels of resource allocation.

Fourth, the national assessment team should ensure that systems and strategies are in place to communicate its findings to institutions and agents who will have a role in implementing policy (for example local administration structures; supervisors, inspectors, and advisers; curriculum development authorities; schools; and teachers). Fifth, after their capacity has been developed, the continuity of staff members in organizing and undertaking assessments and in interpreting fi ndings must be ensured. Turnover of offi cials and technical experts, which is a common occurrence in many developing countries, will have a negative impact on capacity. Finally, the support and commitment of all stakeholders is required. Hence, the purpose, findings, and implications of the national assessment should be clearly communicated, and the concerns of stakeholders who may feel threatened by policy or decisions should be addressed.

Translating Assessment Findings Into Policy and Action Trustworthiness and Relevance of Information Provided by an Assessment

Before considering what decisions they might base on the findings of a national assessment, decision makers should satisfy themselves that the information provided by the assessment is trustworthy and realistic. This conclusion will be warranted (a) if the national assessment team is technically competent, (b) if the knowledge and skills of students that the test was designed to measure are adequately represented in the assessment instrument, (c) if proper procedures have been followed in the collection and analysis of data, and (d) if student learning is described in appropriate detail and in a way that is sensitive to the needs and technical expertise of users (chapter 1; see also Postlethwaite 2004a: chapter 5).

Policy makers and managers require information that addresses their issues of concern and that provides a basis for their policy formation, planning, and decision making. Many examples exist of national assessments failing to meet this need. For example, if the test used in an assessment is too difficult and fails to discriminate at lower levels of achievement, the assessment will not provide the information that a senior ministry offi cial needs to make decisions about resource allocation for lowachieving students. Assessment results that provide only mean scores or a ranking of geographic regions based on such scores without an analysis of the achievement that the scores represent will not provide sufficiently detailed information for curriculum developers and teacher trainers. Finally, assessment reports that take three or four years to complete are unlikely to be of much interest to policy makers because the fi ndings may no longer be relevant to the issues or to the personnel who prompted the assessment.

Procedures to Identify Appropriate Policy or Action Following an Assessment

In some cases, the fi ndings of a national assessment will point directly to action needed to address an identifi ed problem. More often than not, however, the action required will not be obvious. The assessmentwill identify problems, but the information it provides will not generate a solution or select among alternative courses of action. For example, the assessment may provide evidence that students perform poorly in some achievement domains, or it may identify relationships between achievement and background factors, indicating that girls’ achievement is inferior to boys’ achievement or that students in small rural schools perform less well than students in urban schools. However, the identifi cation of factors related to student achievement does not provide any obvious explanation why the genders or students in different locations should differ in their achievements. Many national assessments address this issue to some extent when they collect additional data on the conditions in which learning takes place. The results of analyses in which these conditions are related to student achievement, although not identifying specifi c courses of action, can lead to greater understanding of the factors that affect outcomes and provide a basis for policy formation and decisions (Blalock 1999).

Apart from recognizing the complexity of reaching decisions following an assessment, policy and decision makers have to be cognizant of the feasibility and cost-effectiveness of follow-on initiatives or reforms. Clearly, priority should be given to the identification of inputs that are likely to make a difference to learning. Thus, while some inputs (for example, an elaborate school facility) might be highly attractive, they might not be more effective than more modest inputs. The impact of the cost of an intervention on the education budget is also relevant. Although students might learn better in very small classes, the cost of implementing a smallclass strategy in developing countries may be too high to be considered a viable option (Lockheed and Verspoor 1991).

This chapter identifi es four (not mutually exclusive) procedures in which assessment fi ndings can play a role in suggesting appropriate policy or action following an assessment.

Discussing the Findings

The most important activity following a national assessment is involving stakeholders in a discussion and interpretation of its findings to try to tease out its implications, to suggest causal relationships, and to propose approaches for addressing identified problems. A variety of sources of information and expertise and many stakeholders have a role to play in interpreting the findings and searching for solutions. The contribution of individuals who are in close contact with the day-to-day operation of schools (teachers, inspectors, supervisors, or advisers) will be particularly important. Many countries hold seminars and workshops following a national assessment to allow a variety of viewpoints to be heard. Other, more formal approaches to reviewing fi ndings (for example, in the deliberations of a national commission or in devising a national sector strategy) are also used (see chapter 5).

Letting the Findings “Enlighten” the Policy-Making Process Use of the fi ndings of other policy-oriented educational research suggests that the impact of national assessment fi ndings would be conceptual rather than instrumental. Thus, fi ndings would enter the policy arena, not through the direct application of results, but in the way of generalizations, orientations, and general guidance that “enlighten” the policy-making process, shaping the way that people think about issues and informing general debate and discussion (Husén 1984; Weiss 1979). In this view, research use is a gradual and diffuse process, providing concepts and theoretical perspectives that permeate the policymaking process, affecting understanding of educational issues, and suggesting appropriate solutions to problems.

“Enlightenment” should not be restricted to policy and decision makers. Making public the results of a national assessment can raise consciousness not just among policy makers and managers but also among citizens in general. Raising public awareness, in turn, can serve to underline the important role that education plays in meeting national goals and harness public opinion to support efforts to improve provision.

Although the advantages of enlightenment may be obvious, using research fi ndings in this way can have disadvantages. First, the process is open to oversimplification and distortion. Second, poor research may attract as much attention as good research. Third, some important research fi ndings may never reach policy or decision makers. Although use of national assessment fi ndings to enlighten stakeholders (including the public) about the state of the education system is to be encouraged, more is required. The ministry of education and other key stakeholders should study carefully the fi ndings of a national assessment so that they can devise specific policies and actions to address identifi ed defi ciencies.

Responding to Specific Problems Identified in a National

Assessment

In some national assessments, the action required to address specifi c problems will be obvious. For example, if schools are found not to have resources specifi ed in ministry regulations (for example, textbooks), steps will need to be taken to provide those resources. If teachers’ knowledge of subject matter is inadequate, in-service courses targeted on identified weaknesses would seem the most obvious way to remedy the situation.

Referring to Other Research Findings

A consideration of other research fi ndings can help strengthen the basis for drawing inferences regarding causal relationships that a national assessment might suggest, providing a more secure basis for policy formation. These fi ndings could arise from a variety of studies, including studies of school and teacher effectiveness; studies of the relative infl uence of school and home factors on student learning; and studies of classroom processes, of class size, and of the effects of grade repetition. In suggesting how national assessment fi ndings might be used to improve classroom teaching and learning, chapter 6 looks to the research literature—in particular to studies of school and teacher effectiveness—for guidance.

Caution is indicated in using the findings of research to support inferences drawn from national assessment data. Studies may not be technically adequate, or their fi ndings may not be relevant to the context in which the national assessment was carried out. Potential problems will be magnified if the research was carried out in other countries. In particular, if those countries are economically developed, fi ndings may not be relevant to a developing country. For example, the conclusion that small class sizes are associated with superior student achievement tends to be based on U.S. studies in which a class had 20 or fewer students (see, for example, Finn and Achilles 1990). Classes of this size are rare in developing countries.

Determination of a Systemwide or Targeted Intervention

An important distinction in determining action is whether it will be systemwide or targeted on particular subpopulations or sectors of the education system. Systemwide action is intended to improve the performance of all students, and it includes improved instructional techniques, curriculum reform, and textbook reform. A teachertraining initiative could be systemwide or targeted. Other targeted interventions include a variety of provisions. First, an intervention may provide additional resources for students with special learning needs (for example, early prevention programs or remedial reading programs). Second, an intervention may involve various actions and policies, such as removing or alleviating pedagogical obstacles to performance (for example, lack of learning materials or poor-performing teachers) and economic barriers (for example, school fees or forgone labor). These actions may be designed to directly affect the performance of students in subgroups of the population, such as students from lower socioeconomic backgrounds, children displaced by civil unrest, orphans, linguistic minorities, students in small rural schools, or girls (in some societies). Third, an intervention may be designed to indirectly affect the cognitive performance of students (for example, parent involvement programs or nutritional and school meal programs) (see Willms 2006; World Bank 2004). Targeted interventions often require schools to develop and submit a plan that describes how they propose to use additional resources to improve student learning.

Complexity of Policy Formation and Decision Making

Chapter 1 noted that a range of political factors can play a role in determining the form of a national assessment. This chapter now addresses the complexity of policy and decision making following an assessment in the political environment outlined in chapter 1.

Policy making is not a simple linear process in which a problem is identifi ed, possible solutions are considered, and implementation strategies are devised. Rather, it involves complex political activities, including negotiation, compromise, and responses to pressures and lobbies from a wide range of sources that must fi t into preexisting belief patterns and value systems (see Blalock 1999). In reaching a decision, a policy maker may find the results of an assessment of value, but he or she will also need to take into account numerous political and other considerations on both national and local levels. Such considerations include politicians’ views (ideological positions or concern about voter response and electability), budget availability, views and interests of stakeholders and interest groups, traditional views, and current fashions.

The relationship between a national assessment and political factors has positive aspects. For example, political forces can sometimes be harnessed to support the use of assessment findings. In particular, support for policy and decisions based on assessment findings will be enhanced if key stakeholders have been actively involved in the assessment from the early design stage. Certainly, at all stages, stakeholders should be kept in touch with relevant aspects of policy formation and decision making to ensure that they understand the need and rationale for reform. Representing stakeholders’ interests on a steering committee that supervises the implementation of the national assessment can achieve this end (see Greaney and Kellaghan 2008, volume 1 in this series). Policy makers, in seeking to enlist support for reform, may also need to invoke principles widely accepted throughout the community, such as the provision of equal opportunities for all students, the importance of ensuring that the quality of student learning justifi es the expenditure involved, and the need to provide a firm basis for the development of both individual students and the national economy.

The close connection between a national assessment and the political arena, although holding the prospect of improving policy formation and decision making, is not without its dangers. If an assessment becomes embroiled in political confl ict, it is unlikely to play a major role in improving student learning. In Argentina, reflecting tensions between central and provincial authorities regarding their spheres of influence, information from assessments was used primarily to legitimate national policies and reforms and to regulate provincial authorities, rather than, for example, to design compensatory policies (Benveniste 2002; Gvirtz and Larripa 2004). A somewhat similar situation arose in the Arab Republic of Egypt, where local directorates of education refused to cooperate in a nationally designed assessment because they perceived it to refl ect a hidden agenda to increase the control of central government (Carroll 1996).

The situation in Uruguay was different but also involved a confl ict between powerful stakeholders. Fearful that the results of a national assessment would be used for teacher accountability, teachers’ unionsrefused to cooperate until agreement was reached that reports on school performance would not be published, that the infl uence of student background on achievement would be given due recognition, and that teachers would not be held directly accountable for student performance (Benveniste 2002; Ravela 2005). This confl ict was resolved through negotiation. Results were not used to hold teachers accountable, and Uruguay provides some of the more interesting examples of the use of national assessment fi ndings to improve student learning (see chapter 6).

Conclusion

This chapter’s description of the translation of findings into policy and action following a national assessment has identifi ed two major issues. The first relates to the complexity of policy and decision making, the institutional capacity to absorb and use information, and the need to take account of a variety of vested interests in the process. The second major issue relates to the evidence used to interpret the findings of an assessment and to reach a conclusion about the most appropriate ways to proceed in designing policies or interventions that will address problems identified in the assessment with the objective of improving student learning.

One could argue that the basis for decision making would be strengthened if strategies to improve student learning suggested by national assessment fi ndings were evaluated in an experimental or quasi-experimental study (see chapter 8). For several reasons, however, including time constraints, cost, and availability of personnel with the required technical skills, this course of action is unlikely. In most cases, policy and decision makers will, at best, rely on dissemination of fi ndings to promote conceptualization and greater understanding of issues, discussion of findings by stakeholders, and consideration of relevant research, even though research may not have originated in the country in which the assessment was carried out.

Chapter 5

National Assessment Findings, Policy, and Educational Management

Policy makers, a category that includes politicians, educational administrators, and managers such as senior officials in a ministry of education, are the primary audience for national assessment results. Even when the results have implications for work by other stakeholders (for example, curriculum developers, teacher trainers, or classroom teachers), ministry offi cials will likely have a role to play in formulating policy, in issuing information or directives, or in providing resources. This chapter brings to the attention of policy and decision makers a range of potential and actual uses of findings in policy deliberations and educational management. Four of the uses relate to providing information about the state of education and particularly about student achievement: (a) describing achievement, (b) describing resources, (c) monitoring achievement, and (d) reviewing the education system. A further five relate to the use of that information to address defi ciencies identifi ed in the assessment: (e) formulating general policy and assisting in decision making in conjunction with other information, (f) setting standards, (g) providing additional resources to schools (systemwide or targeted), (h) supporting curriculum revision, and (i) revising textbooks.

Describing Achievement

The basic purpose of a national assessment is to provide information on student achievement – particularly its shortcomings – which is a prerequisite for intervention (Aguerrondo 1992). Furthermore, the information is likely to be unique because it will not normally be available from other sources. Although education ministries routinely collect information about inputs to the education system (for example, student numbers, physical facilities, curriculum materials, teacher-student ratio), a national assessment provides information on the outcomes of the educational investment that inputs represent. Policy makers who have read a report of the findings of a national assessment, such as that outlined in chapter 2, will have an overall picture of student learning that, inevitably, even if not explicitly related to expectations, will invite judgment about the adequacy of student achievement. They are also likely to obtain information on specific domains of achievement with which students are experiencing difficulty, as well as data on the achievements of subgroups in the population. A national assessment report usually presents information on achievement in the form of mean scores. Mean scores in themselves, however, provide only limited information for decision making. More meaningful descriptions of achievement, in the form of profi ciency levels (what students know and can do), as described in chapter 2, provide a more informed basis for decisions and action. Useful information can also be obtained when variance in achievement is partitioned into between- and within-school components (see chapter 2). Large between-school differences are sometimes interpreted as indicating disparity in learning opportunities in the education system. Policy makers should treat such an interpretation with caution, however, because it does not take into account differences between schools in factors over which the school may have little control (the characteristics of students when they enroll and the continuing effects of those characteristics on the work of the school). Between-school differences in achievement may still merit the attention of policy makers because they may provide guidance on intervention. When betweenschool differences in a region are relatively low, and if financial resources are limited, intervention in some schools and not in others would probably not be justifi ed. In contrast, targeting low-performing schools would be justified in regions with large between-school differences.

Describing Resources

A national assessment frequently collects information on the resources available in schools. For example, India’s national assessment in 2000 established that over 90 percent of schools had a school bell, a blackboard, chalk, and an eraser; close to three-quarters had safe drinking water; but less than 40 percent had separate toilets for girls (Singh and others n.d.).

Several national assessments in Africa provide evidence of lack of resources. In Kenya, for example, many schools had an inadequate number of desks and textbooks (Nzomo, Kariuki, and Guantai 2001).

Furthermore, radio broadcast programs to schools did not reach a minimum of one-third of students, who did not have access to school radio (table 5.1). In Zanzibar, an assessment highlighted a serious shortage of classroom furniture (for example, desks and chalkboards) and supplies (for example, textbooks and pencils) (Nassor and Mohammed 1998).

Table 5.1

Table 5.2

Source: Postlethwaite 2004b. Reproduced with permission, EFA Global Monitoring, UNESCO. Note: The Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) is an international nonprofi t developmental organization of 15 ministries of education in Southern and Eastern Africa that work together to share experiences and expertise in developing the capacities of education planners to apply scientifi c methods to monitor and evaluate the conditions of schooling and the quality of education. SACMEQ has completed two major education policy research projects (SACMEQ I and SACMEQ II).

In Nigeria, few schools had maps (13 percent), charts or posters (15 percent), or sports equipment (5 percent). Teachers’ responses to questionnaire items also pointed to the presence of high levels of concern over a lack of teaching materials, as well as low regard for teachers, poor conditions of service, and irregular payment of salaries (Nigeria Federal Ministry of Education 2000).

In Malawi, monitoring of changes in the provision of classroom resources between 1998 and 2002 revealed progress for all facilities (table 5.2). In Zimbabwe, a 1990 review showed schools in Matabeleland South had fewer resources than those in other regions of the country. A follow-up study in 1995 found no improvement (Postlethwaite 2004b).

Monitoring Achievement

If data are available from assessments carried out at different times, trends in achievement (whether it is improving, remaining the same, or deteriorating) can be identifi ed (see chapter 2). This information has sometimes been used to monitor the effects on student achievement of changes in the education system (for example, change in language of instruction or increase in class size). In the United States, data from the national assessment (the National Assessment of Educational Progress, or NAEP) have been used to monitor the major reform initiative No Child Left Behind. In addition to the NAEP monitoring, each state is required to monitor the progress of all students in grades 3 through 8 on its own tests of reading, mathematics, and science. Many states reported signifi cant improvement over the years; however, NAEP results did not reflect this improvement. The improvement recorded from 2003 to 2005, especially in grade 8, was much higher on state tests than on the NAEP test. In Maryland, for example, eighth-grade students who had a reported improvement of 12 percentage points in mathematics on the state test showed no improvement on the NAEP test (de Vise 2005). The difference seemed to occur because of the importance ascribed to the state tests, which had sanctions attached to them. Consequently, teachers focused their teaching on state test content, producing the subsequent increases in test scores without a concomitant improvement in the skills that the test was originally designed to measure (see Madaus and Kellaghan 1992).

In Uruguay, considerable improvement has been reported in the performance of sixth-grade students between 1996 and 2002 on the national assessment. Moreover, the improvement was especially marked among students in “very disadvantaged” schools, in which the percentage of students achieving an “acceptable” level on the test increased from 37.1 to 54.8. This increase is greater than the increase over the same period in schools in social contexts described as “very favorable,” which was from 57.1 to 66.5 percent (Ravela 2006).

In some education systems, a national assessment in the same curriculum area and with the same population is carried out every year. If the aim is simply to monitor standards, this procedure would seem unnecessary, as well as very expensive. Most industrial countries that monitor achievement levels do so less frequently. In the United States, for example, where fi nancial and technical resources are much less likely to be a problem than in developing countries, the NAEP in mathematics and reading is carried out every two years.

A perusal of the results of national assessments that have been running for several decades indicates that large changes in the achievements of students in an education system do not occur over a short time, even when efforts have been made to address problems identifi ed in an assessment. Given this situation, a four- to fi ve-year interval between assessments seems reasonable. Indeed, if national assessments administered over a short time span reported large changes, the equivalence of the tests and procedures used in the assessments would be open to question. Moreover, in many countries, the key source of change in performance over time will be changes in who is attending school. Rising participation rates create problems of comparability that require careful analysis. This and other problems in measuring change in performance over time can arise because of changes in curricula, language, and expectations, as well as from technical issues, such as when assumptions in measurement models (especially item response modeling) are not met or when student scores regress to the mean (Goldstein 1983).

Reviewing The Education System

In many countries, the findings of a national (or regional) assessment have been mentioned in reviews of educational policy and provision or have been used to support major reform initiatives (see table 5.3). The Dominican Republic provides a good example of the use of national assessment results in a major review of the education system, which was followed by a series of ambitious strategies to improve educational quality (box 5.1). The review provided the main input to meetings of regional teams of offi cials and members of communities and school networks that explored the reasons for poor performance and proposed strategies for addressing them.

Formulating General Policy and Assisting in Decision Making

Valid objective evidence on the condition of education, which a welldesigned and well-implemented national assessment can provide, should serve to inject an objective component into decisions and help to ensure that cognizance is taken of empirical evidence as well as personal biases, vested interests, anecdotal evidence, myths, and other forms of “accumulated wisdom” (see box 5.2). Furthermore, the concrete and persuasive evidence that an assessment can provide can help highlight problems in the education system related to access, quality, effi ciency, or equity that might otherwise go unnoticed—or at any rate unattended. Nevertheless, the information derived from a national assessment will not identify ready-made policy or courses of action for the policy maker or other potential user. For one thing, as already noted, a national assessment provides just one piece of information to be considered in the context of a variety of other factors, including theavailability of resources (personnel and material) and the vested interests of stakeholders. Considerable variation exists in national assessment reports in the extent to which they draw implications from assessment data and make recommendations for action or leave these activities to users. Some reports actually go well beyond what seems justifi ed on the basis of the assessment findings in their recommendations (for example, in specifying the details of an intervention for students in rural schools or in suggesting approaches to the teaching of reading or mathematics). Such recommendations, if made, would need to be supported by other research evidence. availability of resources (personnel and material) and the vested interests of stakeholders.

Considerable variation exists in national assessment reports in the extent to which they draw implications from assessment data and make recommendations for action or leave these activities to users.

Some reports actually go well beyond what seems justified on the basis of the assessment findings in their recommendations (for example, in specifying the details of an intervention for students in rural schools or in suggesting approaches to the teaching of reading or mathematics).

Such recommendations, if made, would need to be supported by otherresearch evidence.

Policy makers and senior ministry of education officials may need assistance in interpreting the fi ndings of a national assessment. A major aim of the Southern and Eastern Africa Consortium for Monitoring Educational Quality was, in fact, to

Table 5.3 Selected Countries That Used National Assessment Results in Reviewing the Education System

promote capacity building by equipping educational planners in member countries with the technical ability to monitor and evaluate schooling and the quality of education. A particular feature of its approach was its “learning by doing” training for planners, whom it sought to involve directly in the conduct of studies (SACMEQ 2007). The World Bank Institute implemented a similar capacity-building program in Africa and South Asia.

Although a good deal of evidence indicates that national assessments have contributed to deliberations about educational policy and reform, independent objective evidence to indicate that assessment res lts actually affected policy is limited. An exception is found in Chile, where the results of its national assessment were instrumental in 1997 in convincing the National Congress, on learning that 40 percent of students did not understand what they read, that substantial changes in education were required (Schiefelbein and Schiefelbein 2000). Subsequently, national assessment fi ndings played a role in several policy decisions, including the following (Meckes and Carrasco 2006):

– Directing technical and economic support from the national government to the most disadvantaged populations (as defi ned by learning outcomes) to provide a school feeding program and other assistance to poor students (a program that accounts for 5 percent of the overall public sector education budget)

– Defining the criteria for targeted interventions by the Ministry of Education – Developing programs to improve educational quality and equity

– Defining incentives and goals for improvement

– Evaluating specific policies and programs

– Providing data for educational research purposes.