https://doi.org/10.53656/voc23-211keyp

2023/2, стр. 120 - 143

KEY PERFORMANCE INDICATORS – INPUT, INTERVENING AND OUTPUT VARIABLES

Резюме: In this article, the author argues for an open and transparent use of data that already exists but is not collected, collated and analysed within the education system in Bulgaria. This data needs to become Key Performance Indicators that will drive up: – the Quality of Education; and – the Standards of Achievement. The author focusses on the relationship between: – Input Variables (Prior Attainment and Delegated Financial Resources); – Intervening Variables (Patterns of Expenditure) and – Output Variable (Attendance, Standards of Attainment and Post-school Participation). With one exception, real data is collated, analysed and presented using a range of algebraic and graphical approaches. Finally, the author concludes that with greater subsidiarity (i.e., greater delegation of responsibility, authority and accountability); the greater the need for reliable and valid assessment data to hold the education system to account at school, regional and national level.

Ключови думи: education system; Bulgaria; key performance indicators; variables; input; output; intervening

“The system of state education is part of the public sector and uses resources in the form of taxes, forcedly collected from the private sector. Therefore, the effectiveness of the use of funds allocated for education is an extremely important issue for public sector economics.”

C. V. Brown & P. M. Jackson (1991)

Introduction

Data should be publicly available that demonstrate how individual schools and the education system as a whole in Bulgaria are performing. With one exception, the data presented here already exist; however, they are not collected, collated and analysed. There needs to be an open and transparent approach to monitoring and evaluation that translates this data into Key Performance Indicators. These can be quantitative or qualitative:

– quantitative – standards of achievement, pupil destinations;

– qualitative – questionnaires, end-user opinions;

but Key Performance Indicators (KPIs) are the crucial data that will help to drive up:

1. the Quality of Education; and

2. the Standards of Achievement.

This paper concentrates on Quantitative Performance Indicators and uses the approach and techniques borrowed from Black Box and Open System Theories to aid in their interpretation: only passing reference will be made to Qualitative Performance Indicators.

Black box and open system theory

A Black Box is a system that can be analysed in terms of its Input Variables and its Output Variables1 without any knowledge of its internal structure and organisation. In this analysis, only Input Variables and Output Variables are used to infer the structure and organisation of schools and the education system in Bulgaria.

A Black Box was described by Wiener (1961) as an unknown system that is identified using the techniques of system identification. In this case, statistical methods are used to build mathematical models of a school or an education system from measured data. A common approach is to start from measurements of the Input Variables and the Output Variables and to try to determine a mathematical relation between them without going into many details of what is actually happening inside the school or the education system. This approach is known as of Black Box System Identification.

The Open System Theory is the foundation of black box theory. They both focus on Input Variables and Output Variables that represent exchanges with the environment. In Open Systems Theory, the black box is an abstract representation of a concrete open system which can be viewed solely in terms of its Input Variables and Output Variables. The internal structure and organisation of the school or the education system are irrelevant to Open Systems Theory; only the Input Variables and Output Variables are of interest.

The Black Box Approach is based on the "explanatory principle": i.e., the hypothesis that there is a causal relationship between the Input Variables and the Output Variables. This principle states that:

– the Input Variables and Output Variables are distinct;

– the system has observable (and relatable) Input Variables and Output Variables; and – the system is black to the observer (opaque).

Quantitative performance indicators

In this analysis, Quantitative Performance Indicators are treated as Input and Output Variables. They are the critical indicators of progress toward an intended result; therefore, they:

– provide a focus for strategic and operational improvement;

– create an analytical basis for data-led policy formulation; and

– help focus attention on what matters most

– improving the Quality of Education, and

– raising Standards of Achievement.

Leading and managing with the use of KPIs includes setting targets (the desired Standards of Achievement) and tracking progress against that target.

Good Quantitative Performance Indicators:

– provide objective evidence of progress towards achieving a desired result;

– measure what is intended to be measured (Validity) to help inform better decision making;

– can track changes over time;

– can track the efficient and effective use of resources; the quality of teaching, standards of achievement, pupils’ progress and behaviour and governance.

In addition to Input Variables and Output Variables, in the real life of schools and the education system, it is important to introduce the concept of Intervening Variables. This is illustrated in Table 1 and Figure 3. Intervening Variables are usually the outcome of decisions taken by managers that could have an impact on the quality of education or the standards of achievement. Intervening Variables can function as either Input Variables or Output Variables, depending upon the analysis being undertaken. The pupil/teacher ratio, for example, can be:

– an Output Variable in the analysis of Delegated Financial Resources; or

– an Input Variable in the analysis of Standards of Achievement.

In the process of establishing an algebraic or geometric relationship between the Variables, it is important to maintain a consistent direction of causality; i.e., don’t get Input and Output Variables mixed up. This is particularly true for Intervening Variables; in a particular analysis it is important to be clear whether the Intervening Variable is functioning as an Input Variable or an Output Variable. This is illustrated, in general terms, in Figure 4; with more specific examples shown in Figure 5.

Input variables (prior attainment data)

The marks achieved by pupils in the National External Assessment taken at the end of Grade 7 are an important Input Variable – a measure of the Prior Attainment� of pupils entering Grade 8. As an example, Figure 6 illustrates the distribution of marks of all the pupils entering a Mathematics High School. Figure 7, illustrates the same data set broken down into boys and girls. Casual observation indicates that there is little difference in Prior Attainment between the boys and the girls.

This casual observation is reinforced by Table 2 which indicates that the Minimum, the Lower Quartile, the Median, the Upper Quartile and the Maximum of the boys’ and the girls’ marks are not significantly different.

Table 2

Input variables (delegated financial resources)

Prior Attainment Data are, of course, not the only Input Variable. The Delegated Financial Resources that are made available to school directors are another crucial Input Variable, Staribratov, Kunchev and Budgell (2015). Figure 8 illustrates the main components in the scheme of delegation of one municipality:

– Standards per pupil (68%);

– Supplements per pupil (1%);

– Standards per class (15%);

– Standards per school (2%); and

– Supplements per school (14%).

Figure 8 illustrates quite clearly that the Standards per pupil make up by far the biggest component of the Delegated Financial Resources.

Figure 8

Figure 9, however, illustrates the income per pupil for the individual schools that:

– take secondary aged pupils; and

– are financed and administered by the municipality.

It can be seen from Figure 9 that the Standards per pupil is the same for all schools. However, there are significant differences between the schools in the other components:

– for High School 1, the Standards per pupil component makes up 72.8 % of its income; whereas

– for Community School 6, it makes up only 52.7 %.

Figure 9 prompts some serious questions; for example, why does the Supplements per school component make up only 7.5% of High School 1’s income but as much as 29% of Community School 6’s income? There may be a legitimate reason; for example, Community School 6 may have an antiquated heating system that is very expensive to operate.

Figure 9

Intervening variables (patterns of expenditure)

Patterns of expenditure:

– teachers’ salaries;

– other personnel salaries;

– premises related costs; and

– supplies and services provide easily available Intervening Variables. Figure 10, for example, illustrates the pattern of expenditure across the Science/mathematics and the Mathematics High Schools in Bulgaria.

If greater depth of analysis is required, it is important that a common strategy is developed in order to provide consistent data: Tables 3 and 42 provide one example of such a strategy. Table 3, illustrates the Curriculum Notation for Grade 10 in a Mathematics High School. It shows:

– the overall curriculum model for Grade 10;

– the total number of pupils (104);

– the number of classes (4);

– the total number of lessons (163);

– the number of lessons for each subject across Sectors A, B and C (e.g., Bulgarian Language and Literature {3A}, English {2A+1B}, Mathematics {2A+2B}, PE and Games {2A+1C});

– the number of Bonus Periods where classes are split (e.g., Information Technology and Informatics).

A Curriculum Notation is constructed for each Grade; each of which is linked to Table 4 – the Staff Deployment Analysis – and key information is automatically transferred across (Grade 10 is highlighted):

– the number of pupils (702);

– the number of lessons taught (1126);

– the number of bonus lessons (181);

– the average class size (22).

Other information is entered directly into Table 4:

– the full-time-equivalent number of teachers (49);

– the number of persons in the timetable cycle (35);

– the expenditure on teachers’ salaries (914,174lv).

Important Intervening Variables:

– the pupil/teacher ratio (14:1);

– the average percentage contact (66%);

– the average annual cost of one lesson (812lv);

– the annual cost of the bonus lessons (145,950lv) are then calculated automatically

Output variables (standards of achievement)

Figure 11, illustrates a simple but important Output Variable: the marks for all the pupils in a Mathematics High School in the Secondary School Diploma. Figure 12, illustrates the same data set broken down into boys and girls. Casual observation of this data set indicates that the girls outperform the boys.

This casual observation is similarly reinforced by Table 5 which indicates that the Minimum, the Lower Quartile, the Median, the Upper Quartile of the girls’ marks are higher than those of the boys. The Maximum of the boys’ and the girls’ marks are the same (6.0); but that is because 6.0 is the maximum possible score. As Table 2 indicates that there is no significant difference between the boys’ and girls’ Prior Attainment Scores (the Input Variable). Taken together, therefore, Tables 2 and 5 indicate that the girls make more progress than the boys in their time at the school.

Figure 13, illustrates an alternative Output Variable, the marks in the Bulgarian Language and Literature Matura for the same pupils at the same Mathematics High School. Generally speaking, Figure 13 presents the same shape distribution as Figure 11. However, as those marks in the Bulgarian Language and Literature Matura are such an important element in the algorithm that calculates the marks in the Secondary School Diploma, this should not be too surprising.

Figure 14, illustrates the average marks in the Mathematics Matura for all of the Mathematics/science and Mathematics High Schools in Bulgaria. The lack of any Input Variable (Prior Attainment Scores) makes it impossible to draw any conclusions about the Quality of Education or the Standards of Achievement across the schools. Despite the difference in the Output Variable (the marks in the Mathematics Matura), it is not possible to conclude that School 32 (5.69) is a better school than School 1 (4.15) without significantly more information about:

– the Prior Attainment and Ability Distribution of the pupils who attended the school; and

– which pupils elected to study for the Mathematics Matura – School 1 (5 pupils) and School 32 (158 pupils).

Figure 14 does however raise one important question. Given that School 1 is a Mathematics High School, why did so few pupils choose to study for the Mathematics Matura?

Input variables  output variables (prior attainment  standards of achievement)

In the previous section, it was stressed that the lack of any Input Variables (Prior Attainment Scores) made it impossible to draw any conclusions about the Quality of Education or the Standards of Achievement from Output Variables alone. Figure 15 illustrates an Input Variable (the National External Assessment taken in Grade 7) and an Output Variable (the Secondary School Diploma) for the same pupils at a Mathematics High School, (Kunchev and Budgell 2014). Because the marks have a totally different range and distribution:

– 23.84  35.00 in the National External Assessment; and

– 4.60  6.00 in the Secondary School Diploma;

they have been expressed as a percentage of the maximum possible mark in order to make them comparable.

It is immediately obvious from Figure 15, that the distributions have a completely different shape. This is reinforced by Table 6, a far greater number of pupils are awarded high marks in the Secondary School Diploma than achieve high marks in the National External Assessment.

A school director might immediately want to conclude that Figure 15 and Table 6 demonstrate what a good job the school is doing! There are, however, more compelling interpretations.

1. There is an artificial ceiling in the assessment processes that make up the Secondary School Diploma. Good pupils can be awarded top grades and, therefore, the Diploma does not distinguish between good pupils and excellent pupils.

2. Unmoderated teacher assessments are significant elements of the algorithm that calculate the Secondary School Diploma. These unmoderated assessments, together with the pressure on the teachers to grade the pupils as high as possible, lead to the unreliable over-assessment of the pupils’ standards of achievement.

Figures 16 and 17, illustrate two methods of presenting the same data set:

– the Marks in the National Assessment taken in Grade 7; and

– the Teacher Assessment of the standards attained by the pupils in Bulgarian Language and Literature at the end of Grade 12.

Figure 16, presents the data as a scatter graph; including the equation of the ‘regression line’ ( and a measure of how close the data are to the fitted regression line R2 (0.2938). These measures indicate that, although there is a relationship between the Marks in the National Assessment and the Teacher Assessment of the Standards attained by the pupils in Bulgarian Language and Literature, it is not a particularly strong relationship. This conclusion is reinforced by the highlighted data points which indicate that:

– pupils with marks between 23 and 26 in the National External Assessment can be awarded marks from 3.00 to 5.75 by their teachers in Bulgarian Language and Literature;

– those with marks between 27 and 28 can be awarded marks between 3.25 and 6.00; and

– those with marks between 30 and 31 can be awarded marks between 3.75 and 6.00.

Figure 17, presents the same data as a ‘Box and Whisker Plot’ which illustrates the minimum, lower quartile, median, upper quartile and maximum for each subset of the data.

Figure 18, is comprised of simulated data: it presents the data as a scatter graph; including the equation of the ‘regression line’ ( ) and a measure of how close the data are to the fitted regression line R 2 (0.7725). These measures indicate that not only is there a relationship between the Marks in the National Assessment and the Marks in the Secondary School Diploma, it is a strong relationship: the National Assessment dada account for over 75% of all the variability of the Secondary School Diploma data. This sort of analysis that expresses the Output Variable as a function of the Input Variable should be very important in assessing the Standards of Achievement in a group of similar schools like the Mathematics/science and Mathematics High Schools.

6.0

5.5