Is the policy environment for Zambia supportive of a thriving whole-of-government monitoring and evaluation system?

A whole-of-government monitoring and evaluation system (WoGM&ES) is a robust system that not only provides an integrated and all-encompassing framework of M&E practices, principles and standards to be used throughout government institutional structures, but also functions as an apex-level system for information and draws from the component systems in a framework meant to deliver essential M&E products tailored to satisfy information needs of users [1]. To implement a successful WoGM&ES, a supportive policy environment is crucial for any organisation, governments inclusive. The Zambian government is currently rolling out an ambitious WoGM&ES to strengthen its public sector accountability, feedback and learning functions. It was the objective of this study to investigate the policy environment in Zambia in respect of M&E practice in the public sector. In that regard, particular policy aspects were considered and these included assessing the availability of an M&E plan; whether the difference between M (monitoring) and E (evaluation) was recognised; and if the need for M&E autonomy and impartiality was mentioned. Others were to determine if feedback mechanisms were explicit as well as whether there was integration of M&E results in planning and budgeting processes. The study findings have shown that the policy environment to support a thriving M&E practice in the Zambian public sector was still fragmented and weak. While efforts by government to put policy measures were in place, evidence of transformational shift to implement these measures remained weak. However, it is promising that, in many ways, Zambia was on the right path regarding the introduction and articulation of policy provisions in support of M&E and broadly in promoting a culture of results.


INTRODUCTION
Monitoring and evaluation (M&E) has become a common practice among development organisations and governments the world over. Specifically, governments increasingly use M&E as a tool to improve their good governance prospects as they seek to satisfy various stakeholders including citizens, donors and civil society. Both governments of the the developed and developing countries have adopted some form of M&E arrangements to strengthen their tracking, measurement and improvement of interventions under their control [23][24][25][26]. The Zambian government has also embraced M&E in preparation, implementation and reporting of progress made against goals and objectives in the national development plans (NDPs). In addition, M&E practice is also being rolled out to all other decentralised government structures at sector, provincial and district level.
The policy environment, in support of M&E implementation in the country becomes an important aspect towards a thriving culture of results in any nation or development organisation. When a deliberate policy is in place, the conduct and growth of the M&E function and profession tend to increase [2 & 24]. In many countries, and particularly in developing countries where the evolution of M&E is still in its embryonic stage, you find that policies to support M&E lack or remain fragmented and weak [3]. For that reason, it becomes crucial that the policy environment regarding the conduct of M&E is constantly in check to ensure that every country or organisation strives to provide a sound policy environment to assist strengthen the culture of results through the conduct of functional M&E.
Indeed, when there is a supportive policy and legislation, the practice of M&E in a country becomes a matter of interest to all development stakeholders [20,[27][28][29]. In that regard, creating M&E structures and competition in acquiring the necessary skills in M&E by organisations and individuals becomes possible. The opposite may also be true-in the absence of policy provisions, M&E risks being crowded out by other development processes and practices [17 & 32]. Thus, a country that aims to strengthen its good governance record and to achieve sustainable development results would invest resources and capacity in M&E by ensuring that the policy environment is sound and supportive. This was the core of this study.

METHODOLOGICAL APPROACH
The main objective of this research study was to assess the current policy environment for practicing M&E within the context of Zambia's whole-of-government monitoring and evaluation system (WoGM&ES). The study presupposed that favourable policy environment was a prerequisite for successful implementation of a thriving M&E regime in any country or organisation. To undertake this study, a qualitative, broad based, and investigatory research approach was followed involving the use of various methods and tools to put data and information together. Essentially, data were collected from reviews of secondary data sources as well as primary data sources.
The study design for this research took on board elements that were deemed key to answering the research questions. Particularly, the qualitative approach was adopted to guide the overall data collection, collation, analysis and interpretation. To meet the research objectives, a rigorous process was adopted involving desk-based research through consulting a wide range of literature on the topic as well as field-based research which meant acquisition of hands-on information (primary research). Among many others, secondary sources of data for the study comprised key government policy documents such as the national development plans (NDPs), NDP Annual Progress Reports (APRs), several government-wide reports, and the country's Vision 2030. In addition, a wealth of related M&E literature was used from international development organisations and development agencies as well as scholarly books, discussion papers, journal articles, working papers and research papers from experts and practitioners of M&E.
Furthermore, primary data were collected through key informant interviews and focus group discussions (FGDs) with stakeholders, particularly those concerned with public sector planning, implementation, monitoring and evaluation. Various types of respondents provided information for this research study. Respondents were drawn mainly from designated government and non-state institutions, such as government line ministries, parliament, cabinet office, office of the auditor general, provinces, districts, academia, civil society, and development partners and donors. To undertake analysis for the research findings, a number of techniques were employed. The major analytical tool called LEADS which is comprised of a 5-point scoring scale was adopted in this research study to guide collation and analysis. In addition, the Nvivo software and text analysis were too utilised for synthesising and enriching the research discussion.

ANALYTICAL FRAMEWORK
To investigate the prevailing policy environment in support of the M&E practice in Zambia, and thereby come up with analytical positions regarding the subject matter, two frameworks were adopted; (i) the use of an assessment checklist comprised of five (5) components for assessing the status of M&E policy environment. The components include the M&E plan; M versus E; autonomy and impartiality; feedback; and aligning to planning and budgeting. (ii) the use of a five-point LEADS system of scoring for each element in (i). The 5-point categories of the LEADS scoring system are: L (Little action: 1), E (Elements exist: 2), A (Action taken: 3), D (largely Developed: 4), and S (Sustainable: 5).
The LEADS system is a matrix with components that correspond to the components in the assessment checklist. Thus, the two tools were used together. First, the diagnostic checklist was fully administered to all respondents, which included a review of documentation. All the responses under each component in the checklist were compiled in readiness for discussion and analysis. Second, the LEADS system was used to assess the findings by scoring. This was done in accordance with the responses obtained under each component. Although the scoring exercise was relatively subjective, results from the triangulated qualitative data and information from the field, value addition to the data was realised. This helped to enrich the discussion and analysis of the findings thereof. Table 1 shows the details of the questions asked under each of the five components. Using the questions from the diagnostic checklist, data collection was done. Rigorous document review was also used. On the basis of evidence from the study findings, scores were given guided by the LEADS scoring system. In addition, Table 2 below shows the detailed LEADS scoring system which was used for the analysis of the research findings. The system indicates the five scales (1-5), 1 signifying that the policy environment for the country was not fully supportive of M&E while 5 denoting that there was some existence of conducive policy environment to promote M&E practice. -There is no explicit and consistent approach to reporting, dissemination, integration.
-References are made to reporting, dissemination and / or integration, but there is no explicit and consistent approach.
-There is an approach to reporting, dissemination, integration, but it is not explicit and consistent.
-There is an explicit approach to reporting, disseminatio n, integration, but it is not completely consistent.
-There is an explicit and consistent approach to reporting, dissemination, integration.  Table 3 presents the results of the assessment for the policy environment in Zambia's public sector. Overall, the policy dimension scored 2.2. A closer look at the intra sub-component dynamics gives notable aspects of analytical interest. For example, while the topic on feedback has a score of 3 points (action taken), the rest of the sub-components (M&E plan, M&E, autonomy and impartiality, and alignment with planning and budgeting) scored 2 points each. This may mean that although the policy component seems to be fairly or well developed at a 2.2-point score, Zambia's WoGM&ES fared poorly in its accountability function. This is also true of the sub-dimensions of alignment of M&E with planning and budgeting processes (a 2-point score).

DISCUSSION AND ANALYSIS
To appreciate the details of the findings, a fuller discussion and analysis of the results follows. For consistency and in conformity with the study design, the five components are used as headings.

Monitoring and evaluation plan
The total score for this sub-dimension was 2.0, signifying that only elements existed regarding the M&E plan for Zambia's WoGM&ES. The "existence of acceptable national planning, budgeting and M&E systems, or at least observable improvements in such systems, and trust in a recipient country's policy priorities is in principle necessary for the effective and successful move towards a shift from donor control to recipient control" (1). Thus, it was established that Zambia had a number of documents that articulated M&E issues. The National Planning and Budgeting Policy of 2014 provided high-level guidance for M&E practice and implementation for the public sector. However, the policy did not offer clear and holistic guidance on M&E implementation across government [, 13 & 17]. There was also a draft national performance framework (NPF), which articulated strategic objectives and outcomes that were significant in realising Vision 2030. NPF is the framework that clarifies the theory of change (ToC), illustrating how the implementation of NDPs and the measurement of progress were envisaged to happen. Further, [4 & 26] argued that many governments have realised that without a structured results orientation in the manner governments did their business, not much development could be achieved. Hence the focus was on development of national performance frameworks. In that regard, the NPF could map key result areas and outcomes that are cascaded downwards at sector level with KPIs, baseline values and targets to guide the implementation, monitoring and evaluation of programmes [5]. There were also line ministry strategic plans that articulated M&E activities at that level [6-10]. However, some line ministries, provinces and districts had not developed their strategic plans.
It was acknowledged that a whole-of-government monitoring and evaluation system (WoGM&ES) was in place. Nevertheless, the system was not unified. There was no common definition and understanding of the WoGM&ES across the public sector institutions consulted. Zambia has a national long-term vision (NLTV), namely Vision 2030, which expresses citizens' aspirations of becoming a prosperous middle-income country by 2030. The NLTV is the basis on which all plans and budgets should be anchored. Zambia also has a national development plan (NDP), which a five-year medium-term plan is derived from the NLTV aimed at helping to achieve the vision. An NDP is a detailed policy strategy from which monitoring and evaluation (M&E) is derived. Currently, Zambia is implementing the 7NDP (2017-2021), which has defined development outcomes, strategies, programmes and objectives to be achieved by 2021. Unfortunately, government had not defined strategies and objectives to be achieved at province and district level by the end of the plan.
Many M&E systems across government were fragmented. Only draft national M&E policy, performance frameworks and automated monitoring and evaluation information management systems (M&E-MISs) have been formulated so far. Currently there is a mechanism that facilitates the tracking of delivery of public services and assessment of impact and appropriateness of policies and programmes. However, the system is not effective because of the lack of management information systems (MISs) in institutions that were mandated to provide data and information. Therefore, overall, Zambia has an M&E plan in place, but it is not comprehensive enough to state what to evaluate, how, and for whom. For instance, no document explicitly indicated the prioritised interventions for evaluations (that is, no evaluation plan was in place). However, the M&E plan was clear about the reasons for evaluation (that is, to enhance accountability, feedback and learning) [10 & 19].

Monitoring versus evaluation
The notions of 'monitoring' and 'evaluation' were acknowledged and differentiated only to some extent in all government documents. As a result, a score of 2.0 was given, denoting that only elements existed. A section in 7NDP was dedicated to defining and describing the meaning of each concept [10]. In addition, the two functions were not understood to be different in the 7NDP, they were also acknowledged as being complementary [5, 10, 11, 15 & 22]. While this was clear in the NDP, understanding of the differences between the notions at levels such as line ministry, province and district was found to be weak. At those levels, there was a tendency to put them together as though they were synonymous. Further, there was a great deal of effort at all levels to describe in detail monitoring tasks to be undertaken as opposed to those concerning evaluation.
Policy framework(s) in institutions across the WoGM&ES made M&E of institutional plans and programmes mandatory. But there was no framework to ensure data quality and relevance. Nor were there formal structures to facilitate the use of performance information for programme management and evidence-based decision making.
As a consequence, units, section and departments in charge of M&E based their measurement and achievement of objectives and goals on performance indicators (KPIs), though in some cases these were weak or non-existent. In addition, evaluations were ad hoc and rarely undertaken. Likewise, all MPSAs acknowledged that none had an official public (legal framework) document that established the evaluation guidelines with methodologies and technical standards to guide institutional plan evaluations. Project and programme evaluations were rarely conducted across MPSAs, a factor which caused institutions to score poorly in evaluative practice and culture.

Autonomy and impartiality
A score of 2.0 was given to this sub-section, entailing that elements of autonomy and impartiality existed in the WoGM&ES for Zambia. An assessment was make of whether the need for M&E autonomy and impartiality was mentioned explicitly and whether the M&E plan allowed for tough issues to be analysed and reported. Additionally, the assessment investigated whether there was an independent budget or fund allocation for M&E. It was found that the need for autonomy and impartiality of M&E was not mentioned explicitly. In all four NDPs that were reviewed (FNDP, SNDP, R-SNDP and 7NDP), autonomy and impartiality of M&E were not mentioned categorically or acknowledged as being important for a successful WoGM&ES or for good governance [12, 30 & 33]. The annual progress reports (APRs) did not mention the need for M&E functions to be treated with autonomy and impartiality. In all documents and interviews, M&E functions were described as being undertaken by ordinary departments, units and sections within government structures without any demand for autonomy and impartiality. Perhaps, the only element of autonomy and impartiality that was mentioned in some NDPs and APRs was the need for 'evaluation exercises or processes' to be led by external consultants or experts -not necessarily establishing formal external evaluation structures [12 & 31]. In addition, organisational or structural issues of M&E were not covered prominently in the documents. No mention was made of the need to locate the function of monitoring and that of evaluation in one place or in different locations.
With regard to the analysis and reporting of tough issues arising from the implementation of development interventions, there was no mention of what needed to be done. As a result of weak analyses in APRs for instance, details to inform practical correctional actions were lacking. In the documents, there was sporadic attention to budgets that were meant to finance M&E functions and particular activities. There were no independent and predictable budgets across public institutions for M&E activities. Evidence of budget cuts and non-release of funds for M&E-related activities in most institutions was repeated. Whenever institutional budget cuts were done, budget lines for M&E activities suffered most -signifying that less importance and priority were attached to M&E. Except for a few line ministries (National Development Planning, Health, Education, Agriculture, Fisheries and Livestock), the budget allocations for supporting M&E activities in most public institutions were reported to be small and fragmented. More so, even in those few institutions with small budgets for M&E, allocations seemed to be focused only on limited monitoring activities and almost nothing for evaluation undertakings [5-13 & 28].

Feedback
The assessment score of 3.0 was given, implying that action was taken pertaining to feedback loops. Feedback mechanisms constituted another element that was assessed. Here it was interesting to check whether there was an explicit and consistent approach to reporting, dissemination, and integration. The Zambian policy environment and M&E plans and frameworks had mixed positions on approaches to reporting, dissemination and integration of M&E (in all four NDPs -fifth, sixth, revised sixth and seventh). The APR, based on the reporting performance of the NDP was the main feedback M&E output for the WoGM&ES. Once produced, it was disseminated to stakeholders, particularly government institutions, for possible use in organisational development processes. Dissemination of NDPs and APRs was done through meetings, workshops and the ministerial website [7,8,10,12] and occasionally through newspapers, radio and television. Ministry of Finance (MOF) indicated that budget information was available to the public through the Internet by the time that budget proposals were presented to parliament. MOF also posted this information on its website for the public.
However, although dissemination to stakeholders and integration of M&E results into decisionmaking processes were mentioned, no details were given on how this was done [7, 8, 10, 12, 13 & 14]. Quarterly and annual reports were available from MPSAs, but dissemination to stakeholders was said to be limited. Further, data dissemination was reported to be done through media briefings, posters, reports and postings on the institutional (CSO) website. Other disseminations were done at stakeholders' request. Some products were disseminated to MPSAs and to other non-state actors such as universities and parliament. Nonetheless, it was acknowledged that these platforms catered only for urban-based stakeholders, while those in rural set-ups had no easy access.

Alignment to planning and budgeting
A total score of 2.0 was given for this sub-dimension. It means that only some elements of alignment to planning and budgeting existed. The integration of M&E products into the processes of planning and budgeting was found to be mixed. Some M&E integration was traced or mentioned in the process of designing NDPs. For example, APRs, evaluations and reviews were used to inform the formulation of the SNDP and 7NDP. But this evidence seemed to end only at the planning stage -and not the decision-making level.
However, the most significant problem was with the budgeting. Although there was mention of attempting to use the NDPs to inform budgeting, evidence was weak or missing altogether. In some cases, budgeted and funded programmes and projects were not contained in the NDPs or line ministry budgets [8,9,13]. APRs showed that most fund releases from NDPs in the budgets were unreleased by MOF. There was no evidence of integration of M&E information into the resource allocation. MOF rarely or never used M&E feedback to determine fund allocation and release, if so, the link was weak [11, 13 & 14]. Further, MPSAs were required to present M&E information in support of their budget and medium-term expenditure framework (MTEF) submissions -but to a lesser extent and it was characterised by a weak management structure. Every year, MPSAs were asked to submit a policy brief for their previous budget's expenditure performance to MOF. In those briefs, institutions presented their budget performance reports with some semblance of M&E information.
Nevertheless, there was no strong evidence of integrating M&E information, for instance in informing critical decisions across government processes, such as budgeting and resource allocation by MOF. With regard to whether programme/project output information was used in decision making across government structures, a number of MPSAs acknowledged that they did so, but did not give details of how this was done. At the same time, some institutions stated that the use of output and outcome information in decision making was not regular, coherent or consistent.
In addition, Medium Term Expenditure Frameworks (MTEFs) were not implemented effectively for they were changed or adjusted annually, depending on resource availability. Further, budgetary decisions were carried out without necessarily taking into account the results produced by the performance indicator-system of the NDP(s). Budget decisions were driven by the resource availability in a given year and based on the guidance of cabinet. No incentives were in place to encourage the demand for M&E information by MOF from agencies to accompany their budgetary requests or support. MPSAs were not obligated to present their M&E information in support of their budgets and MTEF submissions. It was reported that there was no such requirement by MOF. The biggest challenge was that most (if not all) MPSAs lacked robust M&E systems to deliver this kind of information. There was also a lack of M&E champions in MPSAs to demand M&E results to inform planning and budgeting decisions and processes. Lack of incentives was said to have led to delays in institutionalising M&E in most MPSAs. Only to a certain extent was it acknowledged that MOF engaged line ministries and other MPSAs in dialogue on their policy choices, based on performance information. This was done through the policy and budget hearings at which MPSAs were invited to dialogue with the treasury on their proposed policies. This gave MPSAs an opportunity to justify, and seek clarity on their proposed policy priorities. In addition, it was reported that, despite such efforts, this did not influence significant policy choices, as did the availability of resources in the treasury.
For the limited engagements by the MOF with MPSAs, it was confirmed that the nature of information required when submitting budget proposals included retrospective and prospective information on ministry spending; information on ministry outputs; and to a limited extent on outcomes and impacts. Also, infrequently information on results of formal evaluations and reviews was requested. It was gathered that these engagements were never results or performance based. No evidence existed of linking performance information of MPSAs and policy hearings by MOF. It was not even clear if MOF had a results approach in the implementation of the short-and medium-term financing frameworks. The MNDP was mandated to coordinate national development planning and it was reported to demand various types of information from MPSAs. Such information included prospective and retrospective information on ministry spending; information on ministry outputs; information on institutional outcomes and impacts; and on results of formal evaluations and reviews, though in many cases outcome and impact level information was missing.

CONCLUSION
In order to develop and implement a thriving WoGM&ES, having in place a supportive policy environment is a crucial prerequisite. Policy provisions offer practical guidelines on why and how governments and development organisations need to structure their M&E functions in order to achieve good governance based on results. This research focused on appreciating the existing policy environment in support of M&E practice in the Zambian public sector. To determine how rooted the practice of M&E was (or was not) on policy provisions becomes a salient starting point towards building and strengthening a thriving culture of results for any country.
The research established that a number of positive strides to support M&E in the Zambian public sector exist. The current policy environment was promising, although largely still in its embryonic stage. The existing national planning and budgeting function is one positive aspect found in support of M&E practice in Zambia. Equally, the recently launched National Monitoring and Evaluation Policy puts Zambia and particularly the public sector in good perspective regarding the enhanced practice of M&E. While many other promising efforts were noted, the policy environment was still lacking in several ways too. The absence of an integrated and well-coordinated planning and budgeting system undermined a thriving M&E practice in Zambia. Also, some policy provisions on planning and budgeting still required refinement and strengthening. There are possible repetitions, redundancies and conflicts which need correction and reinforcements in present policy and legislative provisions. Study findings have shown that development plans at national, line ministry, provincial and district level were not well linked, coordinated nor solidly integrated. Worse off, progress tracking, measurement and learning at all these levels still lacked a great deal.
To resolve most of the current challenges, the Zambian government would do well to embark on a clear policy review and strengthening process. The effort should be aimed at ensuring that M&E was entrenched in national policy documents, practices and processes. All the three (3) arms of government-the executive, legislature and judiciary will need some clear policy and operational guidelines on how M&E functions were to be structured and implemented in the Zambian public sector and beyond.