Evaluator – Supporting Criminal Justice Reform and Strengthening Anticorruption Efforts program in Moldova

Program:

Supporting Criminal Justice Reform and Strengthening Anticorruption Efforts in Moldova

Evaluation Type:

Final Evaluation

Evaluation Purpose:

To study the program’s achievements, lessons learned, best practices, and key takeaways for future programming by conducting a thorough external final evaluation.

Evaluation Period of Performance Start and End Dates:

May 1-July 15, 2024

The American Bar Association Rule of Law Initiative (ABA ROLI) is a non-profit organization that implements legal reform programs in roughly 50 countries around the world. ABA ROLI has nearly 500 professional staff working abroad, and in its Washington, D.C. office. ABA ROLI’s host country partners include judges, lawyers, bar associations, law schools, court administrators, legislatures, ministries of justice and a wide array of civil society organizations, including human rights groups.

ABA ROLI seeks an Evaluator (Local Consultant) for its Supporting Criminal Justice Reform and Strengthening Anticorruption Efforts in Moldova program

1. Background

The Supporting Criminal Justice Reform and Strengthening Anticorruption Efforts in Moldova Program (from here on forward referred to as “the Program”) is a 3.5-year program implemented by the American Bar Association Rule of Law Initiative (ABA ROLI). The Program started in February 2021 and will end in June 2024.

The Program’s goal is to support the continued development of the criminal justice system and actors in Moldova, and to strengthen targeted anticorruption efforts. Specifically, the Program objectives are listed below:

Objective 1: Increased integrity, professionalism, and reform, within the Prosecutor General Office (PGO), Superior Council of Prosecutors (SCP) and Superior Council of Magistrates (SCM

Objective 2: Build capacity of key justice sector entities to combat corruption.

Objective 3: Support law reform initiatives meeting European standards.

Objective 4: Support legal professionals in improving professional standards and capacity

Cross-Cutting 1: Promote inter-legal profession dialogue and concerted action

Cross-Cutting 2: Support reform-oriented associations of young legal professionals and others.

1. Evaluation Objectives

The evaluation is commissioned by ABA ROLI’s Europe and Eurasia Division to comply with the commitment set under the Program. The evaluation will provide evidence-based assessments of the Program’s achievements, lessons learned, best practices, and key takeaways for future programming.

Specifically, the evaluator will determine whether there is suggestive evidence that the Program contributed, in whole or in part, to outcomes prioritized for evaluation under the Program. Specifically:

The evaluation will respond to the following questions:

1. How effective were trainings, mentorship and technical assistance provided under the Project in ensuring improvements in knowledge, skills, and practices/behavior? Specific areas of inquiry include the following:

• Under Objective 2: Handling of corruption, cybercrime and cryptocurrency cases by inspectors, investigators and prosecutors trained and supported
• Under Objective 3: Drafting and supporting priority law reforms by MOJ officials and personnel trained and supported
• Under Objective 4: Legal aid provision by lawyers and legal clinic staff trained and supported
• Under Cross Cutting Activity 1: Interprofessional communication and synergy-building among justice sector professionals
• Under Cross Cutting Activity 2: Strategic planning, project operations and management, and law association management for young and female professionals trained and supported

1. Where there have been observed or felt changes in individual knowledge, skills and practices/behavior, have these individual changes led to positive improvements at the institutional or societal/ community levels?

2. To what extent has the Program improved the capacities of the pilot prosecutors’ offices supported in key areas of excellence of the Model Prosecutors Office Framework?

3. What implications would the findings have for future efforts to support the development of the criminal justice system and to strengthen targeted anticorruption efforts, or of projects in the same context and conditions?

The evaluation shall be guided by the American Evaluation Association’s principles for evaluators [1].

1. Evaluation Method

The evaluation will combine the Kirkpatrick Model of Training Evaluation[2], performance evaluation (using the Model Prosecutor’s Office Framework), and contribution analysis to describe the results and assess the level of contribution to observable or felt benefits.

In questions a and b, and where the type of intervention is training or mentoring. the evaluation will focus on assessing against the four levels of the Kirkpatrick Model of Training Evaluation: (1) Did the learners enjoy the training?; (2) Did learning transfer occur (i.e. changes in knowledge and skills)?; (3) Did the training change behavior?; and (4) Did the training influence performance/ how much benefits were observed or felt by the institution or the community as a result of the training?

In question (c) the evaluation will utilize the Model Prosecutor’s Office Framework to assess progress against the baseline indicators and/or against a comparison group (i.e. a prosecutors’ office that has not received assistance under the Model Prosecutors’ Office (MPO) Framework[3] component of the Program). The evaluation assesses within the areas of excellence under the Framework that was chosen for each prosecutor’s office.

Contribution analysis[4] will be used to assess the Program’s contribution to observed or felt changes. The process involves six steps: (1) define the attribution problem or the contribution questions; (2) develop a theory of change and identify the risks to it; (3) gather existing evidence on the theory of change; (4) assemble and assess the contribution story, or performance story and the challenges to it; (5) Seek out additional evidence; and (6) revise and strengthen the contribution story.

1. Uses and Users of this Evaluation:

This evaluation will contribute to better understanding of the program’s achievements and will document lessons learned and key takeaways for future programming to better inform managers on future program design efforts.

1. Scope

The evaluation cover program activities and results from 2021-2024.

The evaluation will utilize document reviews, surveys, and interviews to gather primary data but will also maximize the use of data collected by the Program as secondary source, especially to establish longitudinal comparisons. Program documents to be reviewed will include program progress reports, partner reports, reports on the criminal justice system and anti-corruption policy priorities, and local partner outputs completed under the Program. Post-tests and other surveys completed following training activities will be sorted and reviewed.

The evaluation requires assessments in most if not all pilot MPOs to observe changes in most if not all MPO Framework areas of excellence to address question (c). The evaluation of pilot MPOs may utilize surveys and checklist templates under the MPO Framework with other quantitative and qualitative tools for assessing outcomes that were unexpected and contributions of efforts and activities under the Program on observed results.

1. Deliverables

2. Desk review and interviews conducted in the 1-2 weeks after start date;

3. An Inception report, detailing the data collection methods and tools to be used for the evaluation.

   • The Inception Report is to be submitted to ABA ROLI no later than 1-2 weeks after the start date (or May 31, 2024)
   • The Inception Report must include a work plan prepared in consultation with ABA ROLI’s project team and should incorporate adjustments to the initial workplan developed considering a desk review of program documents. The inception report should include the data collection and analysis methods and the draft tools (which may include, but is not restricted to survey questionnaires, interview and FGD scripts, data collection tables, or policy analysis sheets) to be used for the evaluation.
   • The Inception Report will be reviewed in detail with ABA ROLI to ensure consistency with the context of the Program and the terms of reference for the assignment.
     1. An Evaluation Report in English of not more than 35 pages, including an Executive Summary.
        1. a draft report to be submitted no later than June 21, 2024
        2. a Final Evaluation Report in English to be delivered to ABA ROLI no later than July 15 (in Microsoft Word, Times New Roman font size 12, single-spaced), that incorporates feedback provided by ABA ROLI.
        3. A briefer on the evaluation findings and conclusions to be presented in a 3-pager document and a presentation material. The evaluator may be called upon to present the highlights of the report to key stakeholders internal or external to the Program.

4. What We Are Looking For

The consultant should be proficient in conducting mixed research and analysis, with relevant experience in conducting advocacy and policy evaluation. He/she must have:

• Local (Moldovan/ Romanian) consultant;
• At least 7 years of experience undertaking training and performance evaluations using mixed research methods;
• Required proficiency in speaking, reading, and writing in Romanian and English.
• Experience in the evaluation of trainings and organizational development efforts for the justice, anti-corruption and human rights sector is preferred.
• Proven ability to organize and coordinate evaluations or studies in the field, and with a group of program implementers in various locations
• Experience working on evaluations of programming in Eastern European countries is preferred.
• Proficiency in the use of Zoom and other secure online meeting platforms to conduct and transcribe evaluation key informant interviews (no travel to the field foreseen under this engagement); and
• Proficiency in the use of MS Office and other applications for online surveys and data analysis.

[1] American Evaluation Association Guiding Principles For Evaluators July 2004 http://www.eval.org/p/cm/ld/fid=51

[2] Kirkpatrick, D. (1998). Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler.

[3] The nine areas of excellence in the MPO Framework include: leadership, management of resources, plans and prosecutorial policies, efficiency of criminal proceedings, quality of prosecutorial work, effective cooperation with other justice sector actors, public trust and confidence, and integrity and accessibility.

[4] https://www.betterevaluation.org/methods-approaches/approaches/contribution-analysis