Final Evaluator

ABOUT THE OPPORTUNITY

Internews is seeking a qualified consultant to conduct a final evaluation for its project, the Journalist Security Fellowship (JSF), during February to June 2024. The evaluation will be conducted fully remotely/online.

ABOUT THE PROJECT

Journalists and independent media outlets based in Southeastern and Central Europe work in challenging environments characterized by media ownership concentration, job insecurity, and hostility from state and non-state actors in the form of digital targeting and espionage. Many of these journalists, particularly freelancers, do not have support to address such threats, and as a result, are often ill-equipped to respond to increasingly complex online attacks. Internews launched an initiative designed to support local journalists (with a focus on freelance journalists) and small independent media outlets in Southeastern and Central Europe (specifically Albania, Bulgaria, Croatia, Czech Republic, Hungary, Poland, Romania, Slovakia, and Slovenia) by helping them prevent, mitigate, and respond to online attacks by combining proven digital safety training methods with state-of-the-art, innovative scenario-based trainings (include virtual reality, tabletop exercises, and technical simulation). To learn more about the Journalist Security Fellowship, visit our blog.

LOGISTICS

Internews has offices in Arcata, CA, Washington, DC and London, UK. Ultimately this is a remote-based role and candidates based anywhere in the world will be considered. However, there are some locations in which Internews is not able to support remote work.

Applications will be reviewed on a rolling basis, so candidates are encouraged to apply soon.

This is a limited-duration position, with an expected duration of approximately 4 months.
The estimated budget for this evaluation is 17,000-20,000 USD.

OUR COMMITMENT TO FOSTERING A CULTURE OF BELONGING

We are an organization of dynamic, mission-driven individuals who are passionate about our core values and about supporting positive change in the world. We pride ourselves on our commitment to innovation and flexibility. We believe that diverse teams are strong teams and work to support an ethic of belonging, dignity, and justice for all people. Our current team includes a mix of genders, parents and non-parents, and people of multiple races, nationalities, ages, sexual orientations and socioeconomic backgrounds. We are an EEO employer and encourage candidates of all races, genders, ages, orientations, ethnicities, and national origins to apply, and welcome those with alternative backgrounds and experiences.

Evaluation Overview

The final evaluation purpose and objective is to:

• Assess the project results on key target beneficiary groups which include project fellows and journalism communities in Central and Southeastern Europe (including journalists, newsrooms, and other media-focused organizations), with an emphasis on beneficiaries from at-risk and/or marginalized backgrounds.
• Assess the effectiveness and sustainability of using scenario-based training methods in digital security trainings for journalists and journalism students, and whether those encourage participants to implement learnings from the trainings in their day-to-day professional activities. Analyze whether this approach to training could be expanded, adapted, and/or scaled.
• Assess performance of the project in terms of OECD-DAC criteria such as relevance, effectiveness, efficiency, impact, and sustainability.
• Analyze underlying factors beyond Internews control that affect the achievement of the project results.
• Provide recommendations for potential future interventions.

Additional components are to include the following:

• Researching and producing a report on the experiences of participants from scenario-based trainings held in Albania, Bulgaria, and Croatia.
• Producing at least three ‘spotlight’ case studies which showcase the successes and stories of select project fellows for funder and external audiences.

Key Evaluation Questions

In line with the evaluation purpose and objective, the following key evaluation questions are proposed for the final evaluation. However, these can be expanded and modified by the evaluator in consultation with Internews.

• How has the project influenced journalism communities in Central and Southeastern Europe, particularly focusing on journalists, newsrooms, and other media-focused organizations?
• To what extent has the project influenced individuals from at-risk and/or marginalized backgrounds within the target beneficiary groups?
• How effective are scenario-based training methods (as supplemental to traditional training methods) in enhancing digital security skills among journalists and journalism students?
• To what degree do participants from the trainings implement the learnings into their day-to-day professional activities?
• Can the scenario-based training approach be expanded, adapted, or scaled to other contexts or regions?
• How relevant is the project to the needs and context of the journalism communities in Central and Southeastern Europe?
• In what ways has the project demonstrated efficiency in resource utilization and implementation?
• What measurable impacts have been achieved by the project, and how sustainable are these impacts?
• What external factors, beyond the control of Internews, have influenced the achievement of project results?
• How have these external factors impacted the overall success and challenges of the project?
• To what extent are there indications that the program’s results will last after its completion?
• What recommendations can be made for future interventions in similar contexts or with similar objectives?
• What were the experiences and feedback from participants in scenario-based trainings held in Albania, Bulgaria, and Croatia? Did these vary along the lines of country or gender of participants?
• What are the specific successes and stories of select project fellows that can be highlighted in the case studies?

Evaluation Approach and Methodology

The evaluator has the freedom to utilize any relevant evaluation approach with relevant quantitative or qualitative methods that would deliver on the objectives of the evaluation. A blend of both primary and secondary data is recommended.

The evaluator should, at a minimum, in their methodology, include the following data collection methods:

• Document review: Thorough review of pertinent program-related materials, including but not limited to, monthly, quarterly, and annual reports narrative reports, M&E reports, survey results, and content produced by the program.
• Data research: Thorough review pertaining to relevant external sources to ensure appropriate level of knowledge of the context of the project.
• Qualitative Interviews (Focus Group Discussions and Key Informant Interviews): The evaluator is expected to hold interviews and meetings with relevant staff of the program, partners, and beneficiaries
• Online Survey with fellows to be able to collect quantitative information [recommended but not mandatory]
• Evaluators, proposing the integration of specific evaluation approaches such as Most Significant Change, ripple effect mapping, Outcome Harvesting, Outcome mapping, etc., would be advantageous.
• Proposals which include a quasi-experimental approach to Evaluation Question #3 (see previous section) will be given preference. However, this is not a required component if the overall proposal is strong.

Estimated Timeframe and Budget

The final evaluation report should be submitted June 21, 2024. The evaluation will comprise of an estimated 30-41 days of work and involve the following elements (please note that deadlines may be subject to change):

REQUIRED TASKS

ESTIMATED Level of Effort (DAYS)

Intro meeting to set goals and priorities

1

Draft and submit of inception report including evaluation framework with Internews template provided

1-3

Conduct desk review

4-5

Finalize evaluation design and methodology, and along with preparation of data collection tools

2-4

Conduct data collection

6-8

Conduct data analysis

6-8

Debrief Internews team on preliminary findings

1

Update the initial evaluation outputs and prepare the final outputs

7-8

Finalize and submit the final outputs based on Internews’s comment

1-2

Final debriefing on key findings with the Internews team and the donor

1

Expected Tasks and Deliverables

• Submission of an inception report outlining detailed evaluation methodology design.
• Conduct a comprehensive desk review that involves reviewing programmatic documentation produced by the project team to grasp the structure, evolution, and needs of the project. Documentation may include quarterly reports, research reports based on interviews and focus group discussions, notes from meetings with partners and fellows, project M&E mechanisms and reporting, funder requirements, etc.
• Conduct data collection, which will include the preparation of data collection tools, conducting interviews, focus group discussions, and/or other data collection activities with project staff and partners, fellows, and other stakeholders in the project countries.
• Conduct a comprehensive data analysis by bringing together the relevant data collected in the previous phases to respond to the evaluation framework and develop strategic and operational recommendations. Discussions with partners and our team will help confirm, challenge, or contribute to additional insights.
• Submission of a final evaluation report (25-40-pages (not including appendices)) in English with findings and recommendations. The body of the report will provide a narrative description of results achieved, operational lessons, and recommendations. Executive summary or similar document should be included. Where appropriate, it will also include illustrative graphs and charts or infographics that provide useful summaries of project data. Appendices will provide detail such as cross tabulations from reviewing quarterly reports and interviews, as well as a list of people interviewed, and survey results (if any), so readers can clearly see the data that supports the findings and conclusions.
• Submission of report on the experiences of participants from scenario-based trainings held in Albania, Bulgaria, and Croatia. This report should be separate from but accompanying the final evaluation report. It should be based on interviews and/or focus group discussions with training participants (aim for at least 5 participants from each country).
• Submission of at least three ‘spotlight’ case studies which showcase the successes and stories of select project fellows for funder and external audiences. These should be separate from but accompanying the final evaluation report.
• Presentation and debrief with Internews staff on the evaluation findings and recommendations.

QUALIFICATIONS WE’RE LOOKING FOR

Required

• Demonstrated experience conducting evaluations of advocacy and human rights programs, particularly for US donors.
• Experience with qualitative and quantitative M&E data collection and analysis methods
• Strong facilitation, presentation, writing and communication skills.
• Fluency in both written and spoken English.
• Commitment to conducting work in a way that fosters Belonging, Dignity, and Justice (BDJ), particularly in engaging directly with Internews partners, staff, and consultants, as well as any other individuals involved in the evaluation process.

Preferred

Note: Candidates who do not have these preferred qualifications, but who are interested and willing to learn, are encouraged to apply.

• Experience in the field of digital rights and internet freedom.
• Formal education (e.g., bachelor’s degree, graduate degree, professional certification, etc.) in a related field.
• Experience working in or with communities in Central and/or Southeastern Europe (specifically Albania, Bulgaria, Croatia, Czech Republic, Hungary, Poland, Romania, Slovakia, Slovenia).
• Fluency or advanced proficiency in at least one commonly spoken and/or minority local language spoken in Albania, Bulgaria, Croatia, Czech Republic, Hungary, Poland, Romania, Slovakia, and/or Slovenia.

To Apply

Applications will be reviewed on a rolling basis with preference for applications submitted by January 31, 2024.

Applications should include the below documentation:

• A 2–3-page statement of interest/technical proposal, describing how your profile meets the qualifications outlined above and highlighting how your methodological approach will address the scope planned for this evaluation.
• A short cost justification (not included in the 2–3-page limit)
• A CV or resume
• Evidence of evaluation work (e.g., link to evaluation reports or PDF attachments), highlighting evaluation theories and approaches.
• If you move forward in the selection process, be prepared to share the names and contact information of at least two references (these should be clients for whom you have conducted external evaluations previously)