Agenda – New York City

The NYC Deep Dive brings together 40 thought leaders and decision makers who are using technology to increase M&E impact in development.

Wednesday, October 1st

4.00-5.30pm

Salon:  Using mobile data for better impact and scale
Adam Wills, GSM Association M4D Impact
Invitation only

5.30-8.00pm

ICT4Drinks – M&E Edition, hosted by ThoughtWorks.
Join the New York City ICT4D/M4D/M&E Community for casual drinks and conversation. This event is open to anyone, but we have a space limit so be sure to RSVP here.

Thursday, October 2nd

9.00-9.30am

Welcome and coffee
Nancy MacPherson, The Rockefeller Foundation

9.30-10am

Moving ICT-enabled M&E forward 
Kenneth Chomitz, World Bank and Linda Raftree, Kurante
ICTs are being used in new and exciting ways to keep pace with the changing and increasingly complex dynamics of development evaluation. Yet these new approaches and tools bring with them methodological limitations and new challenges that need to be overcome. Rather than sticking our heads in the sand, the time has come to address the barriers head-on in innovative ways that can advance the field. What can we do as a sector to move evaluation forward and embrace these new tools and approaches? What ideas and issues came up at the M&E Tech Conference the previous week in Washington, DC? How can we use the next 2 days to advance the wider field?

10.00-10.30am

Lightning Talk 1: Organizational readiness to adopt ICT for M&E
Kerry Bruce, Pact
What did research with staff from more than 40 INGOs and participants from over 50 countries tell us about organizational readiness to adopt ICT for M&E? These highlights in a nutshell can help your organization to adopt new approaches more effectively.

Lightning Talk 2:  Can low-tech really deliver high-value, real-time M&E results?
Yvonne MacPherson, BBC Media Action (TBC)
Not only are simple mobile handsets helping BBC Media Action encourage behavior change in India, they are providing real-time data that is used for ongoing project strategy changes.

Lightning Talk 3:  Using Open Data Kit for a governance accountability measurement system in Mali
Abdoul Karim Coulibaly, The MasterCard Foundation
A decentralized governance system introduced in 2002 in Mali failed to deliver tangible results due to poor governance, lack of accountability and poor citizen engagement. This talk will share AFRIK-POLL’s experiences, findings, benefits and challenges when using Open Data Kit (ODK) and Google App Engine, to develop and pilot the Governance Accountability Monitoring System (GAMS), which uses composite index indicators including citizens’ assessment of their leaders’ performance; communes’ performance against key governance benchmarks.

Lightning Talk 4: Mobile Surveying 2.0 Version – Developing panel surveys through the mobile phone
Amy Sweeney, GeoPoll
In collaboration with global market research companies, GeoPoll is taking mobile surveying to the next level. Rather than one-off mobile surveys, GeoPoll has developed panels through the mobile phone to conduct longitudinal studies to see how changes over time affect groups. GeoPoll will share how and why this private sector approach can be applied to development projects.

10:30-10.45am

Morning break
Connect with your friends and peers in M&E and technology and reflect on the morning’s inspirations and ideas

10.45-12.45pm

Workshop 1:  No-tech and low-tech last-mile participatory M&E
Megan Colnar and Fernando Gaal, The Hunger Project 
Are ICTs useful in communities that have limited access to technology? Can they support participatory M&E approaches? When and how should ICTs be integrated in participatory M&E and at what levels (participant level? staff level?)? This session will involve small groups learning about and trying out some participatory M&E methods and identifying when, where, how and if ICTs could/should be integrated.

Workshop 2:  Transactional data: how operators leverage data to deliver a better service and what it means for M&E
Adam Wills and Caroline Sheldon, GSM Association
GSMA will first aim to demystify data and illustrate how M4D managers sitting within mobile network operators can use transactional data to better understand and serve customers. We will then explore how such exercises can influence the development of M&E approaches.

12.45-1.30pm

Lunch

1.30-3.00pm

Discussion 1:  Should we be using ICTs for M&E in constrained or high-risk environments?
Rahel Dette, Global Public Policy Institute; Gwen Bevis, Management Systems International; Antoine Heuty, Ulula; Anna Levy, SIMLab and Diana Chigas, CDA Collaborative
How should we be using ICTs for M&E in constrained or high-risk environments? Should we be using them at all? This session will explore the tensions and lessons of using ICTs in high-risk environments. We’ll consider a range of platforms (including mobile and mapping), sectors (including extractive industries, humanitarian, development and peacebuilding operations), and human rights protection efforts, as well as diverse geographic perspectives. Are some technologies more useful or risky according to the type of ‘high-risk’ environment? What data quality issues, technical and non-technical, are particular to high risk environments and how can we address them throughout project life cycles? The conversation will draw on the insights and experiences of all in attendance, focusing on challenges and opportunities, various conflict- and risk-sensitivities, navigating tradeoffs, and applied lessons learned.

Discussion 2:  How can we build organizational readiness for ICT-enabled M&E?
John Feighery, mWater; David Garrity, GVA Research and Kerry Bruce, Pact
How can we get our organizations to adopt ICT for M&E technology? What are the barriers and what are the best practices? Join us for a World Café session where we investigate how we have all created momentum for organizational buy-in.

Discussion 3:  Are ICTs an effective tool for qualitative M&E methods?
Emily Tomkys, Oxfam GB; Lynnae Day, Oxfam America and May Miller-Dawkins, CoreLab
Are you wondering when and how ICTs can be used for qualitative M&E? Do you have experiences you want to share or reservations/concerns about their use? Come along and join our discussion on when ICTs can and cannot be an effective tool for qualitative M&E. We will be exploring the ethical issues and considerations as well hearing about new evaluation methods that technology can enable. This is a new and exciting area that organizations are beginning to think about so join us to kick start this conversation and share your ideas.

3.00-3.30pm

Afternoon break
Connect with your friends and peers in M&E and technology and reflect on the day’s learning

3.30-4.45pm

How-To Session 1:  Developing relevant, quality frameworks for evaluating ICT4D and M4D efforts
Kenneth Chomitz, World Bank; Anna Levy, SIMLab and Linda Raftree, Kurante
This session will be a participatory exploration of relevant frameworks for better understanding and measuring the impact of ICT4D and M4D programs and initiatives and the specific contribution of ICTs to program impact. Some of the areas we hope to explore and ‘crowd-source’ from participants include ideas and examples of: What methods and approaches can help to better isolate and understand the impact of using technology in different kinds of programs? How can small programs with small (or non-existent) M&E budgets determine whether integrating technology into their work is proving useful? How can we better understand the impact of a particular technology/digital tool/digitally-enabled approach on wider outcomes like improved health or education? How can we measure the organizational change that a digital system, tool, application or platform is having in terms of staff behavior, cost savings, decision making, budget allocation and other areas?

How-To Session 2:  Build vs Buy: Using ICT for Organization-Wide Data Management
Jamie Lundine, Spatial Collective and William Lester, NPOKI
Does your company or program operate in multiple countries with various stakeholders who have different data needs? Have you solved the problem of collecting the data you need across multiple projects, programs, organizations and countries? Have you struggled with Buy verses Build? How do you manage data once you have it? If these questions and/or challenges sound familiar, this session is for you! We will use a case-study approach to examine the experiences of organizations using ICT (or thinking about using ICT) to manage both operational and field-level data.

How-To Session 3:  What is “real-time” data, why do we need it, and how could we use it if we had it?
Wayan Vota, FHI 360 and Rob Baker, Ushahidi
We often claim to want “real-time” data, but what does that mean? How can we be sure of its quality? Do we even have the systems and organizational culture to make real-time decisions based on it? And finally, what tools could we use to be more “real-time” in our existing processes?

4.45-5.00pm

Wrap up and close

Friday, October 3rd

9.00-9.30am

Welcome and coffee
Adam Wills, GSM Association

Summing up Day 1
Facilitated by Wayan Vota, FHI 360 and Linda Raftree, Kurante
What key points from the first day resonated the most with participants? Were there any broad commonalities in terms of opportunities and challenges? What gaps in knowledge, practice and policy or standards need filling? What should we be thinking about as we enter into the sessions on Day 2?

10.00-10.30am

Lightning Talk 1:  Gamification of NGO performance management
John Hecklinger, Global Giving
NGOs around the world struggle to conduct M&E that improves their organizational and programmatic effectiveness.  GlobalGiving will describe their “Effectiveness Dashboard” that links effective M&E with financial incentives to encourage nonprofits to continuously improve their social impact.

Lightning Talk 2:  Conflict reduction in extractives through realtime community feedback
Antoine Heuty, Ulula
Ulula is establishing community-based SMS feedback systems in areas where the extractive industry is present and monitoring them to detect unrest before it potentially erupts into violence. Using this information, companies can adapt their social programs to meet the needs and demands of workers and their communities. Ulula is partnering with Innovations for Poverty Action to set up a randomized control trial to measure the effectiveness of the program across four mining areas and a potential reach of 100,000 people. This is one of the few early pilot programs investing in using RCT to assess the effectiveness of ICTs to prevent social conflicts and foster community resilience.

Lightning Talk 3:  Using SAVIX to measure Village Saving and Loan Association (VSLA) performance
Abdoul Karim Coulibaly, The MasterCard Foundation
To assess the performance of Village Saving and Loan Associations (VSLA), the SEEP working group on Saving Groups developed a set of standard ratios looking at members’ satisfaction, financial performance and operating efficiency. At the initial stage (in 2006) the data were entered using an EXCEL platform with automation macro to compute the key ratios at group level, field officer and program level. The system has evolved progressively, and as of 2013 is uses a direct online reporting and analytic system with enabled mobile phone data collection application. This system, called SAVIX, was developed by VSL Associates and the Software Group and Funded by Bill and Melinda Gates Foundation and The MasterCard Foundation.

Lightning Talk 4:  9 Principles for Better ICT in M&E
Eric King, USAID Global Development Lab
Multiple major donors have signed on to the Principles for Digital Development, translating the Greentree Consensus into practical action. What does it mean for ICT and M&E?

10:30-10.45am

Morning break
Connect with your friends and peers in M&E and technology and reflect on the morning’s inspirations and ideas

10.45-12.45pm

Workshop 1:  M&E of complex interventions in the age of ‘big data’: obsolescence or emergence?
Emmanuel Letouzé, Data-Pop Alliance; Brian Dalessandro, Dstillery; and  Bridget Sheerin and Evan Waters, Thoughtworks
We’ve all heard the term “big data”, but how to use it in M&E is still a big mystery. Can big data help with M&E of complex interventions? Will it make M&E as we know it obsolete? Or is big data just another trend whose bubble will burst? We’ll explore what Big Data is, how it might apply to M&E, what pitfalls it brings, and how we can start to experiment with it to realize its promise.

Workshop 2:  Whose data? Whose privacy? What can M&E practitioners do to ensure ethical standards and mitigate privacy risks when using ICTs for M&E?
Jennifer Schulte, Independent Consultant and Linda Raftree, Kurante
The nature of data changes when it is digitized,  and new risks for privacy breaches, retaliation and harm arise. M&E practitioners and those who decide on and commission M&E activities need to better understand the emerging risks to their organizations when digital data enters the picture. In addition, and more importantly, they need to be aware of potential risks to their program participants. This session will explore the emerging challenges that arise as we move towards M&E data becoming fully digital. We will take participants through some of the ethical considerations, the potential privacy and security risks and ways to mitigate them, share tools that can help to overcome risk and engage program participants in greater control of their own data and their own risks, and provide tools and resources for further exploration of this critically important aspect of ICT-enabled M&E work.

12.45-1.30pm

Lunch

1.30-3.00pm

Plenary panel:  Methodological challenges in M&E. What can ICTs resolve? What new challenges do they bring?
Kenneth Chomitz, World Bank; Lindsey Shaughnessy, Innovations for Poverty Action, and Julie Peachey, Grameen Foundation; moderated by Nancy MacPherson, The Rockefeller Foundation
ICTs have huge potential to help us to overcome some of the existing operational and methodological challenges that evaluators face, yet there are plenty of questions that arise when it comes to the soundness of some of these new approaches. What should evaluators be doing to ensure that integration of new ICTs into their processes do not taint or invalidate the findings? What can the field as a whole do to provide guidance in this area?

3.00-4.00pm

Matchmaking and afternoon break
Connect with your friends and peers in M&E and technology, hold your side-meetings, follow up with interesting people and ideas

4.00-5.00pm

ICT-enabled M&E:  Where to from here?
Facilitated by Wayan Vota, FHI-360 and Linda Raftree, Kurante
We’ve now had a full four days (two in Washington, DC and two in New York City) with some of the top thinkers and doers in the field of ICT-enabled M&E. We’ve debated and learned of a whole range of new ideas, new possibilities and new challenges. What should we be doing as a sector to take ICT-enabled M&E to the next level? What leadership, standards, guides, good practice and incentives need to be in place to do so?

5.00-5.30pm

Wrap-up and close