Over the last few years there has been a growing drive for development organisations to make better use of technology in their monitoring and evaluation (M&E) systems. This topic has featured at conferences, courses, online discussions and in a number of reports. However, I've seen less discussion or guidance to describe the different types of monitoring and evaluation technology, how they fit into an overall M&E system, and how they relate to the different needs of an organisation.
Coming from the programme management space, I bring a very specific perspective to this question. Hence, I thought it might be useful to set out some initial ideas and questions in this blog and then see what views others have. Hopefully this discussion will help create the beginnings of a typology of M&E technology, which in turn can help organisations find options that best fit their own specific needs.
What areas should an M&E system cover?
Before considering the technology, what are the core areas that a good M&E system should cover? Starting with a framework makes it's easier to see which areas can be supported by which technology. This in turn helps development organisations to think about where they would get the greatest return for their investment.
There are many articles and guidance documents describing how to set up an M&E system. Depending on your viewpoint this is interwoven with the project activities. While I prefer the following approach from the adaptive management school there are many alternatives, so see this as one example only.
Planning - What is your project or programme trying to achieve and how will it do so? There are many approaches to planning and many tools to help you do this. At the simplest level this might include:
- A set of activities with a budget that will result in a number of outputs
- A set of assumptions that explain why those outputs will result in outcomes (ie broader social changes)
- A list of things (often called indicators) that you will measure to track whether the activities, outputs and outcomes are happening or not
Implementation and monitoring - The stage of implementing the project activities and measuring what you planned to measure. A good M&E system will include a clear plan that sets out:
- Definitions for what you will measure
- How often it will be measured
- Who is responsible for measuring it
- What action is needed to assess data quality and follow-up on issues
This is quite simple on a small project, but gets more and more complex as number of activities and the project team grows. When working on larger programmes, there are additional management related activities that the system should address, including:
- Are activities being implemented according to the plan?
- Is monitoring data being collected on time by those responsible?
- Are any parts of the programme blocked and in need of more support?
Analyse, reflect and evaluate - This can be the hardest stage, as the time needed to collect data can limit the time available to use it. This is further compounded if the data has been collected in a way that takes time to process and analyse it. This could mean transcribing data into a spreadsheet or cutting and pasting between files to aggregate data.
Clearly the questions you want to answer with the data collected should tie back to the goals and assumptions set out in the planning stage. This way the data you are collecting is used to regularly review your assumptions and assess if a change in course is needed.
Update plan based on new knowledge - Based on reflection from the data collected and analysed, what changes are needed in the activities? This feeds back into the planning cycle and the process begins again.
Types of M&E technology
With a framework set out, let's see how the types of technology out there can support these areas. Here is where I can offer a starting draft only. There are bound to be things I have missed that or don't know about. Please let me know in the comments so I can edit the post if something important is missing. Clearly many options overlap several of these categories.
(1) Indicators, dashboards and performance
From my experience this is by far the most common focus area for M&E technology. It includes tools focused on high-level performance related data (typically indicators) and ways to collect this data (either by directly entering the indicator values or via surveys). Related tools make it easy to visualise performance data in ways that help donors, managers and stakeholders to interpret it. Tools in this area tend to focus more on the high level data needed to measure performance. There is typically less or no focus on the management of the project or programme activities.
Tools to define and measure indicators. This could include anything from:
- Defining indicators that the project or programme will measure
- Collecting data for each indicator for each reporting period (ie a month or quarter)
- Assigning responsibility for indicators to specific people
- Dissagregating how data for an indicator is collected (ie by age range, gender or project site)
- Forms to collect the data needed for each indicator
Common to these tools is that they typically support the indicator and montioring part of M&E systems by providing a clear framework to define which indicators are being measured and to help collect the data needed to measure them. Sub-categories include:
Survey tools have a wide range of applications. However, in the M&E space they are most often used to collect data needed to measure indicators. These tools could include the following:
- Tools to design new or modify existing forms
- Option to link survey data to indicators
- Mobile app to collect survey data using tablets or smartphones
- Option to collect data without an Internet connection and syncronise back to an online database
Survey tools also often focus on the indicators and how to monitor them. They can help M&E systems by eliminating paper forms, which speeds up data collection and reduces transcription errors.
Analysis and visualisation
Analysis and visualisations tools focus on one specific area of M&E systems. If you choose a standalone tool you should consider also how to get data into the tool. Typical features include:
- Pivot and other tools to cross-tab data
- Charts, maps and other options to visualise data
(2) Activity or project level management
This category of tool focuses on the opposite end of the spectrum. It looks at how teams implementing projects collaborate and manage the project. Teams on smaller projects might use email, Word, Open Office, Excel or Google Docs. Larger teams might use project management software to help. In specific areas (as per those below) there is also specialist software designed for managing specific types of activities or projects. Such tools are often used in combination with the kind of M&E tools discussed above.
Project management tools often focus on facilitating collaboration among a team implementing a project. They are typically open-ended in the way they work, making it easy to add tasks, events, documents and share notes with team members. Such tools typically include:
- Option to quickly create new projects and invite team members
- Ability to create tasks, documents, events, photos and videos
- Discussion tools (sometimes email-based)
Project management tools can (and are) used to support all areas of the M&E system. While they can help with coordination, they are less good at collecting data in structured ways that make analysis and reporting faster.
Case management tools are designed to help coordinate services provided to people who may be considered a case in the context of social programmes. This could cover health care, nursing, rehabilitation, social work, disability insurance, employment, and law. This space overlaps to an extent with existing customer relationship management tools. These tools could include the following:
- Centred on recording information about an individual
- Open, manage and close 'cases' for individual contacts
- Track relationships between contacts
- Tracking relationships or milestones between people and goods or services that they have received
- Data quality tools to avoid duplicate contacts and hence avoid double counting
Case management tools focus more on tracking activities related to individuals and use this data to enable reporting on indicators related to indivudals. They can help M&E systems by providing robust ways of tracking interactions with individuals.
Facility management tools are typically specialised around particular types of organisations, like health facilities, schools or municipalities. They are designed with the needs of these organisations in mind and focus on collecting data needed to manage and report on the activities of that organisation. They might include:
- Staff management tools
- Patient or student tracking tools
- Tools to track health, education or other services provided
- Reporting tools that make it easy to see the data collected
Facility management tools also typically operate at the activity level. As with case management tools, they make it easy to aggregate data to measure indicators related to facility activities.
(3) Programme management
Programme management tools are often also referred to as management information systems. They typically cover both ends of the spectrum, with a focus on the management, M&E and process side of the programmes activities. It's the process dimension that makes these tools interesting (and I would say essential) when working at scale.
The process aspect brings an understanding of the steps followed when implementing activities on one project or site as part of a larger programme. If well implemented this can incorporate both good practice from the programmatic perspective and good data quality practice from the M&E perspective.
Programme management software typically include:
- Tools to easily create custom forms to collect any kind of data
- Strong ways of tracking programme participants (similar to case management software)
- Taxonomies or other standardised ways of cross-referencing data
- Workflow that guides project staff through the activities required to implement the programme
- Option to use data collected to measure programme indicators
- Data warehouse tools to create custom reports or dashboards
- Operational indicators that track implementation of the programme and identify sites or projects that are behind schedule
Programme management tools typically focus on collecting data at the activity level and aggregating this to measure indicators. This gives a much larger and richer source of data then would be available from indicators.
Are there any other categories that you know of? Please mention them in the comments below so I can incorporate into this post. If this way of looking at M&E technology is useful, then an obvious next step would be to map existing tools to relevant categories. This typology of M&E technology would offer a useful starting point for organisations to map their M&E needs to the tools available.
Where does Kwantu fit into this picture?
Our focus is specifically on the programme management (or management information system) area. We're best placed to help those running programmes that are implementing the same type of activity across a number of sites or countries. We've invested in developing BetterData, a fully configurable platform that makes it possible to create a custom system for your programme at a fraction of what it would normally cost.