Quantizing Development
September 20th, 2006
Arriving back in Banda Aceh, it realized it really was my second home. It’s not only familiar and comfortable here, but beautiful. The palm trees, the rice paddies, and the people. The Acehnese people are just so friendly. Walking down the street I am greeted with confetti of smiles and “Hello Misterâ€s. And I was coming back to all the friends I had made while I was last here.
In Banda Aceh I am working on a Monitoring and Evaluation (M&E) database. I have talked about this before in my blog, but I’ll go over it again, and try to relate it to my quantization rant. I’m no M&E expert, but this is the way I see things:
Development is a tricky, complex type of work, which is different than emergency relief. With emergency relief it is a matter of getting food, water, shelter and medical care to people to met there basic needs. It is often a matter of life and death. It is much easier to measure this sort of work. Development is about building communities and increasing the capacity of the people, generally trying to improve the quality of life for people. Development NGOs will have a long term aims. They will carry out various activities designed to have certain outputs. These outputs will hopefully which achieve certain objectives, which fulfill the overall aims.
In each step of the process, numerical indicators are used to measure the progress. The entire development process is quantized down into a number of defined variables on some matrix.
Now, this is only part of M&E, there are other reports which gather which contain more qualitative information, to minimize the quantization losses. But as an IT guy, I’m focusing on the quantized information. Even though this information will loose some of the big picture, I think that it is vital to have quantitative information in order to analyze the performance of the whole process. With numerical data, it is very easy to assess if targets are being met, to compare similar activities, and generally determine how efficient the activities.
My challenge is to design a database, which captures all this information, and automates as much of the analysis as possible, without forcing further quantization of the data. I think I have managed to come up with a design, which is suitably flexible, and gives the users freedom to enter their own indicators and variables. This means that they are not limited by the design of the database, and changing information requirement can be represented.
I’ve attached the specification for anyone who is interested. If you happen to also be working in M&E and find this useful please let me know.
M&E Database Specifications.doc
Now the challenge is to work with the local programmer and actually develop it!
September 27th, 2006 at 9:26 am
Glad to hear you’re enjoying being back in Aceh, Mikey, and enjoying your recent posts – you are doing well explaining your work to mere non-engineers like myself.
Your work in Aceh reminds me of my economist father’s defence of analysing complex information by quantifying it into monetary terms. There will always be information lost, square pegs and round holes and all that, but it’s the best way of comparison and analysis for a lot of otherwise difficult information.
Best, K
September 28th, 2006 at 10:54 am
Hi–I do work in M&E and understand your frustrations! Quantifying capacity building and policy changes are a big part of my work, and I often find myself wishing I was on a rural health project or something else that dealt more with solid numbers, rather than murky qualitative measures. That said, thanks for sharing the database specs–it is indeed useful!
September 29th, 2006 at 2:01 am
Thanks for the comments.
Just a thought about the whole quantitative vs. qualitative information, and how to capture it in a database:
More advanced technology allows databases to capture and organize larger amount of information. This blog is all stored in a database! I can add different posts, photos even. I add categories to sort the posts by. You can even search through my post for key words. Admittedly my blog doesn’t really deliver any “metricsâ€, but that was never my intention. However it wouldn’t be hard to write some software, search for key words, and do some simple analysis.
I think that the ultimate M&E database would contain a combination of quantitative numerical data, linked with qualitative information. So that you could enter the information that on the 29th of September a training was held for 100 teachers in Banda Aceh, but you could also include photos and a narrative report about the training. That way you can capture a more complete picture. Also, by allowing the user to enter any field for which to collect quantitative information on, there will be more numeric information to analyze.
Just my thoughts on the subject!
October 4th, 2006 at 9:15 am
I think your vision of the “ultimate M&E database” is entirely possible – in fact, it’s already being done by some organisations, although not necessarily in our sector. I find the quantitative / qualitative divide frustrating, because I think in some cases it’s been used as an excuse for poor M&E.
The important thing for monitoring (for example) teacher training is not the photos or even the narrative report. Neither of those will tell you if the training was successful, only that the training took place. It’s the attitudes and actions of the teachers which will illustrate whether the training was successful; both of those can be measured quite accurately using a variety of quantitative methods – it’s just that we’re not very good at those kinds of measurement.
October 4th, 2006 at 11:18 am
What I am imagining is a database to collect all information related to a program. M&E would be linked with information which is collected for donor reports and the media.
Longer term impact surveys could be linked to various activities, making it easier to piece together all the various factors into a more holistic picture of the program.
October 5th, 2006 at 6:47 am
Probably more important than recording photos and narratives, would be a custom survey options.
Within the database there would the option to generate custom surveys, which would be linked to activities. The user could select what they want to measure in the survey, what questions they want to ask, then the results would be fed back into the indicators for the activity.
Of course in the long term it may be possible to do complex computer analysis on the raw qualitative data, to extract quantitative information. Could an algorithm be used to scan a narrative description of a teacher training to determine how successful it was? Are there image processing programs out there which could determine how many children were smiling in a photo? Maybe I’m getting carried away…
October 5th, 2006 at 8:46 am
We use customer service reviews quite a bit–they are useful in measuring institutional capacity.
Training is tricky. There are the standard output indicators (# of students, etc) but the result or outcome can often not be fully assessed within a project lifespan. One would need teacher performance reviews, or some measure of how training was applied, to gauge the eventual success.
The database sounds great. A lot of work, but no doubt very useful.
October 6th, 2006 at 8:07 am
With any IT project training and implementation is a big part which is sometimes overlooked. You can have the best database out there, but if people don’t know how to use it, or don’t see the value in it, it’s worthless!
Also this post has been linked from Paul Currion’s blog: http://www.humanitarian.info. Paul’s done a lot of work in IT in the humanitarian sector, and has some interesting related info on his blog.
October 6th, 2006 at 1:31 pm
Like I said, the database you describe is entirely do-able, the real problem is getting staff to use it! I did a database in Afghanistan for water points for an NGO where we included timestamped pictures of the points – very useful when you haven’t seen the thing for two years between maintenance trips – but it never took off.
The other problem I’ve found is that NGOs tend to emphasise qualitative at the expense of quantitative, for two reasons:
1. They’re not in the habit of quantitative thinking, and you don’t have to think about qualitative data so much.
2. Qualitative information can be used for raising funds, while quantitative data forces you to examine your programmes.
I might be cynical after too many years staring at the sun, but would be interested to hear what others think.
October 13th, 2006 at 6:09 am
I can relate to what your database in Afghanistan. I think that every IT project has 3 stages: Specifications, Design and Implementation. The Design phase is relatively easy to get right, but if the specifications or implementation isn’t done right, the project will fail!
Here are some of my thoughts on the whole quant vs qual debate:
I think that the difference comes from the fact that NGOs are ultimately pursuing a qual. goal: “To save the worldâ€. (I use that phrase with a certain amount of irony, but I think that to some degree, it is an underlying feeling that a lot of people have – correct me if I am wrong).
Corporations have a very quant goal: $
I feel that this difference also breeds a different way of thinking for NGOs: focusing on people instead of systems, action instead of planning, emotions instead of facts. People/Governments donate $1000 per person in Aceh, but only $10 per person in Congo. They are more interested in seeing pictures of smiling children, than figures on literacy. I think that this applies to aid workers as well as donors.
I can do what I can to try and bring more systems, and quant. analysis to development, but I may just be fighting a losing battle against the intrinsic nature of the beast!
I also thing that this is an argument for supporting the use of private contractors, whole have their own quant. systems in place.
Another idea I’ve been thinking about is Aid 2.0. Kinda like a humanitarian version of Web 2.0. I’m going to blog about it shortly (when I get the time!).
The basic premise is that agencies will operate on a smaller scale, with more direct contact between the donors and the beneficiaries. One example is http://www.kiva.org, where people can donate to micro-enterprise schemes over the internet. This would kinda throw out the whole need for quant. analysis, which would have it’s disadvantages, but it would bring the focus back to people giving to people, which I think would have huge advantages. The Multinational NGOs would still have a place, but not so much in the direct implementation of projects. I’m probably missing some of the bigger picture here, I’m still a relative newbie to this field!
Well, that was a bit of a rant! Maybe I need to spend some more time under the sun to become more cynical!
October 15th, 2006 at 6:36 am
The problem I have always found is that UN and NGOs find it hard to articulate their information requirements, primarily because management skills and structures are weak. Which leads us onto the quant vs qual debate: I disagree with you about the reason for the difference between private and public organisations (although I think your point about the $ bottom line is valid).
I think the difference comes because of the lack of accountability within the UN and NGO agencies. Corporations are accountable (in the broad sense) to two groups – the customers and the owners/board/shareholders/etc – but the primary focus are the customers, who provide the revenue and consequently define most of the strategic and tactical decisions. In NGOs/UN, the customer is replaced with the beneficiary, who has no such influence over the organisation, which removes the need to calibrate activities to customer quality requirements. This is what leads to fuzzy thinking, rather than the fact that NGOs are pursuing a fuzzy goal…
You mentioned an “Aid2.0″ approach – here are a number of projects already looking at this. After Katrina, the Recovery 2.0 movement started (following comments by Jeff Jarvis) – at the time I predicted that it would run out of team, and sure enough it has. The problem is that, when it’s not an immediate need, most people go back to their day jobs. There’s a very small number of projects that go long-term, and even fewer that ever get a foothold in the sector.
People giving to people is a nice idea, and Paolo diMaio has been trying to get it off the ground with P2PAid. But there’s a very simple reason why UN agencies and INGOs continue to get funds, despite continued demonstrations of inadequate delivery: Economies of Scale. The transaction costs of P2P aid are just too big, even taking into account the effect of the internet in minimising them. Big organisations also have the experience necessary to do the work in a more professional manner (not that we’re all the way there, yet).
P2P aid is a nice idea, but if you had been in a car accident and somebody rocked up saying “I’m not a doctor, but I really want to help,” you’d probably tell them to eff off. I value the contributions of volunteers, definitely, but I would prefer to see a professional humanitarian sector that can deliver high quality service to the beneficiary on the ground… and that requires an approach that uses data properly to assess needs and measure impact.
October 15th, 2006 at 8:02 am
You’d know better than me, but it is my understanding that organizations are accountable to their donors. Is this not the case? But to use the corporate example, NGOs are only responsible to the shareholders, and not to the customers (beneficiaries). Some beneficiaries in Aceh who, have not had houses built, because the NGO who declared that it will rebuild in that area have not been able to build the houses yet. Other people already have houses, because their area was being rebuilt by a different NGO. I know that there a huge number of different issues affect the construction in Aceh, but the point is that the beneficiaries cannot choose its NGO.
How do we change this? Does there need to be a change in how donors operate to lead to a more “Free Market†development?
Maybe I was getting a bit carried away with the whole “Aid 2.0â€. After all “Aid†has a very broad definition. It can be the short term emergency relief to disasters, or long term social and economic development. And who are these “Aid-workersâ€? They can be social workers, journalists, engineers, doctors, economists and even computer programmers, sometimes all working together. It is a hugely complicated beastie, and I think that this is one of the problems with trying to measure it.
I understand about economy of scale. But is development work always scalable? Just look at some of the disasters in the reconstruction effort in Aceh (huge corruption and sub-standard houses being torn down). Can development really be looked at on a huge scale, or is this just a (misguided?) attempt to mirror the corporate world.
The other thing is that INGOs are increasingly partnering with local organizations, in order to work more closely with the community.
The Aid 2.0 example I used (http://www.kiva.org) allowed people to make donations directly to beneficiaries, through local Micro-finance organizations, cutting out the INGO middleman. Each donor is personally attached, so will help ensure the accountability of the local partner.
I’d challenge that the transaction costs are too high. What exactly do you mean by this? Micro-enterprise schemes such as the Grameen operate with many small transactions, and some people seem to think that they are doing a good job.
Of course specialists are needed, but sometimes local knowledge is just as important, and are large scale INGOs the best way to utilize this?
October 15th, 2006 at 9:31 am
Phew. We’re unpackaging a lot in this discussion…
Accountability. Yes, we are accountable to donors, and that’s part of the problem. Institutional donors (like governments) have political interests; private donors (the general public) have more humanitarian interests, but limited attention spans. And of course neither of them are usually connected to the beneficiaries. Another unit in the ECB project I’m working on is looking at this issue, and there’s lots of work being done around it – particularly important is the Humanitarian Accountability Partnership, which has done the most theoretical and practical work on this.
Free Market Aid. Interesting that you mention this idea. A lot of discussion after the tsunami identified that many NGOs were faced with beneficiaries telling them to go away because another NGO had made them a better offer. There was so much money floating around that beneficiaries could literally pick and choose in some places. Does it make for better aid? I don’t know, although I wrote a concept note once suggesting that camps could be managed on a marketplace principle; beneficiaries get vouchers directly from donors via the government, and then select their services from a range of UN agencies and NGOs.
“Development.” I think “development” as a concept is utterly bankrupt in both theory and practice. In theory, development is used to cover such a wide range of issues that it becomes functionally meaningless – it simply refers to people getting on with living. In practice, given that we’ve had 60 years of large-scale and small-scale development assistance, I find it hard to identify in what way it’s had any impact. Yes, people live longer (the statistic that always leads the debate), but that’s because of medical advances outside the realm of development; their quality of life seems to be not much better, and for many (in the case of Africa in particular) far worse.
Scalability etc. You’re right, I don’t think that aid needs to be the preserve of huge organisations. Grameen is a great idea, but I can’t help that feel that micro-finance provides micro-improvements. If you’re living in absolute poverty, that kind of improvement is a phenomenal lift and I think Grameen deserved the Nobel for making it work. However I’m thinking about the large-scale societal improvements that are needed to make changes at the macro level, which I think have to come from… well, decent government.
Local knowledge is critical to understand context, absolutely, although there’s a danger of fetishising local knowledge. It’s frequently very limited, as you’ve found in developing databases out there….