No announcement yet.

Update Brief from Working Group on Impact Evaluation

  • Filter
  • Time
  • Show
Clear All
new posts

  • Update Brief from Working Group on Impact Evaluation

    Update Brief from Working Group on Impact Evaluation
    (Mallika, Djelloul, Suzanne Blog)
    1. Background on what IOCE has been discussed on the topic of Impact Evaluation in the past:.
    Date Member Comments and Observations made
    7 April Jim Jim sent 2 versions of the IOCE Statement on Impact Evaluation. One that went through many edits during 2011 (this one by Bob Picciotto). One that Marco presented at NONIE in 2012.

    Burt Perrin wrote a statement on IE that was officially adopted by EES

    Jim has also collected IE-related websites that I collected (also in 2012)
    DfID Working Paper on Broadening the Range of IE Designs by Elliot Stern and others. Elliot and his colleagues were asked in Mar 2014 at the AfrEA's Yaoundé Conference to develop a more “user-friendly” version.

    Important collection of documents related to the series of NONIE meetings, No longer NONIE secretariat
    1. Two divergent viewpoints: Move forward or refrain from resurrecting the old IE Working Group?
    8 April Suzanne Blogg Suzanne proposed a minor editing on the title of the IOCE paper.
    JIm You may decide to draft an updated statement to subsequently be presented to the current IOCE Board for consideration.
    9 April Suzanne Blogg Best way forward is to start with the aims, title and subtitles of the paper which we can all then work on, using the draft to give us some direction?
    Jim replying 2 Suzanne Jim does not think IOCE should bother to try to “resurrect” the IE working group. He pointed out that a huge four-part (OECD-DAC, International Banks, UN agencies, and IOCE representing VOPEs) NONIE coalition was deeply engaged in this topic for about 8 years, with 3-4 big international conference. It has further “run its course” and basically no longer exists – the last “event’ being hosted by IOCE during the AfrEA conference in Yaoundé in March 2014, attended by only a handful of people.
    12 April Suzanne Suzanne was wondering whether we just drop the idea of reviving the IE Working Group
    Djelloul in reply 2 Suzanne It is not question at all to renounce to contribute to the world of impact evaluation! Let's be constructive and see how you, Mallika and Jim think for moving forward to meet the EPMG expectations by assigning to us this assignment on IE.
    13 April Mallika There is no need to drop it at this time as we are still in the process of getting clearer understanding of the concept, methodology and the process of IE. The utilization focus and change overtime of the different stakeholders particularly on the lives of the beneficiaries, hold validity for continuing the dialogue in the context of sustainability of development interventions.
    May 11 Suzanne In reply to Djelloul's messaging dated May 10 and seeking the views of the WG members on the draft he prepared for being posted on the IOCE Board discussion forum:
    · Malikka felt that we should create opportunities for sharing experience in the use of IE as the way forward for the WG. However, since a lot of effort has gone in for IE already as Jim has summarized, Malikka proposed to leave it open for the EPMG for the decision.
    • Suzanne informed that she resigned some time ago from the position in the WG. She further recommended to refer back to Dorothy.

  • #2
    Djelloul is looking for feedback on this topic. Please provide anything input you have here in the forum. thank you.


    • #3
      The last AfrEA conference was "co-sponsored" with NONIE -- but there was no institutional home for it -- so impact evaluation ended up being a strand. These days I would suggest that 3ie might actually the "impact evaluation" home, now that they have acknowledged RCTs are not the only way to establish impact and that mixed methods approaches are sometimes more suitable.


      • #4
        I would suggest that if you are still working on the concept note, you may stress there why do you think this is the momentun for having a WG on Impact Evaluation. I tend to think that the controversy over methods has declined, and I can´t see the role of an IOCE group on this topic now. But perhaps I´m blind to it!


        • #5
          I don't know how useful or relevant the following may be to the discussion at hand nor the extent to which it may be construed as a rant but anyway here goes:

          Impact evaluation is a rather meaningless term loaded with the political baggage that characterised the dynamics of the relationships between the political instigators of this "movement", e.g. the "when will we ever learn" sales people, and evaluation communities of practice who, on seeing some of the patent nonsense being spewed from the randomista pulpits, engaged in the debate with a view to bringing some measure of realism and scientific honesty to the false promises that were being made. As Bob Williams has well pointed out, this has always been mostly about politics and not about methodology.

          Having said that, I question the etymological foundation and coherence of typifying evaluation on the basis of a partial and narrow definition of value, itself contained within a partial and narrow choice of construct, e.g. usually represented by linear causal performance models. Why not then have outcome evaluation, or effect evaluation, or other nameplates along the various artificially named stops in the "production process"? The lack of intellectual effort and of conceptual clarity, as well as the recklessness with which terms of art are used, concern me greatly as it is in these aspects that lies scientific rigour and not in the confused and baseless "rigorous evaluation" or "rigorous impact evaluation" monikers.

          Because there are other evaluation typologies as well, e.g. program evaluation, policy evaluation, etc. where the categorical anchor appears to be in the nature of the evaluand (itself a category). And more. Or are we clear and comfortable mixing and matching, e.g. gender responsive and equity based real time process evaluation of policy outcomes and effects, intended and unintended, positive and negative, in post-conflict settings.

          So this is what it looks like to me on this rainy Thursday afternoon in Paris.