No announcement yet.

IOCE common principles for credentialing approaches?

  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    Greetings all, I am pleased to see evaluators from around the globe joining in this important discussion about professionalization. I would like to ask everyone "What is the problem we are trying to solve with 'professionalization' (setting aside, for the moment, the fact that it is not entirely clear we all have the same understanding of that word)?" Or stated somewhat differently "If professionalization is the solution, what is the problem?" Looking forward to hearing what folks have to say.
    Best regards to all,
    Thomas Schwandt


    • Kate McKegg
      Kate McKegg commented
      Editing a comment
      Hi Tom, I love your question... the 'problems' that I have seen discussed in the literature and at conferences that are giving rise to the professionalisation debates are, in my view, fundamentally to do with the legitimacy of evaluation, as perceived by others. For example, quality assuring, credentialing and certifying evaluators is perceived as a pathway towards improved evaluation quality - the flip-sidebeing that there appears to be a widespread perception, according to the literature I have read, especially among some funders and commissioners (although also in the eyes of many in the field) that there is too much evaluation that is poor quality. Another 'problem' I have come across is the encroachment (or flood depending on your viewpoint) of business analysts, data and impact analysts into evaluation territory. As I read the literature, professionalization is perceived as a necessary path to greater status, recognition and ultimately market share. I'm not yet convinced that embarking on the murky journey of professionalisation is necessary the answer to the problems outlined above.

  • #32
    From Donna Podems:

    In South Africa we have been engaging in this discussion, and conducted nine months of intensive participatory research. Here are some highlights-please forgive brevity---

    . We started by thinking about professionalisation however the
    conversation moved to how to strengthen evaluators
    . Some argued that we are professionalised---we have a conference, an
    association, books, courses, diplomas, etc-what we don't have is a clear understanding of who can call themselves an evaluator, or what it means to be an evaluator---not everyone can call themselves a dietitian, or an economist or a social worker
    . Regarding the problem, most agreed that we have poor quality
    evaluations (we defined "poor" in the research-e.g. not used, no methodology, no evaluation approach/valuing)
    . The research showed two reasons for this
    . (1) Anyone can call themselves an evaluator, for many commissioners
    it is not clear what constitutes a "good" evaluator, and there is "plenty of work", all leading to very little incentive to strengthen evaluation knowledge and skills, yet reasons to call oneself an evaluator.
    . (2) Anyone can call themselves an evaluator. This most often leads
    to "shoddy evaluation", research masquerading as evaluation, and often the lack of a valuing framework; people just give their opinion. This leads to people not finding evaluation useful and lowering the credibility of evaluation, evaluative processes and ultimately those that refer to themselves as evaluators.
    . Why do we need to strengthen evaluators and establish who is and who
    is not one? (for which there are many paths, explored in the research paper as well, with attention to power and other concerns).
    . For a country that is still developing, where service levels are
    often poor, and funding is finite, good evaluation can offer much needed empirical information and useful processes with which to make informed management decisions. Capable evaluators can fulfil an import societal role, while incompetent ones can be a detriment.

    The research contained many more findings---these are just some very abbreviated highlights.

    Donna Podems, PhD
    Director, OtherWISE: Research and Evaluation Cape Town, South Africa
    Cell: +27-83-712-8976
    Phone: +27-21-461-2132
    Skype: dpodems


    • #33
      This is a very interesting and important conversation. I believe that we may want to consider a borader definition of evaluation "professionalization": developing evaluation into a mature profession. One can refer to the following (pretty well-known) criteria of maturity of evaluation profession:

      1. Need for evaluation services and specialists.This is the demand side of evaluation activities and a driving force for the development of evaluation profession. Demand can emerge and grow due to internal and/or external pressures.

      2. Stable career opportunities in evaluation.Opportunity to choose evaluation as a career, long-term plans related to individual professional development and career growth in evaluation, evaluation positions in various organizations.

      3. Body of knowledge and set of skills unique to evaluation. Evaluation becomes a discipline with a unique body of knowledge that grows due to the contributions made by researchers. Development of evaluation theories. Development of applications of evaluation theories and specific sets of skills to practice evaluations.

      4. Educational programs and other professional development opportunities for evaluators. Educational programs for evaluators are conducted by state and private universities. One can get a certificate or a degree in evaluation.

      5. Institutionalization of evaluation. Evaluation becomes a part of a structured and well-established system.

      6. Professional associations for evaluators. Associations develop evaluation guiding principles / standards / ethical codes, influence educational programs, provide certification of evaluators.

      Dependning on the context (country, region), "professionalization" strategies could have different priorities and could be aimed at solving different problems. The overarching strategic goal or vision though might be common for all of us around the world: making evaluation a mature profession.


      • #34
        Dear Colleagues

        I am posting my first comment and contribution to IOCE so bear with me.

        The notion of professionalisation is a topic that everyone is talking about and yet the conversation appears to go around and around. As the Australasian Evaluation Society has a very inclusive philosophy, the idea of professionalisation can at times bring up concerns about exclusivity. So we are keen to find an inclusive model that also brings credibility to our profession.

        I would like to bring to your attention the Higher Education Academy's (HEA) model of professionalisation based in the United Kingdom. With over 70,000 registered fellows this is the most successful example of professionalisation that is incredibly inclusive. HEA have a number of levels of membership from Associate Fellow, Fellow, Senior Fellow and Principal Fellow. These fellowships allow academic and professional staff at university who deal with learning and teaching (and who usually have no teaching qualifications) to have their practice and experience formally recognised. It is based on an applicant presenting a portfolio against an agreed set of criteria at each level and includes advocacy statements from current fellows. This portfolio is then assessed by a senior expert and granted the status of FHEA or SFHEA.

        There is a cautionary note, this is a very big investment, requires a commitment, staff and a budget. However, this highly successful model addresses the higher education sectors' concern that academic staff who engage in teaching generally without teaching qualifications. This is very similar to our discipline where evaluators engage in evaluation generally without formal qualifications in evaluation.

        As the President of the AES I am really keen to begin this work at an international level or else in agreement with Benoit, let's make an early decision to leave professionalisation at the regional / national level.

        My fear is that we can talk about this for a very long time without moving our profession forward.

        AES Evaluators' professional learning competencies are available here:

        Lyn Alderman
        President, Australasian Evaluation Society


        • #35
          Greetings colleagues,

          Many thanks for this developing and most important conversation.

          (Just FYI--I'll admit that I find it a bit difficult to know where exactly to post my comments given the different strands of the 'professionalization' Forum, but I'll keep posting under this "common principles" site.)

          What I find most fascinating and of great interest to date about what has been contributed to the discussion can be expressed in a two-fold observation:

          First, it appears that 'professionalization' is seen as a solution to one or more of the following problems--a jurisdictional problem (i.e., distinguishing 'evaluation' from other similar occupational undertakings); a quality/competency problem (i.e., ensuring that somehow those who do evaluation are qualified to do so), and a credentialing problem (i.e., the lack of well-established credentials for those who claim to be an evaluator). In one sense these three problems are of course related, but they do not necessarily require the same solution.

          Second, what I find really significant is that ‘professionalization’ is being discussed (not, of course, just in this forum but in the literature more widely) largely in the absence of what the notion of ‘professionalism’ means in evaluation. Or perhaps I should say that I assume ‘professionalism’ is implied when the literature refers to specific competencies that are required for evaluation. In other words, it appears that professionalism is a broad term for a set of competencies reflected in the values of the professional (responsibility, maturity, respect for people) and her/his behaviors. Fair enough, but that barely touches the tip of the iceberg of what professionalism means and how is understood in various professions.

          You all have inspired me, and I intend to prepare a paper on this for EES in Maastricht. I’d like to explore different understandings of professionalism in evaluation, e.g., entrepreneurial, unreflective, activist, democratic, etc. and argue that perhaps we ought to get clear about these ideas before we worry about professionalization as a process of certification or credentialing. Just a thought…

          My best to everyone



          • #36
            in order to focus on the underlying principles that should guide us as we are thinking about increasing the credibility of evaluation through professionalization, it might be useful for those of interested to list the top three principles that we consider to be most important. Mine are:
            1. Inclusiveness - insuring that we do not exclude people who could become excellent evaluators
            2. Fun - I doubt this will be on most lists, but I believe that we do our best work when we enjoy it
            3. Excellence - our efforts need to contribute tohigh quality evaluations that are useful, used and make a positive difference for programs and their beneficiaries
            The questions I asked myself:

            What do I value about evaluation?
            What is my vision for evaluation?

            It would be good to restart the conversation on values.

            Martha McGuire