Announcement

Collapse
No announcement yet.

IOCE common principles for credentialing approaches?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • IOCE common principles for credentialing approaches?

    One of the European Evaluation Society’s (EES) thematic working groups focuses on Professionalization. The EES working together with UKES have developed the concept of Voluntary Evaluator Peer Review (VEPR). You can find more information on the EES web site:

    http://www.europeanevaluation.org/co...ion-evaluation

    In this process, the possible role of IOCE had been discussed. As various associations have expressed interest in evaluation credentialing, it would be desirable to avoid fragmentation and encourage coherence. Based on a charter of common principles it should be possible to harmonize credentialing approaches sponsored by diverse evaluation associations. The EES would welcome IOCE leading this at the global level.

    We invite your feedback on the idea. The EES thinks that this should be one of the priorities on the IOCE work plan 2015. In practical terms, how can we proceed?

  • #2
    Let me add some thoughts to this discussion. Donna Podems in South Africa has been asked by PDME (Govt. of SA) and SAMEA to do research on the status of initiatives around the world for developing lists of competencies for evaluators. Here is what she wrote in an email to me (and approved my sharing with you via this forum).:

    I do not think it should a discussion about credentialing; its not really the topic for discussion. For me, it's about competencies. I think as a field we need to move closer and closer to understanding what our core competencies are, or should be, and how that would promote a sense of identity or ability to be an "evaluator".
    The process of credentialing, or "credifying," or whatever, for me is a process decision that has many potential issues that I think countries need to address (or regions perhaps?).
    So for me, I would push for "us" determining what makes us evaluators....and provide a path for continual self improvement and guidance for the young'ins.

    Comment


    • Linda Morra Imas
      Linda Morra Imas commented
      Editing a comment
      Hi Jim, That has always been the core question for me. While each country may have its unique competencies for evaluators, is there a core set of competencies for evaluators that goes beyond borders?
      Linda

  • #3
    CES has taken the competency-based approach as noted on the CES web site; "The Professional Designation Program is founded on three pillars: a code of ethics, standards and competencies. Information about the CES designation can be found at: http://www.evaluationcanada.ca/about-the-ce-designation
    Information about the competencies can be found at: http://www.evaluationcanada.ca/compe...ian-evaluators

    Comment


    • #4
      Indeed the notion of "competencies" have been resonating more and more in the discussions within IOCE and EvalPartners.... Thanks Larry for pointing the inter-linkages between competencies, standards and ethics.

      My fear however would be to find IOCE cornered to come up with some kind of golden standards on competencies (or any other thematic area) Anyone groomed on systems thinking would tell that reality is far more complex, with factors such an interpersonal skills, "insiders" knowledge of the evaluation context, perspectives of the different stakeholders,, boundaries framing the evaluation, etc,,, all adding up to competencies, standards and ethics

      At the same time, the Canadian experience of credentialing evaluators has proven its merits (empirically I would say). From the dozen plus proposals we received for the EvalPartners evaluation, and after a blind review of each of these proposals by 5 persons in North America, Asia, Africa, Latin America and the Middle East, the best four proposals in terms of methodology and overall coherence all came from Canada!

      I believe that we should acknowledge (and commend) EES's recommendation for a coordination role around the emerging initiatives around competencies and credentialing. Should we set-up a working group in IOCE to reflect further on this issue? would EES be interested in playing a coordination role within this working group? would CES be interested to be part in it? UKES? other VOPEs?

      Comment


      • #5
        Hi everyone. EES organised a session in Dublin last October to bring together colleagues from different communities of practice with an interest in evaluation "professionalisation". The session had two main objectives: The first was to bring each other up to date on the state of various initiatives including those of UKES, EES, IDEAS (presentations were made) but as well of UNEG, our South African colleagues, CES, etc. I believe that objective was achieved. EES will be posting on its website shortly a summary of the session together with relevant presentation material. The second objective (which was mine) was to obtain agreement in principle among participants that, to allow for diversity of process along the road to "professionalisation" while making it possible for different organisations to have mutual recognition down the line if they so wished, a common "touchstone" was needed. This touchstone could be in the form of a basic statement, principles or other (this was not delved into) and remains to be proposed and discussed, however it could serve as our communal longhouse, a space in which we can gather, hold forth, discuss, debate and collaborate. This idea was endorsed and supported by the group and Bob Picciotto has given that agreement the very classy sounding name of The Dublin Consensus.

        Comment


        • #6
          Will the Dublin consensus facilitate a strategic convergence among societies with respect to professionalization? Taken as a whole the New Directions for Evaluation issue is an important contribution to the debate.. It shows that even in the United States the wind has begun to shift towards credentialing even more convincingly perhaps than the NDE commentary implies: my reading of the NDE survey numbers suggests that 58% of respondents may already support credentialing. While the NDE analysis does refer to the pioneering Canadian designation system it lacks a review of the sociological literature about the rise of professions, their role in society and the competitive process through which an occupation becomes a profession. The verdict of history is clear: only occupations that control entry and exit into their ranks rise to the top of the occupational ladder. See my 2011 article in the Evaluation journal (The Logic of Evaluation Professionalism).This said, the recent set up of an AEA board task force led by Jean King is very encouraging. It will in the first instance develop a process for the design of evaluation competencies and it will also launch an education campaign focused on evaluators and evaluation commissioners. This is much needed since AEA has yet to catch up with other societies regarding evaluation capabilities frameworks. On our side of the pond the Voluntary Evaluator Peer Review initiative of the European Evaluation Society and the UK Evaluation Society (following adoption of capabilities frameworks that are broadly similar) deserves broad based support from the European evaluation community.

          Comment


          • #7
            I want to add to these discussions as we explore together what would be involved in further promoting evaluation as a profession, in general, and, more specifically, learning what initiatives exist by various national and regional VOPEs as they consider certification or credentialing or accreditation.

            More specifically, is there a need for better collaboration internationally? And if so, should IOCE play a role in promoting such collaboration? I copy below some conceptual thoughts on this issue by Riitta Oksanen, and am attaching a document that provides some background which I hope will serve as a “discussion starter”, including the questions below.

            WHY should the IOCE be involved in this?
            Over the years that IOCE has existed there have been a number of initiatives for IOCE to address issues related to professionalisation. There has been actual progress made by other evaluation-related organizations and currently several new initiatives are planned.
            IOCE's overall strategic priorities include: Provide leadership in evaluation worldwide, encourage and support organisational capacity building for evaluation organisations, and facilitate communications and sharing of ideas across the global evaluation community.

            WHAT CHANGE can be achieved?
            Improved, efficient and equal access to information and discussion on the existing and planned professionalization initiatives, including lessons learned, to all VOPEs form the global North and South.
            Conclusions of the feasibility of a common "touchstone", which could be in the form of a basic statement, principles or charter.

            WHAT is this about?
            Collecting and providing easy access to documents, materials on professionalization.
            Providing a platform for an inclusive discussion on professionalization.

            WHAT is this NOT about?
            IOCE golden standard on professionalization.
            IOCE managing credentialing or certification processes.
            IOCE taking ownership from national and regional VOPEs.


            We invite those of you receiving this memo to:
            1. add to the list of existing or planned initiatives, including links to relevant websites or share documents themselves;
            2. perhaps describe what progress they’ve made; successes or challenges they are facing; lessons learned that would be relevant to others;
            3. let us know whether or not you feel there is a need to develop a more cohesive network for those involved to share ideas, progress and suggestions for furthering the professionalization initiatives globally;
            4. if so, how you would propose that such a network be structured and organized;
            5. what role you would be willing to play in such a network; and
            6. who else you recommend be invited to join this discussion?
            We invite you to respond to those questions and in other ways contribute to this discussion by posting your comments blog-style on this on this Professionalisation Forum on the IOCE website.


            Jim
            Attached Files
            Last edited by Jim Rugh; 04-10-2015, 11:23 PM.

            Comment


            • Jean King
              Jean King commented
              Editing a comment
              Sorry to be so slow to join the conversation--it is dissertation season here at the university. I want to thank the IOCE for creating this opportunity to share information and perhaps find common ground in a discussion of professionalization. As has been mentioned, I am chairing an AEA Board task force that is preparing a proposal for Board review in June. We will likely ask the Board to endorse two activities, which will be undertaken by a larger, more inclusive task force that the Board will identify:

              1. Developing a set of AEA-endorsed competencies- Developing a rationale and crosswalk that would lead to a set of AEA-endorsed competencies, including processes for their initial approval by the Board and AEA membership and for their eventual validation

              2. Preparing informational materials for the AEA membership about the professionalization of the field of evaluation along with opportunities for continuing conversation on the topic- Presenting broad issues of professionalization in the field of evaluation and other professions, including, but not limited to, the development and use of competencies

              In my experience people new to the professionalization discussion tend to leap immediately to a discussion of credentialing, which is controversial, especially given the prospect of the cost of any system, of limiting access to practice (“What about people who are doing brilliant evaluation work but don’t know to call it evaluation?”), or of removing unqualified practitioners. To be clear, our task force is not recommending that AEA move toward credentialing at this time except to hold a meaningful discussion of the components of professionalization in light of what has happened in other professions historically and what is happening in VOPEs around the world right now.

              Here’s to continued conversation and progress!

              Jean

          • #8
            Thank you for making this important distinction, Jean, between professionalization of evaluation and credentialing. I am reminded of the path of executive coaches. Not long ago, anyone could say they are one. Then, several credentialing programs emerged, different from each other. Then, the International Coaching Federation came up with a credentialing on top of the academic programs. Then, clients began making a credential a requirement, and many times, a credential from a limited number of credentialing organizations. I wonder if we will move in a similar direction...

            Comment


            • Ian Davies
              Ian Davies commented
              Editing a comment
              Indeed there is a useful distinction to be made between professionalisation, i.e. the overarching "rise of professions" and its sociology which Bob has researched and written about, and types of professional recognition, e.g. certification, credentialing, licensure and accreditation (J.W. Auschuld, 2005).
              As well there is a fundamental distinction between categories of designations that restrict the professional act as well as the use of the title, e.g. medical doctors (MD), and those that restrict only the title, e.g. certified management consultants (CMC). The Credentialed Evaluator (CE) designation falls into the latter category. I doubt that evaluation related designations whatever their type would be in a position to restrict practice.

          • #9
            To equate professionalization with credentialing is indeed a mistake and the parallel with executive coaching is instructive since it demonstrates that the path to professionalization is not trouble free. Rising to the top of the occupational ladder depends in the first instance on a rising and sustained demand for specialized knowledge. In a troubled world this is currently the case for evaluation. But an occupation cannot aspire to definite professional status (and reap the rewards that go with it for the profession and for the society) without other attributes: (i) public recognition that the occupation promotes the public interest (ethical guidelines); (ii) recognized disciplinary expertise (honed through practice and validated by peers combined with a high quality tertiary education degree); (iii) access to the practice (ranging from credentialing, to certification and licensing); and (iv) autonomy, i.e. public acceptance that the profession itself and no one else controls access to the practice through purposeful, principled action regarding the other attributes. No single attribute (e.g. credentialing) is sufficient or even necessary but a critical mass of the listed attributes that respond to the demands of distinct authorizing environments and cultural contexts is essential. From this perspective evaluation is currently in the middle of a historic transition - the outcome of which is far from certain since the history of professions confirms that occupations face stiff competition on the road to professionalization. Looking back many occupations fail to achieve professional status because of incoherent efforts as Tessie's example demonstrates. We all know that other well established professions are vying to capture the growing evaluation market. The evaluation community needs to respond at national, regional and global levels and this is why VOPEs need to coordinate their efforts under the broad umbrella provided by IOCE.

            Comment


            • Kate McKegg
              Kate McKegg commented
              Editing a comment
              Thanks Bob for your comments. I am currently concerned that in so much of the conversations I read and hear, credentialing or the development of competencies or standards is equated as being the same thing as professionalisation, rather than a subset of a much more complex journey and process. As you know well, professionalisation is a process or journey that a discipline or field pursues for many reasons - some of which are:
              - to build and maintain a specialized and unique body of knowledge
              - to train those in their field in the requisite skills, knowledge and expertise required to practice
              - to ensure quality and safety of those working in the field or profession, as well as the general public - often by developing principles, ethics, values and standards. My view is that the main focus of evaluation's professionalisation journey to date has been to achieve these purposes, with, I would argue, widely variable results to date depending on context.
              At the pointy end of the professionalisation journey, the purposes for professionalisation become:
              - to limit or restrict entry to the field of practice to only those who are able to demonstrate the requisite knowledge, skills and expertise - this requires some kind credentialing, accreditation or licensing process (and these are not the same thing)
              - to control the behaviour of qualified professionals
              - to gain and/or maintaining a certain standing or legitimacy in the labour market.

              The history of professionalization points to the journey towards professional status being a highly contested and negotiated process, played out through a series of complex inter-relationships with the state, clients in the market, practitioners, tertiary institutions, communities etc. The drivers of professionalization are generally both internal to the field (i.e., the aspirations of practitioners themselves) as well external (from the state as well as commissioners and funders or clients in the market).

              The professional identity of evaluators; the language, concepts, theories, characteristics and requirements to practice are still very much contested and the legitimacy of evaluation would appear similarly so.

              Although there is growing interest and momentum in relation to evaluation professionalization – there is yet, as I see it, very little serious grappling with the extensive historical, theoretical and sociological literature and learning on the professionalisation journeys of other professions (those that made it and those that didn't). I'm unconvinced that we collectively have a good enough understanding about what might be required for evaluation to further professionalize, how well positioned – or not – it might be as a field to professionalize, perhaps even whether professionalization is even necessary to maintain evaluation quality and legitimacy in the market place.

              As a field, I'm also not convinced we really know whether our current professionalization efforts are having any influence on the legitimacy of evaluation, or even if they are having any impact on the development of the field for the better. For example, Although the development of evaluation competencies and standards as a professionalization activity would appear to stimulate discussion and debate among those in the field, about their professional identity and practice, I’m not sure we know if having evaluation competencies and standards in place influences evaluation practice or the perceptions of evaluation’s legitimacy by funders and commissioners. In New Zealand, the legitimacy of evaluation in the public domain is being undermined seriously by the growing interest in big data and the supposed potential it has to solve the information needs of decision makers. Evaluation is seen as expensive, time consuming and not 'real time' enough. See the latest report produced by New Zealand's productivity commission http://www.productivity.govt.nz/inqu...ocial-services

              If as a field we are to respond effectively to the pressures and drivers for professionalization, we need to have a much deepr understanding of the theoretical and conceptual issues involved in professionalization. Currently we appear to be responding in a rather localized way, without this. So, I tend to agree with Bob, that some kind of collaborative and coordinated effort may be necessary, so long as this is truly collaborative and not a top down process.

              A few unanswered questions I have about evaluation professionalization, that I feel we should probably tackle include:

              - How might we (the collective, collaborative we) develop a better understanding and critique of professionalisation, as well as the current state of evaluation professionalisation activity - it's forms and approaches locally, regionally and internationally?
              - What can we learn from the experience of other professions?
              - How prepared is the evaluation field for professionalization? Locally, regionally, internationally? For example, do we even understand sufficiently well what the requisite evaluation skills, knowledge and behaviours are for professionalized evaluation practice?

              I have some concern that we may rush headlong (perhaps we already are to some extent) into lots of action without first pausing to grapple with some really important questions.

          • #10
            Hi everybody, I am member of Italian Evaluation Society (AIV). AIV since 2011 created a Register for AIV members who work on evaluation services, REVAP (Register of professional evaluator). The access to REVAP is on voluntary base: AIV members who intend to be registerd must apply using a specific application form (see attachment).
            In the first three years, up to 2014, the access to the REVAP was bounded to a strict definition of profession intended as the prevalent activity carried out in a year with continuity over the years. The admission criteria were set on a minimum of 6 of working months on evaluation services in the last 8 years.
            This criteria had been set according to the definition of professional member as defined in AIV statute.
            We understood that we were moving in a wrong direction, since not only we weren't able to register all those members with high skills and competencie in evaluation, who do not work with continuity on evaluation; but also those members, with a lower level of competencies, who have just started to work on evaluation services.
            On 2014 AIV General Members Meeting approved a change in the Statute, which shift the perspecitve from profession to the professionalisation.
            I have attached two documents, the first "synthetize" the new system of accreditation, while the second is the application form we are using now for REVAP access.
            One more point, the REVAP is an informal (not recognized) system for evaluators accreditation. Is a first step, since without a critical mass of evaluators is not feasible to work on more ambitious goals (such as an official recognition foreseen for the not regulated professions - at EU level see Directive 2005/36/EC).
            We feel happy to share our work with all of you in order to improve our system and hopefully point out a common standard which can be used at national regional and global level.
            We hope that in the following years the international community of evaluators will be able to create a common framework applicable at different latitudes in order to recognized the value of our "profession". We are highly interested in finding sinergies with the VEPR developed by the EES and other models applied by other IOCE partners.

            I apologies for my English
            Fabrizio
            Attached Files

            Comment


            • #11
              Thanks for this timely and important conversation. Kate's thoughts and questions resonate strongly with me. I am struck by the question of how prepared is the field for professionalisation?
              I am involved with the AEA process and the one in South Africa, and both are grappling with these and other questions. I find that my thoughts and considerations vary considerably depending on which group I am working with, as how I think about this varies within the context in which I am thinking.

              I think providing a clear pathway for becoming an evaluator is a good idea; I think ensuring good evaluation practice is a good idea; what that should look like and who decides this, and who holds that power to enforce it once it is decided, worries me. I like the many emerging competencies as they make me think about what can improve my own practice; I would like to be part of the CES b/c I think that their process would improve my path to becoming a better evaluator. At the same time I would not like the World Bank or any powerful organisation telling me what I need to know to be considered an evaluator, as ultimately this would be enforcing their values. How I value an evaluation is often very different.

              What also concerns me is how this process of professionalisation may limit our still emerging field. If we had an established profession, in say 1998, would we have encouraged Systems Thinking of Feminist Evaluation to be a part of what evaluators learn? Would they even have emerged? As Jim quoted me at the start of this forum, the conversation for me is still a very considered conversation about what it means to be an evaluator; thereby also considering who is not one.

              Comment


              • #12
                Donna's comments and concerns spur me on to share with you the following extract from my preface to a report on professionalisation in the UN system that I wrote for UNEG a couple of months ago (and which I understand UNEG will be making public once the final vetting procedure is completed):

                "We have come a long way collectively since my first debates about 25 years ago in the Canadian Evaluation Society about whether professionalising evaluation would be a good direction to go in. I was then vigorously opposed to the idea. Whether that vigour was based on context, deep thought or youth, remains a matter of post-hoc conjecture thankfully muddled by the fuzziness of recollections.

                However context matters, and it certainly has changed dramatically for evaluation in the last couple of decades. I have come around to envisaging that professionalisation might be a good way to go however I retain some of my scepticism and my doubts, and can argue the case both ways.

                In particular I continue to be concerned about professionalisation creating barriers to entry, but not of the economic kind: it is the barriers to innovation, creativity, caring and intellectual openness that concern me most.

                The real challenge is to avoid professionalised ossification and the risk that evaluation be relegated to a set of accountability, control and “knowledge management” techniques, a risk that is particularly present in the dynamics of donor funded development.

                For evaluation holds promise, and value, as a force for social justice, human rights and equity, the strength of which draws on far more than just evaluation’s technical and methodological resources. Evaluation’s fountainhead are values, open mindedness and emotional intelligence, and it is only by making these explicit and necessary foundations of professional identity, that professionalisation may be positive.

                It behoves us, whatever forms professionalisation initiatives take, to make sure that this direction opens, rather than closes, doors to evaluation colleagues globally. I look forward to the day when I will no longer see terms of reference for consultants that make a distinction between “international” and “local” evaluators, with of course different levels of pay. Instead the call will be for “professional evaluators”.

                Finally the doors should be opened wide to the future. Professionalisation should support emerging evaluators and attract those that will follow, build and improve on the foundations we are attempting to lay."

                Comment


                • #13
                  Great discussion, and excellent questions, Donna, Kate, Bob, and other colleagues. A slightly different question is: What competencies we consider essential to do an evaluation, in addition to the competencies needed to call yourself an evaluator? IDEAS and other VOPEs have worked on individual evaluator competencies, yet in an evaluation, you could, for example, "buy" the sampling expertise separately from the development of the theory of change and data collection. So, the evaluation would still have "sampling" even though the evaluator may not have it. In other words, is it useful to think about competencies to lead an evaluation, and a menu of competencies to conduct and complete an evaluation?

                  Comment


                  • Kate McKegg
                    Kate McKegg commented
                    Editing a comment
                    Hi Tessie,

                    In my view, competencies is just one aspect of professionalisation - albeit an important one. The issue you raise is also important because it reveals the rather more complicated 'real world' of evaluation - it is very rarely a simple process - and why the field must be so very careful about how it defines who is competent. When we developed the New Zealand competencies, we were quite clear that no one person was expected to have all the competencies, rather that they were more likely to be found in a team of people - the document can be found at
                    http://www.anzea.org.nz/evaluation/e...ncies-project/

                • #14
                  Different professionalization processes may pursue different aims in diverse fashions. For example the CES credentialing process does not seek to determine who is competent or not to practice evaluation or may or may not use the term evaluator, nor for that matter do the VEPR initiatives that are being piloted by the UKES and by EES.

                  Comment


                  • #15
                    Dear colleagues

                    Very interesting flow of ideas and very legitimate insights raised (competencies vs. professionalization, equity and barriers to entry, support to emerging evaluators, risk of "legitimizing" certain perspectives against others, etc...

                    I often ask myself if and when our "best enemies" the auditors had this kind of conversation it is obvious that they managed to create somehow universal standards and ring-fenced very well their livelihoods niche, but without the thrill and the social change dimension that our profession entails

                    I would like hence to move the conversation to a more concrete level, in the sense of what can IOCE concretely do in this regard, since there seems to be a broad agreement that we should move forward somehow. Assuming that we set a special Task Force on professionalization, what would be the Top 3 items that this Task Force should take on board of its agenda? after that we will look at the operationalization means for this Task Force, but let us first hear the various opinions and suggestions.

                    Kind regards to everyone

                    Ziad

                    Comment


                    • Fabrizio Tenna
                      Fabrizio Tenna commented
                      Editing a comment
                      Dear Ziad and dear all,
                      first of all let me say that I really appreciate all the contributions, the quality of each comment which shed light from different perspectives to all the pieces of the professionalisation mosaic.
                      I try to go along with concrete approach and figure out what would be the 3 top items, most of them are a synthesis of points already raised by others (Roberto Picciotto, Ian Davis, Donna Podems..etc):
                      1. Ethical framework - in my opinion is the pillar of our "profession", since, among all the other worries which can arose (i.e Donna on who exert the power of deciding who is in or who is out), it's the ethical framework that generate legitamacy, public recognition, and most of all a common system of values for all those who practice evaluation. In many circustances this system of value should work as a natural defense towards all those attemps (and in Italy I can tell you are quite institutionalised) to bend evaluation and people who work on it over other systems of "values" (i.e not indipendent performance and spending review evaluating apparatus).
                      2. Professional framework - we can call it in different ways, but surely a core set of competencies should be point out, without binding it to particular techinique, approach, discipline - but "only" with the purpose of recognising the value of our competencies, which are applied knowledge, knowledge in action.
                      3. Community framework - third items but according to me, not less important, is to understand how the system should work: is the governance of the system, probably the most delicate issue since this is the space of hidden tricks. Once the first two points are set (will's optimism) there's a high probability of rejection (reason's pessimism) since a our community is still not a unique body on the professionalisation issue for many meaningful reasons. So the community framework should first become a community approach in dealing with the two first items (Ethical and professional framework).
                      Kind regards to everyone
                      Fabrizio
                  Working...
                  X