Certification development

People contributing to the development process

The PHAP Board of Directors, interim Certification Commission, and management team are all tremendously grateful to the hundreds of humanitarian practitioners and other subject matter experts from all over the world who have contributed to the development of the first set of certifications under the PHAP Credentialing Program.

Initial development of the concept and program

The development of the PHAP Credentialing Program in 2016 was fundamentally driven by the need for a robust, meaningful, and inexpensive means for humanitarian practitioners to demonstrate professional competencies critical to their work and for humanitarian agencies to promote and develop such competencies among their staff and volunteers.

This need was identified and unpacked through extensive environmental scanning and targeted surveying of humanitarian practitioners and employers, drawing on inputs from more than 5,000 people over the course of 2016.

Overall, the expressed needs and concerns of humanitarian sector stakeholders aligned with the typical characteristics and requirements of professional certification programs in other sectors, the established best practices for which are codified in the ISO 17024 standard for personnel certification.

However, while traditional “role-based” professional certification normally takes a holistic approach to defining and assessing the full set of competencies for an entire professional role, the humanitarian sector demands flexibility and modularity to accommodate the vast diversity of roles and responsibilities among those working and volunteering in humanitarian action – but without sacrificing the seriousness and robustness of the program.

Thus it was determined that the PHAP Credentialing Program should develop certifications according to the requirements of ISO 17024 standard, while defining the scope of each certification to cover a clearly defined sub-set of competencies rather than a full set of competencies for any given role in the sector.

The vision of the Credentialing Program is to accumulate over time a portfolio of certifications at the same level of quality of relevance, but covering different areas of professional competence relevant to humanitarian work, developed by PHAP and other relevant credentialing bodies, as appropriate. Practitioners may then choose which set of certifications is most appropriate and useful for them to pursue, depending on their particular role and situation.

Requirements of ISO 17024 for development of certifications

The underlying philosophy of the ISO 17024 standard is that the certification programs must be developed and maintained by representatives of the target group for certification, in this case humanitarian practitioners from all parts of the humanitarian sector. This is to help ensure that the certification requirements are based in practice and take into account the full range of relevant perspectives.

More specifically, the standard requires:

• Impartiality in all aspects of the program
• Strict separation between training and certification
• Security measures to ensure the integrity of assessments
• Overall organizational and process quality management
• Processes for handling appeals and complaints
• A sustainable model for management and further development to ensure that the certifications remain available and are updated over time

The legitimacy and defensibility of the certifications is therefore derived from the process by which they are developed, which is in turn independently verified by a third-party accreditation service which audits the PHAP Credentialing Program to ensure that it meets the ISO 17024 standard. (The PHAP Credentialing Program expects to undergo its first audit in Q4 of 2017.)

For more information about the key characteristics of the PHAP Credentialing Program certifications, in line with the requirements of ISO 17024, see https://phap.org/credentialing-program-key-characteristics.

Main phases of development, 2016-2017

The first three certifications under the PHAP Credentialing Program were developed in parallel during the period July 2016 through March 2017. The three-phase development process was designed and guided throughout by a team of expert consultants specializing in psychometric assessment and certification program development with other professional sectors.

Phase 1: Scoping and scheme development, validation, and finalization

• Defined the scopes for the first three certification areas.
• Conducted practice analysis with 20+ subject matter experts (SMEs) in each area, representing different perspectives.
• Validated the results with a large number of practitioners (more than 1,000 people) representing different parts of the humanitarian sector.
• Developed the assessment blueprint, determined the percentage of the assessment that should cover each area, and developed recommendations for duration of certification validity and re-certification requirements, based on the results of the validation survey.

Phase 2: Assessment item writing and pre-testing

• Reviewed the cognitive level of the resulting statements in each assessment blueprint and determined, based on the recommendation of experienced psychometricians, that high-quality multiple-choice question (MCQ) assessments would be most appropriate means to assess the identified competencies in the defined areas.
• Gathered SMEs for an item-writing workshop where they were trained by psychometricians on how to develop high-quality MCQ items and proceeded to draft and review each other’s items over a three-day period, producing a total of 120-200 assessment items per certification area.
• Continued with additional item writing and review following the workshop, bringing in additional SMEs, and further improving the quality of the draft items.
• Organized face-to-face pre-testing of assessment items with humanitarian practitioners at different levels, with different organizations, and based in different parts of the world, conducting more than 500 pre-testing sessions in total, covering 25-50 items in each.
• Analyzed the results of the pre-testing to for an objective assessment of the difficulty level and quality of each item.
• Made corrections and further improvements to items as needed, with the support of SMEs.

Phase 3: Form assembly and passing score establishment

• Assembled balanced assessment forms for each of the certification areas, ensuring that they map to the assessment blueprints in terms of number of items for each element in the certification outline.
• Worked with yet another group of SMEs to conduct a “standard setting exercise” using the modified Angoff method to determine the appropriate passing score for the assessment form.

Ongoing maintenance and development over time

Ongoing maintenance and development of the certification requirements takes place at two levels: the maintenance of the assessments (according to the established assessment blueprints), and the periodic review and revision of the overall certification requirements (including the eligibility requirements, content outline/assessment blueprint, period of validity of the certification, and re-certification requirements)

Level 1: Maintenance of the certification assessments

With the support of volunteer item-writing committees and expert psychometricians, PHAP is in a continuous process of developing new items for the certification assessments. These are either used to replace specific items in existing assessment forms or assembled into new assessment forms following the same assessment blueprint.

Most items are contributed by humanitarian practitioners, drawing from their experience. Draft items go through rigorous technical, grammatical, and psychometric review, editing, and analysis. This includes pre-testing with certification candidates as experimental items dispersed throughout live assessments. (10 out of the 60 items in each live assessment are experimental items and do not contribute to a candidate’s pass/fail decision.)

Test maintenance is performed at least annually. This process involves both test-level and item-level analysis, including:

• Checking for passing rate consistency over time and across forms (if applicable)
• Analysis of group mean differences at the test level (if sample sizes allow)
• Testing the reliability of each test form – Cronbach’s coefficient alpha
• Item-level analysis of difficulty and discrimination, looking for item drift and possible item exposure
• Identifying any items with negative discrimination and replacing them

In addition to the technical maintenance of the assessments described above, the Certification Commission and item writing committees engage in ongoing review of the assessment content and development of new assessment forms and items to reflect updates in the latest standards of humanitarian practice and bodies of knowledge in the defined areas of the certification schemes.

Level 2: Review and validation of the certification requirements

A higher level review of the certification requirements themselves, including the assessment blueprints, is conducted periodically – at least once every five years – to ensure their currency and relevancy. The Certification Commission is responsible for determining when such a review is warranted.

The certification review process begins with a new practice analysis study for each certification to identify the tasks for successful performance and competence in the defined area and to verify eligibility and re-certification requirements, as in Phase 1 of the original certification development described above.

Based on the practice analysis study, a new assessment blueprint is developed and new items and assessment forms are established as in Phases 2 and 3 of the original certification development process.