How pass marks are set

Every CBA examination has a pass mark. It is set by a documented standard-setting panel, by the process described here, and published on the programme page before any candidate sits the examination, so no candidate is examined against a figure that has not been settled. This page explains where that figure came from, who decided it, what is written down, and what happens when it changes. It covers the examination component of every CBA credential.

What a pass mark actually is

A pass mark is not a standard of achievement in the abstract. It is a judgement about a specific paper: how much of that paper, assembled from those items, at that difficulty, a minimally competent practitioner should be expected to answer correctly.

That makes a cut score a property of the item bank. Two credentials whose syllabuses make comparable demands can carry different pass marks because one bank happens to contain harder items. Rebuild the bank and the same standard of competence is expressed by a different number.

It follows that a single figure applied across every credential a body awards is not the outcome of standard setting. It is a house number, chosen in advance and then applied to whatever the papers turned out to be. CBA sets a cut score for each programme separately, against that programme’s own bank. Where two programmes carry the same figure, it is because two panels arrived there.

What the cut score applies to

The cut score governs the examination and nothing else.

CBA awards at three tiers. Certificate is examination only. Practitioner adds assessed applied workbooks, completed in a spreadsheet and checked to a stated tolerance. Professional adds professional experience confirmed by a named referee and an ethics undertaking.

Those additional requirements are met or not met in their own right. They are not scored, they are not averaged with your examination result, and a strong examination score does not offset them. A Professional credential cannot be issued at all without a verified experience record, and that is enforced by a database rule rather than by a checklist.

The panel

A cut score is set by a panel of at least three subject-matter experts for that programme, convened by the Head of Assessment for the scheme version being set.

Panellists are selected against these requirements:

• current or recent practice in the subject area of the credential, at or above the level the credential claims;
• familiarity with the people who take a qualification of this kind and the work they do, because the judgement to be made is about candidates rather than about the panellist;
• no financial interest in a CBA training provider, and no remuneration of any kind that varies with how many candidates pass.

Panellists are paid a fixed fee for the time the exercise takes, agreed before it begins.

The Head of Assessment chairs the exercise and keeps the record, and does not submit judgements. No member of CBA staff with responsibility for revenue takes part in a panel or attends one.

CBA is a small organisation drawing on a limited pool of experts, so a panellist may have written or reviewed items in the bank being sampled. CBA does not treat that as disqualifying and does not leave it unstated. It is handled in three ways: each panellist declares, item by item, where they wrote or reviewed the item in front of them; the declaration is recorded next to their judgements; and at least one panellist on every exercise has had no part in writing or reviewing that bank.

Defining the borderline candidate

Everything that follows depends on this, so it is done first, and it is done before any item is looked at.

The panel agrees a written description of the borderline candidate for that credential: the person who is just competent enough to be certified, with nothing to spare. Not the strong candidate, not the typical candidate, and not a candidate who ought to fail. Someone whose knowledge a competent practitioner in the field would accept as sufficient, and would not describe as more than sufficient.

The description is written in the language of that programme’s published syllabus and domains, so that it can be applied to an individual item rather than left as a general sentiment. It is agreed before the item set is opened, for one reason: a description written after the items have been read tends to describe the items.

Once agreed, it is recorded word for word and is not revised during the exercise. Every judgement that follows is a judgement about that person.

Judging the items

CBA uses a modified Angoff method.

Each panellist works through the item set alone and answers one question about each item: of one hundred borderline candidates as just described, how many would answer this item correctly? The answer is a percentage, recorded against that panellist and that item.

The first round is taken independently. Panellists do not see one another’s figures and do not discuss the items before submitting.

The panel then meets, sees the full spread of judgements on every item, and discusses the items where that spread is widest. A second round follows, in which any panellist may revise any figure. Panellists are not asked to converge, and no panellist is pressed to justify a figure until it changes. Both rounds are kept.

For a programme’s first cut score there is no candidate performance data to put in front of the panel between rounds, because no paper has been sat. The panel works from judgement alone, and the record says so.

Combining the judgements

For each item, the second-round judgements are averaged. Those item averages are then averaged across the whole item set. The result is the proportion of the paper that the borderline candidate would be expected to answer correctly, which is the cut score, expressed as a percentage.

It is rounded to the nearest whole percentage point, and where it falls exactly halfway it is rounded down. The rule is fixed in advance so that the direction of rounding is not a decision taken once the number is known.

Some bodies then adjust the panel’s figure by an estimate of the measurement error of the paper. CBA does not, because estimating that error requires performance data from papers that have been sat. Whether such an adjustment should be introduced once there is enough data is a decision CBA has not taken. If it is taken, it will be described on this page before it is applied to any live paper.

Where panellists diverge

A wide spread of judgements on a single item is usually a signal about the item rather than about the panel. Experienced practitioners who disagree sharply about how a borderline candidate would perform are often reacting to an ambiguous stem, a second defensible answer, an option that cues the key, or content sitting outside the published blueprint.

Where the first-round judgements on an item span more than 40 percentage points, the item is referred to item review rather than argued to consensus.

If the item is withdrawn or amended as a result, it is excluded from the calculation and the arithmetic is run again on the remaining items. If item review confirms the item as it stands, it stays in the calculation. Either outcome is recorded against that item, with the reason.

The paper you sit is not the paper the panel judged

CBA papers run from 60 to 80 questions and 90 to 135 minutes depending on the programme, each item a single best answer from four options. Papers are assembled automatically for each candidate, drawn from the bank to the published domain weightings, so no two candidates should see the same paper. Every item counts equally towards your score and there is no negative marking.

A cut score expressed as a percentage therefore has to hold for every paper the assembler can produce, not only for the set the panel saw. Two things follow, and CBA states both.

The item set put to the panel is drawn to the same domain weightings as a live paper, and in the same proportions of item difficulty as the bank it comes from, so that what the panel judges is representative of what the assembler will build.

Papers drawn at random from a bank are nevertheless not exactly equal in difficulty. CBA does not currently apply a statistical equating adjustment between papers, because the data required to support one does not exist before papers have been sat. What stands in its place is the blueprint, the difficulty proportions, the requirement that every item is reviewed and signed off by a named subject-matter expert before it can appear in a live paper, and the audit of each bank for answer cueing domain by domain.

What is recorded

The standard-setting record for a programme contains:

• the programme, the scheme version the cut score was set for, and the date;
• the method used;
• the borderline candidate description, in the words the panel agreed;
• the panellists, their relevant qualification or experience, and any interest each declared;
• the item set judged, and every panellist’s figure for every item in both rounds;
• the items referred to item review, and what happened to each;
• the arithmetic, the unrounded figure, and the figure as rounded;
• the role that approved the result.

No candidate sits a programme’s examination until that record exists. The record itself is enforced by the system that publishes programmes rather than by policy alone: a programme whose standard-setting record is missing, or still carrying placeholder text, cannot be set live, and until it is, its programme page gives no pass mark figure.

Panellists are named in the record and are not named publicly. Identifying the people who sign items into a live bank exposes both them and the bank.

The record is retained for as long as any credential assessed under that scheme version remains in the public register, which is indefinitely. Credential records are not deleted, for the reasons given in the privacy policy, and a record that explains a credential is of no use if it expires before the credential does.

Producing it years later

Your own result is tied to the record at two points.

The pass mark in force at the moment you sat is written to your examination session record when you sit, and is never recalculated afterwards. The scheme version in force when your credential was awarded is written to the credential at issue and is never edited. Between them, a result from any year can be matched to the standard-setting exercise that produced it.

If you want to see how the cut score for your examination was set, ask. CBA will send you the standard-setting record for the scheme version you were assessed under within fifteen working days of the request. Two things are removed first: the identities of the panellists, and the text of any live item. The rest reaches you as it stands, including the individual judgements and the arithmetic.

There is no charge for this, and it is not an appeal.

When a cut score is reviewed

A cut score is set for a particular scheme version and does not stand indefinitely. It is reviewed when any of the following occurs:

• the item bank changes to the point where the set the panel judged no longer represents it, through a substantial number of items being retired or added, or a domain being rebuilt;
• the blueprint is reweighted or the syllabus revised;
• performance data from live papers indicates that the paper is not behaving as the panel assumed, for instance an item that almost every candidate answers correctly or that almost none does;
• an appeal or an item review establishes a defect affecting the bank generally rather than a single item;
• three years have passed since the cut score was set, whether or not anything else has triggered a review.

A review is a fresh standard-setting exercise against a fresh sample of items, not an adjustment to the existing figure. A figure that stands unchanged after a review is an outcome of that review, not an absence of one.

A cut score is not reviewed because the pass rate is higher or lower than would be commercially convenient. CBA does not set a target pass rate, does not hold one, and does not move a cut score after papers have been sat in order to produce one.

Any change to a cut score raises the scheme version of the programme.

A change never applies backwards

Where a review produces a different figure, it takes effect from a stated date and applies only to papers sat on or after that date. It applies that way in both directions.

If the cut score rises, a result already achieved stands, and a credential already awarded is unaffected for the remainder of its three-year term.

If the cut score falls, a result already recorded as a fail is not converted to a pass. You were assessed against the standard in force when you sat, and that is the standard your paper was marked against.

Recertification is separate: when you recertify, you meet the requirements in force at that point, as published on the programme page.

The one route by which a past result does change is a defective item. Where an appeal or an item review establishes that an item was wrong, the complaints and appeals policy governs what follows, including the re-marking of every affected paper. That is a correction to how a paper was marked, not a retrospective change to the standard.

Candidates sitting with adjustments

The cut score is the same for every candidate sitting that examination, including candidates sitting with reasonable adjustments. An adjustment changes the conditions under which you sit, such as additional time, alternative formats or rest breaks. It does not change the blueprint, and it does not change the figure your paper is marked against. Requests are handled as set out in the reasonable adjustments policy.

Standard setting is judgement, not measurement

CBA states the following plainly.

The cut score is a considered judgement made by experts. It is not a measurement of a quantity that exists independently of them, and there is no true pass mark waiting to be discovered.

Different defensible methods produce different numbers from the same bank. A method that asks panellists to judge items produces a different figure from one that asks them to sort real candidate performances, and both are accepted practice.

The same panel, meeting again with a different sample of items, would probably produce a slightly different figure. So would a different panel of equally qualified people.

A cut score is therefore not precise, and CBA does not present it as precise. What makes it defensible is different: it is documented in full, it was made by named experts against a definition of competence agreed before any item was seen, and it was reached without reference to how many candidates it would pass or what that would be worth.

No external body approves CBA cut scores. CBA is not a regulated awarding organisation, and there is no statutory regulator above it for assessment decisions, which is stated here as it is stated in the complaints and appeals policy. What stands in place of external approval is the record, the fact that the record can be produced on request, and the right of appeal.

Challenging a cut score

An appeal against your own result runs on the two grounds set out in the complaints and appeals policy: procedural and assessment.

Disagreeing with the level of a cut score is not in itself a ground of appeal, and CBA says so rather than implying a route that does not exist. Evidence that a cut score was set otherwise than as described on this page is a matter for a complaint, and will be answered as one, within the periods that policy states.

Requests, complaints and appeals all reach CBA through the contact page. Quote your credential ID or the programme code where you have one.