Questionmark OnDemand

Introduction

Evaluating questions

The response data is back from a trial of a new set of items (questions) intended for use in a 'high stakes' situational judgement test which will be used for sifting graduate applicants. You have passed the data onto a psychometrician for professional analysis but you have been asked to provide an interim report to your manager.  Please evaluate the following considerations:

Click on the > button to answer some questions as you consider this data:


Question

The question (item) report for a trial assessment reveals that no-one answered a particular question correctly; it is probably scored the wrong way round.

Question

Calculating the correlation of the overall test scores with other methods used to assess the same criteria (such as an off-the-shelf situational judgement test) would be a measure of concurrent validity.

Question

There is an unusually high level of fluctuation (variance) in the question difficulty scores (P-Values); This is more likely to be a problem with this kind of test scenario than in other test scenarios.

Question

Two of the questions have been answered correctly by all participants; if these were removed then the internal reliability of the test would be unaffected.

Question

If 20% of the trialling participants do not speak English as a first language and, as a group, scored 6% lower than the rest of the group, then ‘Adverse Impact’ (test bias) on the basis of language is very unlikely to be an issue.