We have added score reliability warning flags to test-takers’ score reports, visible on the administrator platform (not visible to the test-taker themselves), which indicate irregularities with the test-taker’s assessment. The presence of any single warning should not be taken as invalidating the score, but they do indicate a larger degree of uncertainty in the score than an assessment without warnings.
Warning Flags
The warning flags have been created after conducting an extensive data analysis with a large sample size of Emmersion test-taker sessions to identify certain patterns that statistically reduce score reliability.
The following table contains more information about each potential warning flag and why we consider each warning state potentially problematic.
Warning Flag | Description | Recommended Action |
Assessment duration is too short | When a test-taker moves through the assessment experience too quickly, this indicates that they may not have followed the instructions properly or taken the experience seriously. In these cases, they most likely received a score that is lower than their actual ability. This warning indicates that the testing session didn't reach the duration threshold recommended to produce a reliable score. | • Have the test-taker retake the assessment.
• Encourage test-taker to take more time with their responses and not rush through the assessment. |
Excessive restarts
| Once a test-taker starts a scored section of an Emmersion Assessment (i.e. listen and repeat, open response, open response writing, and multiple choice), we consider the assessment “started”. We keep track of each time the test-taker leaves the assessment experience and returns, and we label this as a “restart”. This warning indicates that a test-taker's assessment experience has been interrupted enough to impact the reliability of their score.” | • Have the test-taker retake the assessment in a more controlled, distraction-free environment.
• Direct the test-taker to this page to ensure they understand the conditions required to get a reliable score. |
