Level 4 unit report

Analysing pass rates on units and programs on units

In year four the most important metric is the pass rate, as this may affect retention which is a KPI.

Analysing raw statistics without using inferential statistics can be misleading due to the sample size issue. Clearly a unit with only one student can only have either 100% pass rate or 0%. Units with small numbers of students will have more variable pass rates from year to year than larger units.

There are several approaches to this issue. One is calculate confidence intervals and the significance of any difference from the overall pass rate using binomial theory. This is the basis of the confidence intervals that are often provided for opinion polls and other similar samples.

Another approach is to use empirical Bayesian shrinkage towards a Beta prior. This assumes that in the absence of any evidence to the contrary each unit would have the same pass rate. Each observation that is available acts to move the estimate away from this value.

In this case the overall pass rate is 91.9%

If a unit has a small number of students there will only be limited evidence to shift the prior expectation for the pass rate away from the empirical value for the pooled units. If the unit is large then the calculated pass rate will not change.

The table below gives

• n = number of students evaluated
• P = number of students passing the unit
• pass_rate = percent of students actually passing the unit
• eb_pass = Empirical Bayes estimate of pass rate
• lower = Lowe 90% confidence interval for pass rate baseon binomial theory
• upper = Upper 90% confidence interval based on binomial theory
• p_val = P value representing the probability that the data could have been obtained if the true pass rate for the unit was actually 91.9% and the underlying assumptions of binomial theory are met.

To use the table first order the units by p-value using the arrow at the top of the table. Units with p-values below 0.1 are highly likely to differ in some respect from the overall pattern. It is advisable to use a rather more liberal cutoff than the 0.05 significance level often used in published papers. Units with less than 10 students are unlikely to be significantly different from the overall expectation even if they have 100% pass rates. On the other hand if they did have very low pass rates this may turn out to be significant. After screening for units that differ in a clear and statistically significant manner from the overall pattern the empirical Bayes values can be used to guide the interpretation of difference between the units. The 90% confidence interval provides a suggestion of the variability that might arise if a different set of students took the unit. The confidence intervals can clearly be very wide in the case of units with only one or two students.

In this case cell biology has a low pass rate that is also significantly different to the overall rate. This is a large unit and the subject matter is challenging. This does not suggest a problem with unit delivery but may simply represent the nature of the unit. Notice that the residential field trip for BSEVSF has a superficially low pass rate, but this is not significant. A different set of students may have led to a different result.

Differences between mean marks awarded in the units

Ideally the raw data should have been used as input to this analysis. A problem arises if the marks do not follow a Gaussian distribution. This can be overcome through using bootstrapped re sampling with raw data. It is not possible to run full diagnostics on data consisting only of means and standard deviations. However confidence intervals based on the standard error can be produced, with an appropriate correction for small sample sizes.

The overall mean score for all the units is 62.5 percent.

Units scores with confidence intervals are shown below based on pooled variance. No significance test is provided due to the fact that the result may be slightly misleading. The scores are only provided as a guide to the variability expected.

Figures

Pass rate

Mean marks