exploring and examining assessment data via a matrix visualisation

Exploring & Examining Assessment Data via a Matrix

Visualisation

Martin Graham & Jessie Kennedy

Napier University, Edinburgh

AVI 2004 - Gallipoli, Italy 2 of 13

Background

• Work part of OPAL – Online Partner Lens• Prospective business partners, employers,

employees assess each other on various characteristics

• ‘Lazy’ users would like to search these assessments rather than perform their own


Why Visualisation?

• Why not just search the numbers?• Simple search – may just pick out lenient

assessors rather than quality candidates• Statistical analysis – can give averages, but not in

context of related assessments• Recommender system - only possible if user has

previously performed assessments of their own• Lack of feedback & freedom to browse

• Use a visualisation to convey context of users and assessments


Why Matrix Visualisation?

• Node-link visualisations give precedence to nodes

• Also clutter rapidly when edges >>> nodes

• In our case, the interesting data is primarily the assessments – the edges

• Matrix visualisations have edges/links at the centre of attention

• Directed nature of edges mean assessors and candidates map naturally to axes


Initial Layout


Assessment Context

• Investigating single assessments doesn’t tell us much as score is product of both assessment and candidate

• Showing related assessments could reveal context of assessment, and of the participants


• Reveal context by overlaying related assessments as ordered bars

• For example, say the candidate crosshair intersects two assessments, both coloured blue

• The assessors who gave these ratings have their other evaluations collected and ordered around these points

• In this case, the bars show the candidate got the worst scores that each assessor handed out

• Not only are the scores poor on an ‘absolute’ scale, they are poor ‘relatively’ too

Assessment Context


Assessment Context

• This candidate has been involved in 7 evaluations in total, all of them poor

• Furthermore, they have received the lowest score each assessor has handed out


Assessment context

• Brushing a point in the matrix will show value bars for assessors and candidates

• Here, we see a low score obtained even though the candidate has a very high average

• The bars along the vertical crosshair show that this assessor has a history of handing out harsh evaluations.


Filtering / Focusing

• Candidates and assessors may be filtered by position in matrix• I.e. remove all assessors with < n

assessments• Assessments are hierarchical in nature

• User may filter matrix to include only attributes they are interested in

• User may zoom on portions of matrix to see assessment details


Specific Attributes


Conclusions

• Collating and overlaying related assessments acts as a context for verifying a candidate’s or assessor’s associated evaluations

• Allows users to see whether a candidate’s scores are consistent or not given the assessors who have applied them


Acknowledgements

• OPAL – EU Project IST-2001-33288

• http://www.opal-tool.net

exploring and examining assessment data via a matrix visualisation

Data & Analytics