26 Business Professionals in a multinational corporation were asked to assess their search skill prior to undertaking 2 exploratory search goal tasks (not a single right result) using their enterprise search engine. Task #1 could have potentially many results, Task #2 very few.
For each task 4 high value documents were hidden in the search index; participants were unaware. Participants had a set time based on past analysis to make it realistic, to gather together the top #10 most relevant document search results based on the simple tasks. The #4 hidden items per task would be expected to be part of these #10 gathered by participants. Participants assessed their own performance (user satisfaction) after the experiment.
- No relationship was found between user satisfaction and objective performance.
- Only 26% high value items were found yet 60% were satisfied/very satisfied
- No relationship was found between self assessed search expertise and actual objective performance.
- An association was found between relative user satisfaction (Task #2 – Task #1) and objective performance.
Those with higher search outcome performance recognised it was harder in a larger Information space to identify the most relevant results (compared to a smaller space) and adjusted their user satisfaction accordingly.
This new user satisfaction metric has a number of applications to help organisations reduce risk and increase search literacy.
A Geodyssey - Enterprise Search & Discovery and Text Mining Research 
Leave a Reply