This page describes activities within the Evaluate phase of the design cycle. These activities help you to evaluate the concepts produced in the Create phase. They examine how well the product criteria are met, taking the needs of all the stakeholders and target users in account.
On this page:
It is important to set criteria for determining if a product is successful. These should be based on the needs that were identified in the Explore phase of the design cycle.
The criteria will typically cover issues to do with People, Profit and Planet. People issues include: User experience, Total cost of ownership and Social impact. Profit addresses: Costs and revenues, Technical risk and Business risk. Planet covers: Depletion of scarce resources, Energy use and Waste impacts.
These criteria have some relevance to any project. However, their relative importance may vary. At this stage in the design cycle, it is important to review the criteria and discuss which are the most important: deal-breakers or unique selling points.
At this point, it is also important to plan how the criteria will be tested. Start by choosing a benchmark to test against. This is helpful becuase it is much easier to judge whether something is ‘better’ or ‘worse’ than something else, than to make an objective assessment of whether something is ‘good’ or ‘bad’. The benchmark could be an existing product or an existing way of accomplishing a similar task.
It is also important to plan how the various criteria will be measured. For example, user experience may be measured using time taken, error rate and satisfaction during a user test. Headline criteria can also be used on their own for quick and initial evaluation during the early prioritisation of ideas and concepts.
In this activity, a range of relevant experts use their skill and knowledge to systematically judge and test concepts against the agreed criteria. Expert judgement is needed because it may be difficult to formally test how well initial concepts could ultimately perform against the different criteria.
A multi-disciplinary team is needed in order to make judgements against all the different criteria. This ensures the complete set of criteria is appropriately represented and prioritised.
The initial descriptions of user journeys produced in the Explore phase can be extended into a comprehensive task analysis in order to provide a framework for the expert appraisal. This task analysis should cover all aspects of the use of the product, including purchase, installation, use, maintenance and disposal. For each task step the concept’s performance should be evaluated against the relevant criteria. The understanding of users gained from the Explore phase can also be used to help evaluate the concepts from different user perspectives.
The comprehensive task analysis and the insight gained from expert appraisal should be used to set the scope for subsequent evaluation activites.
The capability loss simulation tools available on this site can assist with expert appraisals, helping experts to understand how capability loss may affect product use. These include the Cambridge simulation gloves and glasses and the Impairment simulator software.
- James Hom’s Usability Methods Toolbox website describes various ‘inspection’ methods that can be used by experts to help them inspect and evaluate a concept or product in a systematic way.
- The Usability Net website has a section on Heuristic evaluation. This is a particularly popular method of expert appraisal, in which the product is evaluated against established guidelines or principles.
Testing with users evaluates whether they can use product concepts and prototypes, and how much they like using them. It can identify sources of frustration and difficulty so that they can be addressed. Early user testing is vital because it is hard for designers and experts to correctly imagine the details of what users will do.
User testing can take place at different levels of formality and detail. Early in a project, it is often informal and qualitative, providing initial feedback on concepts and ideas. Later on, it can be used with more objective measures, such as timings and formalised questionnaires, to ensure that products fulfill criteria and standards.
The understanding of users and stakeholders, developed within the Explore phase, can help to choose who should be recruited for the user tests. The task analysis produced as part of ‘Testing with experts’ (above) can help to identify the tasks they should perform.
Whenever users are involved, it is important to take a range of ethical considerations into account. These include issues of consent, privacy, intellectual property and data protection. It is also important to consider whether the stakeholders that are planning or conducting the activities have a vested interest in achieving a particular outcome, and whether this could skew the results.
It can be helpful to identify who would be excluded from using a product or prototype on the basis of their capabilities, such as: Vision, Hearing, Thinking, Reach & Dexterity, and Mobility. This can help in evaluating the accessibility of a product for a diverse range of users.
To do this, it is helpful to work through the task steps involved in using the product (such as those identified in the task analysis produced during ‘Testing with experts’ above). For each step, the assessor examines the demands placed by that step on the various user capabilities. Reducing these capability demands (while achieving the same features or functions) should lead to a more satisfying product that can be used by a wider percentage of the population. More information on the various user capabilities and the impact they can have on product use can be found in the About users section.
Estimating exclusion should not be used in isolation but should be complemented by the other activities in the Evaluate phase of the design cycle. In particular, user testing helps the assessor understand how to score the demand levels of each task.
An exclusion calculator is available at calc.inclusivedesigntoolkit.com. This allows you to estimate the demands that a particular task places on the user’s capabilities, and then calculates the proportion of the UK population who would be excluded by that task.
- Keates and Clarkson (2003)’s book ‘Countering design exclusion: An introduction to inclusive design’ provides more background on exclusion calculations. (Published by Springer.)
- A paper by Goodman-Deane et al. (2011): ‘Estimating exclusion: a tool to help designers’ gives more recent information on exclusion calculations. (Published in ‘Proceedings of Include 2011’.)
This activity draws together, summarises and communicates all of the evidence that has been generated from the evaluation activities.
To do this in a systematic way, it is helpful to look back at the evaluation criteria agreed on in ‘Review criteria’, and document the evidence obtained in each of the key areas, such as User experience, Social impact and Costs and revenues. Doing this highlights if evidence is missing in some areas, and can help to determine if more evaluation is needed.
The evidence in each area can then be compared with the criteria to drive the objective choice of the lead concept to focus on in the next cycle of concept design.