User Acceptance Testing

Neil Ivory, Transformation Consultant

User Acceptance Testing (UAT) is a critical part of any successful digital transformation programme, but it’s importance can often be overlooked. It is also usually left to the final hour when it is often then too late, too difficult or too costly to fix any issues. We caught up with Neil Ivory – Transformation Consultant at One Consulting, to find out why effective UAT is so important, and to hear his thoughts on why it should be conducted throughout the project rather than left to the end.

So, Neil, please can you give us a brief overview of User Acceptance Testing – what is it in a nutshell?

“UAT is the stage of an implementation project where users test new software, systems or processes to see if it actually does what it’s intended to do, in a real-life situation. It’s one of the most significant phases in a project lifecycle, with its main aim being to ensure the new software/systems/framework or whatever it is that’s been implemented, adheres to the client’s initial requirements or needs.

It’s often the final stage of any implementation process, carried out to ensure system requirements meet business needs. Basically, does the new software and the system do what it’s supposed to do? If not then it can, in theory, be fixed before go-live. It’s the only way to truly validate usability, performance, compliance and an effective user experience before new software, systems or frameworks are officially deployed into the real world.”

Thinking about the clients you help and the work you do, why is UAT so important in the context of digital transformation?

UAT focuses on the functionality and usability of software, systems, applications or business processes, and the aim is to verify that everything meets the client’s specific requirements, whether they are functional, data, process, technical or business needs. It’s critical to ensure the outcome is aligned to the business’s and individual users’ needs. When working with clients we’re committed to maximising organisational efficiency and success, mitigating risk and ensuring a return on investment. When it comes to the clients we work with, we’re passionate about connecting all aspects of technology, systems and processes, interrogating them to ensure everything is seamless and works in the intended way. We want to make sure everything – people, processes and technology – is aligned and working in complete harmony.  It should be noted however, that this should include the data being recorded and reported on.

Are there any issues or drawbacks associated with UAT?

One of the main issues is that UAT tends to take place at the end of a project. What we witness time and again is that by the end of a lengthy project most people have run out of steam. The excitement or novelty has worn off so often instead of taking their time to write a comprehensive suite of scripts to document what needs testing, it’s rushed, and the test phase doesn’t get the due care, attention, and consideration that’s truly required… and that’s incredibly frustrating to see. If you’ve invested hundreds or thousands or even millions of pounds on the implementation of new software or systems, you need to be 100% sure it’s going to do what you need it to do. We understand that there is testing throughout the implementation plan but this is often not done by the users but by the implementors.  It is often the users that have the detailed knowledge about how a process should work and what data is required so they should be involved all the way through the implementation process.   

Another reason UAT slips is lack of resource availability or poor participation from people involved. End users are often already stretched to the limit with their day jobs, they simply don’t have the time or interest to commit fully to this all-important stage. This comes back to how we ensure that all members of the implementation are fully engaged and enthused throughout the project.

Another major problem is narrow or blinkered thinking, I don’t mean that disrespectfully but often people are too close to a project or a system they’ve been using for years to really interrogate its effectiveness. Here’s an example I like to use:

I want to check if this new system can make a cup of coffee, so my script will say put the kettle on, press boil, get a cup, put the coffee in, pour the water, add milk, stir. Does it make the coffee? Yes. Does it taste nice? Yes it does! Great, job done! Tick that box. Let’s go live. And that’s that. Often, one person will concentrate on one narrow piece of work and get that part to work, without looking and fully understanding the whole process and understanding who else may be affected. They don’t think to interrogate further. What if the coffee runs out? Can the system order more milk when required? What if the kettle breaks? Can the coffee be made with cold water? How will people know when the coffee is ready? How will we know if someone wants sugar or extra milk? Do you want a large mug or a small cup? What if the user doesn’t want the coffee now, but in an hour’s time? What that person is in a meeting, doesn’t get any alerts, and the coffee is left to go cold? What if someone wants one type of coffee, say an espresso, but someone else wants a latte? What if there’s a power cut one day so the milk goes off in the fridge? There are all sorts of scenarios that could manifest and disrupt the system and it’s critical to understand and test them all.

Is there anything we should ensure is included in UAT that isn’t always?

Yes, the big thing is the data element, this has got to be tested in terms of does it exist, by that I mean are we recording everything we need and not just what we’ve currently got?

Is the data in the right format so that it can be included in reports and business intelligence?  Too often we see duplicates of the current system in terms of data and not enough thought given to what we actually need.  We need to remember that data is the bedrock of the system, and we need to be thinking 2, 3 for 4 years ahead both in terms of recording (is there a table and a column to record it in?) and what happens if we need to change it.  Recently a sample check that we asked a client to look into regarding deletion and adding a transaction showed that the system got itself into a muddle – it was easily put right but if the UAT hadn’t picked it up the problems caused were potentially very big.

The next point to get across is where is the data stored and is it accessible?  We came across another scenario where the data was in the wrong format, in the wrong table and not linked across the organisation.  The SQL script to get the data out and make all the necessary BI work was mind blowing.

You’ve mentioned issues with leaving UAT until the end of a project. So when do you think it should take place?

There’s a lot to be said for a phased delivery approach. As we’ve already touched on, user acceptance tests are mostly conducted at the end of a project at a time when the product is just about finished or something – a system or framework for example – is about to go live. This means it’s often rushed, or if problems do arise it’s perhaps too late or too costly to rectify.  Problems often get dismissed, excuses can be made, decisions taken to just live with it or pushed back to ‘Phase 2’ of the project, which sometimes never actually takes place.

My thoughts are that UAT shouldn’t always be at the end of a project – if done throughout the project lifecycle, you can pick up on any discrepancies and put them right. It should be a continuous process… testing as you go through the project itself. Does that add time? Maybe so, but it’s better to discover faults or inefficiencies at the beginning and put them right than at the end when you’ve run out of time or money. Correct or adjust things along the way, while they’re fresh in your mind and whilst you’ve got the enthusiasm and energy.

One Consulting are successfully helping a number of organisations across a variety of industry sectors. What value do you add when it comes to UAT in particular?

It’s simple – we’re independent, and impartial. It’s been proven that testing performed by an outside organisation is more effective than testing performed by those who are potentially too close to the project. We will critically analyse processes, software and systems, we know what to look for and we’ll dig deep and ask the right questions, challenging things from all angles. This means the potential for things to go wrong is greatly reduced.

When you’re inside an organisation or close to a particular system, app or process you tend to think you know everything there is to know about it, whereas an independent consultant will think differently and offer up alternate scenarios. Going back to the coffee analogy, what if your usual supplier runs out of coffee? What if someone in the organisation wants tea instead? What if there are no clean cups?

We boast a depth and breadth of experience which our clients greatly benefit from:

1. All work we undertake starts with a thorough analysis of a clients’ business requirements.
2. We’ll document business intentions and objectives and map out acceptance criteria / identify test scenarios.
3. Create a detailed UAT test plan / project plan in advance, mapping out the various phases necessary from strategic level, logical level and drill-down-in-detail level / create UAT test cases / prepare test data.
4. Produce documentation to support UAT.
5. We’ll identify or help our clients to identify UAT resources i.e. who will be doing the testing (we will always encourage actual users to form at least part of the test team).
6. We’ll appoint or help our clients to agree on a decision making structure.
7. We’ll bridge any communication gaps between teams.
8. We will initiate UAT and provide support / guidance throughout the process.
9. We’ll undertake a thorough risk assessment / determine the test intensity required at different stages – for example some areas require less testing, others may be considered critical and will therefore require extensive testing.
10. We’ll verify the performance and efficiency of the software, systems, applications or processes that are being tested.
11. In terms of verification and validation, we help our clients consider real world scenarios. Verification is connected with specifications and requirements whereas validation is performed on real scenarios or real cases.
12. We’ll provide support should new business requirements arise or when defects or inefficiencies are found.
13. We will evaluate and confirm when the time is right for go-live.

To summarise, what would be your top 5 tips for successful UAT?

1. Make sure your processes are mapped out clearly and documented properly in  the first place.
2. Make sure you’ve thought through every potential scenario. Also consider all wider parties involved, not just your part of the process or the limited aspects of the system you use – it impacts everyone.
3. Don’t leave UAT until the eleventh hour when it’s potentially too late, too stressful or too costly to fix any defects or change plans. Incorporate UAT throughout the process.
4. Seek independent advice – this is most important! An independent consultant will question everything, things you won’t have ever thought of if you’re too embedded in the project.
5. Always do a post go-live review, not right away, but perhaps a month or so down the line. Here’s what we wanted the system to do, is it actually doing it? Could it be improved? Could the reporting be improved? Could the process management be improved? Could the alerting be improved? Review your findings and fix / make changes accordingly. Don’t just settle and agree to put up with something for the next ten years. It’s all about continuous improvement.

If you’d like to find out more

Call us on 0344 326 1221

Drop us an email at info@oneconsulting.uk

Follow us on LinkedIn! https://www.linkedin.com/company/oneconsulting/