What really happened when the RBS payment system failed

At the start of this week, we had a discussion at the Financial Services Club about the issues at the Royal Bank of Scotland (RBS) over the summer when their payments systems failed.

It was a major failure that garnered headlines worldwide, with many banks saying: “there, but for the grace of God, go I”.

Why such a furor occurred is pretty obvious: when a core payments system goes down, people cannot pay for the things they need, companies cannot function, commerce grinds to a halt and economies fail.

Pretty obvious really, and it’s the reason why banks with critical transaction processing operations in any country are ‘too big to fail’.

Joining the debate were:

• Ralph Silva, Director and Broadcast Analyst with SRN;
• Chris Dunne, Strategy Director with Vocalink;
• Roy Vella, former Director for Mobile Payments with the Royal Bank of Scotland; and
• Anand Vyas, Vice President, Sales for UK and Europe with Thinksoft.

(see end of this blog entry for their profiles)

Ralph opened the debate with an interesting take on what banking’s all about.   Asking the audience to tell him what banks do, most responded with comments like ‘provide credit’, ‘process transactions’, ‘move money’, ‘support business’ and more.

In typical Ralph style – I should know, I used to work with him! – he dismissed all of these comments and said, “no, what banks provide is emotion”.

Very true – many negative.

Actually, not that.  Banks, as Ralph clearly articulates, are there to help us live our lives.  They provide a promise that our money is safe and, when we are trying to improve our lives by buying a house or a car, they help us to fulfill oru dreams.

They don’t provide any physical products, so their function is to process promises of the fulfillment of emotions.

And that’s why the RBS payments failure was such a critical issue for all of their customers: because they could no longer fulfill the basic requirement of processing the promises of payment.

This was an interesting take on things, as most surveys say that consumers will switch accounts due to poor service, errors on their account or due to better interest rates (only four percent would switch due to the banks morals and ethics apparently), so the hygiene factor for a bank must be processing payments.

But what actually happened?

Chris Dunne answered this question by pointing to Stephen Hester’s letters to the Treasury Select Committee (TSC).

Stephen Hester, CEO of RBS, wrote two letters to Andrew Tyrie, Chair of the TSC, explaining what happened and what the bank was going to do about it.

The first letter on 29^th June - the failure occurred on 19^th June 2012 – explained some of the background and issues and what the bank was doing about it; the second letter on 6^th July  gave specific responses to parliamentary questions.

As Chris put it, the basic issue was that payment are processed normally in a similar way to traffic on a highway.

The cars travel bumper to bumper (or fender to fender, if you prefer) and all works fine until one car brakes suddenly and has a fender bender (or bumper jumper, if you prefer).

Then everything backs up.

That’s fine and easy to sort out.

You send in a fire engine and cutting crew and sort out the mess and get the traffic moving again.

The problem RBS had is that the accident occurred in heavy fog, rain, snow, ice …  you name it.

The weather was massively inclement and nothing was easy to sort out, so all they could do is get the traffic moving around the accident whilst the forensic fire engine service and CSI cutting crew moved in to work what had happened.

[Ed: for the detailed analysis of what happened, read Hester’s letters]

Roy Vella disagreed, and said that the issue was more down to bad management of systems, too much legacy infrastructure, an inability to adapt and adopt modern capabilities like cloud and internet technologies, and a refusal to break old systems just because they aren’t broken.

Legacy systems are an issue, as I’ve blogged before, and they are becoming more of one.  The basic mantra of if it ain’t broke, leave it, is becoming turned on its head as the view today should be that if it’s older than five years, replace it.

Anand Vyas agreed but turned the subject towards testing and the fact that a bank needs to adequately test and ensure their systems are risk aware.  The flow of systems will be broken if the systems are not robustly and rigorously stressed before change and migration takes place.

In this case, the feeling was that the RBS change was possibly felt to be a trivial upgrade.  If the upgrade or change is significant, of course the bank will test and test but, if it’s just a minor alteration, no-one cares.

This may be why the CA-7 upgrade took place without adequate analysis and stress testing beforehand, and then the corrupt files caused by the upgrade became the major car crash wreck we saw in the summer.

Whatever your view, the fact that banks are seeing more and more technology failures just goes to show that the financial system is running on technologies that are no longer fit for purpose and need more  testing to be robust.

You can watch the debate through these 15-minute clips kindly supplied by Ralph Silva.

Part One

Part Two

Part Three

 

Biographies of the Speakers:

Ralph Silva is a director and broadcast analyst working with SRN out of the company's London office where he covers the innovation and emerging trends affecting Fortune 1000 companies. He has over 23 years of global experience beginning with 12 years as an investment banker in San Francisco managing the IPO process for financial and IT companies. He then moved on to become one of Gartner's most prominent analysts focused on the financial services industry before he moving to become the first full time analyst for TowerGroup in Europe. He now runs his own business providing content for broadcasters andmultinational marketing departments.

Chris Dunne joined VocaLink in 2004 as part of the implementation of the £100 million Bacs technology renewal programme. He then moved to be Director of Customer Services, in charge of customer contact centres and service management across the entire range of services delivered by VocaLink, including the Bacs service, Faster Payments, LINK and ATM managed services. Having completed a successful programme of customer service improvement and cost efficiencies, Chris is now responsible for business strategy for the core Global Transaction Services business.

Roy Vella is a mobile payments expert and the Non-Executive Director at txt2buy Ltd, a new universal mobile sales channel enabling consumers to purchase instantly from any offline or online advertising. Previously, Roy has worked with many respected names in the mobile and payments industry including Visa, Vodafone, RBS and PayPal with experience spanning across 20+ years.

Anand Vyas is Vice President of Sales for the UK and Europe for Thinksoft, a software testing company. Anand joined Thinksoft in May 2010 having previously held senior positions with Birlasoft and HCL Technologies.

 

This Financial Services Club was sponsored by:

 

For details of sponsorship email us.

 

Comments

[Alaric Adair]

So, was there any mention of the loss of skilled UK operations staff due to cost cutting and off-shoring? When the total cost of the fiasco is considered the money "saved" is a pittance.

[Arthur Edwards]

“no, what banks provide is emotion”... at last people are waking up to the fact that banking per se does not exist (except as a legal category). See Beyond Banking - http://www.cfae.biz/about/stop-the-press/article/?tx_ttnews[tt_news]=82&cHash=3aa6a4fb94d29b71ed84e26ae13288f1