Freely Available British and Irish Public Legal Information
[Home] [Databases] [World Law] [Multidatabase Search] [Help] [Feedback]
England and Wales High Court (Queen's Bench Division) Decisions
You are here: BAILII >> Databases >> England and Wales High Court (Queen's Bench Division) Decisions >> Bates & Ors v the Post Office Ltd (No 6: Horizon Issues) (Rev 1) [2019] EWHC 3408 (QB) (16 December 2019)
URL: http://www.bailii.org/ew/cases/EWHC/QB/2019/3408.html
Cite as: [2019] EWHC 3408 (QB)
[New search] [Printable PDF version] [Help]
Case No: HQ16X01238, HQ17X02637 and HQ17X04248
Neutral Citation Number: [2019] EWHC 3408 (QB)
THE POST OFFICE GROUP LITIGATION
IN THE HIGH COURT OF JUSTICE
QUEENS BENCH DIVISION
Rolls Building
Fetter Lane
London, EC4A 1NL
Date: 16 December 2019
Before :
THE HONOURABLE MR JUSTICE FRASER
- - - - - - - - - - - - - - - - - - - - -
Between :
|
|
Alan Bates and Others Claimants |
|
|
|
- and - |
|
|
|
Post Office Limited Defendant
|
|
- - - - - - - - - - - - - - - - - - - - -
Judgment (No.6) “Horizon Issues”
- - - - - - - - - - - - - - - - - - - - -
Patrick Green QC, Kathleen Donnelly, Ognjen Miletic and Reanne MacKenzie
for the Claimants (instructed by Freeths LLP)
Antony de Garr Robinson QC, Simon Henderson, Owain Draper and Rebecca Keating
for the Defendant (instructed by Womble Bond Dickinson LLP)
Hearing dates: 11, 12, 13, 14, 18, 19, 20, 21 and 27 March;
3, 9 and 12 April; 4, 5, 6, 7, 11, 13 and 14 June 2019;
1 and 2 July 2019
Draft distributed to parties on 28 November 2019
- - - - - - - - - - - - - - - - - - - - -
Mr Justice Fraser:
1. This judgment is part of the long-running legal dispute between the claimants, who are all either former or serving sub-post masters and sub-post mistresses (“SPMs”), and the Post Office. All the SPMs in the litigation had contracted with the Post Office at different times to run branch Post Offices in different locations across the country. The Post Office introduced a computer system called Horizon in 2000 across all its branches. That was changed in 2010 to an online version called Horizon Online or HNG-X, and the former version is now called Legacy Horizon. The claimants’ case is essentially that both Legacy Horizon and Horizon Online (which used many elements of the existing system) were or are unreliable, and this led to unexplained shortfalls and discrepancies in their branch accounts. The Post Office denies this, asserting that the systems were and are robust, and extremely unlikely to be the cause of such matters. There are numerous different causes of action brought against the Post Office, and the Post Office counterclaims against the claimants, including seeking damages for fraud. This is group litigation under CPR Part 19.
2. This judgment concerns the operation and functionality of the Horizon system itself. It follows the lengthy Common Issues judgment, which is Judgment (No.3). That judgment was extremely long, for two main reasons. It concerned six different claimants, each of whom had contracted with the Post Office at different times, and each of those six claimants’ cases concerned very different facts. Very few of the material facts relating to contract formation were agreed for any of these six, and therefore factual findings had to be made for each of them in this respect. Approximately 500 paragraphs were necessary to do this. Although first instance judgments must be reasoned so that parties understand the outcome (and so that any judgment can be subject to appropriate review), very lengthy judgments can be difficult to follow. If a judgment becomes too long, or too technical, it can be counter-productive to wider understanding on the part of those not immersed in the fine detail of every aspect of the case.
3. I have therefore included technical detail about the Horizon system, its operation, and some aspects of the technical evidence, in a Technical Appendix to this judgment. The contents of that appendix should not be seen as being of subordinate effect to the contents of the judgment itself; this is done simply for the convenience of readers. It is also intended that readers who do need to immerse themselves in technical matters to a very fine level of detail may not find it necessary to study the appendix, and reading the judgment alone may be sufficient. Notwithstanding that approach, however, this judgment too is extremely long. This is due to the complexity of the Horizon system, which even for computer systems is extraordinarily complicated for the reasons explained below; the period of time over which the complaints range (which starts with the introduction of Horizon in 2000 and runs to date); and due to the way that the litigation has unfolded. There are some matters in this judgment that go to issues affecting the group litigation going forwards, such as disclosure, which have increased its length. It is also the case that this litigation is being very strongly contested on both sides. I have endeavoured to provide a reasonable level of detail to explain my findings on the Horizon Issues to assist both sides as much as possible. Finally, the only findings that are made in this judgment are those necessary to come to conclusions on the Horizon Issues. All other matters in all the claims of the claimants and the counterclaims by the Post Office remain for decision in later judgments.
4. This judgment is in the following parts:
Paragraph no.
C. Features of this Group Litigation 57
D. Evidence of Fact: The Claimants 76
E. Evidence of Fact: The Post Office 202
F. Documents and Available Information 559
G. The Experts’ Agreements 654
H. The Period 21 March 2019 to 4 June 2019 706
J. Conclusion on Expert Evidence 870
K. Audit Data 905
M. Answers to the Horizon Issues 965
Appendices:
2. Summary of bugs, errors and defects
3. Glossary
A. Introduction
5. These proceedings are being conducted pursuant to a Group Litigation Order (“GLO”) made on 22 March 2017 by Senior Master Fontaine, and approved by the President of the Queen’s Bench Division. A more comprehensive introduction to the issues generally between the parties is contained at [2] to [43] of Judgment (No.3) “Common Issues”, which is at [2019] EWHC 606 (QB). Other judgments concerning procedural rather than substantive issues, are the first Judgment at [2017] EWHC 2844 (QB) and Judgment (No.2) “Strike Out” at [2018] EWHC 2698 (QB). During the Horizon Issues trial which is the subject of this judgment, the Post Office issued an application that I recuse myself as Managing Judge in this group litigation, and stop the Horizon Issues trial, so that it could be recommenced at some later date in the future (before a replacement Managing Judge). That application was refused and the judgment is at Judgment (No.4) “Recusal Application” at [2019] EWHC 871 (QB). Permission to appeal was refused by the single Lord Justice on 9 May 2019. There is also Judgment (No.5) “Common Issues Costs” at [2019] EWHC 1373 (QB) which made various orders in respect of the costs of the Common Issues trial, a hearing in respect of that having taken place on 23 May 2019.
6. This trial concerns what the parties referred to at the case management stage as the Horizon Issues. The intention behind the case management of this litigation was that the contractual issues (which were called the “Common Issues” and affected all the claimants) and the computer issues relating to the operation, functionality and reliability of the Horizon system (which were called the “Horizon Issues”, and also affected all the claimants) would be resolved first. The parties needed time to prepare for these two trials, which took place in late 2018 (for the Common Issues) and into the spring of 2019 (for this one), in particular to perform disclosure, and prepare and serve evidence. For the Horizon Issues, this included expert evidence, for which I gave permission, from two IT experts, one for the claimants (Mr Coyne) and one for the Post Office (Dr Worden).
7. The recusal application caused significant delay and disruption to the Horizon Issues trial. Originally, that trial had been programmed to finish (including delivery of oral closing submissions) by 11 April 2019, a date which became unachievable as a result of that application. Following Judgment (No.4) the remaining witness of fact for the Post Office to be called in person, Mr Parker, a senior Fujitsu employee, was called on 11 April 2019.
8. The two IT experts could not be called until 4 June 2019 onwards. The two experts therefore had a period of some weeks, following on from the evidence of fact, before they were called to give their oral evidence. There was therefore an interval between 11 April 2019 and 4 June 2019 when the experts were called. Their evidence collectively took two court weeks.
10. The Horizon Issues trial timetable was therefore different to a conventional trial in this respect, because the vast majority of the oral factual evidence, with the single exception of Mr Parker’s evidence, was completed by 21 March 2019. That is 75 days, or almost 11 weeks, before Mr Coyne was called by the claimants, and even longer before Dr Worden was called by the Post Office. The approach of the experts to the opportunity presented by that interval further to assist the court with their evidence is notable, in my judgment. I will deal with this further in Parts H and I of this judgment. For a more detailed description of the architecture of the Horizon system (both Legacy Horizon and Horizon Online) reference should be made to the Technical Appendix. For a general understanding of the issues in this litigation, an overview is sufficient.
11. Horizon is both an accounting system, and also supports a large number of what are called business applications. Some transactions that a customer might wish to carry out in their local Post Office branch are what are called retail activities, such as buying a Post Office product. One example of this is buying a book of stamps. However, the Post Office has a large number of business customers whose products are offered to the public through branch Post Offices. The Post Office calls these companies “clients”. For every kind of activity which a customer wishes to transact in their Post Office branch, Horizon needs to have the functionality to perform it. This functionality means supporting the counter activity of carrying out the transaction, which is another way of saying Horizon enables the SPM or their assistant to transact the business over the counter. That business may be a combination of Post Office retail activity, and purchasing services or products offered by the Post Office on behalf of its clients. However, in conjunction with the ability of a SPM to serve a customer (what is called counter or front office), there is the associated “back office activity” of settling with Post Office's 'client' organisation, who has provided some service to the customer - such as the DVLA, or a bank. Accounting is a theme or thread that runs through all of these business requirements of Horizon, but it is only a part of them. The number of services provided by Post Office branches is large and has increased steadily from 1998 to the present day. The number of clients has almost certainly increased, and the functionality of Horizon has expanded in line with the growth in service, both on the counter and in the back office.
12. For those activities where the Post Office branch is acting like a retail outlet (such as selling stamps), both the hardware and software is provided by Horizon to support this activity. This is the Electronic Point of Sale Software component of Horizon, that is referred to as “EPOSS”. EPOSS allows the SPM or assistant to record that some goods have been provided to a customer, compute the price of those goods, and allow the customer to pay the money required for all their purchased goods, using either cash or a credit/debit card. Often, a customer may wish to carry out two or more different activities in one visit to the counter - for instance, to settle a utilities bill and to buy some stamps. This can be done in the same activity and so Horizon has the concept of a customer carrying out a 'basket' of activities and settling the total amount due for the basket in several ways - by one single credit card transaction, by a cheque, or by cash.
13. Baskets of Post Office activities and non-Post Office activities are not supported by Horizon. Often, a local Post Office branch will be a retail outlet too, selling non-Post Office goods such as groceries, newspapers or even (as with Mr Bates at Craig-y-Don) a haberdashery. If a customer wishes to buy a newspaper and some stamps, the newspaper is not sold by the Post Office; it is simply sold by the associated, though in accounting terms separate, retail outlet run by the SPM which uses the same premises. So, the customer has to settle in two parts. In some premises, a customer may queue up to purchase (say) greetings cards and pens (from the retail side of the branch Post Office). If they then wish to perform some activities with the Post Office, they may then need to queue up at a separate counter position to do that; that separate and second activity would be transacted through Horizon. In this respect, the National Lottery is an exception and spans the two businesses.
14. The Post Office’s clients include high street banks, Camelot, gas and electricity companies (for paying utility bills), DWP (for paying benefits and pensions) and DVLA (for paying road fund tax). Because of the different nature of the services provided through the Post Office for all of these many hundreds of client organisations, the services provided through the Post Office will be different from the service provided for other clients. There is therefore the need for some unique software functionality within Horizon. This must be provided both in the branch and the back office to support the activities for that client. This is a part of what makes Horizon such a large and complex system. The other reason that Horizon is so complex is because it has evolved over a long period. As will be seen below, the original concept and design for Horizon was for a joint Post Office/DWP project so that welfare benefits could be paid to benefit claimants in a certain way. That changed and the DWP withdrew from the project, but the need to offer other and an increasing number of products has led to many additions being added on to the system, increasing its complexity over time.
15. As well as the counter activities necessary to run a branch Post Office, Horizon also supports the periodic process of balancing and rollover for each branch; essential elements in the accounting to the Post Office performed by the SPM. Every branch operates in Trading Periods, which are either four or five weeks (according to a timetable published periodically by Post Office). At the start of each Trading Period the branch is supposed to be 'in balance'. This means that the physical stock and cash in the branch agrees with the data regarding the stock and cash held in Horizon. Then, during the Trading Period, Horizon records all customer transactions made at the branch, so it records the changes in cash and stock. It is also required to record any replenishments or remittances of cash or stock in the branch. Thus, Horizon records all changes in cash and stock held at the branch during the Trading Period, and should be able to compute, from the starting amounts and the changes, the expected amounts of cash and stock at the end of the period.
16. Without reciting the entire background to the group litigation between the Post Office and the claimants, it is the accounting and functional accuracy of Horizon that is at the heart of the current disputes, which have run for a great many years. The claimants maintain that the Horizon system in operation threw up numerous discrepancies and shortfalls in their branch accounts, for which the Post Office unfairly held them responsible. The Post Office dispute this, and maintain that these occurrences are explicable by carelessness, fault or criminality on the part of the claimants.
17. Horizon, whether in its first incarnation as Legacy Horizon until 2010, or what is called Horizon Online since then (originally HNG-X, now since 2017 HNG-A running on a different windows system), is at the centre of this group litigation. It should also be noted that it is not a system that is operated by the Post Office. It was originally designed, “rolled out” and operated by ICL, a company that was partly owned by Fujitsu, and which is now fully absorbed within Fujitsu, and has been for many years. There is a contract between Fujitsu and the Post Office in respect of Horizon, and perhaps in its current form it is rather different than the original contractual relations between those two entities. The contractual relationship between the Post Office and Fujitsu is not of direct relevance to the Horizon Issues, but arises tangentially only in respect of what is a sub-issue, namely the charging structure operated by Fujitsu for what are called ARQ requests for audit data and whether this inhibited the Post Office in this respect. The operation, functionality and accuracy of Horizon is an extremely thorny issue (or bundle of issues), although given the breadth of allegation and counter-allegation in the group litigation they are not the only issues. Providing the answers to the Horizon Issues will not lead to complete resolution of the litigation. It should however resolve one of the central issues about Horizon’s accuracy and functionality.
B. The Horizon Issues
18. The Horizon Issues are as follows. In the 3rd Case Management Order of 1 March 2018, leading counsel for the parties were ordered to meet and seek to agree the Horizon Issues to be tried in March 2019. This was done, the issues were agreed by the parties and approved by the court. I take the following from the Case Management Order dated 23 March 2018, which was the next order made, and was a Consent Order. The paragraph references to the pleadings were included in the list of Horizon Issues, which were themselves appended to the Order of 23 March 2018 itself. These include at the beginning an agreed definition of “the Horizon System”.
“AGREED LIST OF HORIZON ISSUES
definitions for the purpose of this list of issues
“the Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed, as defined in the Generic Particulars of Claim (“the GPOC”), at §16 and as admitted by Post Office in its Defence at §37.
BUGS, ERRORS AND DEFECTS IN HORIZON
Accuracy and integrity of data
(1) To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?
(2) Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as described in (1) above and if so how?
(3) To what extent and in what respects is the Horizon System “robust” and extremely unlikely to be the cause of shortfalls in branches?
Controls and measures for preventing / fixing bugs and developing the system
(4) To what extent has there been potential for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?
(5) How, if at all, does the Horizon system itself compare transaction data recorded by Horizon against transaction data from sources outside of Horizon?
(6) To what extent did measures and/or controls that existed on Horizon prevent, detect, identify, report or reduce an extremely low level of risk of the following:
a. data entry errors;
b. data packet or system level errors (including data processing, effecting, and recording the same);
c. a failure to detect, correct and remedy software coding errors or bugs;
d. errors in the transmission, replication and storage of transaction record data; and
e. the data stored in the central data centre not being an accurate record of transactions entered on branch terminals?
OPERATION OF HORIZON
Remote Access
(7) Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon remotely (i.e. not from within a branch)?
Availability of Information and Report Writing
(8) What transaction data and reporting functions were available through Horizon to Post Office for identifying the occurrence of alleged shortfalls and the causes of alleged shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the Horizon system?
(9) At all material times, what transaction data and reporting functions (if any) were available through Horizon to Subpostmasters for:
a. identifying apparent or alleged discrepancies and shortfalls and/or the causes of the same; and
b. accessing and identifying transactions recorded on Horizon?
Access to and/or Editing of Transactions and Branch Accounts
(10) Whether the Defendant and/or Fujitsu have had the ability/facility to (i) insert, inject, edit or delete transaction data or data in branch accounts; (ii) implement fixes in Horizon that had the potential to affect transaction data or data in branch accounts; or (iii) rebuild branch transaction data:
a. at all;
b. without the knowledge of the Subpostmasters in question; and
c. without the consent of the Subpostmaster in question.
(11) If they did, did the Horizon system have any permission controls upon the use of the above facility, and did the system maintain a log of such actions and such permission controls?
(12) If the Defendant and/or Fujitsu did have such ability, how often was that used, if at all?
(13) To what extent did use of any such facility have the potential to affect the reliability of the Branches’ accounting positions?
Branch trading statements, making good and disputing shortfalls
(14) How (if at all) does the Horizon system and its functionality:
a. enable Subpostmasters to compare the stock and cash in branch against the stock and cash indicated on Horizon?
b. enable or require Subpostmasters to decide how to deal with, dispute, accept or make good alleged discrepancy by (i) providing his or her own personal funds or (ii) settling centrally?
c. record and reflect the consequence of raising a dispute on an alleged discrepancy, on Horizon Branch account data and, in particular:
(i) does raising a dispute with the helpline cause a block to be placed on the value of an alleged shortfall; and
(ii) is that recorded on the Horizon system as a debt due to Post Office?
d. enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch Trading Statement after 2005?
e. enable or require Subpostmasters to continue to trade if they did not complete a Branch Trading Statement; and if so, on what basis and with what consequences on the Horizon system?
Transaction Corrections
(15) How did Horizon process and/or record Transaction Corrections?”
19. Given the nature of the proceedings, and the disputes about the Horizon system which were apparent on the face of the pleadings, I had indicated to the parties at an early stage that the generic disputes about the operation of the Horizon system, which would need expert evidence, would be resolved after the contractual issues (which became called the Common Issues, and led to Judgment (No.3)). The issues were not foisted on the parties by the court. The court approved the wording of the issues agreed by the parties during the case management stage of the litigation. There was no difficulty about this at the time.
20. However, the meaning of issues 1 and 3 in particular proved, at the Horizon Issues trial itself, to be controversial. This is regrettable. Nor was any controversy aired with the court prior to the actual trial. That too is regrettable. The court is well used to parties who disagree over the answers to certain issues in litigation generally. It is unsatisfactory when they also disagree about what any particular issue itself actually means, or the question that is being posed (including being posed to experts) by an issue, particularly once that issue has already been finalised and ordered.
21. Here, the two main areas of dispute over the meaning of the Horizon Issues were as follows (there were others, but these are the most important ones). The claimants approached Issue 1 as it was worded, requiring consideration of whether bugs, errors or defects had the potential to cause discrepancies or shortfalls in SPMs’ branch accounts or transactions. In other words, this issue was not limited to consideration of whether bugs, errors or defects had in fact actually caused discrepancies or shortfalls. The Post Office, on the other hand, approached Issue 1 as requiring consideration of whether bugs, errors or defects had in fact actually caused discrepancies or shortfalls, and by reference to the claimants’ accounts specifically. This approach by the Post Office sought, in my judgment, to narrow the scope of the Horizon Issues. This was not raised at the case management stage.
22. The second main area of dispute concerned Issue 3. This includes consideration of the concept of “robustness”. This word has been at the heart of the Post Office’s defence of the Horizon system for many years. The Post Office has said publicly, and on many occasions, that no computer system is 100% accurate and/or perfect but Horizon is “robust”. This approach by the Post Office pre-dated the commencement of the group litigation and was at the heart of the Post Office’s response to the increasing criticisms from different quarters about Horizon. I deal with this important concept below in more detail at [36] below. What the Post Office effectively means by this (in outline terms only) is that the Horizon system can properly and safely be relied upon by the Post Office for the purposes for which it is designed and intended. The parties – and this is clear on the pleadings, as well as in the terms of Horizon Issue 3 – disagree over whether Horizon is robust. In the Reply, the claimants challenged this in the following terms:
“It is therefore denied that Horizon 'is robust and [...] is extremely unlikely to the cause of losses in branches' (paragraph 16). In fact, the relatively small chance of errors admitted by the Defendant, would be likely to produce the very picture reflected in the Claimants' case.”
23. Mr de Garr Robinson for the Post Office was somewhat critical at the trial of the drafting of both Issues 1 and 3. He was not involved in the drafting of the Horizon Issues, as the Post Office had used different leading counsel. However, that criticism overlooked that the Post Office had agreed to the wording of all the Horizon Issues through its other leading counsel, and the wording had also been approved by the court. Although there have been a total of three leading counsel altogether instructed for the Post Office thus far at first instance, with a fourth leading counsel used by the Post Office to try and appeal the Common Issues to the Court of Appeal, simply because different counsel are used does not mean that the issues ordered to be tried become different. The Horizon Issues themselves were appended to the Case Management Order itself.
24. Lists of issues, whether in group litigation or generally, are very important. I can do no better than quote Longmore LJ in Scicluna v Zippy Stitch Ltd [2018] EWCA Civ 1320, who said at [14] “ever since the Woolf reforms, parties in the High Court have been required to agree lists of issues formulating the points which need to be determined by the judge. That list of issues then constitutes the road map by which the judge is to navigate his or her way to a just determination of the case
.” In this litigation, the Horizon Issues are not only important, but are vitally important. The whole of the proceedings concerns the operation and reliability of the Horizon system. It could quite easily be called the Horizon Post Office Case. Although there are a great number of different causes of action pleaded in both directions, both by the claimants and by the Post Office, against one another, the central core of the case is about Horizon. I consider that the wording of both Horizon Issues 1 and 3 – indeed, all the Horizon Issues – to be clear and had they not been, I would not have approved the agreed wording.
25. However, some explanation is necessary given the disagreement to which I have referred.
26. The phrase “bugs, errors or defects” is sufficiently wide to capture the many different faults or characteristics by which a computer system might not work correctly. The parties in this litigation, usually but not always for convenience, would often refer simply to “bugs”, and one part of the 2nd Experts’ Joint Statement became known as “the Bug Table”. Computer professionals will often refer simply to “code”, and a software bug can refer to errors within a system’s source code, but “software bugs” has become more of a general term and is not restricted, in my judgment, to meaning an error or defect specifically within source code, or even code in an operating system. Source code is not the only type of software used in a system, particularly in a complex system such as Horizon which uses numerous applications or programmes too. Reference data is part of the software of most modern systems, and this can be changed without the underlying code necessarily being changed. Indeed, that is one of the attractions of reference data. Software bug means something within a system that causes it to cause an incorrect or unexpected result. During Mr de Garr Robinson’s cross-examination of Mr Roll, he concentrated on “code” very specifically and carefully. There is more to the criticisms levelled at Horizon by the claimants than complaints merely about bugs within the Horizon source code.
27. The wording of Horizon Issue 1, “to what extent was it possible or likely for bugs, errors or defects…..” is therefore very wide. Although that wording was proposed and agreed by the parties, I considered at the case management stage when the Horizon Issues were ordered, and still consider now, that these words clearly cover the whole range of criticisms levelled at Horizon by the claimants, and the matters that could potentially be wrong with the Horizon system without restriction. Bugs, errors or defects is not a phrase restricted solely to something contained in the source code, or any code. It includes, for example, data errors, data packet errors, data corruption, duplication of entries, errors in reference data and/or the operation of the system, as well as a very wide type of different problems or defects within the system. “Bugs, errors or defects” is wide enough wording to include a wide range of alleged problems with the system.
28. I also consider that the words “possible or likely” are also wide enough to cover both ends of the spectrum of what the parties would have the court decide. In other words, it is a neutrally worded issue of wide effect which does not, by its phrasing, indicate any particular starting point or end point. “Possible” means something potentially could happen; or whether it could happen or not. “Likely” means that something is, on the balance of probabilities, more likely to have happened than not. The issue uses both terms, and therefore poses a wide question so that the answer provided by the court will be of maximum utility in the group litigation.
29. Finally, the word “potential”, in the clause “have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon…..” clearly means, on its express terms, that this issue is dealing with possible or prospective effects, rather than whether bugs, errors or defects have in fact caused actual or alleged discrepancies or shortfalls. The way the wording of this issue was dealt with in the expert evidence will be dealt with below.
30. If there were any ambiguity in Issue 1, and I do not consider that there is, the issue specifically refers to certain passages in the pleadings. It would therefore be of assistance to consider the pleadings. The relevant passages from the Particulars of Claim as amended are as follows:
“23. However, the Claimants aver that there were a large number of software coding errors, bugs or defects which required fixes to be developed and implemented. There were also data or data packet errors. There was a frequent need for Fujitsu to rebuild branch transaction data from backups, giving rise to the further risk of error being introduced into the branch transaction records. The Claimants understand that Fujitsu maintained a 'Known Error Log' relating to some or all of these issues which was provided to the Defendant but which has not been disclosed.
24. Further, the Claimants aver and rely upon the following:
24.1. Insufficient error repellency in the system (including sufficient prevention, detection, identification and reporting of errors), both at the data entry level and at the data packet or system level (including data processing, effecting and reconciling transactions, and recording the same);
24.1A bugs and/or errors and/or defects in Horizon and any data or data packet errors had the potential to produce apparent shortfalls which did not represent a real loss to the Defendant;
24.2. Horizon is imperfect and has the potential for creating errors (as the Defendant has admitted in pre-action correspondence, in the Letter of Response, dated 28 July 2016, at paragraph 1.3);
24.3. bugs and/or errors have on some occasions produced discrepancies and/or apparent shortfalls (as the Defendant has admitted in pre-action correspondence, in the Letter of Response, Schedule 6) and such shortfalls may also have arisen from data or data packet errors; and, further
24.4. the Defendant sought and/or recovered such alleged shortfalls from Subpostmasters (as is presently understood to be admitted by the Defendant in the Letter of Response, Schedule 6, paragraphs 4.1 to 4.5).”
31. The passages in the Defence which are identified in the Issue are paragraphs 49 to 56. This is a lengthy series of paragraphs but as all are referred to expressly within Issue 1, I will reproduce them all. These state the following:
“Bugs, errors or defects in Horizon
49. As to paragraph 22:
(1) If and to the extent that the Claimants wish to assert that any of the shortfalls for which they were held responsible were Horizon-generated shortfalls, it is for them to make that distinct allegation and seek to prove it. Post Office notes that they do not make the allegation in the GPoC. It further notes that, in paragraph 20 of their solicitors' letter to Post Office's solicitors dated 27 October 2016, the Claimants make it clear that they do not allege that there is a systematic flaw in Horizon or indeed any flaw which has caused any Claimant to be wrongly held responsible for any shortfall.
(2) It is denied that Post Office has unreasonably or otherwise failed to provide "obviously relevant disclosure" in relation to bugs, errors or defects in Horizon. There has been no order or application for disclosure and, in the premises set out above, there appears to be no basis for providing such disclosure.
50. Paragraph 23 is embarrassing for its lack of particularity, in that (amongst other things) it does not identify the errors, bugs or defects on which the Claimants rely or how "large" their number was or the period in which they are said to have occurred and nor does it identify the transaction data that Fujitsu is alleged to have rebuilt, how "frequent" was the need to rebuild it or the extent of the ''risk of error" which is said to have been introduced. In the premises, Post Office cannot plead to the first three sentences of this paragraph. However:
(1) All IT systems experience software coding errors or bugs which require fixes to be developed and implemented. As is noted in paragraph 53 and 54 below, there are robust measures in place in Horizon for their detection, correction and remediation.
(2) All IT systems involving the transmission of data over the internet experience data or data packet errors during transmission and such systems routinely have protective measures in place to prevent such errors creating any difference between the data transmitted and the data received and retained by the recipient. Horizon has robust controls making it extremely unlikely that transaction data input in a branch would be corrupted when being transferred to, and stored in, Post Office's data centre in a manner that would not be detected and remedied.
(3) Like all IT systems, Horizon has backups to guard against any loss of data due to local hardware failure. Where hardware fails, the data on that hardware is recovered from the backup. Post Office takes the term "rebuild" to refer to the situation before the introduction of Horizon Online where a new terminal was introduced to a branch and the data stored on the other branch terminals (or on a disc where it was a single counter branch) was restored to the new terminal. In this context, Post Office does not accept that there was a "frequent" need to "rebuild" data from back-ups.
(4) It is admitted that Fujitsu maintain a "Known Error Log". This is not used by Post Office and nor is it in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a knowledge base document used by Fujitsu which explains how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data.
51. In paragraph 24, the Claimants again bundle many ambiguous and/ or misleading allegations together. Post Office separates out and addresses those allegations in paragraphs 52 to 56 below.
52. As paragraph 24.1 does not explain what is meant by "error repellency", what sorts of errors are referred to, what is meant by "data entry level", what would constitute "sufficient" prevention, detection, identification or reporting of these errors, or in what respects the error repellency of Horizon was insufficient, Post Office cannot plead to this paragraph. However, the general thrust of paragraph 24.1 is denied and the robust controls, procedures and practices pleaded in paragraphs 53 and 54 below are noted.
53. As to paragraph 24.1A, it is a truism that errors or bugs in an IT system and data or data packet errors have the potential to create errors in the data held in that system. However, Horizon has at all material times included technical control measures to reduce to an extremely low level the risk of an error in the transmission, replication and storage of the transaction record data. These have varied from time to time and they currently include the following:
(1) Horizon creates, transmits and stores trans action data in the form of "baskets". A basket is a complete transactional session between a customer and Post Office and may include one, several or many individual transactions taking place within the same session. Horizon will not accept a basket of transactions that does not net to zero (i.e. the value of any sales is set off by the value of any payment made or received). This reduces greatly the risk of any error in the data entered within any given basket.
(2) If a basket of transactions fails properly to complete its transmission to the central database (because, for example, of a power loss), the system rejects any partial transmission and requests the full basket from the branch terminal. This reduces greatly the possibility of baskets of transactions failing to be recorded.
(3) At the point of a basket being accepted by Horizon, it is assigned a unique sequential number (a "JSN") that allows it to be identified relative to the other baskets transmitted by that branch. This reduces greatly the risk of recording duplicate baskets or there being a missing basket.
(4) Each basket is also given a digital signature, i.e. a unique code calculated by using industry standard cryptography. If the data in the basket were to change after the digital signature was generated, this would be apparent upon checking the digital signature.
(5) Initial data integrity checks are undertaken when baskets are received at the Post Office data centre from a branch. Baskets are then copied from the central database to the Audit Store where a digital seal is then applied (the "Audit Store Seal"). If the baskets and/ or the data within the baskets were altered after the application of the Audit Store Seal, this would be apparent when the baskets are extracted from the Audit Store.
(6) Horizon and the above controls are themselves subject to various audits and checks including audits carried out by third parties.
54. Further as to paragraph 24.1A, in addition to the technical controls referred to above, there are several operational procedures and practices conducted by Post Office and Subpostmasters that serve to increase the reliability of the data stored in the central data centre as an accurate record of the transactions entered on branch terminals. These currently include the following:
(1) For many transaction types, Post Office compares its own transaction record against the corresponding records held by Post Office clients. If an error in Horizon were to result in the corruption of transaction data, this should be revealed by the comparison.
(2) There are detailed procedures in place to address the risk of data loss resulting from interrupted sessions, power outages or telecommunications failures in branches. These are set out in the "Recovery - Horizon Online Quick Reference Guide" and Horizon guides the system user through the recovery process (which include completing any transactions that are cut short). These procedures should prevent any data errors arising from interrupted sessions, power outages and telecommunications failures.
(3) The display of the transactions being effected on-screen at the branch terminal allows the user of the system to identify any inconsistency between the information shown on the screen and the transaction that the user has keyed into the system. If, for example, a hypothetical bug in the terminal were to cause a key-strike on number S to be recorded as an input of number 6, this would be detected rapidly by system users, given the large number of system users and the huge number of transactions effected on Horizon.
(4) The accounting and record-keeping obligations placed on Subpostmasters reduce the risk of any errors going undetected. For example, there is an obligation for each branch to count and declare to Post Office the cash it holds on a daily basis, which increases the likelihood of promptly detecting any overstatement or understatement of the cash position on Horizon. If a Subpostmaster detects that an error has been made at an early stage, its cause is more likely to be identified.
(5) Fujitsu operates industry standard processes for developing and updating Horizon and for investigating and resolving any identified potential system errors.
55. As to paragraph 24.2, Post Office admits that, like all other IT systems, Horizon is not a perfect system which has never had any errors or bugs. However, as indicated in paragraphs 53 and 54 above, it has robust systems in place to identify them, fix them and correct their consequences (if any).
56. As to paragraphs 24.3 and 24.4:
(1) There have been occasions on which bugs or errors in Horizon have resulted in discrepancies and thus shortfalls or net gains in some branch accounts, as outlined in Schedule 6 of the Letter of Response. It is denied (if it be alleged) that such bugs or errors have affected any of the Claimants.
(2) On each occasion, both the bugs or errors and the resulting discrepancies in the relevant branch accounts were corrected. Post Office took steps to ensure that it had identified all branches affected by the bugs or errors and that no Subpostmaster was ultimately held responsible for any resultant shortfalls.
(3) Paragraphs 4.1 to 4.5 of Schedule 6 to the Letter of Response relate to the so-called Suspense Account Bug. Without prejudice to the burden of proof, none of the branches affected by the Suspense Account Bug are branches for which the Claimants were responsible.
(4) None of the Subpostmasters whose branches were affected by the Suspense Account Bug were ultimately held responsible for the shortfalls that it generated. The Claimants are therefore wrong to understand Post Office as having admitted that it "recovered such alleged shortfalls from Subpostmasters". Where Subpostmasters in the affected branches had made good or settled centrally shortfalls that were later corrected, those Subpostmasters received a payment or credit in the amount of the shortfall.”
32. It can be seen that the word “robust” is used throughout these passages of the Defence, and “robust systems” in paragraph 55. Given “robustness” forms part of Issue 3 there is no need separately to consider it at this stage when analysing the differences between the parties about what Issue 1 means.
33. The passages in both parties’ pleadings which I have reproduced above show that it is the potential, in general terms, of the Horizon system to have the effect complained of that is the subject matter of Horizon Issue 1. In order to read Issue 1 in the manner contended for by the Post Office, one would have to delete entirely the phrase “to have the potential” from the issue. One would also have to substitute the term “Subpostmasters’ branch accounts or transactions” with “the claimants’ branch accounts or transactions”. That is not what Issue 1 states, it is not what the parties agreed the issue would be, and it is not the issue that the court approved and ordered. In my judgment, the approach of the Post Office to what Horizon Issue 1 actually means is too narrow.
34. In my judgment, the correct construction of Horizon Issue 1 is that contended for by the claimants. In other words, it involves a two-stage process. Firstly, consideration of whether there were, or are, bugs, errors or defects in the Horizon system as alleged by the claimants. Secondly, if the answer is that there were, or are, such bugs, errors or defects, the second stage is to consider whether these have (or did have previously) the potential to cause apparent or alleged discrepancies in SPMs’ branch accounts generally. The issue is not whether such bugs, errors or defects did in fact cause such discrepancies or shortfalls in the claimants’ accounts specifically. That separate or different issue – the effect upon claimants’ branch accounts - is a more claimant specific one. It will have to be determined at some stage, for any of the claimants whose individual claims come to trial in the future. It may require expert forensic accountancy evidence. It was not ordered to be dealt with in the Horizon Issues trial. The Horizon Issues were intended to be, and in my judgment on their wording are, generic issues relating to Horizon and its operation. However, if a bug, error or defect is shown to have had an actual impact on any SPM’s accounts, then by definition it has had the potential to have such an impact. Actual impact on branch accounts can therefore be of assistance in considering Issue 1 as ordered, namely including potential.
35. The passages in the pleadings quoted above do show that the Post Office had, in Schedule 6 to its Letter of Response, accepted the existence of two bugs or errors in Horizon, one of which was called the Suspense Account Bug, but denied that the latter had affected any branch accounts of any of the claimants. The passages in paragraph 56 of the Defence that explained that no SPMs had been “ultimately held responsible for the shortfalls that it generated” reads a little differently with the hindsight provided from evidence in two lengthy trials. This is because the Post Office’s Defence effectively accepts that “Subpostmasters in the affected branches had made good or settled centrally” the sums in question – that is the shortfalls in their branch accounts that Horizon showed – but states that these were later corrected. Both the expressions “making good” and “settled centrally” means that the Post Office had originally held the SPM responsible for the losses, as the shortfalls formed part of their branch accounts. These terms mean that the SPMs had either paid the money (making good) or been given time to pay sums which the Post Office treated as a debt, even if they were disputed sums (settling centrally), meanings given to these terms that were confirmed by Ms Van Den Bogerd in her evidence for the Common Issues trial. However, the pleading means that the Post Office had then, on the Post Office’s case, subsequently corrected this impact on branch accounts which was why it pleaded no SPMs were “ultimately held responsible”. Therefore there might prove not to be as much between the parties on this point in reality, as the group litigation unfolds, as there appears to be on the pleaded case. The Suspense Account Bug undoubtedly had actual impact upon SPMs’ branch accounts.
The meaning of “robustness”
36. Turning to the disagreement about Issue 3, given the parties disagreed about whether the Horizon system is (or was) “robust”, it is a fairly elementary step to consider the meaning of that term, and how it is being used by the parties. Context is important so far as the meaning of the word “robust” is concerned. If someone is in robust health, it usually is taken to mean that they are healthy, or even very healthy. A robust exchange of views can be a polite way of referring to an argument. Given the importance of the concept to the Horizon system, its prominence in the Post Office’s defence of the system, and its express inclusion (admittedly in inverted commas) in the Horizon Issues, I asked each side in the litigation during oral closing submissions for a reference from their pleadings or submissions for the meaning which they ascribed to the word. I referred to this as their benchmark definition. Robustness was referred to by both sides in the litigation in numerous places, but not always in the same precise terms, and clarity is to be welcomed.
37. The claimants answered this by reference to the remainder of Issue 3, namely “extremely unlikely to be the cause of shortfalls in branches” and explained that the claimants had found the word robust “difficult to define” other than by reference to this. This would mean therefore that it had no separate independent meaning other than as a summary of the longer second part of Issue 3. In other words, a robust system would be one that is extremely unlikely to be the cause of shortfalls in branches. The claimants also implicitly, if not expressly, criticised use of the term both by the Post Office in its pleadings and written submissions as being more aligned to public relations than as a performance standard.
38. The Post Office asked for some time to provide the reference that I requested. Given the meaning of “robust” is so central in the Post Office’s defence of the Horizon system, I granted the Post Office the time that was requested.
39. The Post Office subsequently, after the trial ended, submitted a short document entitled “the Post Office’s case on the meaning of robustness”. This was not what was intended when I sought a reference from the Post Office to their definition, and the document submitted went rather further and made wider ranging submissions. The document did state, so far as the meaning of the word is concerned, the following:
“In Post Office’s submission, the meaning of robustness is a matter for expert opinion. Robustness is a well-established concept in the IT industry and is the subject of academic study: see para. 361 of Post Office’s written closing”.
“There are different dimensions of robustness, such as robustness against hardware failure, software defects and user error. The robustness of the system also depends on the processes around it.
Robustness does not mean perfection; but that the consequences of imperfection must be managed appropriately. If the extent of imperfection is too high, this would be very difficult to do which would imply less robustness.
Horizon has evolved since its inception. Therefore, its robustness may have varied throughout its lifetime. The level of robustness may have increased or decreased as the system was changed.
The existence of branch shortfalls is agreed. The experts do not agree at this point as to whether this indicates any lack of robustness.”
“For the purposes of addressing the robustness of Horizon, I have applied the following definition of robustness:
‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling.’ ”
In consideration of the likelihood of Horizon to be the cause of shortfalls in branches, Horizon is not determined to be robust in this regard because:
(a) it contained high levels of bugs, errors and defects as set out under Issue 1 above which created discrepancies in the branch accounts of Subpostmasters;
(b) it suffered failures of internal mechanisms which were intended to ensure integrity of data;
(c) the system did not enable such discrepancies to be detected, accurately identified and/or recorded either reliably, consistently or at all;
(d) the system did not reliably identify ‘Mis-keying’, which is inevitable in any system with user input, and did not reliably have in place functionality to restrict users from progressing a mis-key;
(e) it required numerous processes and workarounds to be in place to allow Fujitsu to modify data already recorded by Horizon, which would not be required in a “robust” system; and/or
(f) there were weaknesses and risks of errors and other sources of unreliability within Horizon.”
(italics present in original)
42. Dr Worden stated in the same Joint Statement:
“The definition of 'robust' proposed above by Mr Coyne is not adequate, for reasons given below. The term 'robust' is not, as implied in para 3.1 of the outline, either ill-defined or a piece of IT public relations. Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?
In my report I shall survey the evidence I have found that Fujitsu paid sufficient attention to the dimensions of robustness, and that they did so successfully. I shall also address evidence from Mr Coyne implying that Horizon fell short of its robustness objectives.
In my current preliminary opinion, Horizon is a highly robust system, and this has important implications for the other Horizon issues, notably issue 1.”
43. It can be seen therefore that Dr Worden in the Joint Statement did not agree Mr Coyne’s definition, and expressly said it was not adequate. In any event, the meaning of any word – even “robust”, or “robustness” – ought to be capable of description by the parties themselves. Although on its face it did not appear that Dr Worden agreed with Mr Coyne’s definition, a footnote in the Post Office first set of post-hearing submissions suggested that Dr Worden was not disagreeing with the first part of Mr Coyne’s text, in other words that part of the text that contained his definition of robustness (which was in italics in the 1st Joint Statement). Obviously if the parties (or their experts) could agree the definition to be applied so far as the Horizon System is concerned, that ought to be identified. I therefore asked the Post Office via email whether it agreed with the definition adopted by Mr Coyne, and if not, what its alternative definition was.
44. This led to a further document being received from the Post Office dated 18 July 2019. It referred to the passage in the 1st Joint Statement (which is quoted at [40] above) as “the agreed definition”. That rather overlooks that Mr Coyne identified the definition of robustness which he was applying, and Dr Worden expressly disagreed with this in the same Joint Statement under the heading “Areas of Disagreement”, and stated “the definition of ‘robust’ proposed above by Mr Coyne is not adequate, for the reasons given below”. It also overlooks that in the 3rd Joint Statement, paragraph 3.1 had an agreed entry which stated the following:
“Irrespective of how you define the detail of robustness, in line with most other large-scale computer systems, Horizon's robustness has generally improved.
From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events.
This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”
(emphasis added)
45. Later in the same document of 18 July 2019 the submission was made by the Post Office that “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. It was also submitted that “As Post Office understands it, this is what Mr Coyne meant when in his comments in [the 1st Joint Statement] he defined robustness as “the ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.
46. This was precisely the definition which Dr Worden, in his areas of disagreement on the 1st Joint Statement, described as “inadequate”. The end position therefore is as follows.
47. The claimants found “robustness” difficult to define in the abstract and tied it in with the other wording of Horizon Issue 3; a robust system would be “extremely unlikely to be the cause of shortfalls in branches”. That however is a consequence of how a robust system would operate, not a definition of what robustness means.
48. The Post Office defined it as follows: “the robustness of a system is the effectiveness of the system in managing the risks of imperfections (which are inevitable in any system) and their consequences”. The Post Office was also prepared to accept Mr Coyne’s italicised definition in the 1st Joint Statement, namely ‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform correctly in any scenario, including where invalid inputs are introduced, with effective error handling”.
49. Mr Coyne applied the definition he set out in italics in the 1st Joint Statement, quoted in the immediately preceding paragraph of this judgment and at [41] above.
50. Dr Worden’s definition was as follows:
“Robustness (which is closely related to resilience) is an engineering objective, and large parts of project budgets are devoted to achieving it. It receives its meaning in the phrase 'robust against... [some risk or threat]', and there are a large number of risks that business IT systems need to be robust against - such as hardware failures, communications failures, power cuts, disasters, user errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean 'be perfect'; it means 'address the risks of being imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon robust? and: in each dimension, how large were the remaining risks?”
51. The Post Office also submitted that Mr Coyne’s definition was not “materially different” to that of Dr Worden.
52. The Post Office made submissions in paragraph 3(b) of the written submissions dated 18 July 2019 on robustness that stated that Mr Coyne cannot have intended to exclude the effect of countermeasures when he considered the concept of robustness. I shall return to this topic when dealing with countermeasures. This is because some of the countermeasures considered by Dr Worden are not parts of the Horizon System at all, such as SPMs noticing adverse entries in their branch accounts, and the manual issuing of Transaction Corrections (TCs) by the Post Office (which both parties agree are outside of the Horizon System).
53. I do however accept the Post Office’s submissions that there is not a great or material difference in the definitions of robustness adopted by the parties’ experts. I do not accept the claimants’ submission that robustness is difficult to define. Dr Worden defined robustness by using what he termed as “the dimensions of robustness”. It is rather circular to describe the meaning of robustness as being “robust against” some particular risk. Although Mr Coyne provided his definition in the 1st Joint Statement, the statement by Dr Worden that this was “inadequate” may only have been aimed at the entirety of Mr Coyne’s entry in the areas of disagreement, as effectively accepted by the Post Office in their most recent written submissions on the subject. Whether that is an explanation of the lack of agreement in the Joint Statement, I also agree with the Post Office that Mr Coyne’s definition is not materially different to that used by Dr Worden.
55. The exercise necessary above, to arrive at the definition of robustness in [54] above, is not judicial pedantry. Given the central importance of robustness to the disputes about the Horizon System, and the Horizon Issues, it is in my judgment essential. It is mildly surprising, given how central the assertion of robustness has been to the Post Office’s defence of the Horizon System, that Dr Worden’s interpretation of the term has been relied upon so heavily by the Post Office, given the term was used by the Post Office for some years prior to his involvement.
56. However, regardless of that passing observation, I find that both experts correctly understood what robustness in fact means, and applied the definition at [54] above in considering their expert evidence. I will return to the expert evidence in some detail later in the judgment, including in the Technical Appendix.
C. Features of this Group Litigation
57. There are many different ways of managing group litigation. The subject matter of such litigation is different from group to group, and what is appropriate in one set of proceedings will not necessarily be the best approach in another set. Group litigation may involve many claimants – even tens of thousands, in some cases – all with what is essentially the same type of claim, governed by a number of common issues that apply across all or most of the cases. Some group litigation may involve similar numbers of claimants, all with similar claims that are factually different. In this case, there are 584 claimants, which in the context of some group litigation, is not very many, although it is still in the many hundreds, and the period of time over which the relevant events are said to have taken place is about 16 or 17 years. There are different aspects to each individual case, but to deal with the litigation efficiently, cost-effectively and proportionately it is simply not feasible for the same judge to try all the claims, one after the other, in full on all of their respective merits. Such a process would take several years. It is not what group litigation is intended to achieve.
58. The concept of group litigation is that the Managing Judge, whoever he or she may be, with the assistance of the parties insofar as that may be available, selects the most suitable mechanism for that particular set of proceedings in order to achieve compliance with the overriding objective. That will – or should – lead to cost and time saving. That does however require a high degree of co-operation from the litigants. Here, my intention of holding a substantive trial each judicial term onwards to resolve as much of this group litigation as possible, as quickly as practicable, became simply unachievable as a result of the issuing of the recusal application by the Post Office, explained at [5] above. A further trial in the autumn of 2019 also became undesirable as a result of the parties wishing to have time to consider mediation, and to find out whether the Court of Appeal would grant the Post Office permission to appeal on the contractual issues in Judgment (No.3). Over the numerous hearings and two full substantive trials that I have conducted, I have gained the distinct impression that the Post Office is less committed to speedy resolution of the entire group litigation than are the claimants, but it is not possible to state with finality whether that is correct.
59. All litigation is important to those involved in it. In this group litigation, all the claimants are of the view summarised at [10] in Judgment (No.3) in terms of the behaviour of the Post Office. Recitation that the claimants are of that view does not mean that any similar views are held by the court, or that any decisions have yet been made by the court on that point one way or the other. The total sums claimed by way of liquidated sums (excluding heads of general damages not yet quantified) are approximately £18.7 million. The Post Office counterclaims for certain sums against the claimants and also alleges fraud. Reputations are plainly at stake on both sides. That is the case in many types of proceedings, particularly high-profile ones that attract press interest. However, that sum of money is not large, in the context of large scale and protracted litigation. The parties’ joint costs are approximately now £27 million, and because of the notification provisions in one of the earlier Case Management Orders I made, regular notifications of the total sum of costs are made to the court. The Post Office alone spent over one million pounds in little more than a month earlier this year; the notification letter of 13 May 2019 from its solicitors stated that its costs were in excess of £12,800,000; by the time of its letter of 25 June 2019, the Post Office’s costs were in excess of £13,900,000. Both this level, and rate, of expenditure is very high, even by the standards of commercial litigation between very high value blue chip companies.
60. I have already explained that the Post Office has now used four different leading counsel, and it also engaged a second commercial firm of solicitors to act for it on its appeal. The claimants are funded by means of litigation funding, explained in Judgment No.3 so far as it is relevant. The claimants are ordinary individuals who ran branch Post Offices, and the Post Office is publicly funded, so none of the parties are high value FTSE 100 companies. Not all of the Post Office’s costs will be recoverable costs regardless of the outcome, not least due to the existence of two Costs Management Orders which are in smaller amounts than the Post Office’s rapidly growing grand total, but the total figures still represent real expenditure of actual money.
61. These proceedings also have the additional feature of criminal convictions on the part of some SPMs. As explained at [10] in Judgment (No.3), there is a Criminal Cases Review Commission (“CCRC”) review underway in respect of the convictions of some claimants. I have been told that these are subject to a stay pending judgment upon the Horizon Issues. I have explained this before in open court, but matters such as criminal convictions are no part of this group litigation. This court has no jurisdiction over such matters.
64. Prior to considering the “rule” to which he was referring – it is not one of the Civil Procedure Rules, so far as I am aware – it was therefore necessary to identify exactly what “different proceedings” were being referred to in this disagreement between the parties. There are five elements to this matter.
(1) Firstly, it had never been raised by any party at any stage during this group litigation that there were any criminal prosecutions currently underway such that it was necessary to consider any reporting restrictions for the group litigation. I expressly raised this with the Post Office on the occasion referred to at [62] and [63] above, who confirmed there were no such prosecutions underway. The “different proceedings” to which Mr de Garr Robinson referred were the proceedings already before the CCRC.
(2) Secondly, the possibility of future (as opposed to current) criminal prosecutions, or the potentially criminal impact upon individual SPMs, did more than hover in the background to the Horizon Issues trial. Some claimants who gave evidence in this trial were expressly accused by the Post Office of criminal offences in cross-examination in this trial, something which had also occurred in the Common Issues trial. It should be clearly understood that any findings I make in respect of any witnesses do not determine with finality any issues to be tried in any particular trial to follow, including the trial(s) of their individual claims.
(3) Thirdly, the rule to which leading counsel for the Post Office referred was not identified. Journalists must be alive to the risk of serious prejudice when criminal proceedings are live, as a result of the Contempt of Court Act 1981, and are thereby restricted in reporting matters generally in that respect. That statutory provision replaced what had been referred to as the sub judice rule, but this did not relate to statements made in evidence in open court. The principle of open justice is a most important one and there were no restrictions imposed on any reporting of the Horizon Issues trial (nor the Common Issues trial) at all, nor were any sought by the claimants or by the Post Office.
(4) Fourthly, there was no specific restrictions imposed on the parties by the court in terms of the evidence they could adduce and what could be said by way of submissions (written or oral), due to potential impact upon any “different proceedings”. I have not heard any argument on the matter, but I do not consider it would be proper to do so in any event. It was also not argued before me that the CCRC proceedings, which are subject in any event to a stay pending at least part of the outcome of the group litigation, are in any way of such a character that reporting of the Horizon Issues trial ought to be restricted in any way, or such that they could be prejudiced by the group litigation. Indeed, the presence of a stay of those proceedings pending the outcome of some of the group litigation issues would suggest directly to the contrary. The objection therefore by the Post Office to the way in which the claimants sought to make closing submissions on the actual impact upon different SPMs of what were said to be bugs, errors and defects in the Horizon System was, so far as it could be understood, not a well-founded one.
(5) Finally, the claimants by their counsel were trying to draw parallels between what had happened to one particular SPM regarding “doubling up”, and numerous entries in internal Fujitsu and Post Office documents prior to that of similar “doubling” occurrences being caused by bugs, errors or defects. Those parallels are obvious on the face of the documents. The degree to which the CCRC find such parallels of assistance, if at all, is a matter entirely for them. The Post Office submitted most strongly that these different references were not to one single bug, but were references to a number of different issues or bugs that had been experienced in Horizon over the years. Whether that makes the claimants’ point for them or not, it is important to remind all the parties that the issues in this litigation are not going to be decided with sympathy, or lack of it, coming into account in the analysis in any respect whatsoever.
65. Mr Green for the claimants submitted that the Post Office was “trying to distract from the stinging nature of the underlying documents by objecting” to the submissions and that the Post Office may be finding it “uncomfortable” to have its own documents, in a particular date range, highlighted by reference to particular criminal cases, prosecutions mounted by the Post Office, and the experiences of SPMs in those cases. Certainly the Post Office and Fujitsu have been exposed to a degree of scrutiny in this litigation which does not appear to have occurred before; however, there has not been litigation of this type on these issues before.
1. The Senior Courts of England and Wales (who acquired this name by reason of the Constitutional Reform Act 2005) comprises the Court of Appeal, the High Court and the Crown Court.
2. The Court of Appeal has two divisions, Criminal and Civil. Appeals from the High Court are to the Civil Division. Appeals from the Crown Court are to the Court of Appeal Criminal Division, also known as the CACD.
3. The High Court is the highest court of first instance in civil cases.
4. The Crown Court is the highest court of first instance in criminal cases. Some criminal matters are dealt with by the High Court (by way of Divisional Court) but these are narrow in scope and do not arise in any respect concerning the group litigation. All appeals against both convictions and sentence from the Crown Court are dealt with by the CACD.
5. The Criminal Cases Review Commission, or CCRC, is a statutory body responsible for investigating alleged miscarriages of justice in England, Wales and Northern Ireland. It was established by section 8 of the Criminal Appeal Act 1995. It is an independent non-departmental government body, and has the power to refer cases to the CACD.
6. This group litigation is concerned only with the issues arising in the civil claims being brought against the Post Office by the claimants, and the Post Office’s counterclaims. It will result in a series of judgments on those issues which are public. What, if anything, the CCRC do in any respect following any of the judgments is entirely a matter for the CCRC and forms absolutely no part whatsoever of the group litigation.
7. This court has no jurisdiction in respect of any of the convictions of those SPMs who were successfully prosecuted by the Post Office. Although the presence of criminal convictions does have evidential effect in respect of individual claims by individual claimants who have been convicted of false accounting, these have not arisen in either of the two substantive trials held to date (Common Issues and Horizon Issues) nor will they arise in either of the next two (the principles governing Heads of Loss, and then some individual claims).
67. There has been no restriction imposed on any party, or any witness, in this group litigation by the court in terms of the evidence that could be adduced, or submissions that could be made, with one exception. Mr Henderson of Second Sight was subject to a restriction upon his evidence by reason of the terms of an agreement that he had entered into with the Post Office when Second Sight were engaged in what was originally intended to be consensual resolution of the claims. This was called the Second Sight Mediation Scheme. This feature of the evidence of Mr Henderson is a matter which is dealt with in more detail in Part D of this judgment. It was a restriction imposed upon him, or agreed, by the parties, and the court was not involved in that restriction in any way.
68. Turning to a different matter, I have already made certain criticisms of the Post Office in Judgment (No.3) in terms of how it had conducted itself in the litigation, and I had also made criticisms of some of its senior witnesses. One of those witnesses, Ms Van Den Bogerd, was also called as a witness in the Horizon Issues trial. So far as I am concerned, she came to the court for the Horizon Issues trial with a clean slate in terms of whether her evidence on the Horizon Issues would be accepted or not. A different way of putting the same point is that simply because certain adverse findings had been made concerning the evidence she had given in the Common Issues trial, this did not mean that I adopted any particular starting position so far as her factual evidence was concerned for this trial. The court was entirely neutral in terms of starting position. Although it was certainly not a point that went to her credit that I had already made adverse findings of the accuracy of her evidence in the Common Issues trial, that did not mean that I started with any fixed view of the likely accuracy of her evidence in the Horizon Issues trial. Her evidence in the Horizon Issues trial is dealt with at [203] below.
69. Further, before turning to the detail of each side’s evidence, an approach was adopted by the Post Office on occasion of seeking to adduce what was (or should have been) in reality evidence of fact, but by way of submission, or points made “on instruction”. Sometimes, depending upon the nature of the subject matter, such an approach is understandable or unavoidable, and may be unobjectionable. It is not therefore sensible to state that this should never be done in any conceivable circumstance in any trial. However, on important points that have been dealt with by a particular witness in their evidence of fact, it is not a suitable device to adopt. This was particularly done in terms of the cross-examination of Mr Coyne concerning evidence already given by Mr Godeseth in his cross-examination, about alteration by Fujitsu of a particular branch account. I deal with that in detail at [376] to [379] below.
70. This was also done in Appendix 2 of the Post Office’s Closing Submissions, where (sometimes detailed) factual explanations were given in respect of bugs in the Bug Table. That appendix was compiled by different teams of solicitors and counsel, something explained by the Post Office when, some months after it was submitted, they discovered that three pages were missing and sought permission to serve them rather late. I granted permission for them to be added, as they had been prepared before the deadline for service and omitted due to an administrative oversight.
71. Submissions should not contain evidence, or positive evidential assertions, that are not present in the evidence served in the trial. This is a fundamental point. I provide some examples in the Technical Appendix by reference to specific entries for specific bugs. Blurring (or ignoring) the lines between submission and evidence is entirely unhelpful. Evidence is something that comes from a witness (lay or expert) and which the opposing side is entitled to test by way of cross-examination. It is not appropriate for detailed factual assertions to be made in closing submissions that are not directly referable to evidence in the case. There is no way such factual assertions can be tested; if they come in closing submissions, there is no way that the opposing party can deal with those assertions in their own evidence, or even put relevant points to witnesses for the other party in cross-examination.
72. Further, this is not a case that is being tried in a Specialist List, such as the Technology and Construction Court – it is a general Queen’s Bench Division case – but it readily could have been tried in such a list. It contains a great deal of technical subject matter, particularly in this trial dealing with the Horizon Issues. The two IT experts have each given evidence in other computer litigation before. Such subject matter, and such expert evidence, is readily suited to analysis by the parties and precision, which is the usual approach of courts generally. In my judgment, this is particularly important in technical matters such as these. Bluster, unfounded assertion in cross-examination, detailed technical explanations of fact given “on instruction” and submission are rarely helpful.
73. Finally, there has been a vast amount of highly detailed material deployed by both sides, not simply evidence of fact, and expert evidence, but also reference to a great many documents. The use of an electronic bundle has made this easier to manage than otherwise, but written submissions alone were in excess of 1100 pages in total from both sides. It is neither possible nor desirable to recite in this judgment, or in the Technical Appendix, or resolve, every single disputed item, or every single disputed fact, no matter how minor. I only make findings in this judgment that are necessary to enable me to resolve the Horizon Issues themselves. Simply because I do not specifically refer to a particular submission or piece of evidence, it should not be thought that I have not had regard to it. I have considered all the material, evidence (both factual and expert), submissions, and passages in contemporaneous documents, multiple times. The experts’ agreements in particular have been of great assistance, but everything has been considered. This judgment, together with the Technical Appendix, will be of substantial length, and to recite everything would very probably make it of unmanageable proportions. It is in any event far longer than is ideal, but that is partly explained by the fact it deals with the life of a complex computer system that spans some 18 years.
74. In adopting this approach, I have borne very much in mind the overriding objective in the Civil Procedure Rules, the need for proportionality, but also the obvious need to provide a reasoned judgment. I have also taken specific account of the dicta of Males LJ in Simetra Global Assets Ltd and another v Ikon Finance Ltd and another [2019] EWCA Civ 1413. That case concerned foreign exchange trading, and claims for dishonest assistance and damages for deceit and conspiracy against a total of 12 different defendants, both personal and corporate. Although it concerns those matters, plainly very different to the Horizon Issues, it does state generally what ought to be included in a judgment. The requirement for a judge to give adequate reasons in a judgment is analysed at [37] to [46], with the expression “the building blocks of the reasoned judicial process” (used by Henry LJ in Glicksman v Redbridge Healthcare NHS Trust [2001] EWCA Civ 1097) approved at [42] by Males LJ. Four points are summarised at [46] of Simetra:
75. I am acutely conscious that the first of those points, succinctness, is not likely to be achieved in this judgment. This is not only due to the nature of the subject matter, the Horizon Issues, but also the fact that both Legacy Horizon and Horizon Online are involved, and the system was brought in some time ago, namely the year 2000. Due to the nature of the claims brought (and the limitation issues certain to arise), the functionality of the Horizon system(s), robustness, and the other Horizon Issues had to be dealt with at an early stage in the group litigation. I will return to the other three of Males LJ’s four points after my review of the evidence, as well as his comments on the importance of contemporaneous documents at [48] and [49] of Simetra. I do this at [937] and following below.
D. Evidence of Fact: The Claimants
76. The claimants originally served statements from a greater number of witnesses of fact than were in fact called at the trial. This was because, just before the Pre-Trial Review on 14 February 2019, the Post Office objected to the evidence of two particular individuals. Very shortly before the Common Issues trial, the Post Office had issued an application to strike out a significant number of passages in the six Lead Claimants’ witness statements, and that application had been dismissed in Judgment (No.2) “Strike Out”. On this occasion, although no application to strike out was issued, the Post Office sought a ruling at the PTR in respect of the evidence of Mr McLachlan and Mr Henderson.
77. Mr McLachlan had been called as an expert witness in the criminal trial of Mrs Seema Misra, a SPM at West Byfleet in Surrey who was charged both with theft from her branch, and also false accounting. He was called for the defence. The sums in question were approximately £74,000. Mrs Misra pleaded not guilty, and her defence was that the Horizon system was to blame. She was convicted by a jury after a trial in late 2010 at Guildford Crown Court, and was sentenced to a term of imprisonment of 15 months. She was pregnant at the time of her conviction and imprisonment. Expert evidence was called by the Post Office at her trial from Mr Gareth Jenkins from Fujitsu. Mr Jenkins was not called as a witness by the Post Office in the Horizon Issues trial before me, but a large amount of the evidence from the Fujitsu witnesses was attributed to information directly given to them by Mr Jenkins. This was a controversial matter between the parties at the Horizon trial. The Post Office did not proffer an explanation for Mr Jenkins’ non-appearance as a witness; they were not obliged to do so. However, in their closing submissions and after all the evidence had been called by both parties, they did so. The explanation provided was by way of submission and not evidence. This is a matter with which I deal with further at [509] to [513] below. Prior to the PTR, the Post Office objected to Mr McLachlan’s evidence on the basis that it contained opinion evidence and the claimants did not have permission for it, which is something required under the CPR. This was resolved by the claimants deciding not to call Mr McLachlan, and explaining at the PTR itself that they would not call him. On the face of it, the claimants were wise to concede this point as if his statement had included opinion evidence, the Post Office would have had a valid point of objection. I have not therefore considered the evidence of Mr McLachlan which forms no part of the case.
1. Interim Report dated 8 July 2013;
2. Briefing Report Part One dated 25 July 2014;
3. Briefing Report Part Two version 1 dated 21 August 2014;
4. Briefing Report Part Two version 2 dated 9 April 2015.
80. Firstly, the answers to the different Horizon Issues would not be determined based on the accuracy or correctness of the contents of the Second Sight reports themselves. The court was to hear detailed expert evidence from two IT experts whose evidence would go directly to the Horizon Issues. Whether the Second Sight reports were, or were not, correct in their conclusions (which were generally critical of Horizon) would not form part of that process, and would not even qualify as a sideshow. Certainly their contents would not be determinative of the Horizon Issues in the group litigation. I have had no regard to the contents of the different Second Sight reports, nor to the Second Sight conclusions, in arriving at the answers to the Horizon Issues. I am aware that all of the Second Sight conclusions were challenged by the Post Office.
81. Secondly, it is not for the court in civil litigation to identify in advance to any litigant that it need not cross-examine upon particular evidence of fact of the opposing party. This is particularly so in a time-limited trial such as this one, but in my judgment is a general point of principle. The court was effectively being asked to direct the Post Office as to how it should conduct its case, and also consequentially, about how much of its time at the trial should be used for particular witnesses. This is not the function of the court. I was aware that Mr Henderson’s evidence was challenged by the Post Office. It is for this reason that he was to be cross-examined. How the Post Office chose to do that, the trial time its advisers chose to allocate to that exercise, and which (if any) parts of the statement were to be challenged and how, were all matters for the Post Office to decide.
82. Thirdly, it would in any event be wholly unconventional in a time-limited trial dealing with the subject matter of this group litigation for the court to give any weight to any submissions by the claimants that particular points of criticism or detail in such detailed documentation as the Second Sight reports were essentially “accepted” by the Post Office because they had not been specifically challenged in cross-examination. There is never sufficient time, in any time limited trial, to cross-examine upon everything. A time limited trial in the 21st century is not conducted in the same manner as a trial would have been many years ago, particularly in a detailed technical dispute such as this one. There was no forensic route available to the claimants whereby a failure to cross-examine upon the Second Sight conclusions or reports, whether due to the length of the trial or otherwise, would or could lead to a conclusion by the court that some aspect of the Second Sight reports was agreed or admitted by the Post Office, when it plainly was not. The period in question in the group litigation spans some 18 years. If attention is to be paid to other court users, the overriding objective and the Civil Procedure Rules, it is simply not possible (nor desirable) for any trial judge to permit the parties to have unlimited time to cross examine upon everything. The quid pro quo of that is a party cannot be expected to cross examine upon everything. There should have been no concern on the part of the Post Office that the Second Sight report(s) would be taken as not challenged by the Post Office unless their contents were subject to cross examination. Given there appeared to be such a concern, I explained the position at the PTR. The claimants also expressly said at the PTR that no such point would be taken. That latter element was unnecessary, as whether the point was taken by the claimants or not, it would not have been accepted.
83. The position was therefore clarified or explained at the PTR and the Post Office, notwithstanding that no formal ruling was made on those specific parts of Mr Henderson’s statement that it had to challenge in cross-examination, appeared to be content. A transcript of this hearing is available. The contents of the Second Sight reports were not cross-examined upon when Mr Henderson gave evidence, but in my judgment there was no need for them to be. As it happened, Mr Henderson was not cross-examined for very long, but that does not matter. I have not taken any account of the contents of the Second Sight reports in deciding the Horizon Issues.
84. The claimants therefore called the following witnesses of fact. For the reasons that I explain below in relation to Mr Roll for the claimants, and Mr Godeseth for the Post Office, I consider these two witnesses in particular to have been of the greatest assistance in resolving the Horizon Issues. However, all of the factual witnesses contributed to my understanding of the system over the years, both generally and specifically. The specific experiences of the claimant witnesses who gave evidence about what occurred in their branches are, if I accept them, specific examples of the working of the Horizon system at the branch Post Office end in practice. I will deal with each of the witnesses in turn.
Mr Latif
85. Mr Adrees Latif was the SPM at Caddington Post Office in Caddington, Bedfordshire, since 2001 until late September 2018. His appointment with the Post Office therefore ended after the litigation commenced, and he was subjected to an audit in September 2018. He was cross-examined by video link from Islamabad, as it had been necessary for him to travel to Pakistan as a result of a family bereavement. He left the UK on 19 February 2019, and he travelled from Kashmir to Islamabad to give his evidence by video.
86. He gave evidence about two specific incidents. One occurred in July 2015 and related to the transfer of £2,000 from the AA stock unit to the stock unit designated SP1. Originally his statement had said it was the SJ1 stock unit but this was a typographical error which he readily accepted. He successfully transferred the £2,000 from AA, but when he went to the SP1 unit, the same sum had not transferred into that unit successfully. There was no explanation for this that he could come up with, including having checked his own CCTV, and he was sure he had carried out the transaction correctly – it was not an unusual transaction. He described the sum of £2,000 as having “simply disappeared from Horizon” and explained that this would lead to a shortfall in the branch account for that sum.
87. The second incident occurred in January 2018 in relation to Camelot and scratch cards. His heading for this he corrected to “Transaction Acknowledgement Issue”. He explained that certain information was sent by Camelot to Horizon twice, and the Post Office sent out a notice stating that due to this mistake by Camelot, a Transaction Correction or TC would be issued. The correction occurred and he accepted it, but the stock figure for scratch cards in his branch on Horizon remained unchanged. This therefore showed his branch as having £1,000 more in scratch cards than was actually present in the branch. This issue was still outstanding as at the date of his amended statement which was 1 March 2019, and remained outstanding as at the date of his cross-examination which was 12 March 2019. This is, obviously, a period well in excess of one year.
88. The Post Office’s case on the £2,000 was put methodically to Mr Latif. That regarding the stock transfer is best summed up in the evidence of Ms Van Den Bogerd, who stated that provided certain steps or action were carried out correctly, what Mr Latif had said happened simply would not occur. She said, inter alia “providing these two actions are completed, the stock unit from where the cash is transferred should not show a discrepancy" (emphasis added). As she put it, her “strong belief is that Mr Latif has recalled these events incorrectly”. She also said that “the records that Post Office has reviewed do not support what Mr Latif has said and I believe that he may have mis-recollected events from 3 years ago”. The Post Office’s case was essentially that Mr Latif had not done the steps correctly, because had he done so, what he said happened could not have happened.
89. Mr Latif’s response to that was both consistent, considered and credible, and was best summarised in one of his answers:
“A. I'm experienced -- I have been running a post office for 17 years, sir. I have also worked for the Post Office on training other offices how to run a post office. I was also involved in running and introducing the new Horizon software changes in 2006 onwards, where I went to several offices on behalf of the Post Office to give them training. So I'm an experienced, trained subpostmaster and I ran my business successfully for 17 years. So I may have been a bit brief in the statement but obviously I can run through those -- exactly those steps that we would take to make sure that there is no operator error on our behalf.”
90. The conflicting evidence on this particular point is a good illustration of the “poles apart” position to Horizon by the Post Office and the claimant SPMs in this litigation. Because of the Post Office’s position on Horizon, almost all and any of the criticisms or accounts of factual events which the claimants made, or make, about how this system worked in practice are attributed to fault or carelessness by the SPM or their assistants. Indeed, without fault or carelessness by an SPM, the Post Office simply cannot explain these occurrences. The Post Office’s position is therefore to challenge the factual account – which it is entitled to do – because if the factual account by an SPM is accepted as truthful and accurate, then the Post Office would have to accept that there must be a fault or faults within Horizon. Therefore, the Post Office cannot accept that the factual account is truthful and/or accurate. Thus the dispute goes around and around in endless circles. This litigation is aimed at breaking that deadlock.
91. It is also the case that Mr Latif had the following point positively put to him about why he had checked the CCTV that he had within his branch. “So you now say you looked at the CCTV because your colleagues were concerned that you hadn't done the transaction properly?” even though Mr Latif had said no such thing. Mr Latif was subject to a fairly robust attack, not only on his account, how it matched up with other records which the Post Office said contradicted it, what he and his assistants had or had not been doing, and indeed upon the full scope of his evidence and his credibility – as shown by the question I have reproduced. That question was framed as though even his own colleagues had concerns about what he had done. It was positively put to him by the Post Office that he had not even complained to the Post Office, although he provided the name of his Area Manager Mr Navjot Jando and said he had complained to him many times. The Post Office did not call Mr Jando to rebut this. One exchange will suffice as an example of the type of attack upon Mr Latif:
“Q. You don't say anywhere in your witness statement that the £2,000 physical cash also somehow disappeared, but that seems to be what you are now saying, is that right?
A. Well, the system gave a shortfall of £2,000 and that's been my statement all the way through, sir, so I don't know what you're trying to confuse me, but there's a shortfall of £2,000 in stock unit AA and there should not be a stock shortfall. The money is physically there.”
92. Mr Latif’s evidence had never been, so far as his witness statement and evidence orally before the court, that £2,000 in cash had physically disappeared. His evidence was that there was a shortfall of that amount shown in Horizon as a result of what he had done. There were some aspects of the cross-examination of Mr Latif which were simply unhelpful. Firstly, extensive spreadsheets were put to Mr Latif which he had not seen before. They are plainly not in chronological order, and had in any event been what was called “filtered” by the Post Office legal team; they were not agreed by the claimants, nor was the “filtering” process explained at any point. There is a limitation on the degree of assistance to be obtained by such an exercise. Further, there was no agreed exercise by the experts whereby the two of them had gone through all the records directly relating to this specific instance and agreed what the records did, or did not, show. The Post Office’s case amounted, literally, to a pure challenge of fact that what Mr Latif said had occurred, simply did not occur in fact.
“Q. On the basis of that, Mr Latif, Post Office says there was no failed transfer such as that described in your witness statement and that you are simply wrong about that, it never happened.
A. So you are calling me a liar?
Q. Mr Latif, you may be mistaken or you may be lying. I put the question that it didn't happen.
A. Well, I state that they did.”
93. So far as the other issue experienced by Mr Latif in January 2018 was concerned, the Post Office accepted, as Ms Van Den Bogerd’s original written evidence (prior to later correction) put it, that its data showed “that the branch received two TAs (meaning Transaction Acknowledgement) on 18 January 2018. However, due to an error by Post Office, instead of increasing the scratch-card stock, the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with Horizon. Horizon processed the TAs accurately. I note that the TAs were accepted by the branch, which could have been challenged at that point if the user had noticed that the TAs were not for a positive number, as they should have been.”
(emphasis added)
94. However, this is notable for the following reasons. The first was that the blame is shifted back to Mr Latif for not spotting the mistake and challenging the TA. His evidence was that it was not possible to challenge a TA. They simply have to be accepted. This was accepted by Ms Van Den Bogerd when she came to give evidence orally. Therefore the notable point made by Ms Van Den Bogerd, a director of the Post Office, in her written evidence, to shift the blame back onto Mr Latif, is simply wrong in fact. TAs cannot be challenged, they have to be accepted.
95. The second is that the Post Office evidence entirely omits any reference to the “memo view” – a type of communication used by the Post Office on Horizon which leads to a “pop up” on the terminal the next time each user logs on. There was simply no reference to this at all by Ms Van Den Bogerd, and Mr Latif said that this was sent out by the Post Office “to everyone” saying the error on the Lottery would be corrected by means of the issuing of a TC. Finally, Ms Van Den Bogerd corrected this passage in her evidence in chief by way of printed corrections prepared before she was called. She said that these corrections had been handed to her solicitors prior to the trial. If that is correct, it is surprising that Mr Latif was cross-examined on the basis of the original evidence in her statement, as that was not going to be her evidence in chief. However, regardless of that, she readily agreed that there is no choice regarding the acceptance of TAs – they simply have to be accepted by the SPM. In those circumstances, there is no explanation as to how the emphasised sentence in [93] above came to be in her statement at all, and it was plainly incorrect. Also, the call logs (about which Ms Van Den Bogerd was cross-examined) concerning this incident entirely support Mr Latif’s evidence.
96. Finally, although this is a minor point compared to the ones in the preceding paragraph, the tenor of her witness evidence that Horizon processed the TAs in this respect gave the impression that there were no problems with the Lottery and TAs so far as Horizon were concerned.
97. Not only did the steps taken by the Post Office – the TAs – not correct the issue that occurred in Mr Latif’s branch, but in cross-examination Mr Latif explained further:
“A. Can I just confirm, there was an audit done in September of this year, an audit by a Post Office trained auditor, and my stockholding was still showing negative. And a Jane Lawrence is the auditor and she has – still could not resolve this matter, so the problem hasn't gone away, the problem is still there. And there have been a number of calls to the helpline to resolve that negative stock and it hasn't worked. They haven't come back with a response.”
Mr Latif was very clear about this:
“A. I still state that an independent audit was done by the Post Office, a Mrs Jane Lawrence, in September 2018 the stockholding was still negative and as the branch was handed over to another subpostmaster that is going to be investigated, it's going to be investigated. If you are now coming back with this evidence, I still say that there is a problem somewhere and I don't know what's happened but we have still got a negative stock figure within our branch.”
He returned to this again, in the light of being pressed yet further on the point that it was human error at the branch that had caused the discrepancy or shortfall.
“So the fact remains there was an independent audit happened by your Post Office -- by Post Office's auditor, her recommendation was that they look into it as is (inaudible) the strategy is still there. And there were a number of calls to the helpline pleading with them to resolve this issue before the audit and there will be a complete trail of that, sir.”
He also referred to the what had happened as being “a glitch”.
99. In due course Mr Latif will have his own individual claim, and the Post Office with have its individual counterclaim against him, tried. The only findings I am making in respect of his evidence are those necessary for me to resolve the Horizon Issues. All other issues remain to be tried in those later proceedings. Those later proceedings may explore in some detail not only the two specific matters in respect of which he gave evidence in the Horizon Issues trial, but any others which are relevant to both claim and counterclaim. They will do so by reference to other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the findings of fact that I make affect my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Tank
100. Mr Jayesh Tank was the SPM at Fleckney Post Office, in Fleckney Leicestershire, from 4 May 2006 to 15 March 2017 when it was closed as part of the Network Transformation Programme. He gave evidence regarding certain issues with Horizon that he had experienced, which included the effects of a complete power failure to the building, which occurred mid-transaction. This occurred whilst a customer was making a withdrawal in the branch from her Post Office card account. On his evidence, this led to a shortfall in his accounts of about £600. Mr Tank paid this sum to the Post Office, and the letter from the Post Office stating how it came to be owed described it as a “branch discrepancy”; it was in fact £660. He paid it by way of a deduction from his remuneration.
101. The second issue was one concerning mail labels, when Horizon would (as he put it) “jump ahead” to the end of the transaction and no label (which is what would or should be affixed to the packet to be posted) would be printed. This would cause a loss in the branch accounts.
102. Ms Van Den Bogerd again gave evidence about these specific instances. Her witness statement led Mr Tank to correct the date when he remembered the power failure occurring (he accepted it was 2014, not earlier in 2010 or 2011 as he had initially thought), and he corrected this in a supplemental statement. Her statement had helped him pin down the date. This incident became the subject of a PEAK at Fujitsu, although Mr Tank did not know this at the time. This supplementary statement also exhibited to it old “posts” he had made contemporaneously on an internet forum on the website Yahoo in respect of another loss, £195, which he had suffered in 2011.
103. Mr Tank had been interviewed by the Post Office investigators in 2015 and in the Horizon Issues trial it became clear that certain matters extremely critical of his conduct were going to be put to him in cross-examination. He was therefore given the warning against self-incrimination under the Civil Evidence Act in the same way that the two claimants in the Common Issues trial, who were accused in cross-examination of criminal offences, were given it. Mr Tank answered all the questions that were put to him. Mr Tank had, eventually, used the icon on Horizon in his branch for “Official Postage” incorrectly, and his explanation for this was that this was deliberate and was done in order to get the attention of the Post Office. He said at his 2015 interview that all his attempts by way of phoning the helpline and contact with his area manager had simply not resolved the numerous complaints he had made. He frankly accepted that he had not sent the post by official means which he had entered on the Horizon system as official postage, doing what he did by way of protest. It was put to him in cross-examination that he had “helped himself” to official postage, which on the evidence available was a suggestion that did not seem to have any basis in fact, if he had been doing what he said he was doing. There was certainly no evidence before the court to substantiate this assertion.
104. In his interview on 5 November 2015 by the Post Office investigator Mr Bridges, it had been accepted by Mr Bridges (who interviewed him) that Mr Tank had reversed the entries for official postage in any event. This acceptance at the time by the Post Office investigator was directly contrary to the way that the points were put to Mr Tank in his cross-examination, which were that he had “helped himself” to postage and “improved his financial position” by acting as he did. Detailed findings on this will have to wait for the full trial of his claim and counterclaim, but it is notable, in my judgment, that the attack on the credibility of this witness in the Horizon Issues trial, and positive allegations to him of criminality, were not consistent with the contemporaneous record of the acceptance by Mr Bridges for the Post Office in the investigation interview in 2015. Mr Bridge clearly accepted that the entries had been reversed. Indeed, Mr Bridges introduced this subject in that interview as follows:
105. “Okay thank you. Let's move on to the postage claims for the moment. So in my letter I gave details of the claims and the reversals that you completed since 25 August. I think my first question would be that I know that you reversed them but why undertake them in the first place bearing in mind these are, in effect you are stating the transactions which took, well say the transactions have taken place which you have used official postage for which in effect did not take place.”
106. The questions in cross-examination to Mr Tank, which must have been put on instruction, did not take account of what the Post Office had accepted at that time in the interview. Another feature of the interview is that Mr Tank was told by the interviewer “there are no issues with Horizon”. That exchange is as follows:
“KB (Mr Bridges) I can confirm that there are no issues with Horizon and as my letter stated there is no issue with the production of labels.
Jay (Mr Tank) Then obviously you conducted your investigation into my concerns without asking me for my evidence. I've got the evidence here to show you. I'm happy for you to take copies but I'm going to hang on to it but how can you say something is fully investigated if you've not looked at all of the evidence.”
On the face of it, that is a valid point that Mr Tank made. Had there been an investigation, his evidence would plainly be something that ought to have been taken into account by any investigator.
107. The Post Office relied, again, upon evidence from Ms Van Den Bogerd. Mr Tank was cross-examined by reference to this and also by reference to a technical explanation concerning the fact, it was said in cross-examination, that the power outage must have occurred at a particular point in time. The cross-examination was as follows:
“MR HENDERSON: So what appears to have happened was a transaction from the Post Office card account was in the middle of being processed and so it was in the stack presumably.
A. Yes.
Q. But had not yet been posted to Horizon. So you hadn't cashed out on that transaction, you hadn't completed everything to do with that transaction?
A. On the stack –
Q. It's on the stack.
A. It's on the stack, but the stack has a balance of zero, so to clear the stack you just press "enter" and it goes straight –
Q. But you hadn't got to the point of clearing the stack?
A. I'm not sure.
Q. Okay. My suggestion is that there was probably an outage at just the point where the money had been taken from Post Office card account but had not been processed onto Horizon. That's my suggestion to you.
A. Okay.
Mr Justice Fraser: Well, is the witness going to be in a position to agree or disagree?
Mr Henderson: Well, he might be if he recalled.
Mr Justice Fraser: Do you recall that happening when there were outages?
A. No. I cannot recall.
Mr Justice Fraser: Were you aware of when outages would occur like that?
A. Not all the time.
Mr Justice Fraser: Do you want to put the question again?
Mr Henderson: Yes. What I'm suggesting is that the cause of this problem was that an outage occurred at a particular point in time.
A. Yes.
Q. You were in the process of effecting a transaction from POCA?
A. Yes.
Q. It was in the stack and it had cleared from POCA?
A. Actually you mentioned the word "outage". I'm not -- was there a power outage?
Q. I'm not sure if it was a power outage, but I think it may have been a problem with the system.
A. Ah, okay.
Q. The system went down in some way.
A. So -- yes, because you said that if there's a power outage then there's evidence when you have to log back in, so did that happen on this occasion?
Q. Okay, I want to come to all this and I'm doing this clumsily. What I'm suggesting is that what may have happened -- and if you don't recall, you don't recall, but what may have happened is that the transaction was in the stack, the money had been taken from the Post Office card account and before you cleared the stack there was an outage.
A. Possibly.
Q. Okay.”
(emphasis added)
108. I have reproduced this passage because it demonstrates the following. The Post Office put a possible explanation, based on technical grounds, to the witness who did not have either at the time, or at the trial, the technical expertise, or the relevant technical records, or the data, to enable him to agree or disagree. Even that explanation itself accepted “there may have been a problem with the system” and “the system went down in some way”. The witness’ answer, which in my judgment is the only sensible answer that could sensibly be given in these circumstances, was “possibly”.
109. This type of evidence obtained on cross-examination is of no assistance in resolving the Horizon Issues, other than an implicit acceptance by the Post Office (on this occasion) that outages could, potentially, in technically terms lead to what Mr Tank experienced. In any event, the Post Office’s explanation accepted that “there may have been a problem with the system” which is equally consistent with the claimants’ case, as it is with the Post Office’s case, if not more consistent with the claimants’ case than with that of the Post Office. Power outages do happen, and there is a process to be followed. Given Mr Tank was using Horizon Online, he was then in his cross-examination taken to Version 5 of a HOL quick reference guide. The version to which he was taken was not, however, the version available to him at the time, as that was a single A4 double-sided version in his branch, and the one put to him at the trial was longer than two pages and available online. The date attributed in the electronic trial bundle to Version 5, the version put to him in cross-examination, is 30 July 2015. His forum posts and his supplementary witness statement made it clear that the incident occurred in September 2014, a later date admittedly than given in his first statement, but still somewhat earlier than July 2015. This was however the version of the guide that Ms Van Den Bogerd relied upon in paragraph 78 of her 2nd witness statement to challenge Mr Tank’s account. I find the version that he would have had available at the time was not Version 5, and was the single A4 version that he had available to him in his branch in September 2014. However, and regardless of this, Mr Tank accepted that recovery receipts would or should be generated and set out the procedure. He also accepted that giving all the receipts to the customer was not the proper procedure.
110. In the PEAK dealing with the shortfall of his accounts of £195, which is PEAK PC0214226 and is headed “Failed Recovery Transaction(s)” Fujitsu recorded:
“Date:14-Dec-2011 10:43:53 User: Wayne Bragg
Summary:
The banking transaction had completed (A3 rec'd and authorised @ 13:33:37), including the receipt print (13:33:42), and money should have changed hands.
The basket settlement failed from 13:35 with 'No response received from data centre', and the two retries also failed, and the attempt CANCELLED at 13:37.
The Disconnected Session receipts show "Cash TO CUSTOMER 195.04" so the customer's account should be correct but the branch will have a shortage (for a withdrawal) because the session hasn't been recorded.”
(emphasis added)
111. In my judgment this entry clearly supports Mr Tank’s evidence about how Horizon operated on the occasion to which he referred. The heading of the PEAK was “failed recovery transactions”. The PEAK clearly states that the customer’s account would be correct – they were given the cash of £195.04 as they expected, but “the branch will have a shortage (for a withdrawal)” as the session was not recorded. A transaction correction was issued. There were related PEAKs, and also there was an entry “KEL acha959T may be relevant” on the PEAK, and Anne Chambers was referred to. The PEAK also recorded “Sending to SSC for investigation.” That KEL was raised by Anne Chambers on 28 February 2010, and the title was “HNGX banking reconciliation – state 4”. This is an important KEL, and I deal with its further detail in the Technical Appendix.
112. There was a dispute between Mr Tank and counsel for the Post Office about whether this was an example of “Horizon working as it is supposed to”, which Mr Tank did not accept, and also whether he would have been refunded the sums (which he accepted had occurred) had he not phoned in and reported the problem. It was put to him that he would have been refunded even had he not called in to report, although how this would have occurred given the way PEAKs are initiated was not explained to him. In any event, the PEAK demonstrated that what Mr Tank had said occurred, had indeed occurred. Transaction Corrections are issued outside the scope of the Horizon system.
113. The position regarding the procedure for spoiled labels was, again, a situation whereby the Post Office’s explanation was this simply could not have happened. Ms Van Den Bogerd’s evidence was that there was a procedure available that Mr Tank should have used, yet the document put to Mr Tank that was said to support this stated expressly in terms that “the label could only be spoiled if the label was on hand” and Mr Tank’s point was that no label was “on hand”, given the problem he had was that the label did not print at all. Using the procedure in the document therefore would not be possible, and indeed would be contrary to the Post Office instructions that it could only be used if the label was on hand. These instructions also required the SPM to write “spoilt” on the label and keep it with the Horizon receipt for two years. This could not be done if the label had never printed. The explanation by Ms Van Den Bogerd, put to Mr Tank, was also directly contrary to the contents of a letter dated 7 September 2015 from the Post Office to Mr Tank which stated:
“Your enquiry has been investigated and I can confirm there are no issues with Horizon online or the production of postage labels which would cause the situation you described. However if a user either pressed the yes or return key quickly before screen messages appear this can lead to a user confirming a postage print has happened when in rare circumstances it may have failed and put the cost of the failed print into the basket. In this circumstance Subpostmasters should contact Network Business Support Centre to arrange a credit for the spoiled postage.”
This means that the Post Office in that letter accepted that in some circumstances, depending upon how quickly a key was pressed, the printing of a postage label could fail and cause an impact to branch accounts. The SPM must make a telephone call to the helpline, to correct the fact that the branch account would show a charge for the postage even though no label had been printed and hence could not be provided to the customer. This letter suggests not that Mr Tank had not followed a particular procedure for dealing with spoiled (rather than non-existent labels) but that the system would, depending upon how quickly a user pressed a particular key, not print a label, even though the cost of the failed print would go in the basket. By going “in the basket” this means the branch accounts would include the cost of that label as a debit to those accounts. In other words, this letter supported Mr Tank’s direct evidence.
114. The Post Office in cross-examination put the following to Mr Tank:
“
Q: There were procedures built into Horizon to cater for the situation that you explained -- I have to say in the vaguest of terms, but as I understand what you are
saying, there were procedures in place which ensured you could deal with the situation, weren't there?
A. No.
Q. We will have to differ.”
I would express this rather differently. If the speed of pressing a key could lead to the cost of a printed label being added to the basket (and hence branch accounts) even though that label had not printed, it is difficult to see that there is a “procedure built into Horizon” as defined to cater for this. On the contrary, it appears from the Post Office’s own letter in 2015 that there was no such procedure. For the avoidance of doubt, I find that there was no such procedure built into Horizon. The evidence demonstrates to the contrary. Phoning the helpline and asking for a refund is not “a procedure built into Horizon”.
115. Further, it is difficult to see how the explanation proffered by Ms Van Den Bogerd regarding how to deal with a spoiled label could apply to Mr Tank for two reasons. Firstly, I accept that as a matter of common sense interpretation and language, a label cannot be “on hand” if it was never printed. Mr Tank did not consider he could properly use the procedure in the guide for the problems he experienced, and I agree with him. Secondly, the Post Office’s own explanation to him in 2015 was very different to what he was told in cross-examination in 2019 that he should have been doing. Even were I to assume (in the Post Office’s favour) that there must have been instructions from the Post Office somewhere regarding how a SPM should behave when a label failed entirely to print, as print failures (which would not produce a label “on hand”) were plainly not unheard of, it remains to be seen what those instructions were, and whether those instructions conflicted with what he was told in the letter in 2015. No such instructions have been produced by the Post Office in any event.
116. I consider Mr Tank to have been a credible witness. I find that Mr Tank’s experience in branch was not an example of Horizon working “as it was supposed to”. I find that the creation both of a PEAK, and reference to a KEL within that PEAK, to be consistent with my conclusion. The proposition that Horizon worked as it was supposed to is an obviously flawed one. Indeed, that proposition is also inconsistent with common sense. PEAKs and KELs are not created for situations where Horizon is working “as it is supposed to”. They are created to deal with errors. The acronym KEL is for Known Error Log. Nor are matters referred to the SSC for investigation when Horizon is working correctly. Whether what occurred afterwards was an example of Horizon working correctly depends on the categorisation of Transaction Corrections and whether they are part of the Horizon system or not. I explain later in this judgment that they are not, therefore the issuing of a TC cannot be Horizon working as it should. Mr Tank also gave evidence, which I accept – and this is made out in the contemporaneous documents – that he had some difficulty in advancing this matter through the correct channels. The helpline operator with whom he was dealing initially refused to “pass up” the matter to a more senior person. Whether this was an isolated instance of unhelpfulness in this single case, or a more generally obstructive approach across the helpline, will have to wait for future trials in this litigation, and I recite it for completeness only in respect of Mr Tank’s experience. It does not form part of my consideration of the Horizon Issues.
117. Mr Tank was accused of criminal offences and it was said that he had “helped himself” to official postage and had “taken official postage”. These accusations were not put to him at his interview in 2015 and indeed the text of that interview shows that the Post Office interviewer expressly accepted that he had reversed the transactions, which is not consistent with Mr Tank “helping himself”. There are 10 pages of detailed closing submissions by the Post Office about Mr Tank’s evidence, and although he is criticised for being vague, confused, imprecise and for not having prepared his evidence with care, the allegation of criminality is not raised. However, the point was positively put to him in open court, and Mr Tank is entitled to a finding on it. In my judgment, it would be quite wrong to leave such an accusation hanging in the air, unresolved in this way. I find that on the material before the court in this trial and on the evidence available and put to him, these serious accusations are not made out. That is not to say I have made binding findings on the full details of his claim which remains to be tried in due course. The Post Office submitted, both in respect of Mr Tank and other of the claimants’ witnesses, that they confined their evidence to what was described as “a small sub-set of his complaints, presumably because it was felt that this threw light on the Horizon issues – and Post Office’s evidence was similarly confined”.
118. However, there are two important points that must be borne in mind about that submission, both as it is made in respect of Mr Tank, and indeed the other witnesses called for the claimants. This has not been a trial of their individual claims. There may be other issues about Mr Tank’s branch accounts, raised both by him and by the Post Office, at the full trial of his claim and the Post Office’s counterclaim. That does not mean that this specific evidence, about specific incidents experienced on Horizon, should be discounted. Indeed, this specific evidence is not only directly relevant to the Horizon Issues, but is, in my judgment, very important evidence going to those issues. The second point is the parties’ agreed wording of the Horizon Issues does not allow for any wider issues going to Mr Tank’s branch accounts to be considered in this trial. All the other issues between Mr Tank and the Post Office remain to be tried in the later proceedings. The only point of difference between them on one of his two issues concerning Horizon was whether what occurred when a transaction correction was issued could properly be described as “Horizon working as it is supposed to”, and certainly the KEL referred to in the PEAK appeared during the Horizon Issues trial more than once. The creation of a PEAK, and the KEL to which reference was made within it, in my judgment demonstrates that this is not Horizon working as it should, as does the issue of a TC. The wider consideration of these matters that are dealt with both by the experts and later in this judgment. I reject the suggestion that Horizon was working as it was supposed to on this occasion. So far as Mr Tank’s experience with the £195 shortfall is concerned, there was a Horizon Online failure; three identical receipts were printed, which should not have happened; the receipts all showed a disconnected session; and there was a loss in the branch accounts that evening for the amount paid out to the customer. This was even though Mr Tank had settled with the customer for the amount specified on the receipt. There was no record of the transaction at all on the transaction log that was produced for the period of the Horizon Online failure. None of these, in my judgment, are examples of “Horizon working as it was supposed to”.
119. The operation of the helpline is not part of the Horizon Issues, so it is not necessary to consider and make findings on what Mr Tank said the helpline told him, which he explained in his posts was “that the loss is mine unless I can sort out with customer directly.”
120. The later proceedings concerning Mr Tank’s claims (and any counterclaim) will probably hear further evidence which are relevant to both claim and counterclaim. They will do so by reference to other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Anup Patny
121. Mr Anup Patny became the SPM of Spencefield Post Office in Leicester in October 2014. He is currently suspended by the Post Office, having been suspended on 17 August 2016. His son, Mr Aakash Patny, also gave evidence. I shall refer to Mr Anup Patny as Mr Patny Senior for this reason. I shall also deal with his evidence, and that of his son, together for reasons that will become apparent.
122. There was what he described as a “major system outage” on 9 May 2016 in his branch, which was a single counter branch run in conjunction with his retail business in the same premises. This was just a few months before he was suspended. This outage led to the closure of his branch and he believed that this had affected the whole network. At the end of the next trading period, which was on 11 May 2016, there was a shortfall of almost £17,000, predominantly made up of a shortage of 16,000 coins of £1 denomination. This was a very large number and he said that he knew he would never have such a large amount of £1 coins in his branch. He was also the person in the branch who had dealt with the cash delivery that took place on the relevant morning, which is called “remming in” the cash. This means accepting cash into the branch from the cash delivery that Post Office makes to the branch, and entering the cash received into Horizon. His son, who also gave evidence, followed this matter up with the Post Office. On 19 May 2016 the Post Office contacted him about a discrepancy in respect of stamps, which so far as he was concerned related to the £16,000 regarding coins.
123. The Post Office accepted that there had been an outage but disputed that this either did, or could have, resulted in the discrepancy which Mr Patny said occurred at his branch. Ms Van Den Bogerd gave evidence about this, and the Post Office’s position is best summarised in the explanation put to Mr Patny Senior in his cross-examination:
“Q. Well, what I'm suggesting is that the most natural explanation for this, whatever adjustments were made, is that at some point on 11 May someone hadn't counted a big pile of £10 notes. They had been put in a safe and forgotten about, which is understandable, and that they were found the next day, or located, and there was an accurate cash declaration on 12 May.
A. I don't know about that, sir.
Q. It's perfectly plausible, isn't it?
A. I can't say anything to that.”
124. This suggestion by the Post Office is, in my judgment, somewhat fanciful. A “big pile of £10 notes” is not the sort of thing that ordinarily gets “put in a safe and forgotten about”, and then suddenly “found”. It was also put to the witness that “things were pretty chaotic in your branch when it comes to these sorts of things” and “the cash declarations look like they are all over the place”. Mr Patny Senior did not agree with this characterisation of the way his branch was run. He certainly did not agree with the “big pile of £10 notes” theory the Post Office put forward. His evidence was also that it was his son who had contacted the helpline, and who had made the adjustments to Horizon that he was advised to do by the helpline. Initially the Post Office sought to challenge this account through cross-examination of Mr Patny Senior only. Counsel for the Post Office did not initially intend to put questions on this to his son, even though his son was plainly the correct witness to answer the detailed questions, as he was the person who said he had done certain things, and he was about to be called as a witness. The reluctance of the Post Office to put questions to the person who was plainly the relevant person to ask about this was said to be because “the records are the records” and the Post Office did not intend to repeat cross-examination that had already been done. I was not prepared to permit this. I required questions about what Mr Patny had, or had not, done, to be put to him directly, and not have the cross-examination conducted by proxy through his father. Mr Patny Senior’s son had given a witness statement in respect of these matters, and was plainly the correct witness to whom the questions should be put. He was the next witness listed on the trial timetable, and was going to be called next for the claimants. The notion that questions challenging what he had done, or not done, should more properly be directed to his father is contrary to how cross-examination should be conducted.
Mr Aakash Patny
125. Mr Aakash Patny is the son of Mr Patny Senior, and worked in the branch and retail premises with him, as did Mrs Patny, the wife of Mr Patny Senior and mother of Mr Patny. He is not a claimant. He was given the warning against self-incrimination under the Civil Evidence Act. He did however answer all the questions put to him.
126. Mr Patny assisted his father, particularly with balancing. He would usually arrive at lunch time, and so was not present when the national outage occurred on the morning of 9 May 2016, which had been resolved by the time he arrived that date. He gave evidence concerning the same amount of money, which both he and his father considered a shortfall, and which the Post Office had claimed was due to it and was a branch discrepancy for which the branch was responsible. It had been settled centrally but had not been paid to the Post Office.
127. So far as 11 May 2016 was concerned, when Mr Aakash Patny had arrived at the branch he became aware of the shortfall of over £17,000, which was a “complete surprise” to him and which he considered could not be explained. He had contacted the helpline, and had followed the steps he was advised to take, which he thought had resolved the issue. These steps were things he did on the Horizon terminal, and sequences of entries he was talked through over the phone. The documents available supported his evidence that he had made such a call, although some of the entries did not match what he said he had told them. There was no record of what he was told to do, nor was there any record put to him showing what key strokes had been inputted at the branch.
128. The Post Office’s case on the shortfall was that the outage on 9 May 2016 did not lead to any discrepancy. It was not put to Mr Patny that he had not made key strokes as advised over the telephone, or that he made different keystrokes to those advised. All that was put, somewhat unclearly, was that “the problem” was resolved, cash had been physically “found” that had been missed at the time of the previous cash declaration, and also that a figure showed on a cash declaration of “plus £17,000 odd” had the effect of “cancelling out” the previous discrepancy and showed that “there was no longer a cash problem”. Mr Patny denied any money had been found that had been lost; denied that if a discrepancy had been cancelled out the figure would have been plus £17,000, rather than zero; and said that although he thought following the key strokes he was told to input had resolved the issue, it had not. He was told to “readjust” the cash stock figure, something he had never done before, and relied upon the instructions he was given by the helpline. The Post Office’s cross-examination was done by reference to the cash management report, not the audit data. I will return to audit data later in this judgment in Part K, but the audit data would have showed what keys were pressed, in what sequence and when. Such data would have been of great assistance, but was not produced by the Post Office.
129. On 19 May 2016 the branch was contacted by the Post Office, in respect of an over-declaration of stamps. The amount of this was approximately the same value as the amount of the discrepancy of cash had been a few days earlier. He had done the declaration for stamps. A person called Debra Lambley phoned the branch. He was struck by the co-incidence in the amount of the stamp discrepancy, and told Ms Lambley the stamp declaration was impossible due to the number of stamps this represented. To assist a reader of this judgment on that latter point, but something that was not explored in the evidence, to obtain a broad idea of how many stamps that would represent, the following points are relevant. On 28 March 2016 the Royal Mail announced an increase to take effect that week in the price of stamps. 1st class stamps became 64p each, and 2nd class stamps became 55p each. This shortfall would therefore have represented 25,000 1st class stamps, or over 29,000 2nd class stamps. It goes without saying that this is a great deal of stamps.
130. He was instructed to re-adjust the stamp stock figure, followed her instructions as to how to do this, and the system then showed the £16,000 shortage. It was by now past 7.00pm, he phoned the helpline, they contradicted Ms Lambley and he then had to re-declare the following day, being unable to do so that day as it was after the 7.00pm cut off.
131. The Post Office’s case on this, as it was put to Mr Patny, was firstly that he had made a mistake in declaring the stamps. He denied this. The circularity of the Post Office’s case can be seen from a lengthy passage of evidence, which I will not reproduce, from page 26 on that day’s transcript to page 35. Mr Patny explained, basically, that the helpline gave instructions to resolve the cash discrepancy; this resulted in the figure for stamps being wrong by the same amount; Ms Lambley was not prepared to have this, and following her instructions this resolved the stamps situation, but the cash discrepancy reappeared. Then the sequence happened a second time. Mr Patny was prepared to accept he may have made a mistake once, but denied he could have made exactly the same mistake the same way a second time.
132. It was also then positively put to him that one way of disguising a shortfall in cash would be to over-declare the number of stamps. A positive allegation of dishonesty was put to him, which he rejected.
133. The premise behind this suggestion of dishonesty was that it was said by the Post Office that if the stamps figure went up, the cash figure would automatically go down by the same amount. He did not agree with this. It was put as a positive statement in the following terms:
“Q. I suggest finally, Mr Patny, it is rather surprising, given that you do all the balancing, or did all the balancing in the branch, that you don't understand that the effect of increasing the declaration of stamps is to have a corresponding effect on the declaration of cash?
A. I wasn't aware of that, no.”
134. There was no direct evidence to support the proposition that by increasing a declaration of stamps, it would have a corresponding effect in the opposite direction upon the declaration of cash held in the branch. In any case, in the Common Issues trial, there was evidence that making a cash declaration required specifying how many notes of each denomination was physically held in the branch. An SPM had to declare both the cash and the stock that was held in the branch. It is difficult to see how changing the number of stamps held in a branch would, or could, affect the physical counting and declaration of the amount of cash held in a branch by way of specific denominations. However, and in any event, the allegation of dishonesty against Mr Patny was a line of cross-examination that was not supported by evidence from Ms Van Den Bogerd, who was expressly asked about it, and who accepted that she had not made an allegation of dishonesty in her witness statement and said she did not offer an opinion on whether Mr Patny was dishonest or not. That means that there was no evidence from any witness called for the Post Office to support the allegation of dishonesty.
“On 23 February, the branch declared cash holdings of £25,803.87, a net downward movement of £8,601.59. The net value of transactions during this period resulted in a £1,806.71 decrease in cash. The Moneygram transaction described above would account for a further £3,100, bringing the total explainable cash movement to a £4,906.71 decrease. However, this leaves £3,694.88 of cash movement unaccounted for. I cannot say for certain what caused this additional loss of cash but there is nothing in the accounts that suggest a problem with Horizon. It appears more likely to me to be a problem with cash handling in the branch or a user error when making cash declarations.”
(emphasis added)
136. This statement does not answer the evidence of Mr Patny on the point, and the more that counsel for the Post Office tried to press the point(s) in cross-examination, the more stark it became that this evidence by the Post Office was no answer. Firstly, it assumes that the MoneyGram transaction was only responsible for £3,100, not the figure of £6,200 which Mr Patny stated. It assumes that £3,694.88 does not include the lower figure of £3,100, which in my judgment, on the documents deployed, it plainly does. Secondly, Ms Van Den Bogerd cannot say what caused the additional loss of cash. Thirdly, it states that that there is “nothing in the accounts that suggest a problem with Horizon” (emphasis added). That is rather stating the obvious. If all the problems complained of by the claimants with Horizon could be seen in the branch accounts, this litigation would not be occurring. Fourthly, she retreats to criticism of the way cash is handled in the branch, or other mistakes – “user error”. It is a perfect summary of the Post Office’s case, but it does not provide an answer. Other than vague “user error”, there is no explanation provided. I find that there is nothing in any of the documents to suggest user error.
137. Additionally, Mr Patny explained that he had fully explained all of this to the area manager Mr Irwin, who agreed with him that the MoneyGram transaction had appeared to “double up”. Further disclosure by the Post Office when the trial of Mr Patny Senior’s claim comes to be tried, together with evidence from Mr Irwin if he is called, will provide further information. Finally, the exercise undertaken by the Post Office in cross-examination failed to take account of the fact that one had to consider both the cash figure, and the shortfall, from day to day to make sure that one is comparing apples with apples. I consider the exercise undertaken in cross-examination to be flawed.
138. I cannot resolve this purely on the factual material put before the court by the Post Office. The audit data showing the actual key strokes inputted will be required in order to do that, together with findings on the expert evidence and my findings on the existence of bugs, defects and errors within Horizon. However, I do find that Mr Patny came across as a careful person, and as a credible witness, and I accept that he took the steps he did as explained in his evidence. He accepted that he had not reversed the transaction until later, rather than immediately as he ought to have done. The resolution of the movement of the cash position, and what if any explanation is available, simply cannot be done with finality at this stage on the basis of the factual evidence before the court alone. The Post Office had, according to Mr Patny, failed to provide the Credence data which would make it clear what had happened and how. The helpline logs show that Mr Patny had chased this on numerous occasions, and also the entries suggested that a Credence report was being prepared. The audit data showing what key strokes he had undertaken and when, was not produced by the Post Office.
139. There was no evidence before the court to support the allegation of dishonesty that was expressly made against him, and it was not supported by Ms Van Den Bogerd. On the material before the court in this trial, I find the allegation of dishonesty against him not to be made out and I reject it. That is not to say that the subsequent trial dealing with Mr Patny Senior’s claims will automatically accept all the evidence given by his son, if he calls him as a witness. All future issues in individual claims will be dealt with, and resolved, on the evidence led by the parties in those subsequent trials. I am only deciding matters in this judgment that are necessary to resolve the Horizon Issues. However, in circumstances where positive allegations of criminality are expressly put to a witness in a trial such as this, it is only fair to deal with those allegations. They cannot as a matter of fairness be left hanging in mid-air, particularly where there is no evidence provided in support from the party making the allegation in any event.
140. The later proceedings concerning Mr Patny Senior’s claims will explore in some detail not only the specific matters in respect of which he and his son gave evidence in the Horizon Issues trial, but any others which are relevant to both his claim and counterclaim. They will do so by reference to the evidence advanced in that later trial and the other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which the evidence of Mr Patny Senior and his son affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mrs Burke
141. Mrs Angela Burke had worked for the Post Office in various roles since she was 16. Most recently, she worked in the Newport Post Office in Brough where her husband was the SPM. She was a Branch Assistant and had worked as an assistant for about 15 years, and had also previously been a SPM herself. She and her husband had an associated greeting cards and stationery business, and the branch was closed in October 2017 as part of Network Transformation. The card business has also closed.
142. Her evidence related to events of 9 May 2016 when the national outage occurred, the same date as the occasion about which both the Mr Patnys had given evidence. She gave evidence about the impact upon the branch business, the way she was serving customers and how Horizon was being very slow that day, with a sand timer appearing on the screen for a very long time. She served one customer, who was making a cash withdrawal, as she obtained the relevant messages and approvals on the screen. However, after they had left, a receipt printed saying “Recovery failed” and the withdrawal of £150 was not shown. She then later studied the transaction log and this latter transaction did not appear.
143. She realised what was happening and decided to close her branch that morning. She sat down and worked through all the transactions in her branch, using what was available to her. She was given an explanation by the helpline which she did not accept, as she knew and could remember that the payment to the customer had been authorised, which is what led to her handing over that sum. She also was engaged in a contemporaneous exercise whilst it was very fresh in her memory. She said there was no alert from Horizon, and “there was no means through the Horizon system for the discrepancy to be identified or for its cause to be established in my situation”. The transcript of this call was available – she and her husband had obtained it using a subject access request from the Post Office.
144. Mrs Burke then went to extraordinary lengths. She also proved herself very tenacious, as many people may well have simply given up on the sum of £150. She identified the customer, and she tracked him down. She went to his house and explained what had occurred. He happened still to have the receipt from the transaction at her Post Office. It entirely matches her account. She went with the customer to the customer’s bank, which was the TSB in Goole. She explained with the customer to the bank cashier what had happened, and the cashier printed out the bank statement and showed that the sum had been withdrawn from the customer’s bank account. The customer permitted Mrs Burke to have this.
145. She and her husband then pursued this through the helpline and their call was escalated. They did receive a TC for £150, but actually that referred to an amount in respect of Lloyds Bank and not to TSB. This was pointed out to the Post Office who claimed there was no code on Horizon for the TSB. If that is correct, I find it highly surprising that there should be no separate code for the TSB. The bank currently known as Lloyds used to be known as Lloyds TSB – there are cases on bailii that use that name in relation to other litigation, for example – but the TSB was split from Lloyds TSB, which then became simply Lloyds Bank. This split occurred some years before the events of which Mrs Burke gave evidence. TSB and Lloyds Bank had been separate for some time by May 2016.
146. Mrs Burke’s evidence was explored in very considerable detail in cross-examination, including the fact that she chose to keep the stack open in order to serve more than one customer at once, how she had served them and the other details of what had occurred. She was challenged on what was said to be a “relatively small” point, of whether she would have received a TC from the Post Office without having done all the work that she had done, in tracking down the customer and going to his bank. I reject that categorisation of the point by the Post Office. It is not a small point. It is, in the context of the overall litigation, a major point. It is difficult to see, given the evidence collected by Mrs Burke which was provided to the Post Office, how they could have failed to issue her with a TC. Whether this TC would have issued absent the work that she did to demonstrate the fault and its effect is not a “small point”.
147. There are really two points that are relevant. The first is a broader one: is the process for issuing TCs part of the Horizon system? The second is a case specific one: would, on the balance of probabilities, the Post Office have issued a TC in her case without the evidence that Mrs Burke herself collected and provided to the Post Office?
148. So far as the first point is concerned, the process for issuing TCs is not part of the Horizon system. That is addressed elsewhere in the judgment. So far as the second point is concerned, the documents put to Mrs Burke demonstrate that Fujitsu had worked out what had occurred; that the internal document attached to an internal email of 12 May 2016 did not accept that Newport branch had in fact paid the money to the customer, even though Mrs Burke had told them this on 9 May; and that by 16 May 2016 the decision had been taken that the branch would receive a TC anyway. Her response to the “small point” was “possibly, yes”.
149. The conclusion that I draw from this is that the Post Office required something more from Mrs Burke than her word that the money had been paid out by her branch to a customer – the internal documents refer to “not knowing” if the money had been paid, even though she had already told the Post Office this. I find that the fact that this is a Fujitsu document rather than a Post Office document does not matter for this purpose. That something more that was required, in her case, was the evidence she had personally compiled that was of great weight, and was accepted by the Post Office, as the TC was issued. Whether evidence of lesser weight would have been accepted by the Post Office is not possible to say, and is entirely hypothetical. Fujitsu had, though, identified this particular problem experienced at the Newport branch by the national outage. The information of what occurred, and the problem, was not visible to the SPM or their assistants in the branch.
150. Counsel for the Post Office’s “relatively small point” put to Mrs Burke, that the Post Office would have issued her with a TC anyway, is in fact not entirely consistent with the Post Office’s own evidence for this trial. Mrs Van Den Bogerd stated in paragraph 110 of her witness statement that “Once Post Office was presented with evidence that the customer had received the cash and the customer's bank had recorded the withdrawal, a transaction correction was issued to bring the branch.” She makes it clear, in my judgment, that it was the work that Mrs Burke did personally that had led to the issuing of the TC. She also makes it clear that she believes that what happened occurred as a result of what Mrs Burke did:
“Following Mrs Burke's investigation, Post Office generated a transaction correction for the £150 withdrawal that had not been recovered at the time. If Mrs Burke had not done this herself, there is a process built into Horizon for flagging non-recovered transactions which would have prompted an investigation and I'm sure would have led to the same outcome.”
151. She explains that had this not been done, more would have been required from the Post Office – namely an investigation – before the Post Office would have reached the decision to refund the £150, although she does say that she is sure the same outcome would have been reached. This is rather different to the way the case was put to Mrs Burke, which did not include the requirement for any such further investigation by the Post Office being required. The impression sought to be given was that on the Fujitsu documents already available, a TC would have been issued anyway to Mrs Burke a few days later. I do not accept that. Something more at the Post Office end would have been plainly required, as Ms Van Den Bogerd explains. Mrs Burke was the person who had in fact provided proof of payment out, and so no investigation was undertaken.
152. It is in my judgment likely, on the basis of all the evidence in the litigation to date, that the sum of £150 would have showed up as a shortfall at the end of the next branch trading period, as any further investigation was not likely to have been completed by the end of that period. However, if Mrs Burke had not acted as she had, and if an investigation had been done by the Post Office, if that had been resolved in her favour, and if that had led to a TC being issued, then the shortfall in the branch accounts would have been corrected in a later branch trading period, namely the period during which the TC was issued. No loss flowed to Mrs Burke in respect of this incident, as a result of her own investigation. In any event, this is not a point of difference between the parties that it is necessary to resolve in order to resolve the Horizon Issues. There was plainly a potential impact to her branch accounts in any event, and that potential impact was caused by the Horizon system.
Mr Roll
153. Mr Richard Roll had worked at Fujitsu between 2001 and 2004. This was during the days of Legacy Horizon and he had no experience of Horizon Online. He provided two witness statements, the first of which was dated 11 July 2016. This was obviously very much in the early days of the group litigation. The second statement addressed certain factual matters in Dr Worden’s first expert report. I consider Mr Roll to be an important factual witness in this group litigation. He has no personal interest in the litigation and is not a claimant. He has never worked for the Post Office, although whilst at Fujitsu working on Legacy Horizon he obviously had a great amount of involvement in the Horizon system. However, given the expert evidence and particularly the degree of agreement between Mr Coyne and Dr Worden on the number of accepted bugs in Horizon, which even on Dr Worden’s position was 11 different ones, in addition to Mr Roll’s evidence there was ample other evidence in relation to my findings on the Horizon Issues. Mr Roll’s career has started in the Royal Air Force, which he joined in 1976 and left in 1989. Whilst in the RAF, which he had joined as an avionics engineer, he worked on mainframe computer systems and was selected for a software development team working on aircraft control and attack systems. After he left the RAF, he worked in various roles in development and support, and joined Fujitsu in January 2001. There, he worked in the Software Support Centre, or SSC, in Fujitsu at Bracknell in 3rd line support.
154. After he left Fujitsu he changed career direction entirely and attended the University of Southampton, from where he obtained a BSc in Podiatry. He worked in the NHS until 2011 and then went into private practice, where he remains, now with his own clinic. The claimants described Mr Roll as a “whistleblower”, effectively making public allegations about how Fujitsu dealt with Legacy Horizon and the access which was available to Fujitsu to SPMs’ branch accounts. This was access of a far wider nature than admitted to by the Post Office or Fujitsu publicly.
155. It was Mr Roll’s evidence that was substantially behind the eventual acceptance by Fujitsu witnesses of the ability to obtain remote access without SPM knowledge or permission, and the injection of messages into the counter. In particular Mr Parker had, in his first statement, described that Fujitsu lacked this “power” and had said that Mr Roll’s evidence was inaccurate and misleading. Mr Parker later corrected his 1st statement, and I deal with his explanation about this change – and what he said he meant in his 1st statement – in my summary of Mr Parker’s evidence below at [464] to [498].
156. Mr Roll described himself as an IT specialist, a description with which I agree, and one which is demonstrably correct given the number of years he performed his role at Fujitsu in the SSC in 3rd line support. Problems or issues would only reach 3rd line support if 1st or 2nd line support did not resolve them. His IT experience includes, in my judgment, a considerable degree of expertise in software. Indeed, although I am critical of Fujitsu elsewhere in this judgment, that criticism does not extend to any suggestion that they employed personnel within SSC who were not experienced and sufficiently expert in software matters to perform that role. Had Mr Roll not been sufficiently qualified to work in 3rd line support, he would not have been there for the period that he was.
157. The Post Office in cross-examination attacked both his recollection, and seemed also to challenge his expertise. This the Post Office is entitled to do. Given he was 3rd line support, in my judgment he obviously had a high degree of expertise. There was a debate at one stage about whether he was “elite” or “super elite”, and he put himself in the former category. The Post Office also concentrated on a narrower band, or sub-set, of the issues regarding which he gave evidence, focussing more on potential problems with code and Mr Roll’s experience of those, rather than the wider issues generally in respect of Horizon of which he gave evidence. He had made clear very early in his cross-examination that, so far as he was concerned, data corruption was an issue in Legacy Horizon as well as software issues which were related to the code.
158. Mr Roll had left Fujitsu in 2004 and there will inevitably be difficulties in recollection by any witness of detailed points after such a passage of time. He was willing to indicate when he could not remember something, and would agree with points put to him by the Post Office when relevant. These were often on the basis of hypotheses which he was asked to accept, often in relation to a spreadsheet which Mr Parker had prepared showing how relatively few software issues Mr Roll was said to have worked upon. Mr Roll also, again inevitably, was not submerged in the fine details of the subject matter of the litigation, in the same way as some of the Fujitsu witnesses such as, for example, Mr Parker himself, who is the Head of Post Office Application Support and the SSC, and who has worked on the Post Office account at Fujitsu since 1997, before Horizon was even introduced. Given Mr Roll left Fujitsu in 2004, it is 15 years since he worked on Horizon.
159. His two statements were fairly short (4 and 8 pages respectively) and given at a high level. One example can be provided from his cross-examination. He was of the view that problems referred to 3rd line support, which by definition would mean that 1st and 2nd line support had not been able to deal with them, would not include problems caused by mistakes by SPMs. Mr Parker disagreed with this and this was put to Mr Roll.
“Q: Then the next unnumbered paragraph, Mr Parker says:
"If NBSC were unable to identify the cause of a discrepancy they would often fall back on a default statement along the lines of 'this looks like a software issue' so that the SSC would investigate it. However, Mr Roll's statement that 'if an error was referred to us then it was extremely unlikely to be due to a mistake made by a postmaster' is not correct. The vast majority of discrepancies investigated by the SSC as pseudo 'software issues' were (and are) not caused by software issues."
I would like to suggest to you, Mr Roll, that that's true, isn't it?
A. The way that I remember it, it was issues to do with the software that were causing the problems. Whether that was the programme that had been written or data corruption, that's what I remember as our problems being.
Q. You remember there being problems with data corruption?
A. Yes.
Q. I'm not asking you about data corruption, Mr Roll, I'm asking you about software issues. The claim I would like to put to you again is the last sentence of that paragraph:
"The vast majority of discrepancies investigated by the SSC as pseudo 'software issues' were (and are) not caused by software issues."
Are you in a position to agree to that?
A. From my recollection I would disagree with that, but it was a long time ago.
Q. I'm grateful.”
160. This passage of cross-examination, and there are a number of other similar examples, shows the following:
1. Mr Roll’s conclusion was a general one, with which Mr Parker disagreed. Assertion and counter-assertion are not, of themselves, of assistance to resolving the Horizon Issues. His conclusion, as with that of Mr Parker, is simply their point of view.
2. If a problem made it to 3rd line support, then by definition it had not been capable of being resolved by 1st and 2nd line support.
3. Fujitsu would, even at the 3rd line support level, attribute some problems which they could not understand or resolve to SPM error, and Mr Parker’s evidence is consistent with this approach by Fujitsu. This is made clear in the text of PEAKs put to Mr Parker. There is a very stark example in one of the PEAKs where the Post Office’s own auditors have specifically ruled out user error, yet Anne Chambers chose to close the PEAK with the defect cause being user error.
4. The Post Office would try to keep the evidence in cross-examination of Mr Roll within very narrow bounds. This passage shows that Mr Roll expressly gave data corruption as one of the problems of which he had experience. Horizon Issue 1 included the expression “bugs, errors and defects”. In my judgment data corruption is included within that phrase. Mr Roll was kept very close in his cross-examination to evidence concerning code alone. Bugs, errors and defects are far wider than simply code. The Horizon Issues were not specifically restricted to software issues as narrowly re-defined by the Post Office, nor are they restricted to code.
5. Mr Roll would sensibly concede that the events of which he gave evidence were a long time ago.
6. As shown by the answer “the way that I remember it, it was issues to do with the software that were causing the problems”, even at this remove of time, Mr Roll had a specific recollection that there were “issues with the software” that were “causing the problems”. This is as long ago as 2001 to 2004.
161. Some of the cross-examination was of no assistance in resolving the Horizon Issues, for example an exercise with a spreadsheet which Mr Parker had been involved in preparing, showing the percentage of calls to 3rd line support and the broad categories into which they had been placed (by Mr Parker, as it turned out, and inaccurately at that). Mr Roll was in no position to give any helpful evidence in respect of this, and the categories were in any event misleading as headline descriptive terms. A witness who had left Fujitsu’s employment some 15 years ago would not, sensibly, be in a position to agree or disagree with such detailed points or collation of data in such a detailed spreadsheet in any way that would provide the court with useful evidence, particularly when it emerged (as it later did when Mr Parker was cross-examined) how misleading the Fujitsu headings were.
162. Mr Roll had looked at Mr Parker’s statements, although he said only briefly, and he did not know how many statements Mr Parker had made. He could not remember some of the details, such as the detailed descriptions of 1st and 2nd line support at SSC, and was frank about this. He could not recall the split between 1st line support by Fujitsu and the Post Office’s own 1st line help desk called the National Business Support Centre (or NBSC). He acknowledged that he worked in 3rd line support, rather than “third/fourth line support” as he had put it in his witness statement. He did not however over-state his abilities; for example he said some members of 3rd line support were “super elite”, but he would not count himself as falling within that description. Part of his cross-examination which I consider fairly encompasses his approach to giving evidence is the following, where parts of Mr Parker’s witness statements were being put to him.
“Q. And if we can pick it up at paragraph 28 of Mr Parker's witness statement, you worked in third line for between -- well:
"Between 1 January 2001 and 31 December 2014 ...” and those are the years that you worked for Fujitsu "... the SSC received a total of 27,005 calls, meaning that on average 563 calls per month were dealt with over this 4-year period."
And he refers to a spreadsheet setting that out and he then analyses the data in that spreadsheet. Would you accept that that is a fair reflection of the amount of calls coming in, the amount of incidents coming in to SSC when you were there, third line?
A. I can't really remember. I know there were periods when it was very busy and periods when it wasn't so busy. Sometimes we had three or four jobs on the go at once, other times we were given other work to do from the manager.
Q. And paragraph 29, Mr Parker says:
"Transferred calls (ie those not resolved by the SSC) are of interest." He says:
"A very small proportion of calls transferred to 4th line support would have concerned software errors requiring resolution ..."
Stopping there, Mr Roll, that's true, isn't it?
A. Yes.
Q. So he then says:
"... it would be interesting [therefore] to know the number of calls transferred to fourth line."
Would you agree to that? It would give some indication of the extent to which incidents coming into the SSC properly, genuinely represented software areas that required fixing? That would be a useful way of -- a touchstone of trying to work out --?
A. Yes.
Q. Thank you. He then says that, unfortunately:
"... while the SSC have records of the volume of transferred calls, we do not retain records of where they are transferred to and it is not the case that all of these would have been transferred to 4th line support. For example incidents would often arrive at SSC from internal teams for routing back to help desks."
Do you remember that?
A. I don't remember that particular ...
Q. But would it be right to say that of the calls coming into third line support, a significant proportion would go -- of calls that would then be transferred out, would go to places other than fourth line support? Would that be fair? Does that accord with your recollection?
A. The way I recollect it is that calls would come in and we would work on them, either fix them, in which case they would go back to the originator, or we would pass them on.
Q. To other people?
A. To other people.
Q. Depending on the nature of the problem?
A. Yes.
Q. So if there was an infrastructure problem you would probably pass the call on to the infrastructure team, it wouldn't go to fourth line support, would it?
A. Probably.
Q. And there are a number of other teams that calls coming into third line would be passed to who would not be fourth line support?
A. Probably. I can't remember the full details.”
163. This can be seen all to be at a very high level. Mr Roll is not likely to have known the total number of calls that came into SSC during a 4 year (or indeed any) period; the average number of calls per month; or of how these were dealt with. He would also not be likely to be able to recall these details, even had he known them at the time. The analysis which was put to him would have been “useful to know” – the percentage, or even total number, of calls transferred from 3rd line support to 4th line support – would not have been something he would have known at the time either. Interestingly, not only would such information have been something that Fujitsu could have readily recorded at the time, as both 3rd and 4th line support at SSC are both Fujitsu functions, and not only was this plainly accepted as something that would be of assistance, but Fujitsu “did not retain records of where they are transferred to”. Thus the absence of a useful record in this respect is a result of how Fujitsu retained its records.
164. Mr Roll was then taken to an analysis of calls coming in, and calls going out, which had been done by Mr Parker. This related to some 3,764 calls. Mr Roll was never going to be in a position to dispute that figure; to dispute or agree the percentages given for how the calls were dealt with; or provide any meaningful evidence about Mr Parker’s analysis at all. It was presented to him for the first time in the witness box. Sometimes unreasonable or unhelpful witnesses take issue with matters of which they have no knowledge; Mr Roll did not do so. If he did not know, or was not in a position to answer substantively, he would simply say so.
165. Another passage of his evidence was as follows, when a passage of Mr Parker’s evidence was put to him that stated "From the SSC, only a tiny proportion of incidents were escalated to the 4th line support team. It follows that only a tiny fraction of incidents raised actually needed to be looked at by the only team who might potentially effect changes in software." He was asked if he accepted the conclusion:
“A. When you take it as an average then yes I suppose so, but the system evolved from -- I started there in 2000 I think it was and left in August 2004. The nature of the work changed over time and the ones that stick in my mind are the ones where there was sort of the fire-fighting efforts where it was -- there were difficult periods where there were software issues, so ...
Q. Yes, Mr Roll. We could by all means go back to the graphs because you will have seen from the live PEAKs into and out of the SSC it actually gave monthly figures, but what I'm suggesting to you, Mr Roll, is that -- although you may well have been busy on all sorts of things, the fact of the matter is that software problems requiring a software fix represented a tiny fraction of the work that was handled by the SSC third line support.
A. Yes.”
166. This theme of the way that the Post Office put its case carried on into the evidence of Dr Worden and his Section 8 statistical exercise, which I address below. The expressions “tiny proportion”, “tiny fraction” and other terms used sought to demonstrate, basically, that Horizon worked very well most, if not all, of the time. Given the use to which the Horizon system was put by the Post Office, namely monthly balances which governed the branch accounts of SPMs, with shortfalls and discrepancies being “made good” – which means paid – by the SPMs in question, this approach can be seen to have limited utility. Firstly, these are subjective terms. Secondly, given the PEAKs show that a SPM may report a single unexplained discrepancy for (say) £25,000, or even £1,000 or £500, the fact that this may be only a tiny fraction of the number of calls the SSC worked on that month might come as cold comfort to that specific SPM. The number of calls, and how they were dealt with, is of some relevance to robustness, but such high level and subjective points were not likely to be of enormous assistance, one way or the other, in challenging Mr Roll’s evidence.
167. It was put to Mr Roll that his role, which he described as “product specialist”, was “the junior level of people working” in SSC, which he disputed. He explained that most people in SSC were “product specialists”. 25 of the 30 or so total number of people were, according to Mr Roll, all product specialists.
“Q. -- the SSC? Of that 30 people how many people were at your level?
A. When I worked there I -- it was two or three people were senior levels, I think Mr Parker was one, a couple of others, then there was Mike Peach and I believe the rest of us -- 25 or so.
Q. The figures I would like to ask suggest to you, Mr Roll, are about 25 people had the junior level and there were about five people who were true specialists who I think you fairly described as specialist earlier on in your evidence. Would you accept that that might be the case?
A. Junior level seems to -- it doesn't fully explain the complexity of the system or the knowledge of the system required, but yes, I suppose ...
Q. Well, let's agree on ordinary -- I'm not seeking to cast any imputations and it is right that you should – words do have implications. So the ordinary level was your level –
A. Base level.
Q. -- and there was this perhaps five or so people above -- the senior people above you and what did they do? Did they do the more challenging work?”
An interesting post script to this line of questioning is that later in the trial it emerged that all of the SSC personnel had the very powerful APPSUP privileges at this time. It is highly unlikely that this was given to those at the “junior level”.
“3rd line support
3rd line support groups within RMGA include:
SSC – 3rd line support for RGMA written application code.
MSS – 3rd line support for software distribution and event management
3rd line support staff apply analytical skills to the symptoms and evidence gathered by 1st and 2nd line and undertake in-depth investigation into incidents. They have detailed knowledge of the system based on documentation and source code inspection.
Trained on operating systems, COTS packages that underlie the application and the coding languages used within the application. They are also expected to self train by examination of support guides, design documentation written for the components of the end user application. They will also have access to development and package management tools to allow the production of specialised diagnostic code, scripts and support tools.
It is incumbent upon the 3rd line support unit to produce a work around and on 4th line to produce the final code solution to any software problem. This does not preclude the production of a workaround by other units or negate the requirement for 4th line to provide assistance in the generation of a workaround.
The SSC are responsible for the implementation of any workarounds that require data changes to the live system. They are the only unit with authorisation and sufficient physical security controls to perform this function.”
(emphasis added)
169. This gives a more comprehensive picture of the type of expertise required to be in 3rd line support, which is where Mr Roll worked whilst he was at Fujitsu. This is of more assistance than the terms deployed, such as “the five true specialists” (for the senior managers) and “the junior level” for Mr Roll. It shows that 3rd level support have a high level of necessary expertise. They are the only unit with the necessary authorisation and security controls to be permitted to implement workarounds that require data changes to the live system. This is not the role of junior personnel.
170. This area of questioning also focused on code, and introduced another concept, the “core specialist team”.
“Q. I'm really trying to grope towards what the five true specialists, what kind of work they were doing. They were doing the more challenging kind of work, weren't they?
A. I find it difficult to answer that from what I remember of the way we worked. There were some areas where some of the senior people where I would perhaps have been more -- had more experience because of previous work and previous programming that I would have been better off and had more knowledge than they would have done, but in other areas then they would be far superior to me.
Q. Well, let me suggest one area where they would have been far superior to you, Mr Roll, or at least it was perceived within the organisation -- I'm not trying to have a debate with you about your own perceptions of yourself, that would be completely unfair, but the perception at the SSC was that were there software errors or potential software errors that required large amounts of code to be examined, the people who would generally do that examination would be those five people, people like Mr Parker. Would you accept that?
A. Generally I suppose, yes, although –
Q. And it would be relatively rare for someone outside that core specialist team to be doing that kind of work -- not impossible but relatively rare, yes?
A. Several of us looked at the code on occasions. I was a C programmer in that and other languages before I moved to Fujitsu so there were areas of knowledge we had from previous areas, but I suppose most of it would then be passed on to those people.
Q. I see. So would you accept from me then that generally it would be them that would look at lots of code but occasionally there might be occasions when someone else would look at lots of code?
A. From a PinICL perspective yes, but some of us looked at code more often, just out of interest.”
171. This series of questions introduced another subjective variable, “large amounts of code”. Mr Roll, in his answer above, explained that he had been a C programmer before he moved to Fujitsu. C is a general purpose programming language which is widely used. Mr Roll, as someone who had been a programmer both in C and other languages before he went to Fujitsu, had both written code himself (as a C programmer), and I find would have been very familiar with code in its different guises. Indeed, given the Fujitsu documentation describing the expertise of 3rd line support in [168] above, which stated 3rd line support would have “detailed knowledge of the system based on documentation and source code inspection”, it is clear to me, and I find, that Mr Roll was amply experienced to give evidence to this court on the matters that he did concerning the Horizon system (which was at that point Legacy Horizon), the problems that were experienced with it, and what occurred at SSC. Source code is a term used to describe a version of software, or a description, as it is originally written. There are different definitions of source code in different computer authorities (such as the Linux Information Project) but the precise definition is not important for the purposes of the Horizon Issues trial. The Horizon Issues do not concentrate solely on problems either with code, or source code. The phrase is “bugs, errors and defects” and the Horizon System is defined in the Horizon Issues.
173. Mr Roll’s evidence was that errors made by SPMs were relatively easy to pick up at 1st and 2nd line support level; that most errors he dealt with were coding errors or data corruption; that issues were identified that required software “fixes” to be written by Development; and – importantly in my judgment - that the type of issues that were routinely encountered at SSC could and did cause financial discrepancies to branch accounts. He also stated that:
“If we were unable to find the cause of the discrepancy then this was reported up the chain and it was assumed that the postmaster was to blame.”
174. He said that even if software fixes were developed the problem would sometimes reappear several weeks later. He also stated that remote access to branch systems was possible; the ability was extensive; that this was done without the SPMs being aware; that data and transaction information could be changed by Fujitsu; and that sometimes SSC would log into a branch system whilst it was switched on but not in use.
175. The final paragraph of his 1st statement said:
“In summary, the issues with coding in the Horizon system were extensive. Furthermore, the coding issues impacted on transaction data and caused financial discrepancies on the Horizon system at Branch level. It was those issues that I, and other colleagues at Fujitsu, were routinely working on daily. Furthermore, remote access to the Horizon system at Branch level was extensive, as was the ability to change data and change transaction information, even while the postmaster was working, without the postmaster being aware of this.”
176. He had also said that “during the course of resolving the software issues, we would frequently access a Post Office counter IT system remotely”.
177. His use of “frequently” and “routine” are, in my judgment, subjective, and as explained above in terms of “tiny fraction”, subjective terms are not entirely helpful. What he meant by this is it was not unusual for this to occur. It is difficult to judge, at the remove of 15 years from when Mr Roll left Fujitsu, just how often something that he remembers as frequently or routine in fact occurred. He sensibly accepted that the more regular or mundane matters in which one is involved tend not to be remembered, with rather more unusual events sticking in the memory. In terms therefore of how often this occurred whilst he was there, it is not possible to come to a concluded view. However, I consider it important that the occasions to which Mr Roll refers were not isolated incidents on his evidence, nor were they unusual, and that financial discrepancies on the Horizon system at branch level was something which he recalls working on daily. It is also important that he gave evidence that if Fujitsu could not track down the cause then it was assumed the SPM was responsible, in other words user error would be used as a default setting for investigations. This matches the evidence of an enormous number of PEAKs in the Technical Appendix.
178. He also gave evidence about hardware failures, and gave specific responsive evidence in respect of Dr Worden’s expert report.
179. I accept Mr Roll’s evidence, which is supported by the contemporaneous documents. The Post Office in its closing submissions accepted that he was a careful and precise witness, and also that “he was at pains to assist the court and to give accurate answers”. It also, however, submitted that his evidence “was, unsurprisingly, hazy in many respects”.
180. Notwithstanding the limitations on his evidence due to the passage of time, I found Mr Roll to be a reliable and helpful witness. I do not consider it was hazy in any important respects. I also found his evidence to be very important. The Post Office set out in cross-examination to demonstrate that he did not have the expertise that he claimed to have; that he was at too low a level in 3rd line support to have been involved in the matters which he described, certainly so far as software issues were concerned; and that Mr Parker’s evidence in particular about what he had been doing should be preferred. On those two former points the Post Office failed, in my judgment. The success or failure of the latter point also, but not exclusively, depended upon the evidence of Mr Parker. After Mr Parker was cross-examined, it was clear to me that Mr Parker’s exercises that were put to Mr Roll could not be relied upon to demonstrate what they sought to demonstrate. I prefer and accept the evidence of Mr Roll to that of Mr Parker and the other Fujitsu witnesses (with the exception of Mr Godeseth) by some considerable margin. I find that during the years when Mr Roll worked at Fujitsu, 2001 to December 2004, SSC were not infrequently involved in attempting to remedy unexplained shortfalls and discrepancies in branch accounts reported by SPMs. If they were unable to find the cause of the discrepancy then the assumption would be made that it must be the SPM to blame. This is clearly shown in my analysis in the Technical Appendix particularly on the different entries in the Bug Table, and my findings on the number of bugs present in the system.
“15. During my time at Fujitsu I know that there were budget pressures and redundancies which impacted system development and testing. The test team felt they were under enormous pressure to complete the testing within certain timescales which negatively affected the test regime. Meanwhile, the development team had to balance time spent on fixes, with time spent on developing new features for Legacy Horizon and time spent developing a new system which I believe later became Horizon Online.
16. In my first statement, I refer to the pressure that the SSC team and Fujitsu were under generally due to an awareness of the financial penalties imposed by the service level agreements between Post Office and Fujitsu (paragraph 12 of my first statement). I believe that although individual penalties were quite modest, when applied across multiple counters/post offices the cumulative figures involved were very high, potentially amounting to tens of millions or more. I disagree with Stephen Parker’s statement that these potential financial penalties were not a factor for the SSC (paragraph 43 of Stephen Parker’s witness statement) as we were aware of them and often commented on them, e.g. “That’s saved Fujitsu another £25 million”.”
182. I do not attach much weight to this as pressure in the workplace is such a subjective matter. Some members of SSC may have felt under great pressure; others less so. This is a subjective account of what it was to work at SSC at that time from the point of view of Mr Roll. The type of statement to which Mr Roll refers in terms of millions of pounds saved does not advance matters one way or the other. Given the type of problems that 3rd level support was involved in attempting to resolve, there would have been an inevitable pressure involved in attempting to resolve such issues speedily. Mr Roll chose to leave Fujitsu, and therefore he left those pressures behind. Putting it at its mildest, it would be extraordinarily disappointing if PEAKs were closed, attributing fault to a SPM, simply because that was the easiest and quickest way for a SSC product specialist to keep on top of their workload. Without separate cross-examination of each product specialist who chose to close PEAKs where prior entries suggested user error had been ruled out, it is not possible to determine the degree to which this was their motivation in any individual case. However, it is not relevant. It is no part of the Horizon Issues to determine why, or whether, individual personnel at SSC acted in a particular way – budgetary concerns for their employer; exercises of judgement; even laziness; or any of the other many possibilities. Mr Parker gave evidence that “the possibility of financial penalties was never a factor for the SSC”, and Mr Roll said that it was. Mr Parker’s evidence on this suffers from a wholesale attribution of this lack of motive to every member of SSC, as indeed does Mr Roll’s, in the other direction. Everyone is different, and there will have been a range of different reasons operating on each member of SSC each day. Any sensible business will, in any event, always have at least part of its attention on financial performance and this is understandable. The Horizon Issues trial is not an inquiry into how Fujitsu manages its personnel, or its business. I do question the allocation of Category A, B or C to some of the PEAKs that were used in the trial, but this is something that was pursued (though not very far) with Mr Parker and I deal with it there.
183. The way that Dr Worden was simply not prepared to accept Mr Roll’s evidence at face value, and set out to disprove his factual evidence, something which in my judgment an independent expert should never do, is dealt with in the section of this judgment where I deal with Dr Worden’s evidence. The degree to which the evidence of Mr Roll, and his experience of Legacy Horizon between 2001 and 2004, affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
184. The fact that I find Mr Roll reliable does not mean that I automatically accept all his conclusions. I have given a specific example at [181] about financial pressures. However, Mr Roll’s evidence was supportive of the claimants’ case and it was necessary for the Post Office to render his evidence unreliable, insofar as they could, in order to damage the claimants’ case and bolster their own. In my judgment, this the Post Office failed to do. Mr Roll’s evidence was also what directly led to the true picture emerging of remote access. However, of itself this does not provide an entire answer to the Horizon Issues.
Mr Henderson
185. As explained in [78] and [79] above, Mr Henderson is a director of a company called Second Sight which was involved in a mediation scheme, until the Post Office withdrew from it. Initially Second Sight was engaged to perform an investigation, but after its interim report its scope of involvement changed. It ran from 2012 to 2015, when Second Sight’s appointment was terminated. The investigation and subsequent mediation scheme were set up in the light of disquiet about the complaints of various SPMs, the operation of Horizon, and concerns that were publicly discussed, not least by Members of Parliament. All of the documents and emails which Second Sight had at that time were returned to the Post Office. Mr Henderson estimates these as 16,500 emails and 18,500 other documents, which he said were sensibly organised in 1,700 folders. They were in good order.
186. The documents in that set that were provided by the Post Office to the claimants for this litigation were not, according to Mr Henderson, provided in the same way. He criticised them for having no folder structure, having had metadata removed, and lacking the original date and time stamps which were present on the original documents with which Second Sight had been provided by the Post Office. Some important documents, such as the Fujitsu XML transaction reports, were described by the Post Office as “unreadable”. He had however reviewed some, but not all, of the XML data when inspection was provided of some of the unreadable documents. He did not believe that the documents provided in the Horizon litigation were the complete set of data provided to Second Sight, which he had returned. I accept his evidence on this. He was clear in his recollection about how the documents were organised, and approximately how many there were and their type. I do not know why the claimants were not given, in disclosure, the same documents, prepared and collated in the same way, as the Post Office themselves had received them from Second Sight.
187. He described Mr Jenkins, whom he had met, as the Fujitsu lead engineer on the Post Office contract. He had met him in September 2012. He explained in his witness statement that he had been given sample XML data by Mr Jenkins, and had used this to see a level of detail that neither SPMs nor, interestingly, the Post Office, had available to them to view. He had also been told at the same meeting that remote access was “occasionally used” by Fujitsu. He asked for complete email records for 2008 for the 7 employees known to be working at Bracknell at the time, so he could satisfy himself as to whether such remote access was done without the SPM’s permission or knowledge. He was not given these, and the partial records he was given were inconclusive. His reverse engineering of the XML data he was given gave him grounds for concern but Second Sight’s engagement was terminated before he could reach any concluded view.
188. He had also identified what he considered to be particular difficulties that Horizon had with foreign currency – it is (or was) a single currency system and only used pounds sterling – and also the National Lottery. In view of the evidence of Ms Van Den Bogerd in the Common Issues trial and the Ping fix, that latter point may not be particularly controversial in any event as this litigation progresses. Foreign exchange is the subject matter of bugs 14 and 23 in the bug table. Bug 14 is now accepted by the Post Office in Appendix 2 of its Closing Submissions as one of a number of “bugs with lasting impact (although they were resolved)”. I find in the Technical Appendix that Bug 23 is a bug. The contents of the Technical Appendix are therefore consistent with Mr Henderson’s views on this.
189. A working group was involved in the Second Sight scheme, and the progress of cases all the way through the mediation scheme was slower than the working group wanted. Second Sight’s appointment was terminated before Mr Henderson could reach finalised conclusions on most of the issues that he said had been uncovered. I have already stated the approach that I adopt in relation to the technical nature of the Horizon Issues, and how the evidence of the experts in IT who gave evidence before me effectively takes precedence over observations by other witnesses on technical issues, which must include Mr Henderson.
190. Mr Henderson explained in his oral evidence that he considered his evidence to the court not to be entirely unrestricted. He considered it was subject to a restriction that had been imposed upon him. There was a letter of engagement in this respect, which was dated 1 July 2014 and accompanied by a document which was signed by someone called Chris Aujard for the Post Office. The letter of engagement set out several pages of terms governing Second Sight’s engagement. Clauses 2.1 onwards set out what Second Sight were to do:
“2.1 The Scheme has been set up to try to achieve the mutual and final resolution of a Subpostmaster's concerns about Horizon and any associated issues.
2.2. Second Sight Support Services Limited ("Second Sight" or "you") has agreed to be a member of the "Working Group" whose role it is to oversee the Scheme and to assist in investigating individual Subpostmaster complaints.
2.3. This letter and its schedules will form the basis of the terms of Second Sight's engagement by Post Office Limited ("Post Office") to provide Services to the Working Group in relation to the Scheme.”
191. Clause 6.2 stated:
“6.2 Second Sight will not, and will ensure that the SS Directors and any SS Personnel will not, act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees save to the extent a) that it is expressly agreed in writing by Post Office that the work proposed to be undertaken will not have a material adverse effect on Post Office's commercial or financial interests or reputation, or b) as required by applicable law or by the mandatory rules or requirements of any regulatory authority, government department or agency to which Second Sight is subject or c) as required by an order of a court of competent jurisdiction.”
Clause 6.3 imposed this for a period of 15 months.
192. The shorter one page document which Mr Henderson and his co-director were asked to sign and return contained different terms. The letter of engagement set out the terms upon which Second Sight would act in the mediation scheme. The shorter document stated the following:
“I will not act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees save to the extent a) that it is expressly agreed, in writing by Post Office that the work proposed to be undertaken will not have a material adverse effect on Post Office's commercial or financial interests or reputation, or b) as required by applicable law or by the mandatory rules or requirements of any regulatory authority, government department or agency to which Second Sight is subject or c) as required by an order of a court of competent jurisdiction.”
The period of that was stated to be 15 months, corrected by hand from the period initially typed in of 12 months. This obviously therefore, as signed, represented if not the same then almost identical terms as those contained in paragraph 6.2 of the letter of engagement.
193. It is a curiosity that the detailed terms in the letter of engagement were produced by the Post Office – as Mr Henderson explained – about half way through the process. It is also the case that the services to be provided by Second Sight were to be provided to the Working Group, which also included members of the Justice for Subpostmasters Alliance or JFSA, although the services were to be paid for by the Post Office. I have outlined these terms above to make it clear, on a transparent basis, what restrictions were imposed upon Second Sight. The Post Office’s wish to avoid any “material adverse effect on Post Office's commercial or financial interests or reputation” is expressed in very wide terms. Its concern about its reputation is similar to a provision included in its funding agreement with the National Federation of Subpostmasters which is referred to in Judgment (No.3).
194. The restriction to which Mr Henderson referred in his oral evidence, which he considered limited his evidence, was an agreement between the Post Office and the claimants about what Mr Henderson could state in evidence in this litigation, which he referred to as the “protocol agreement”. The Post Office was therefore in the position that it could, to a certain extent, control the scope and extent of Mr Henderson’s evidence. He also thought that there was a restriction on the length of the witness statement he could provide. For clarity, there was no such restriction imposed by the court.
195. The Post Office, in its closing submissions, submitted the following:
“Mr Henderson was asked whether confidentiality restrictions had caused him any inhibition in answering the questions put to him in cross-examination. He said that he had the issue in the back of his mind and that he had tried to make sure that his answers did not infringe the protocol. It is understandable that Mr Henderson would wish to be careful, but the idea that he was restricted in answering Post Office’s questions is bizarre.”
196. The submissions then went on to deal with the questions put to Mr Henderson, and how there was no restriction that could have impacted upon his answers.
197. The actual question posed to Mr Henderson was not as set out in the above extract of the submissions at [195] above, it was rather wider. It was posed by me and it was at the end of his cross-examination. It was in the following terms:
“I just want to be clear: is it your evidence therefore that because of that protocol agreement your evidence of fact to this court is narrower in scope than it would be absent the protocol agreement?”
198. His answer was “yes, it is”. He was then asked by counsel for the Post Office whether it had inhibited him in answering questions, and he said it did not. I do not consider that the closing submissions are an accurate summary of both questions put to him about this, or the restrictions he considers had an impact upon his evidence. The restriction was wider than impacting upon the questions he was asked, and his answer showed that his evidence as a whole had been affected. In any event, those questions were posed to him by the Post Office, as he was called as a witness for the claimants.
199. The mediation scheme was brought to an end by way of notice given by the Post Office to Second Sight on 10 March 2015. The cross-examination on this was as follows:
“Q:…. If we could just look at it very briefly, the termination letter to which you have just referred is at {F/13/24.1}. That's the termination letter I think to which you at least indirectly referred.
A. That's correct.
Q. If we could look then to {F/1324.2}. This is a much longer letter from Post Office that came to you on the same day. Do you recall this longer letter? If I maybe tell you broadly what it is to do with. It's a letter in which Post Office sets out a plan for how Second Sight could finish its outstanding work. Do you remember that letter?
A. Yes, I do.
Q. And the second paragraph makes clear that Second Sight was expected to continue working during the notice period and that even beyond the notice period there would be a proposed future role for Second Sight. That's what this document dealt with, isn't it? It's a fairly long document.
A. It is, but there's another document of that date or very close to that date which you haven't mentioned which was the press release from Post Office announcing the winding up with immediate effect of the mediation scheme itself. That I understand was the primary decision and our termination was a consequence of that, not a separate issue.
Q. Well, I think when you say mediation scheme it's fair to say, isn't it, that this winding up process, as you describe it, did not put an end to mediations; in fact –
A. It put an end to the mediation working group with, as I understand it, no consultation and the announcement was made I think the day immediately before the next planned meeting of the working group, so it was a considerable sort of shock to everybody.”
The following points can be made in respect of the evidence from Mr Henderson:
1. Second Sight was engaged to act in a mediation process. Mediation is consensual dispute resolution, and such processes require a high degree of confidentiality within them if they are to be effective.
2. The termination of that process unilaterally by the Post Office, with or without consultation, is, so far as this group litigation is concerned, something that may or may not in the future call for further consideration in terms of costs. It is not at this stage relevant to the answers to the Horizon Issues.
3. Reasonable and conventional terms of confidentiality were to be expected in any terms of engagement agreed between the Post Office and Second Sight, given the scheme was a mediation scheme.
4. Terms seeking to protect the “Post Office's commercial or financial interests or reputation”, which were included, whether justified or not in all the circumstances of the role being performed by Second Sight, are not relevant to the answers to the Horizon Issues.
5. In Farm Assist Ltd (in liquidation) v Secretary of State for the Environment, Food and Rural Affairs [2009] EWHC 1102 (TCC) a mediator, who had been issued with a witness summons compelling her attendance to give evidence, failed in her bid to have the summons set aside. Ramsey J gave a judgment refusing to do so, having considered the position on confidentiality, without prejudice privilege and other privilege. He stated, in his conclusions at [44]:
“The court will generally uphold that confidentiality [of mediation proceedings] but where it is necessary in the interests of justice for evidence to be given of confidential matters, the Courts will order or permit that evidence to be given or produced.”
The point has not, however, been argued in this case in any respect, and it may never arise.
6. In this case the restriction has been imposed by the Post Office and agreed by the claimants. It is regrettable, in my judgment, that any witness of fact feels their evidence to be restricted by any existing agreement with a party to that litigation. Apart from anything else, it is something of a contradiction for a witness, who swears or affirms that their evidence is “the truth, the whole truth and nothing but the truth” to then add that there is such a restriction. This appears to contradict the requirement to tell the whole truth. However, the court has never been asked to become involved in resolving any dispute between the parties in this respect.
7. I do not consider that any such restriction – the scope of which I am in any case unaware – will have had any effect upon my consideration of the correct answers to the Horizon Issues, or the answers themselves.
200. Mr Henderson seemed to me to be a careful and honest witness. Although his involvement in the Second Sight scheme may have led to his having particular personal views about Horizon, he was measured about how he expressed these. His views about concerns over issues in particular affecting the National Lottery and foreign currency exchange are, to a certain extent, corroborated both by the Ping fix and the position of the Post Office on bug 14 in the bug table. However, whether they are corroborated or not, it is not necessary to come to a concluded view on Mr Henderson’s evidence about these, as it is the two IT experts whose evidence will be of most assistance in this respect.
201. The degree to which the evidence of fact by the claimants affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
E. Evidence of Fact: The Post Office
202. The Post Office called the following witnesses. For the reasons explained further below, I consider Mrs Van Den Bogerd and Mr Godeseth to be the most important witnesses called by the Post Office.
Mrs Van Den Bogerd
204. I have already explained that Ms Van Den Bogerd’s evidence was considered completely afresh in this trial, and that simply because I had made the findings that I had about her previously, did not mean that I would adopt the same conclusions in this trial. Her written evidence for the Horizon Issues trial was still substantially of the same tenor in relation to individual SPMs, in terms of widespread attribution of fault to SPMs as a default setting, but I did detect a change of approach in Ms Van Den Bogerd in this trial, compared to the Common Issues trial.
205. Originally, for example, in her statement Mrs Van Den Bogerd had specifically attributed Mr Patny’s loss on MoneyGram to user error, based on what she said the data showed. This passage, in paragraph 72, was corrected by her on a sheet of corrections of a number of the Post Office’s witnesses, and in that correction (there were a number of others) she explained that Mr Patny had cancelled the transaction but had not reversed it. She therefore accepted that the transaction had been cancelled.
206. She generally demonstrated in her cross-examination a more realistic approach to the accuracy of her evidence than she originally demonstrated in the Common Issues trial. By the time she was cross-examined she would have had the benefit of reading Judgment (No.3), and that might explain her new approach.
207. She had also amended, in her sheet of corrections, certain statements that had been included in her Horizon Issues witness statement that were simply not sustainable on the facts. One example of this was in relation to Mrs Stubbs (a claimant and witness in the Common Issues trial, though not in the Horizon Issues trial) whom Mrs Van Den Bogerd had said “chose to settle centrally” items that were, in fact, obviously and plainly disputed by Mrs Stubbs; and in respect of which no SPM had any real “choice”. Their choice, such as it was, was either paying immediately or settling centrally, which meant not paying immediately, but seeking time to pay. Mrs Van Den Bogerd’s explanation for this was that she had not known these sums were disputed by Mrs Stubbs. Quite how that could be, given the extended saga in relation to these sums, the involvement of Mrs Stubbs’ MP on her behalf (Sir John Redwood, a former Cabinet Minister), the Post Office’s promises both to Mrs Stubbs and her MP of an investigation (the results of which, if one was ever done, have still not emerged, so far as I know), and indeed Mrs Stubbs’ own evidence in the Common Issues trial, is entirely unclear.
208. She also explained that in relation to Mrs Burke “ I have looked at other evidence in relation to Mrs Burke and what was very clear to me is that Mrs Burke had done absolutely nothing wrong in that situation.” This was not at all how Mrs Burke had been cross-examined. Mrs Van Den Bogerd said that the corrections she had made to her statement had been communicated to the Post Office’s solicitors before the Horizon Issues trial had started. If that is true, I do not see how counsel for the Post Office would have cross-examined on the basis of her un-corrected statement.
210. I am most surprised that Mrs Van Den Bogerd could not remember seeing this PEAK before she was shown it in cross-examination. It is a very important PEAK. It is PEAK number PC0065021, dated 17 April 2001. The reason it is important is as follows. It relates to multiple branches. It concerns phantom transactions. It identifies dissatisfaction from more than one SPM as to how phantom transactions are being investigated and resolved, or more accurately, how they are not being. It shows calls being “closed” – ie brought to an end - without the permission of the SPM, even though that should not be done. It also shows at least one SPM threatening not to comply with their contractual obligations due to lack of confidence in the system, and also threats of legal action. Further, in one branch, where items had been the subject of phantom transactions (according to the SPM) ROMEC, the Royal Mail’s own engineers, attended that branch to fit suppressors and other equipment in an effort to rectify this.
211. The PEAK plainly records the involvement of ROMEC, the Royal Mail’s own engineering personnel, as follows. “ ROMEC have been to site and state that they have actually seen the phantom transactions, so it is not just the PM's word now.” (emphasis added). The significance of this entry is obvious, and notable. Mrs Van Den Bogerd agreed that this was “independent site visit corroboration of the problem by Royal Mail’s own engineers at the branch”, and she also agreed that this was “clearly not user error any more”. I do not understand how the master PEAK containing such important information could not have been at the forefront of Mrs Van Den Bogerd’s mind. It is, in my judgment, important corroboration from those with experience of Horizon (the Royal Mail’s own engineers) who stated they had actually seen the phantom transactions.
212. However, the conclusion reached by Fujitsu and recorded in the PEAK was as follows:
"Phantom transactions have not been proven in circumstances which preclude user error. In all cases where these have occurred a user error related cause can be attributed to the phenomenon"
The PEAK also concludes “No fault in product".
214. Another PEAK Mrs Van Den Bogerd was asked about demonstrated the lack of accuracy in Fujitsu’s characterisation of the type of problems that made their way to the SSC (and which formed the basis of the Post Office’s cross-examination of Mr Roll based on the Fujitsu/Parker spreadsheet). This was PEAK PC0208335 of 14 February 2011, also headed “Phantom Stock Declaration”. This related to withdrawn stock discrepancies, and was summarised as "Branches will be forced to declare stock when they don't want to. Apparent reappearance of withdrawn stock may cause spurious discrepancies." It was recorded in the PEAK that this could affect 10 branches per week over the next few months. The PEAK showed that a SPM was told to declare the correct stock, which that SPM disagreed with, and which Mrs Van Den Bogerd agreed would mean the SPM entering an account into the Horizon system with which the SPM disagreed. Mrs Van Den Bogerd had not seen this PEAK either, before she was asked about it in the witness box. She agreed that this PEAK appeared to be a software problem, and also that it was in a sense easier for the SPM because the phantom transaction related to stock that was not even any longer in use. The SPM in question simply could not have dealt in that stock.
215. In any event, I found this PEAK of great assistance, not only due to the content I have summarised above. A fix was developed by Fujitsu but Anne Chambers, a Fujitsu employee who had also given evidence in legal proceedings in court before (in at least one case, the Castleton case) had entered the following in the PEAK which stated that “this fix and the MSC already applied doesn't remove all old declarations”. Further, Fujitsu chose to categorise this PEAK as “Administrative Response”. That term does not begin, in my judgment, properly to describe or summarise the problems to which the body of the PEAK referred. I find that this PEAK clearly related to a software problem, regardless of the misleading way that Fujitsu chose to categorise it as “Administrative Response”. I consider that anyone reading this PEAK at the time could only sensibly conclude that this was a software problem.
216. Mrs Van Den Bogerd was taken through a number of examples of real-world situations, recorded in a variety of contemporaneous documents, where a wide variety of SPMs reported a very wide range of problems. In one an internal Post Office email reported that a SPM had “found sensitive issue with Horizon when the system put a phantom cheque on the cheque line in July 2013. Claims to have evidence to support his claim. Although he himself did not suffer a loss, thinks that Horizon is flawed. Did not ask to be contacted about this. Just wanted to say that he had this information and threatened to go to MP as a result.”
217. The question was posed internally at the Post Office:
"Given the current media and in particular the BBC's attention on Horizon, do you think it is worthwhile looking into this 'alleged flaw' with Horizon that this SPMR has highlighted to pre-empt any enquiries from his MP?"
218. The ultimate response from Andrew Winn of the Post Office was that the claim could not be investigated without further details and Fujitsu involvement, that Mr Winn did not understand the purpose of the call by the SPM, and also stated:
“My instinct is that we have enough on with people asking us to look at things.”
219. Mrs Van Den Bogerd agreed that this was an inadequate response. She said it would have been very easy for Mr Winn to have contacted the branch and obtained further details, and he should have done so. In my judgment, the stance taken by the Post Office at the time in 2013 demonstrates the most dreadful complacency, and total lack of interest in investigating these serious issues, bordering on fearfulness of what might be found if they were properly investigated. This SPM, whose branch was known to the Post Office, should obviously have been asked for further details (if further details were required for an investigation), and the Post Office and/or Fujitsu should plainly have investigated the matter as a matter of some importance. By 2013 Horizon was an extraordinarily controversial subject; there can simply be no sensible excuse for the Post Office’s failure to try and understand this particular subject. This is particularly reprehensible given that an internal Post Office document in August 2013 showed that Mr Winn’s involvement in this was because his area of responsibility was as follows: “also responsible for resolving specific branch accounting issues.” It was his specific job to resolve specific branch accounting issues, yet he decided at the time that “we have enough on”. I agree with Mrs Van Den Bogerd that this is inadequate – that is putting it at its most favourable for the Post Office. Somewhat stronger terms are also justified.
220. Mrs Van Den Bogerd had only learned of the ability of Fujitsu in terms of remote access, namely the insertion of transactions at the counter under Legacy Horizon “within the last year or so”, which given she is central to the Post Office’s position in the group litigation shows, in my judgment, a remarkable situation. It is not necessary, in order to resolve the Horizon Issues, to go further into why she has found this out so belatedly, or why that might be.
“However, due to an error by Post Office, instead of increasing the scratch card stock, the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with Horizon.”
Actually, given the process of issuing TAs for the lottery was entirely automated, and done in conjunction with the Camelot system and by Horizon, she accepted that the passage should be corrected. The cross-examination was as follows:
“Q: So would it be fair to correct that part of your statement –
A. It would be actually, yes.
Q. -- to say "To be clear, this is an issue with Horizon and not a data entry error by Post Office"; is that a fair correction to make?
A. I have made a mistake in that the way I have worded that, absolutely, yes. So, yes, it is fair.
Q. Would you agree with the formulation I have given, or would you prefer something slightly different? What would you want the court to note as your evidence?
A. So the automated -- my understanding is the transaction -- the TA is the information that comes from Camelot to us and then it is passed through into Horizon, so in that respect Horizon just conveys it, is my understanding, and the information that's come from Camelot in that respect would be incorrect.
Q. So the point is that either way, it is not a manual data entry by Post Office?
A. No, it's not.
Q. Whatever it is, it's definitely not that, you agree with that?
A. Yes, absolutely not. Agree, yes.
Q. What it could be is some problem with the information somewhere between the terminal in the branch –
A. The lottery terminal, yes.
Q. The lottery terminal in the branch and the matters showing up on the face of the Horizon terminal in the branch?
A. Yes, I agree that.
Q. Somewhere there?
A. Yes.
Q. So it's definitely not a user error, is it?
A. No, that's not user error.
Q. Right. And on the face of it, it at least suggests some doubt as to the robustness and integrity of the Camelot data coming through in that automated system, doesn't it?
A. I would say yes.”
222. It can therefore be seen that her written evidence – “To be clear, this was a data entry error by Post Office and not an issue with Horizon” – could more accurately be stated, after she answered these questions, as “to be clear, this was not a data entry error by Post Office and was an issue with Horizon”. There is a world of difference between those two statements, which are, in my judgment, poles apart both in their content and effect. Indeed, the accurate statement is the exact opposite of how it had been put in her witness statement.
224. She had also said that Mr Coyne had relied upon extracts from the Rose Report which were “taken out of context and mistakenly claim[s] that the relevant reversal was issued in error by Horizon and not the” SPM. She then went on to explain why this was not correct. The Rose Report is explained in more detail both below and at [940] onwards where I consider contemporaneous documents. However, in her cross-examination, she accepted that this part of her statement was not correct. The Rose Report, in fact and in express terms, stated that the reversal was done by the system, something Mrs Van Den Bogerd accepted; she said that it was “done by the system, absolutely yes”, that it was “definitely generated by the system” and there was “no question” it was generated by the system. She stated she was not “making herself clear” in her witness statement.
225. Mrs Van Den Bogerd was re-examined on this specific point and the question and answer are as follows:
“Q: The extracts taken from the report by Helen Rose referred to by Mr Coyne are taken out of context and mistakenly claim that the relevant reversal was issued in error by Horizon not the subpostmaster."
I just want to give you an opportunity -- Mr Green was pressing you to accept that actually what you had said in that sentence was wrong. I would like to give you an opportunity to explain to the court what you actually meant.
A. What I meant was that the actual reversal was part of that recovery and it had actually taken place as it should have taken place, which is what I meant in that. So it wasn't a failed reversal because it actually had happened as it should have happened, but I accepted in there that the -- it wasn't obvious to the postmaster at the time that what had happened -- that he hadn't -- because it didn't show that he had actually -- it showed that he had done it and he knew he hadn't done what we referred to earlier was an explicit reversal. That's what I meant in that.”
"Q:
“Just one question from my part - if the reversal is system created but shows as an existing reversal, could this not be reflected with a different code ie SR (system reversed) to clear up any initial challenges.
My feelings at the moment are not questioning what Horizon does as I fully believe that it is working as it should, it is just that I don't think that some of the system based correction and adjustment transactions are clear to us on either Credence or ARQ logs."
That's what she is saying, yes?
A. Yes.
Q. And that was a fair observation, wasn't it, by her?
A. That was a fair observation, yes.
Q. About the shortcomings of Credence and ARQ logs, yes?
A. Yes.
Q. And Gareth Jenkins' answer:
"I understand your concerns. It would be relatively simple to add an extra column into the existing ARQ report spreadsheet, that would make it clear whether the reversal basket was generated by recovery or not.
I think this would address your concern. I'm not sure what the formal process is for changing the report layout. Penny, can you advise as to the process: is this done through a CR?"
Do you know what a CR is?
A. Change request.
Q. Change request, okay. Then at the bottom:
"I do believe that the system has behaved as it should and I do not see this scenario occurring regularly and creating large losses. However, my concerns are that we cannot clearly see what has happened on the data available to us and this in itself may be misinterpreted when giving evidence and using the same data for prosecutions.
My recommendation is that a change request is submitted so that all system created reversals are clearly identifiable on both Fujitsu and Credence."
Do you know if that change request was acted on after that?
A. I don't believe it has been acted on.
Q. You don't believe it has been?
A. I don't believe so.”
(emphasis added)
228. Another express passage or conclusion in the report was also put to her:
Q. "The reversal was due to recovery (Counter Mode Id= 8 118) so this was not an explicit reversal by the clerk. This scenario is fairly rare so it is certainly quite easy for the clerk to have made a mistake and either he or the customer could be in pocket/out-of-pocket (depending on exactly what happened!)."
Then this:
"The system is behaving as it should."
So there were issues, weren't there, where a problem could arise for a subpostmaster by design of the system; were you aware of that? There was a whole category of PEAK codes for faults which are agreed between Fujitsu and Post Office to just stay like that as part of the design?
A. Okay.”
These two questions were then put individually:
“Let's take it in stages. You can see there it says "The system is behaving as it should"?
A. Yes.
Q. That would be problematic for a subpostmaster, wouldn't it?
A. The way Gareth [Jenkins] describes it here would be, yes, because what he says is that it would have printed the session receipt but it doesn't seem as if it did, when actually disconnection transaction receipts were actually printed in this example and a recovery receipt was printed. But that's not referred to in here.
Q. Well, let's just have a look at that. Let me just ask you my second question and then we will go on to probe that with more care. The second point is are you aware of a closure code for Fujitsu for PEAKs which refers to faults which are known in the Horizon system but agreed between Post Office and Fujitsu to stay there?
A. I'm not aware of a closure code.
Q. You didn't know about that?
A. No, I don't know about that.”
229. The following important points arise in respect of this:
1. Mr Jenkins thought that “the system had behaved as it should”. Given Mrs Van Den Bogerd accepted that this situation could lead either to an SPM, or a customer, either being in pocket or out of pocket, I disagree that an accounting system should work in that way. This is not only an optimistic description, it is in my judgment entirely wrong.
2. The conclusion was that the scenario might not occur regularly and create “large losses”. The size of the potential losses was plainly relevant to Fujitsu. Given how the Post Office used the Horizon system vis à vis its SPMs, sums that were not “large” to Fujitsu does not mean that smaller losses would not still be appreciable sums to individual SPMs, who would have to pay for them if there were branch shortfalls. There is also no way of discerning what Mr Jenkins meant by “regularly” in terms of the risk of re-occurrence. Both regularity, and whether potential losses were large, are subjective. The Lepton incident showed that the Post Office required the SPM initially to pay the shortfall, which he did.
3. The Credence data was inadequate to show what had actually happened. Indeed, not only was this data inadequate, it made it look as though the SPM had done something that he or she had not done, and which the system had done.
4. It was expressly accepted that more was required in terms of the required accuracy for “giving evidence and using the same data for prosecutions”. It was concluded that system created reversals had to be identifiable, which they were not as at the date of the Rose Report and these emails.
5. A change required a formal Change Request, and that was neither initiated nor acted upon.
6. Finally, and in my judgment also importantly, any risks that Fujitsu and/or the Post Office “ cannot clearly see what has happened on the data available to” them and “this in itself may be misinterpreted when giving evidence and using the same data for prosecutions” is a serious matter. I do not understand how a report containing such a reference to such a serious matter could be mis-summarised by Mrs Van Den Bogerd in her witness statement dealing with Mr Coyne’s analysis of this.
230. A large number of documents, including operational documents, were put to her, many of which it appeared she had not seen before. For example, she was shown documents in respect of MoneyGram dealing with a software release in January 2016 which showed, to use the IT speak of the document, “multiple instances of system latency” – or, to use more normal language, problems in the operation of the system. These persisted through 2016 until nearly the end of that year. The internal summary of the problem was:
"For the last several months Post Office has experienced a live operational issue with MoneyGram transactions across the branch network. In the event of a transaction timing out at the counter, a system error message is displayed to the user ... and the transaction is aborted. This leaves no record of the transaction at the counter and the transactions and funds may or may not have been committed in the MoneyGram domain. This causes significant issues for Post Office and MoneyGram and for customers."
(emphasis added)
231. Mrs Van Den Bogerd said that she had not seen the internal document setting all this out until she gave evidence in the witness box. One of Mr Patny’s issues in his evidence was about MoneyGram.
232. She had also been involved in a meeting on 17 May 2012 with James Arbuthnot MP, Oliver Letwin MP, and others from the Post Office, including the Chairman and the Chief Executive of the Post Office. The documents, including the briefing pack, demonstrate that the MPs were told that the Post Office wished to be “open and transparent”. This was not accurate. The MPs were told that the Horizon system had undergone an upgrade in 2010 and had the full support of the NFSP. In reality, its functionality for SPMs had not been upgraded, and the NFSP had privately expressed its concern to the Post Office about Horizon. There is nothing open and transparent about telling these MPs information to the contrary. James, now Lord, Arbuthnot had been the Minister for Defence Procurement under the Conservative government of Sir John Major. Oliver, now Sir Oliver, Letwin was Minister of State for Government Policy under the Coalition government, which had come to power on 11 May 2010. These gentlemen had become involved in this matter on behalf of their constituents. They were entitled to expect accurate information from the Post Office. They did not receive it.
233. A large number of different issues with both Horizon Legacy and Horizon Online were also put to Mrs Van Den Bogerd, only some of which she knew about. She has been closely involved with the defence of Horizon for some years. To be fair to Mrs Van Den Bogerd, she has also been responsible for some changes over time which have made Horizon easier to use, more reliable and (to use the wording of Horizon Issue 4), more robust. Horizon as at the date the experts were looking at it is a more robust system than it was, say, 5 or even 3 years ago. A great deal of this is likely to be due to the efforts of Mrs Van Den Bogerd.
234. Equally, however, the contents of some of the internal Post Office documents that were put to her were, in my judgment, very damaging to the Post Office’s case on the Horizon Issues. In an “Extracts from Lessons Learned Log” document of 11 November 2015, which was heavily redacted (including the redaction of single words), one entry under “issues identified” was as follows regarding the Post Office:
"Failure to be open and honest when issues arise eg roll out of Horizon, HNGx migration issues/issues affecting few branches not seemingly publicised."
(emphasis added)
This frank expression in a document authored by Mrs Van Den Bogerd herself as recently as 2015 is consistent with the general tenor of the claimants’ case and their criticism of the Post Office. This entry in this contemporaneous internal Post Office document is, in my judgment, inconsistent with the picture which the Post Office continually seeks to portray, namely that it wishes to be open and transparent about issues with Horizon, and is as interested in getting to the bottom of any problems with Horizon as anyone else. It is an internal recognition of a “failure to be open and honest when issues arise.”
235. There were two periods that she was asked about which clearly showed the problems experienced by Horizon almost from the outset. These were the introduction of Legacy Horizon, in late 1999 and 2000 (the national roll-out); and the migration to Horizon Online in 2010 (including the pilot scheme in 2009). Mrs Van Den Bogerd gave some evidence in her statements about both of these, in terms complimentary of Horizon, although she was far more involved in the latter than the former. At the time of the introduction of Legacy Horizon she was only responsible for a smaller number of branches in her then role.
236. In PEAK PC0033128 an entry in November 1999 shows that the branch at Dungannon experienced a discrepancy of £43,000. The PEAK states:
“PM - Dugannon PO £43k discrepancy
Outlet has a discrepancy of £43,000 after balancing SUs and doing office snapshot.
Phil Turnock POCL BSM has advised outlet on this week’s balance. Steve Warwick development is investigating why this mis-balance occurred.
Immediate impact of this week’s balance has been addressed but POCL are concerned that the cause is still unknown and this will affect this and other outlets.”
237. After some investigation the following entry appears:
"I have talked with development ref this problem. It is seen as a one-off. No fault can be found and developments do not expect to be able to find a fault with the evidence available. There is no additional information available as evidence. I suggest this call be placed on monitor for 1 month."
The approach to this discrepancy mirrors so much of the case. A discrepancy occurs; Fujitsu cannot find a fault; and say they do not expect to be able be find one “with the evidence available.”
By February 2000 Fujitsu stated “counter 5 looks suspect” and continued:
“Further examination of the event logs for these two counters indicate that counter 5 looks suspect (C drive nearly full and big gap of no messages).
Calls from PO into HSH for period between 30-Oct and 10-Nov indicate a reboot (counter not specified, but would tie in with counter 5 event log) on Saturday 30-Oct-1999.
The evidence in the message store was that messages continued to be written to the message store but that all the 'Payment' transactions which should have been recorded in the rollover trailer messages failed to appear (although others did, such as the Rem OUT and Transfer OUT totals).
This indicates that the problem was not one of running out of Disk space but of failing either to retrieve, or write out, transaction totals for one particular node in the node hierarchy.
Given that there were known problems with corrupted Persistent Object indexes at about this time, it is possible that an update to an EPOSSNodes object failed to be registered correctly at the outlet, causing the node accumulation to fail.
It was decided to prove this out by deleting the 'Payments' node in the node Hierarchy and then running the SU balance, to attempt to identify the root cause of the problem. Call passed to testing to be scheduled.
Update 18th Feb 2000
The test was carried out on 16th February as follows: delete the Payments EPOSSNodes object before producing a SU balance, on a version of the current live system (CI2_2R).
When trying to print the Payments part of the SU balance, the missing node is detected by the system and an error tablet with message "A system error has occurred whilst printing. Please ring the helpdesk. Error at 67640." is generated. So the balance could not be finished.
This type of error trapping error trapping was introduced at the end of last year when resolving AI298 issues and we are investigating if the outlet did not have such error handling when the problem occurred.
Certainly, with the current system, a missing Payments node now would not go undetected.
The problem is currently back with development for further investigation.”
238. On 17 March 2000 there is another entry that says there has been another incident “of a very similar nature” at another office. This can therefore be seen to be something that was not, as recorded initially, a “one-off”, even if that were acceptable given the discrepancy of £43,000. Mrs Van Den Bogerd agreed that this showed that the underlying information was not checked until February 2000, even though the discrepancy occurred in November 1999. She was asked about a later entry in April 2000 that referred to another occurrence, this time in Appleby for £9,000 approximately. The cause for this was recorded in July 2000 as “data trees failing to build” and that entry states “there have been a number of calls relating to this kind of issue. A fix has been put in at C14 which will prevent this happening”. Mrs Van Den Bogerd accepted that this was an example of an error occurring, and the system itself failing to spot that the error had occurred.
239. She was shown PEAK PC0027887 which was dated July 1999 and related to “receipts and payments misbalance”. This showed an escalating misbalance that increased, in my judgment dramatically, in steps up to over one million pounds:
“On week 9 receipts and payments misbalance of £1337.05, week 10 misbalance of £24000, week 11 misbalance of £12000, week 12 misbalance of £1051111.48 and week 13 misbalance of £17426.05, she has a difference on week 11 of balance due to post office and balance brought fwd on week 12 of £1082544.32
Overall these weeks net out a difference of £27343.84. she needs business support (reconciliation) to look into this.”
240. Mrs Van Den Bogerd agreed that £1.05 million was a large amount for any branch and could not be a “real amount of cash” that “the SPM has put in her pocket”. A later entry in the same PEAK of 27 July 1999 was:
“Balance brought forward was multiplied twice due the known software error. The initial balance brought forward for this CAP was £1196622.72. This was multiplied twice to give a total BBF of £2279189.04.
The discrepancy was therefore £1082540.28. This was due a known software error which has no been resolved.”
241. She did not know if “no resolved” meant “now resolved” or “not resolved”. That PEAK was not closed until 31 August 2000, and was categorised as “Administrative Response”. One of the latest entries in this PEAK, which was not put to Mrs Van Den Bogerd but in my judgment is relevant to Fujitsu’s approach generally, in particular to categorisation of PEAKs (as this had already stated there was a “known software error”), was dated 31 August 2000 (one year after the original issue occurred) and it is most convenient to reproduce here in this judgment. This stated:
“I have now taken over analysing this problem from Steve Warwick who requested additional information on 13/10/1999 17:49:21. I have looked at the information and I am not convinced it can possibly be complete.
For example he said he wanted information from 20th May until 26th May. I opened the spreadsheet mc20may.xls and observed it only had 6 counter 32 transactions in - is this correct?
In any case my team is not used to dealing with data in this form (spreadsheets). We have developed techniques to look at problems using a full message store plus audit and event logs from the failing counter. Even if all the relevant spreadsheets could be obtained I do not think it would be worthwhile me getting to grips with this new method of analysing problems.
I see this is a very old problem (21/07/1999) and there have been many software updates since then.
May I suggest we discontinue investigation of this particular problem but that if a similar problem occurs again you send full message store plus audit and event logs from the failing counter.”
And
“Closing call on basis of insufficient evidence. As this is such an old call I have not contacted the call originator. I suggest that this call remains closed!
INSUFFICIENT EVIDENCE - ADVICE GIVEN”
(block capitals present in original)
To close such a PEAK as “Administrative Response” is, in my judgment, remarkable.
242. The number of different incidents recorded in the monthly incident review increased in November and December 2000, which is shortly after (for example) the date when Horizon was installed in Mr Bates’ branch office at Craig-y-Don, as set out at [105] of Judgment (No.3). By then the Horizon System had been rolled out nationally. As the ICL Management Summary that was put to Mrs Van Den Bogerd states:
“The most frequently occurring incidents in November were both types of Receipts and Payments Incidents (Migration and Post Migration), with 31 incidents per category. The Migration incidents have remained at the same level, whereby the Post Migration occurrences have increased. This was followed by 17 Transactions Polled by TIP but not by HAPS, these were due to delayed transactions as reported on APSS 2133c. These transactions are added back into normal processing.”
243. Mrs Van Den Bogerd was not aware of these problems going on at the time; she was a Regional Network Manager and did not have a national role then. No background information was provided to her then about problems that branches other than her own were experiencing, which were being passed back to ICL at the time. She therefore had no knowledge of the type of problem experienced by Mr Bates in June 2001, which was recorded in a Post Office audit document of his branch that was put to her which stated “A correct assessment of cash holdings could not be made because the Horizon system intermittently adds the previous days cash holdings to the daily declaration.”
244. By 2010 and the introduction of Horizon Online, she did have a national role and broader responsibility. She agreed that Horizon Online did not upgrade functionality, and documents were put to her that showed, as Mr Green put it “a litany of issues” with Horizon. The pilot scheme went sufficiently badly that Fujitsu put it on “red alert” and the high volume pilot was suspended in April 2010. Mrs Van Den Bogerd had not even mentioned this in her witness statements, and said she did not know this had occurred. The NFSP are recorded as “raised concerns but remain supportive”. All migrations were suspended as from 25 March 2010. There were database errors recorded in a Post Office document at F/614 called “Horizon Online Programme Update” which required an Oracle patch set to be supplied by Oracle, and on one occasion for a limited period 50% of MoneyGram transactions failed.
245. A Network Functional Report of May 2010 was put to her. Over 5 pages of the 6 pages are redacted. Mrs Van Den Bogerd could not recall if she had seen this document, and said that given the extent of redactions, it would be hard for her to recognise it, an observation with which I agree. She said that she was not aware of there being issues with Horizon over Easter 2010, which the document recorded.
246. Redactions had been incorrectly applied by the Post Office and/or its solicitors to some of the relevant documents. This can be seen from the following example. I asked leading counsel for the Post Office to review certain redactions and having done so, an unredacted version was disclosed of a document dated 25 June 2014 entitled “Branch Support Programme”. This had been co-authored by Mrs Van Den Bogerd. The purpose of the paper was to “Update the Post Office Executive Committee on the progress of the Branch Support Programme.” Each one of the 6 key performance indicators against which the programme would be tracked had initially been redacted by the Post Office, and following the review by counsel as requested by the court they were unredacted. They were as follows:
“* Reduction of operating costs by £3m per annum
* Reduction in net agent debt by £1m
* Reduction in subpostmaster suspensions as a result of audit shortages to a level of 60 per year
* Reduction of calls into NBSC by 25%
* Reduction of audit losses of £10k of over by 50%
* Satisfaction with on-line training models of 95%”
247. Apart from the first bullet point, three of the other five are consistent with an aim by the Post Office to make improvements to reduce the amount of debt incurred by, and suspensions, of SPMs, and reduce the audit losses. The other two are related to training. This is not consistent with a view that the debt/suspensions/audit losses are incurred by carelessness on the part of SPMs or criminal activity. It is also hard to see how it could be justified that these had been redacted originally.
248. So far as her evidence about Mr Latif was concerned, after Mrs Van Den Bogerd had been taken through the features of his experience with the failed stock transfer and the similarities with accepted bugs in Horizon, namely the printing out of two receipts very close to one another (in his case, 4 seconds apart), and asked if this worried her, she said she “would want to go back and have a look at” it. In my judgment, part of the way through the Horizon Issues trial in 2019 is rather too late for that. I conclude that there was no real answer available to Mrs Van Den Bogerd to the points put to her concerning Mr Latif’s experience. Had there been, I have no doubt she would have provided that answer in evidence.
249. Mrs Van Den Bogerd was in the witness box for in excess of one day, the longest period of any of the witnesses of fact for either the claimants or the Post Office. Her cross examination led to a far greater understanding of the Horizon Issues on the part of the court, although her written evidence was, as originally drafted, extraordinarily one-sided. She minimised any reference to problems or issues with Horizon, and reverted to potential user error whenever possible as a potential explanation, an approach which she explained in her written statement as providing “plausible” explanations. Her witness statement also stated, in terms, the exact opposite of what the reality of the situation was, and I have given examples at [221], [223] and [226] above. Witness statements are supposed to be factually accurate, and care must be taken in future rounds of this group litigation that they are drafted in accordance with the rules. Making statements that are the exact opposite of the facts is never helpful, to put it at its mildest. It is also the opposite of what witness statements are supposed to be.
250. However, during her cross-examination Mrs Van Den Bogerd readily conceded a number of points which were put to her. Her accuracy in her oral evidence was also more satisfactory than it had been during the Common Issues trial. She accepted her written evidence suggesting a causative user error by Mr Tank was not correct, for example. I do not consider that her written evidence had provided plausible explanations. It provided explanations that the Post Office wished to advance, as these explanations would, if accepted, provide a defence to the claimants’ factual case. These explanations were not based on the facts.
251. There were no evident attempts on this occasion to mislead me in her oral evidence, or avoid uncomfortable points through a claim of ignorance (with the possible exception of the PEAK referred to at [210] above) or of not having seen a document before. When she told me that she had not seen a particular document before, I accept that. However, she also told me that she had been assisted in preparing her evidence by a team of ten people, and the Post Office had devoted “high level resources” to her evidence, as one would expect. It is therefore very surprising that she had not seen many of the important documents that Mr Green put to her in cross-examination. Some, such as the master PEAK on phantom sales, one of the recurring issues on Horizon, would have been the obvious place for anyone, still less a team of ten, to start when considering preparation of evidence for a witness statement. She explained that there was some pressure of time in terms of how long was available to her to prepare the statement, and this was further explained in supplementary re-examination, but I do not accept insufficient time as a valid explanation for her lack of knowledge on such important points. For example, she told me that at the time of preparing her witness statement, she had not even heard of the Callendar Square bug, one of two bugs that the Post Office accepted some time ago had been present in the Horizon system. This is an extraordinary gap in her knowledge. She did not know that there was a KEL dealing with failed recoveries, originally raised by Anne Chambers in Fujitsu as long ago as 2010, which is called KEL acha959T and which was updated most recently in 2017. This described failed recoveries, and seemed on its face to accept that these would recur, and was very close to the experience of both Mr Tank and also Mrs Burke. I do not see how Mrs Van Den Bogerd (assisted by her team of ten, and with the benefit of the Post Office’s considerable resources) could seek to give accurate evidence in the Horizon Issues trial without referring to this KEL, still less without even knowing about it. I am also somewhat disappointed – putting it at its very best for the Post Office – that a team of ten could have assisted Mrs Van Den Bogerd in preparing a witness statement that was so inaccurate on such important points as I have identified above.
252. Although my findings on her evidence in the Common Issues trial cannot be ignored, I am of the view that her approach in the Horizon Issues trial to answering questions was far more constructive and aligned to what is expected of any witness giving evidence in court, particularly a senior witness of an organisation such as the Post Office. I do however consider that this litigation, and indeed her cross-examination, is a very expensive way for a senior director at the Post Office to become educated about the myriad issues contained in the documents that were put to her. Either the team of ten people assisting her with her evidence had the aim of producing entirely one-sided evidence in chief, or they were unaware of all the documents relied upon by the claimants. Either alternative is highly regrettable.
253. Finally, and this is a point in Mrs Van Den Bogerd’s favour, there is no doubt that the Post Office is not as sufficiently close to the detail of what has occurred over the years on Legacy Horizon and Horizon Online, as Fujitsu. As will be seen from my analysis of the Fujitsu evidence of fact, I have certain views about the lack of accuracy on the part of Fujitsu witnesses in their evidence. If that lack of accuracy has also been included in reporting to the Post Office by Fujitsu, then that goes some way to explaining the Post Office’s lack of grasp of so much material that is consistent with the claimants’ case. As at late 2019, the date of this judgment, the Post Office also has the added benefit of the views of both the IT experts in the litigation, their four joint statements, the agreed number of bugs in the bug table (12, if one takes the number accepted by Dr Worden), the total number of bugs Mr Coyne says he has discovered (which so far as Horizon Issue 1 is concerned became 21), and collectively my findings on the Horizon Issues. These are not restricted solely to the number of bugs which I have found to exist which relate to Horizon Issue 1, but also those that relate to Horizon Issue 4. There are multiple bugs, errors and defects in both Legacy Horizon and Horizon Online in its HNG-X form.
254. The degree to which the evidence of Mrs Van Den Bogerd sits with my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Ms Phillips
255. Ms Phillips is a Team Leader at the Post Office for Agent Accounting and Santander Banking. She was originally engaged by the Post Office in 1999 through an employment agency, and has worked in a number of roles since then. In her current role, she oversees the processes of recovering losses from SPMs that they have declared in their branches. The department used to have “debt” rather than “accounting” in the name, but its function remained the same.
256. Ms Philips confirmed what had been in dispute for so long during the Common Issues trial, namely that SPMs had no option but to accept the figures provided to them, even though they may have been “settled centrally”. This is notwithstanding that her terminology in her witness statement was that SPMs “chose to accept”. This terminology was used throughout the witness statements for the Post Office, and it has already been seen that Mrs Van Den Bogerd accepted TAs had to be accepted, and there was no choice in this respect, as Mr Latif had explained. SPMs clearly did not have a choice, and the wording “chose to accept” is not an accurate description. It obviously states or suggests that SPMs had a choice, when no such choice was available to them. The only choice they had was whether to pay the amount immediately, or settle centrally which means obtaining time to pay. It had been necessary for me to make findings on what “settling centrally” meant in the Common Issues trial, and the Lead Claimants in that trial had been directly challenged on this very point in their cross-examination.
257. There is no reference in the letters sent to the SPMs in question relating to any ability to dispute the sums which the Post Office treated as debts. Ms Phillips’ evidence went to the way that the Post Office seek to collect sums from SPMs who have settled centrally, even though they may have earlier raised a dispute (or sought to raise a dispute) with the helpline. Sometimes amounts above £5,000 would be checked by reference to the helpline call logs, but the amount was not a set threshold and it depended upon how many sums had been settled centrally as to whether this happened. This checking did not therefore always occur, even above that figure.
258. Her evidence demonstrated that there was no dispute function within Horizon. Essentially, her task was debt collecting, as the Post Office saw amounts “settled centrally” as debts. Her concern was to obtain payment, or agreements to pay, and if not, then to escalate the matter to a SPM’s contract adviser. Usually, she said, the first or even second letter would lead to payment by an SPM. Neither of those standard letters referred to disputes. Mr Godeseth confirmed that the lack of any dispute button or function on Horizon was, as he put it, “by design”. This shows that the Post Office had been advancing a case, at least for a substantial part of the Common Issues trial, which was directly contrary to the evidence of its own witnesses of fact in the later Horizon Issues trial. I find this difficult to understand or explain. However, Ms Phillips’ role was fairly clear and she explained the process of branch dispute forms which would be used when a SPM raised a dispute directly with her team. This elementary system seems sensible, but it was only introduced in early 2018, and it would lead to a seven day period after which (if the form was not returned) the “block” placed on the amount would be lifted. She estimates that it would be used at least 5 times a month.
259. Ms Phillips gave useful evidence on the system used by her team, and I found her credible. The evidence did however go more to the Post Office’s accounting functions, and how it pursued sums that it considered to be due from SPMs. The reforms introduced in 2018 in terms of the branch dispute form are part of that process, but are not part of the Horizon system.
260. She had also provided some evidence to Mr Smith (who was also called as a Post Office witness) in terms of the number of Santander TCs that were issued, which had not been described correctly in his witness statement and which had been misunderstood by both the experts. This was clarified, and also she explained that of the number of TCs issued in one year in her particular area, 2016-2017, which was 3,968, the Post Office could not say how many were disputed because that data was not kept. This would appear to be a fairly elementary piece of information for the Post Office to monitor or record. The failure to do so is surprising.
261. Ms Phillips’ evidence in her witness statement did not directly impact upon the Horizon Issues to any appreciable degree. Her evidence went to wider issues in the group litigation. Her figures for Santander TCs, which had been given in evidence (for some reason) by Mr Smith and not by her, were considered by the experts.
Mrs Mather
262. Mrs Tracy Mather is a Team Leader of the Finance Service Centre or FSC. She started working for the Post Office in 1987 and has been in the FSC throughout, becoming a team leader in 1999. She has managed teams in Cheques, Postal Order, Pay-out and MoneyGram. Her witness statement was served in response to Mr Coyne’s 1st Expert Report, and explained why the Post Office used Credence to investigate discrepancies. She explained the difference between Credence, which records the transactional data, and POLSAP, which is what is called a “back end” accounting system. She explained the latter is on a higher level. Credence is used by her team to investigate differences between what a branch says has happened to a transaction through Horizon, and as she put it, what a different source of information might say. Credence was adopted in 2009.
263. She confirmed that Credence does not record key strokes, it records “transactional data as in sales and non-sales”. Mrs Van Den Bogerd had stated that Credence records actual key strokes by stating in her written evidence that Credence “records all key stroke activity performed in that branch by the user ID, date and time.” This was not correct, and was corrected by Mrs Mather in chief. Her team does not have access to the ARQ data, which has to be obtained from Fujitsu. This is also called audit data. Credence is not used for MoneyGram. I accept her evidence on this point and find that Credence does not record all key stroke activity.
265. She gave written evidence about what was called the Rose Report, although she did not actually know about it herself, and she also was unaware of anyone being deterred from making ARQ requests due to the cost to the Post Office (charged by Fujitsu), which was a point Mr Coyne had raised. Given her team did not make ARQ requests, her evidence on this was unsurprising. She was aware that a certain amount of requests were allowed, and above that amount a charge was raised by Fujitsu; however she did not know what the charge was.
266. There was one passage of her witness statement which went plainly beyond any direct evidence she could usefully give. It was in paragraph 20 and gave vague evidence about her understanding of some confirmation given by Fujitsu to the Post Office’s solicitors about the contractual limit on ARQ requests, when or if that had been exceeded, a “new commercial term” between Fujitsu and the Post Office about what was agreed and the cost of these. Her conclusion was “any requests above the agreed limit are chargeable but I understand that the terms depend on the details of the requirement.” She was not cross examined about this, and given her answers in cross examination about her understanding of ARQ requests (which was minimal) and her experience of having made such requests (which was never), that is understandable. However, in terms of providing guidance to the parties on the contents of witness statements, to assist the efficient conduct of these proceedings going forwards, witnesses should not have paragraphs of this nature in their witness statements about matters in which they have had no involvement at all. Mrs Mather – and there is no criticism of her in this – has simply never been involved either in making ARQ requests, or in the contractual arrangements between Fujitsu and the Post Office, or the “new commercial term” between those two entities in relation to the extra data required for the litigation. She stated that she had “no idea” about the charges that would be raised.
267. I found Mrs Mather to be a credible and helpful witness. Her evidence was very useful regarding Credence, and otherwise of less than central relevance to the Horizon Issues. The degree to which the evidence of fact supports my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Smith
268. Mr Paul Smith is the Operations Support Manager at the FSC and has been since November 2016, having started at the Post Office in 1996. Originally he started in pensions and then moved to human resources and payroll, until in 1999 he moved to the Network Business Support Centre helpline, or NBSC. In 2000 he moved to work as a Problem Manager, dealing with problems that came through the helpline. He led the team that updated the knowledge base for the helpline, and was Lottery Manager in 2004. After performing different roles he became a Change Analyst in 2011, and then the Santander manager. He returned to Lottery within the Post Office in 2013 and worked there as a Team Leader. His existing role has a number of different activities including handling specific product based problems. His evidence dealt with volume of TCs and a suggestion made in Mr Coyne’s report that TCs for a branch in Potters Bar were initially made in error.
“TCs are issued by FSC. I understand from Post Office’s solicitors that the processes by which FSC determines whether a TC is required is outside the scope of the Horizon Issues trial.”
270. The reason that this is important is as follows. Dr Worden relies upon TCs as one of his “countermeasures” that goes to the robustness of the Horizon system. Mr Coyne does not, and the claimants submit that the whole process of TCs is entirely outside the scope of the Horizon system. This evidence from one of the Post Office’s own witnesses makes it clear that the claimants’ point of view is shared (or at least was when Mr Smith’s statement was prepared) by the Post Office’s solicitors, and the decisions regarding whether TCs are issued are outside the scope of the Horizon Issues. This evidence by Mr Smith supports the claimants’ position on this, and also supports the approach of the expert evidence of Mr Coyne, and is contrary to the position adopted by Dr Worden.
271. A case management system was introduced by the Post Office in September 2018 to record each individual challenge to a TC. This is called the Dynamics System, and is still in what is called “roll out”. Individual challenges to TCs were not recorded prior to that. Further information about how this system was working was obtained in cross-examination, and leads me to the conclusion that the way that TCs are recorded now is far improved from what it was before the Dynamics System was introduced. That does not however take one very far. Mr Smith knew of reports that were available from that system but he had not produced any. Those reports would be the best evidence of the type of points Mr Smith was seeking to make for the most recent periods at least, but given he had not used them, it is not possible to speculate on what they might show.
272. Mr Smith had obtained almost all the data contained in his statement from others, and some was plainly incorrect, and other data so vague as to be wholly unhelpful. For example, for Santander, originally he had provided a table with a column stating that 2,890 disputes were received from branches in the 2016/2017 financial year. In fact, that column should have been disputes received by Santander from the Post Office. This was corrected in his second witness statement, but Mr Smith was unnecessarily combative, in my judgment, about the need for the correction, maintaining that “the facts and figures were correct” in the first statement when they plainly were not. The figures were correct as figures but the descriptions of what the figures were, were plainly wrong.
273. Even after the correction, the various figures extracted by him from information provided by others was not particularly useful. For example, the figures for the numbers of TCs disputed, and for compensating TCs, for BOI Retracts and the Lottery, were both estimated at 1,500 and 500, for both line entries. This estimate was plainly of the most vague type, and even Mr Smith could not comment on the reliability of either of those estimated figures for either line entry, although he maintained the figure for issued TCs would be reliable. This evidence from him, was of the most general type, and was not of great, or indeed any, assistance. For example, in respect of MoneyGram his evidence was that one or two disputes a month were received but they were rarely if ever accepted. Given he had no idea at all of the overall total, this evidence does not assist me in resolving the Horizon Issues one way or the other.
274. His explanation for the TCs at Potters Bar was that a cheque that was deposited for £90,000 was mis-keyed, and entered for £900,000. Accordingly TCs were required to correct this at the branch, and the need for two was because the payment was made by cheque.
275. I found Mr Smith’s evidence, with three exceptions, to be of no assistance whatsoever. The two exceptions are his evidence I have summarised in [269] which is relevant to the scope of the Horizon Issues and Dr Worden’s countermeasures, and his explanation about Potters Bar and the two TCs. I accept his explanation about the Potters Bar TCs. Both of those exceptions are relevant to the Horizon Issues. The third exception is his evidence about the Dynamics System. This goes to the robustness of Horizon as of 2019 and is therefore relevant. However, the absence of such a system also goes to robustness prior to its introduction in 2018, and is also therefore relevant. However, his wider evidence is of limited relevance at best. His knowledge of the figures contained in his statement, and the vagueness of his evidence generally, were such that the utility of the exercise which he presented in his statement is de minimis. In any event, the substance of that evidence went to an attempt to demonstrate that some TCs are challenged by branches, and fewer challenges are upheld than are made. The figures vary inevitably from year to year, and this case is not about general statistical trends in any event. In some areas of business TCs have fallen to a very low level, for example the DVLA, which Mr Smith explained was only 4 in the last year, none of which have been upheld. However, the records that were put to him show that there were 2,717 in the period 2010/2011. All this evidence does is show that far fewer TCs are now issued by the Post Office in relation to the DVLA business, which as Mr Smith explained, has fallen considerably in any event. None of these points are of particular relevance to resolving the Horizon Issues. The experts are agreed that Horizon as it is in mid-2019 is more robust than it was in the past. One would expect, therefore, fewer TCs to be issued now.
Mr Johnson
277. He was cross-examined about screen layout and design, the risk of mis-keying (which did not seem to concern him, although there were internal Post Office documents showing it was a known problem) and also the reports and functions available to SPMs to investigate shortfalls and discrepancies, rather than those which the Post Office itself used such as Credence. He agreed that the ways available to SPMs to investigate in branch were “not the most user-friendly way of investigating”.
278. He agreed with a statement in a 2016 internal Post Office document entitled “Network Development Programme Operation Simplification” that stated:
"There are a number of branch operations processes, especially around branch accounting and reconciliation, which operate using legacy processes. They are unnecessarily complex and detract Post Office branch resources from serving customers. Stock Unit Management and accountability is very poorly controlled and is operated on very complex business rules. The lack of accountability and visibility of cash and stock transfers between Stock Units can lead to errors, rework and provides opportunities for fraud."
(emphasis added)
279. He also stated that the following entry was “not completely unreasonable” although he said that his experience of suspense accounts in branches was that they worked and did what they were supposed to do:
"Similarly, Suspense Accounting is based upon legacy cash accounting practices, ill-defined and out of date processes. Inefficiencies lead to poor utilisation of resources both in Post Office branches and Support Services."
280. Mr Johnson is a highly experienced person, as one would expect given that the Post Office chose him to train the legal teams and experts on Horizon. All his experience, so far as using Horizon is concerned, is in Crown Offices, which are not the same as branch Post Offices, in that the Post Office operates Crown Offices itself. This means that a SPM in a branch Post Office is personally responsible for any losses in that branch; whereas a Crown Office manager is an employee of the Post Office. This therefore gives him a somewhat different angle of approach to the use of Horizon, but I accept him as an accurate and helpful witness.
281. His evidence was very useful in terms of how Horizon works. The Horizon Issues are not about the actual design of the Horizon terminals in branch, and whether (say) this could be improved by having larger buttons or the screen layout adapted differently. In some of Mr Johnson’s evidence, the claimants’ counsel strayed into the areas of potential improvements to functionality in branch and design. These are not part of the Horizon Issues. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Dunks
282. Mr Andy Dunks is at IT Security Analyst at Fujitsu Services Ltd, and was the first Fujitsu witness to be called for the Post Office. He has been employed at Fujitsu on the Post Office account since 2002. He has working knowledge of the Horizon System. His evidence went to audit data extraction and the integrity of the data during that process. His evidence related to the process as it was when he made his statement, although given he had been involved for some 17 years his experience also related to Legacy Horizon and his evidence about data extraction covered that period too. As emerged in his cross-examination, he had also given a witness statement in the criminal trial of Mrs Seema Misra.
283. He agreed with the definition of data integrity put to him, which is that data integrity is the overall completeness, accuracy and consistency of that data which you can measure by comparing between sources. He said the 12 controls that he had listed in his statement explaining these he had compiled as a list from his recollection, and that he had not worked off a document to come up with the list in his statement.
284. Mr Dunks gave every indication, in the first part of his cross-examination, of being helpful and frank, and explained the process of data extraction and answered questions put to him openly. However, this encouraging start came to an end when he was asked about two curiously worded paragraphs in his statement. These stated as follows:
“8. There is no reason to believe that the information in this statement is inaccurate because of the improper use of the system. To the best of my knowledge and belief at all material times the system was operating properly, or if not, any respect in which it was not operating properly, or was out of operation was not such as to effect the information held within it.
9. Any records to which I refer in my statement form part of the records relating to the business of Fujitsu Services Limited. These were compiled during the ordinary course of business from information supplied by persons who have, or may reasonably be supposed to have, personal knowledge of the matter dealt with in the information supplied, but are unlikely to have any recollection of the information or cannot be traced. As part of my duties, I have access to these records.”
285. Before I come to his evidence about this, it is obvious that the wording of paragraph 8 is almost that of a legal disclaimer (or a legally worded claim of accuracy, to be more precise), rather than a witness’ actual evidence. It would be very curious for a witness of fact to decide to put such a formally (and rather clumsily) worded paragraph in their witness statement. However, he was asked about these passages. He was asked what he meant by “the system”. The transcript records the following:
“Q: What is the "system" there? Is that the system of the process of extracting audit data, or is it something else?
(Pause).
A. Good question. There's no -- I'm not sure what I was meaning by that, "There is no reason to believe ..."”
286. He was asked if, in relation to the part which states “the system was working properly or, if not, any respect in which it was not operating properly, or was out of operation” the following:
“Q: Again, pausing there, it is slightly confusing. Are you aware or not aware of any instances where that system was not operating properly?
A. No, not really, no.”
287. He was asked if that part of the statement had been written for him by someone else, and he said he could not remember. He was also asked if it was a “Fujitsu party line” to put this in statements when it comes to extracting data and he said no, and that he was not aware of a Fujitsu party line. He accepted that gaps and duplicates in audit data were problematic, and he explained that they had to be investigated by audit support. It was clear to me that Mr Dunks considered audit data that had gaps and duplicates should not, in that form, be used. He said he would contact audit support if these occurred.
288. However, some of Mr Dunks’ evidence, in particular what he had to say about paragraph 8 of his own statement, proved in my judgment to be somewhat misleading.
289. He was shown a witness statement from Mr Gareth Jenkins from the Misra criminal trial in 2010. He said he had not seen this statement before. The statement put to him is some 8 years older, approximately, than his own statement. Mr Jenkins’ statement stated at the end:
“There is no reason to believe that the information in this statement is inaccurate because of the improper use of the computer. To the best of my knowledge and belief at all material times the computer was operating properly, or if not, any respect in which it was not operating properly, or was out of operation was not such as to effect the information held on it. I hold a responsible position in relation to the working of the computer.”
290. The similarities between this passage, and the one in paragraph 8 of Mr Dunks’ witness statement almost 9 years later, are somewhat striking. Given he had not seen the statement before, there was no obvious explanation. His answer was as follows:
“Q. And you're not sure why it would largely replicate your paragraph 8?
A. No, I mean, we do have a standard witness statement that we produce for ARQs. When we supply ARQs we are sometimes asked for a witness statement to go through the process and verify as far as I'm aware that the data I supplied is accurate. Now, we use that quite a lot and it may actually be in that statement.
Q. I see. So when I asked you earlier whether it was something of a Fujitsu party line you said you didn't think it was, but actually it looks as though it is on the basis of what you have just told me and on this document as well.
A. It could be. It may be part of our standard witness statement that we supply.”
291. Mr Jenkins had given evidence in his witness statement from 2010 about duplicate entries in the audit data that had had to be removed. Mr Dunks was then taken to the witness statement of another Fujitsu witness in the Misra trial, called Penelope Thomas. She had given evidence about data extraction and listed a number of controls, which were remarkably similar to the ones that Mr Dunks gave evidence about (although Ms Thomas had omitted one). This is hardly likely to be a co-incidence, if Mr Dunks really had compiled his list of controls from memory or experience. She also had the identical passage in her witness statement as Mr Dunks’ paragraph 8.
292. Mr Dunks also said in respect of this, which in my judgment painted a rather different picture to that portrayed in his witness statement, the following:
“A. The statement again is to verify the integrity of the data once extracted and given. We don't control what's in the data. Our process is about extracting it and securely passing it over to the Post Office. It's not our concern of what's in the data.
Q. It's not your concern what's in the data?
A. No, it's what we're -- we're process-driven to extract certain types of data for certain requests.
Q. And so whether or not that might match another record or not, or replicate or duplicate or have gaps, that's not part of your remit, that's not really part of your concern?
A. No, it's not.
Q. I see. So reading that and reading the paragraph 8 as it was in your witness statement, it doesn't give much comfort to somebody that's then trying to rely on ARQ data as being a gold standard to compare and investigate anomalies, does it?
A. Possibly not.”
293. The reference to “gold standard” was the expression that had been used by Dr Worden in his report to describe the quality of the audit data that was held by Fujitsu. It should be a very great concern to anyone tasked with extracting audit data, should that data prove to have gaps and duplicates in it. This is because gaps and duplicates in the data affects the accuracy of that data. I consider that such a cavalier approach to whether audit data has gaps and duplicates, as evidenced by Mr Dunks’ saying that such things were not part of his concern, to be entirely contradictory to a statement verifying the accuracy of such audit data.
294. My findings in relation to this are as follows:
1. Mr Dunks expressly sought to mislead me by stating that there was no “Fujitsu party line” when it came to the contents of drafting witness statements about audit records for legal proceedings. There plainly is; it was used in the Fujitsu statements in 2010 and it was used by him in his statement for the Horizon Issues trial.
2. The passage included in paragraph 8 of his statement is plainly a standard form of words, and reads as though it is a legally drafted assertion of accuracy. The witness’ assertion that the system was at all material times working properly, or if it wasn’t, this did not affect accuracy, does not sit consistently with Mr Dunk’s acceptance of gaps and duplicates in audit data.
3. That Mr Dunks did not really know very much about paragraph 8 of his own witness statement was confirmed by the way he was puzzled when he was asked about it.
4. However, Mr Dunks’ acceptance of standard wording provided by others (probably within Fujitsu) is less important than the fact that he originally tried to mislead the court.
5. I do not accept that he identified the 12 controls in his statement from his recollection. He must have been working off another document, which he was not prepared to identify. I draw this conclusion because of the marked similarity with the controls included in Mrs Thomas’ statement from 2010; the wording of the controls themselves; and the fact he was prepared to mislead me about the paragraph 8 point.
6. In re-examination he asserted that the statements in his paragraph 8 were true, and that he would not have signed his statement had this not been the case. I find that he cannot have known whether those statements were true or not. There is no evidence of any steps he took to check whether or not they were true or not. I find they were simply standard sentences supplied to him, and I find that he signed his statement without any independent knowledge of whether they were true or not.
7. I find that passage in his witness statement simply to be meaningless assertion.
295. He said that he was unaware of Post Office internal concerns that there was “a lack of system audit trail” in a document of 22 October 2016. I find his lack of knowledge about this surprising, given his role.
296. I found Mr Dunks very unsatisfactory as a witness. He was both plainly aware of the Fujitsu “party line”, or corporate position, regarding the words asserting accuracy of audit data, and he was very anxious to keep to it, whilst initially denying that there was one. He sought to mislead me about both his paragraph 8 wording, and the way he had compiled his list of controls. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Godeseth
297. Mr Torstein Godeseth is employed by Fujitsu and is the Chief Architect on the Post Office account. He had provided three witness statements, as had Mr Parker. After graduating from Oxford with a degree in physics in 1974, he worked for Rolls Royce as a combustion engineer and then joined the Royal Navy in 1977 as an Instructor Officer. Whilst in the Royal Navy, he started his career in IT working in systems programming. It will be remembered that in the 1970s computing and IT was a relatively young field, and Mr Godeseth has therefore been involved in that industry since its relative infancy. He left the Royal Navy in 1981 and worked in systems programming and technical support for the IT at Forward Trust Ltd. He joined the Post Office IT department in November 1987. Initially this was to work on a project to introduce technology into Post Office branches.
298. He was working with the Post Office as a technical advisor when the Post Office and Benefits Agency procured the Horizon system originally. He remained involved in different roles, including being outsourced from the Royal Mail IT department to Xansa in 2003, and contracted to the Post Office to act as the technical adviser interfacing with Fujitsu amongst others. Although an independent contractor, he worked between 2005 and 2010 with the Post Office as the technical advisor on IT projects including the change from Legacy Horizon to Horizon Online. He joined Fujitsu as a full employee in 2010. He described himself in cross-examination as the designer of Horizon.
299. He had a greater knowledge of Horizon Online than of Legacy Horizon, although given his involvement in Horizon from its early days he had significant knowledge about both. Horizon was initially, in the late 1990s, called the Pathway project and was a joint effort between the Post Office and the Benefits Agency, including a way to computerise the payment of benefits to those entitled to them, through payments to benefits claimants which were to be made through the Post Office by means of a swipe card method. This project ran from approximately 1996 to 1999 and was to be provided by ICL (which Fujitsu partly owned and later fully acquired). At some point during the development of this project the Benefits Agency withdrew, and what had been the Pathway project, essentially a tri-partite venture (the parties being ICL, the Post Office and the Benefits Agency) became Horizon, now called Legacy Horizon.
300. Although this litigation is not about the way that the Horizon system was designed, it is about its functionality and robustness in use, Mr Godeseth said that it was a specific Post Office decision not to have any dispute button/function for SPMs built into the Horizon system.
301. He said:
“I think the basic argument was that disputes -- we wanted the flow of data through the system as quickly as possible because that keeps our books tidy and it was an inference that there was always the -- you had to press a button to take things through, but then you would pick up the phone to NBSC and say that wasn't right.”
(emphasis added)
302. I accept this factual evidence. It is, however, directly contrary to a major part of the Post Office’s case in the Common Issues trial, namely that by reason of an SPM accepting items at the end of a branch trading period, that gave the branch trading statement the effect in law of being what is called a “settled account”. I rejected that argument by the Post Office then, unaware when I did so of later evidence of fact that would emerge of the specific decision taken at the time by the Post Office that expressly did not include any feature within the Horizon system for a SPM to dispute items with which they disagreed. My finding in the Common Issues trial is consistent with Mr Godeseth’s evidence on this point.
303. I deal with Mr Godeseth’s evidence within this judgment in more detail than any of the other witnesses of fact from either the claimants or the Post Office. This is because his evidence was considerably more detailed, and of more direct assistance in resolving the Horizon Issues, than any other witness of fact. He has been involved in Horizon in both its iterations for 20 years and has a vast amount of knowledge of its operation. I found the majority of his evidence reliable. I provide my conclusions regarding him as a witness at the end of this section below at [453] to [463].
304. In Legacy Horizon, a messaging system was responsible for storing all data in Post Office branches and replicating it to data centres. This was called Riposte. Mr Godeseth decribed himself as being “on the other side of the fence” during this period and he had consulted Mr Jenkins for the section of his evidence dealing with when Riposte was in use. However, Mr Godeseth did describe himself as “having a pretty good knowledge of Riposte”.
305. When it was brought in, Horizon Online was aimed, not at improving functionality of Horizon, but at reducing cost, and re-used (the term put to Mr Godeseth was “recycled”) many application components of Legacy Horizon. The data was no longer to be held at the branches (on the counters), it was to be held centrally in the branch database, or BDRB. Accordingly, harvesters were required to extract transactions from the BDRB. The branch database receives information from different sources. Data in Horizon Online is obtained from different sources.
306. As an example with the lottery, if a person buys some lottery tickets in their branch Post Office, the information about a particular transaction starts at the lottery terminal in a branch; goes directly from there to Camelot; from Camelot it goes to Credence; and then from Credence it goes to the branch database, which is effectively that information arriving in Horizon. It is then transmitted to the terminals (in the branch) and the SPM would see a series of TAs on the terminal in the branch, which that SPM would have to accept; these TAs would relate to the lottery tickets sold at the beginning of this short history. The transaction of the purchase of the lottery tickets by this route enters the branch accounts when the TAs are accepted, and the data is captured by the audit system. This is how lottery transactions work after what was called the Ping fix. Prior to that it was somewhat less streamlined, although given the process I have described is the post-Ping fix streamlined version, it is obvious that pre-Ping fix, the route for the data was even more convoluted.
307. Other data for other transactions for other products goes into the branch accounts from what is called the basket. That basket is compiled by, as an example, a customer at the counter purchasing various Post Office and other client products; the SPM or assistant serving that customer and pressing an icon on the screen (for example for stamps, and other purchases); or for other products, from a PIN pad, scanning a barcode or weigh scales. The cost of a book of stamps is taken from the reference data table and the terminal shows the SPM or assistant how much money the total number of transactions are worth and what the cost to the customer is to be. It is basically the shopping basket which is put together by the SPM (or assistant) serving the customer and adding, by means of pressing buttons on the screen, different items to the basket as the customer goes through the items or products they wish to buy. At the end of this hypothetical transaction, if the cost of the basket is (say) £21.98, the customer will pay that amount (say by cash) which will be taken by the SPM and put into the cash amount held in the branch.
308. In Legacy Horizon, the data was held in the branch on the counters in a message store.
The message system called Riposte was how the replicating of the data to data centres occurred. The data was then stored on the correspondence server message store. This was explained by Mr Godeseth in the following way.
309. All counter data was held in a bespoke message store (which was part of the Riposte product supplied by another company called Escher Inc.). This data was replicated within each branch to all counter positions and from each branch to the data centres where it was held in the correspondence server message stores. Similarly, any data inserted into the message store at the data centre (for example reference data or authorisations for banking transactions) would be replicated back to the branch counters. Selected data was then extracted from the correspondence servers to update Post Office's back end systems.
310. Users with sufficient access permissions could inject additional messages (i.e. data) at the correspondence server. Any additional messages injected at the correspondence server by users with sufficient access permissions included information including the identity of the user. That information would not be visible in the standard audit extracts, but it would be visible in a detailed examination of the raw audit data. An SPM did not have access to audit extracts.
311. There was some disagreement between Mr Godeseth and internal Post Office documents about whether the audit data from Riposte was held in Riposte attributable language (his view) or whether an audit conversion tool was required to convert existing audit data from Riposte to another readable/searchable format. This ultimately may not matter, and it may be that the author of the internal document suggesting otherwise did not understand that Riposte attributable language was a readable/searchable format. This difference in view does not matter for the purpose of resolving the Horizon Issues. The Riposte software was provided by another entity called Escher. There was no evidence about the contractual relationship between Fujitsu and Escher, and that does not matter for the purpose of resolving the Horizon Issues.
312. Mr Godeseth had studied a number of problems in Horizon, and knew there were problems with Riposte, having seen numerous PEAKs referring to this. He had also been responsible for some testing (for example potential failures of controls, such as a basket not balancing to zero). He was not aware of items with the same Journal Sequence Number (or JSN) having been committed to the BRDB, and agreed that this should not be possible. This is because the whole way the BRDB works (or one of the principles of it, at least) is that each item committed there should be given a unique JSN. It should not be possible for two different items to be given the same JSN.
313. Mr Godeseth explained that “the legacy version of Horizon was far more susceptible to communication glitches” and agreed that there were two improvements made when Horizon Online was introduced in that respect. These were that the susceptibility to communication glitches was reduced; and also the quality of the communications infrastructure was improved. It should also be added that because the storage of data was moved to somewhere not in branch (somewhere which had not existed in Legacy Horizon, namely the BRDB), and because Horizon Online was an online system, the communications infrastructure had to change substantially in any event.
314. One internal document from as long ago as 2008 stated: “In addition, in common with many elderly systems that have been subjected to a succession of major changes, it [ie Legacy Horizon] has become increasingly difficult to make those changes, and expensive to operate." This was put to Mr Godeseth:
“Q: Now, that's a fair description of how the system originally was perhaps designed jointly with the DSS at the beginning and launched and then over the years, between then and 2008, there have been lots of sort of bolt-ons and additional things that have been changed on the system, haven't there?
A. Yes, I think that's fair. The major one probably in my mind would be banking.”
The reference to the DSS meant the Benefits Agency. In other places this was referred to as DWP (or Department of Work and Pensions) but all three references – DSS, Benefits Agency, DWP – meant the department involved in paying welfare benefits. Different documents and references would use different acronyms to refer to the same entity.
315. He explained that a small group of Fujitsu users from the SSC (about 30 users) had the ability to inject additional transactions into a branch's accounts in Horizon Online, using a designed piece of functionality called a Balancing Transaction. This was approximately the whole number of personnel who worked in SSC.
316. He had given information in his first witness statement, namely that in Legacy Horizon, any transactions injected by SSC would have used the computer server address as the counter position which would be a number greater than 32, so it would be clear that a transaction had been injected in this way by someone other than the SPM. This is important because it would be consistent with the case originally advanced by the Post Office that any such injections would be entirely visible as having been done externally (ie, not within the branch) due to the counter number used.
317. However, this important information was simply incorrect, and was corrected both by Mr Godeseth and Mr Parker in subsequent statements before they were called, and as a direct result of Mr Roll’s evidence. The information that was incorrect, and therefore had to be corrected, had come directly from Mr Jenkins. This shows that Mr Jenkins did, in at least one very important respect, give Mr Godeseth directly incorrect information about the visibility of injected transactions, which not only could have an effect on branch accounts, and whether this would show (or rather, not show) that the impact on those accounts had come from injections made outside the branch. Mr Godeseth only found out the true position when Mr Parker was preparing his subsequent witness statement in the weeks prior to the commencement of the Horizon Issues trial, in other words in 2019. He had not known that before. His explanation about this was as follows.
“Q. You were finding out a detail that you didn't know before in quite a controversial area, weren't you?
A. It was clearly an area that was going to be of interest because of the fact that we were inserting transactions into Riposte. It was an operational necessity and it was done in a controlled way. I had believed that the way that transactions were being injected would give them a counter position greater than 32 because the correspondence servers basically had nodes or addresses which were above 32, there was a special address for the gateway server, there was a special address for the extra disc in a single position branch and I had basically expected messages to be introduced using a different counter position and having read a whole number of PEAKs, I can quite clearly see that the standard practice in Fujitsu was to label something which was being inserted into Riposte so as to make it as clear as possible that it was not being done -- it was being done as something out of the ordinary, it was being inserted because of a problem.
So we had techniques for doing that. You could put in an attribute because this wouldn't be visible to a subpostmaster, I fully understand that, but it would be visible in the audit trail, when you ever come back to pull out the audit trail, you could put in an attribute to say "This was done under PEAK 75". Subpostmasters would never see that. They would not see it in their account in their branches and I'm fully aware that that is the case. It was a better audit than Mr Roll was alluding to when he said that it was left in a PINICL, because that would have been an audit which is separate from the actual data that we would be looking at should we ever need to pull stuff out of the audit trail and the intention was always to make it as clear as possible that this had been done under exceptional circumstances.
The techniques used to make it as visible to the subpostmaster as possible would be to put in references which referred to a counter that didn't exist in the branch, such as -- I saw a technique described in a number of cases which said put in a -- you know, if you are correcting something for counter 1, call it counter 11; if you're correcting something for counter 2, call it counter 12. These things would have been visible to a subpostmaster and the reason that you had to do it that way was to make sure that these transactions also got picked up and dealt with because these were legitimate counter numbers.
If I start to put in data with a number which is not a legitimate counter number then it's going to be ignored by systems further down the track.
Q. So which were legitimate counter numbers?
A. Up to 32.”
(emphasis added)
318. He also said in relation to Riposte that “Riposte was responsible for actually wrapping the message that we were looking to insert at the counter, and in doing that, Riposte will tell you the counter ID, or technically it was a stream, it would pick up the user ID, it would pick up the time, so this was effectively the envelope which wrapped the payload that we were looking to inject. If there was no user logged on at the counter then Riposte would introduce a blank user ID and that would be picked up in later processing.” This therefore meant that if, say, a SPM (or an assistant) was logged on at the time that the message was inserted at the counter, it would appear next to that SPM’s user ID (or that of the assistant). It would look as though the SPM had been responsible.
319. This was made crystal clear in different passages of cross-examination:
“Q: You realised for the first time that it was possible to inject or insert a transaction with a counter position less than 32 when Mr Parker was preparing his second statement?
A. Correct.
Q. And you knew that that was a contentious issue in this litigation, yes?
A. Yes.”
320. And further:
“Q. Now, it would be possible, would it not, to use a counter number of 1, or 2, or 3?
A. It would.
Q. And if that counter number was a counter number actually in use by the SPM, it would appear to the SPM, from the records they could see, that it was a transaction which had been done in their branch, by them or their assistants?
A. Yes.”
(emphasis added)
321. I consider this to be extremely important evidence, both in resolving the Horizon Issues, and indeed in the whole group litigation. Its import is obvious. It means that Fujitsu could remotely insert a transaction into the accounts of a branch using a counter number which was the same as a counter number actually in use by the SPM (or an assistant). This would appear to the SPM from the records that they could see (and anyone else looking at those records) as though the inserted transaction had been performed in the branch itself. This information was only disclosed by Fujitsu (and therefore the Post Office) in this group litigation in January and February 2019. Even Mr Godeseth, a very senior person in Fujitsu so far as Horizon is concerned, said that he did not know this before.
322. Mr Godeseth was also shown a Fujitsu document which was "Fujitsu restricted" and "Copyright Fujitsu Limited 2017". The title of the document was "Post Office Account – Customer service problem management procedure" and its purpose is described as "To describe and document the customer service problem management process." He was not sure he had seen the document before, and he was certainly not familiar with its contents. Both experts had seen the document and proceeded on the basis that what it recorded had in fact been adopted. “The customer” for the purpose of the document was the Post Office.
323. He sensibly accepted that to have a robust system it was important to make informed assessments of where problems lie, based on the relevant information that was available, and also that it was important to capture and track that information in a way that could be readily analysed. Indeed, Mr Godeseth’s role included awareness that changes being made to the system were implemented without prejudicing the continued operation of the system. The document itself expressly defined “problem” as the unknown underlying root cause of one or more incidents. That, to me, seems a sensible definition. An unknown root cause of even one incident is a problem in a system such as this one. This is accepted by the document’s definition of “problem”.
“The following metrics, to be reported monthly, will be used to measure effectiveness of the process and drive performance of the process and overall service in general:
Number and impacts of incidents occurring before root problem is identified and resolved
Number of repeat incidents following corrective action
Number of problem records arising from pro-active actions and trend analysis
Number of changes arising from proactive actions
Percentage of problem records without an action plan
Average length of time to resolve problems
Number of incidents closed without a KEL
The number of Problem Records arising from Managed Changed (MSC, CP) activities
The number of Problem Records arising from the implementation of new services/major releases.”
"I have spoken to my colleague Steve Bansal, Fujitsu's senior service delivery manager, who has informed me that the Post Office account customer service problem management procedure document was introduced by Saheed Salawu, Fujitsu's former Horizon lead service delivery manager and that Saheed Salawu left the Fujitsu Post Office account in around February 2013, before the new procedure had been implemented. I understand from Steve that Saheed Salawu's replacement did not wish to implement the changes and therefore the records referred to by Mr Coyne in paragraphs 5.157 to 5.159 of his report do not exist, as we continued to follow the previous existing reporting methodology.”
(emphasis added)
326. Mr Godeseth did not know who Saheed Salawu’s replacement was, although he thought it might be a Mr Wicks. He did not know what the “previous existing reporting methodology” to which he referred in his own witness statement in fact was. He could not explain why the report which was said to have been abandoned, or at least not pursued after February 2013 when Saheed Salawu had left Fujitsu, was issued for approval in July 2014, over one year later. Indeed, he could not really explain why the records sought by the claimants did not exist at all. He thought the reporting might be done at meetings with ATOS, but he could not remember going to one. In fact, his evidence on this was so very vague as to be somewhat alarming, given his role within Fujitsu. He did not know if there were any records of the particular type apart from between the years 2014 to 2017, and he only knew that because Mr Bansal had told him.
327. In another document, he was shown an entry which recorded that First Rate, a joint venture between the Post Office and the Bank of Ireland dealing with foreign exchange (also called bureau de change or bureau services), had identified an anomaly over the way that Horizon reversed transactions that were recorded and polled through to them. This was fixed in something called counter release 9, but although he said this happened “on his watch” this had not been brought to his attention.
328. In the 2015 Problem Management Problem Review, the Dallmellington bug (which is an acknowledged and agreed software bug) was referred to, but without identifying how many branches were affected, for how long they were affected, or the amounts by which their branch accounts were impacted. He accepted that it was fair to describe the report as “not a particularly rigorous or robust treatment of recording the problem, its extent and duration and effect.”
329. The following conclusions can be drawn from this evidence by Mr Godeseth. Fujitsu should, had it been interested in providing a robust system to the Post Office, been collating and reporting problems, which should obviously have included software bugs, errors and defects, against the metrics contained in the Customer service problem management procedure document, or very similar ones. On the Fujitsu evidence before the court, it did not do so. The discrepancies in the dates of distribution and issue of the document entitled "Post Office Account – Customer service problem management procedure" and its "Copyright Fujitsu Limited 2017" do not tally with the account in Mr Godeseth’s witness statement that this process was abandoned or not pursued when Mr Salawu left Fujitsu in 2013. The dates just do not match up. The document was issued for approval one year after that gentleman left Fujitsu. Even the evidence that it was not pursued or implemented after Mr Salawu left was second hand from Mr Bansal, the source of Mr Godeseth’s knowledge.
331. In the Post Office’s closing submissions, an explanation was given in submission in relation to the explanation given by Mr Godeseth in his witness statement which I have set out in [325] above about why the records did not exist, that explanation being that the procedure was not implemented. The Post Office submitted that it was only section 1.4 of the document that was not implemented, and not the whole document. This was in paragraph 147 of its written closing submissions, which dealt with what the Post Office submitted were “errors” made by Mr Godeseth which it submitted (and which the Post Office accepted) were “significant” and which it also stated had taken the Post Office “by surprise”. It sought to correct these errors, by way of submission rather than evidence, and one of those corrections was in relation to the implementation of the document in question. The claimants objected to this, and said the explanation that was provided was nowhere in the evidence, and the Post Office should not be permitted to give evidence by way of submissions.
332. Leading counsel for the Post Office was asked about this specific part of the submissions at the end of oral closing submissions and said that the explanation in paragraph 147.4 of the closing was “my instructions but, my Lord, it is based upon a previous version of the document and then an amended version of the document. And I will undertake to give your Lordship the two references.” In an email of 10 July 2019, which provided some references, further explanation was given which were said to support the submission that it was only section 1.4 of the document that was not implemented. The Post Office relied upon the following four points in support of that submission:
1. The passage in Mr Godeseth’s witness statement I have reproduced at [325] above.
2. That there were different versions of the document that were disclosed, but only one had been put in the trial bundle. The first version containing section 1.4 was dated 6 June 2010 and was authored by Mr Salawu.
3. This indicated therefore that the change was introduced by him. Most of the procedure document had been in existence for a number of years before that.
4. Earlier versions had been disclosed, none of which contained section 1.4, and none included section 1.4 and none were distributed to or even mentioned Mr Salawu.
333. The following points can be made about this.
1. Witness statements are expected to be factually correct.
2. Each party in this litigation who has submitted witness statements had ample opportunity to ensure those statements contain accurate evidence before they were served. Further, each witness is asked, at the beginning of their evidence when they confirm the contents of their statements in their evidence in chief. Apart from qualifications that were sometimes made, in this case (and in the great majority) the statements are usually stated to be true to the best of that witness’ knowledge and belief.
3. If a witness’ written evidence is shown to be incorrect (as Mr Godeseth’s, along with other of the Fujitsu witnesses, was) then the appropriate way to correct this is either in re-examination or by way of supplementary evidence. Here, Mr Godeseth was asked about this document and gave evidence in respect thereof on 20 March 2019. Mr Parker, also a senior witness from Fujitsu, was not called to give evidence until 11 April 2019. That was, again, ample time for this self-contained point to have been dealt with by supplementary evidence from Mr Parker, if that was thought by the Post Office to be important. In any event, there is no reason for it having been incorrect in the first place. Whether a procedure was, or was not, adopted by Fujitsu internally, is not something that ought to be capable of misinterpretation in any event, let alone on a subject as important as recording problems in the Horizon system.
4. I do not accept a matter such as this is suitable to be corrected by submission “on instruction” in any event.
5. However, even if it were, I am afraid that my view of information emanating from Fujitsu is, based on numerous other instances of such information being wrong in the actual evidence, to be somewhat sceptical of its reliability. It would also be potentially unfair to the claimants to allow this type of “correction” as they would be deprived of the opportunity of testing the new explanation. Absent some corroboration of this type of information coming from Fujitsu that supported it, I would not accept it in this case on this issue.
6. Further, and even if I were to take a contrary view of the points at 1 to 5 above, and accept that the Post Office should be given the opportunity of correcting Mr Godeseth’s evidence in this way, the explanation given by way of closing submissions does not assist the Post Office, for the reasons explained in the following paragraphs, [334] and [335] below.
334. This is for the following reasons. Mr Coyne’s point, to which Mr Godeseth was responding, was that the Fujitsu document identified metrics and KPIs to measure/control and reduce the risk of failure to detect, correct and remedy Horizon errors and bugs. He was basing this on section 1.4 of the document. His conclusion was that “From the above, it is my opinion that Post Office should be aware of all recorded bugs/errors/defects in addition to those previously acknowledged by them, from the process metrics compiled above.” These records were sought by the claimants and the Post Office explained that Fujitsu’s position was that the records did not exist. They were not therefore provided to the experts. Mr Godeseth’s cross-examination showed that the Fujitsu explanation was wrong (and the Post Office’s closing submissions accepted it was wrong). The records however were not produced.
335. Finally on this, after receiving the email of 10 July 2019 to which I refer at [332] above, I asked in an email of 12 July 2019 for hard copies of the documents referred to in the relevant paragraph of that email (and hence relied upon by the Post Office in this respect) to be delivered to me. This was done, and a file served on the court containing four versions of the Post Office’s Customer Service Problem Management Procedure. They are version 0.1 dated 13 November 2007; version 2.0 dated 22 April 2008; version 2.1 dated 6 June 2010; and the version put to Mr Godeseth which was in the trial bundle, dated 5 September 2017. Given the additional versions produced all pre-date Mr Salawu leaving Fujitsu in 2013, this means that the points I have identified at [329] and [330] are still valid ones, and are still in the claimants’ favour.
336. Even on Dr Worden’s evidence and his views on the lower number of bugs in the Bug Table (he accepts that there are 11, although Mr Coyne’s figure is somewhat higher), there are plainly more bugs in Horizon than Fujitsu itself was aware of. Over the years of both Legacy Horizon and Horizon Online, the total number of software bugs, defects and errors in Horizon considered by the experts is far greater than the number to which Fujitsu have admitted. This is shown in the appendix to this judgment summarising the number of bugs, errors and defects, and their years of operation. The total number of software bugs, defects and errors in both versions of Horizon is very important information. Why Fujitsu chose not to collate and report these in the manner that even a Fujitsu internal document stated would (or should) be done, is wholly unclear. A comprehensive record, whether using the metrics identified by Mr Salawu or other similar ones, would surely only assist both Fujitsu and the Post Office in its administration of Horizon. Given that such a comprehensive record has not been produced to the experts for their consideration, notwithstanding the express request made on behalf of Mr Coyne, the court is entitled to conclude that no such comprehensive record exists.
337. Mr Godeseth was also asked about the BRDB transaction correction tool. This applies to Horizon Online. The original design document for this stated that it was to allow the SSC to correct transactions by inserting balancing records to transactional/accounting or stock tables directly into the BRDB system, and also to audit the changes made. It was not designed to delete or update the records. However, in cross-examination the claimants’ leading counsel analysed, and put to him, some careful points on how this tool had been used in practice.
338. The tool works by means of an SQL insert into a variety of tables in the BRDB. SQL is an abbreviation for Structure Query Language, and is a domain-specific language used in programming and designed for managing data. It is very widely used. Although Mr Godeseth would refer simply to “Oracle” as the programme, Oracle Database or Oracle is a database management system which is produced and marketed by the US corporation which is also called Oracle. Although that is the name of the company, and SQL was developed (or invented) before the Oracle Corporation was founded, in this trial the terminology sometimes became a little shorthand on the part of those IT professionals very experienced in such matters. The Oracle Corporation is very well known, not only in the computing field, but through its co-founder Mr Larry Ellison who, amongst other achievements, has backed and competed in various America’s Cup races, including winning the 33rd America’s Cup in 2010 in the trimaran USA 17 with Oracle BMW Racing. Mr Godeseth, and the experts, are all highly experienced in the field and shorthand references to well-known languages or applications such as SQL and Oracle are to be expected. This use of shorthand did not lead to any lack of clarity in the evidence, and I refer to it only for completeness.
340. The design document made clear that:
“The following transaction tables have been granted INSERT privileges to OPS$SUPPORTTOOLUSER.
The transaction correction statement is only allowed to insert into these tables.”
These 9 tables were then listed. The process would use a "Transaction file containing an SQL INSERT statement that creates the required balancing transaction.", which is an Oracle command, and the SQL INSERT statement would provide a missing half of a transaction, where only one half was present. As it was put to Mr Godeseth, “the "SQL INSERT" statement effectively goes in and puts in the missing other side of that transaction?”, a point with which he agreed. The SQL INSERT command would put that missing other side into the records.
341. The SSC members would log into their own UNIX user, and then change directory and place their transaction file into the sub-directory. The document then records “They [ie the SSC member using the tool] will then invoke BRDBX015 manually. The shell script module will be owned by the UNIX user 'supporttooluser'."
342. However, the explanation that was provided by the Post Office’s solicitors, of which Mr Godeseth said he was fully aware, about the use of the tool was as follows:
"Each document is associated with a single SQL statement which made a database correction. There are two different types of correction shown in the files - the SQL statements for each are of the form:
"1. Update OPS$BRDB.brdb_rx_recovery_transactionsSET_ settlement_complete _time stamp = ..." And then the "INSERT INTO" command.
343. The explanation of Type 1 was "Type 1 reflects the action taken to reset the recovery flag on a transaction. This will have no effect on branch accounts….”. It was also stated expressly that "Type 2 reflects the action taken to insert a Balancing Transaction where it changes transaction data in the main transactional tables. This will affect branch accounts."
345. This was further confirmed – the mandatory use of insert, and the fact that an update command was not included in the design intent of the tool – elsewhere in the design document where it stated under method “The module will read the contents of the input transaction file, which will be in the form of an SQL insert statement. Only a single insert statement is allowed and (after an optional introductory comment) it must start with the 'insert into' clause." The insert would be in the SQL script, and Mr Godeseth expressly confirmed this. This is another example of Fujitsu failing to observe its own design intent for the use of a tool that can have an impact upon branch accounts, and in my judgment is a factor that goes to the robustness of Horizon as well as the accuracy of data. As was put to Mr Godeseth, the use of the tool had gone “way beyond” the intent in the design document, and he agreed. He said that “there is tooling which is based on this” – by which he meant based on the transaction correction tool - “which has two aspects to it, certainly, so I think I’m agreeing with you”. It is agreed that there was only one occasion when what was called Type 2 was used. This evidence goes to the wider point that I have explained, namely Fujitsu failing to observe its own design intent.
346. Mr Godeseth was taken, very carefully, through a specific use of the transaction correction tool in 2010. In PEAK 0195561, a problem was reported to the SSC on 4 March 2010 where a SPM had tried, on 2 March 2010, to transfer out £4,000 (referred to in the PEAK as 4,000 pds, which means either pounds (plural) or pounds sterling) from an individual stock unit into the shared main stock unit when the system crashed. The SPM was then issued with 2 x £4,000 receipts. These two receipts had the same session number. The PEAK, as one would expect, records various matters in note form and also uses informal shorthand. However, the main thrust is that when the SPM did the cash declaration, although the main stock unit (into which the £4,000 was being transferred) “was fine”, the unit from which the cash was taken “was out by 4000 pounds (a loss of 4000 pds)”. This is very similar to what Mr Latif said had happened to him, although the transfer in July 2015 to which he referred was £2,000. The PEAK related to Horizon Online and was the admitted occasion when the Balancing Transaction tool had been used.
347. Mr Godeseth had obtained the information in his witness statement dealing with this from Mr Parker. He also said he thought it was puzzling, and he was not sure that the PEAK “would be a totally accurate reflection of what happened” and was “an interpretation put together by a developer who is investigating the problems.” He suggested the original log files would be better (by which he meant better sources of information) and still exist, but they were not before the court. These log files would be held by Fujitsu, so any failure to have them before the court was no fault of the parties in this litigation, and that can only be laid at the door of Fujitsu. If a senior Fujitsu witness thought they would be better records, they should have been accessed and evidence about them could have been given directly. Certainly Fujitsu witnesses could have given evidence that relied upon them, had they so wished.
348. The PEAK records the following:
“After discussion with Gareth Jenkins, the suggested correction is to negate the duplicate transfer out by writing 2 lines to the BRDB_RX_REP_SESSION and BRDB_RX_EPOSS_TRANSACTIONS tables, with:
1) Product 1, Quantity 1, Amount 4000.00, Counter mode id 7 (TI)
2) Product 6276, Quantity -1, Amount -4000.00, Counter mode id 7 (TI)
This should be done using the Transaction Correction tool. An OCP approved by POL will be needed.”
349. This concerns Horizon Online, due to the references to OCP and OSR, and shows extensive efforts being made at Fujitsu to try to understand why this has occurred, and general doubt that the efforts to reproduce the fault in testing has worked. Some at Fujitsu wanted the priority to be downgraded; other entries resisted this, shown by some of the entries such as:
“What is missing from this Peak is an explanation of the events in terms of the requests, how they were ordered and when any was committed. Only then can we qualify the priority. The assumption is that we have a fix. The facts are –
1: A settlement request to timed.
2: A retry of request timeout occurred.
3: According to the DB entries both later succeeded.
Now unlike other reconciliation Peaks this stands out…..
We can’t reduce the priority unless we understand what is going on.”
350. The PEAK is lengthy and demonstrates a degree of frustration within Fujitsu at their own failure to get to the root cause. One entry on 24 March 2010 states inter alia “…..this shouldn’t have happened”. One might observe that this is stating the obvious. A later one on 25 March 2010 states “advised on the latest update in the call she states she was going to check if it was the same thing that happened before. [The Post Office] are want to know why this has happened? - why does it keep happening? - can you advise on this.”
(emphasis added)
351. One of the reasons or concerns expressed in the PEAK is that JSN entries – which should be unique and in respect of which no duplicates should be permitted in the journal table - appear to have occurred. Mr Godeseth confirmed that this should not have happened. Duplicate entries should not be permitted in the database. The documents showed that this did occur. Mr Godeseth said that the “JSN is part of the primary key into the message table, I cannot see how an Oracle database would allow that to happen” and “that would have to be a bug in Oracle, and it was certainly not the bug that I was looking – it was not the bug that I was looking at for the red alert”. He also said: “If this in fact happened as written down here [ie in the PEAK] then it would have had to have been a bug in Oracle and I certainly don't remember any such bug.”
352. The PEAK also shows computer script being used, both in an attempt to correct, and understand and analyse, what occurred. On 23 April 2010 an entry states:
“I think we have done as much as we can on this one. In conclusion although we haven’t been able to totally explain the behaviour, the risk of this type of PEAK occurring again has been minimised in live due to a change of behaviour in the BAL with respect to transactions…..This may now be marked as a duplicate of PC0194893”.
Rather concerningly – but entirely consistently with what I consider to be Fujitsu’s general approach to closure codes - the following entry appears that same day in another entry stating that the “Defect cause” is “updated to 40 – General – User”.
353. However, later the same day the following entry is made:
“I am sending this call back with Response Rejected.
Closing a call as 'Duplicate Call' results in a black mark against me. It basically means that I should not have sent the call over since the same problem has already been sent over in a previous call.
PC0195561 (duplicate transfer of 4000.00 cash) may have been caused by the same underlying fault as PC0194893 (banking reconciliation), however I could not have been reasonably expected to link the 2 calls and take the decision that it was not necessary to send PC0195561 over for further investigation.
Please close this call with category 'Advice After Investigation'
[End of Response]
Response code to call type L as Category 52 -- Pending -- Response Rejected
Response was delivered to Consumer.”
354. Eventually KEL cardc262s was updated with the information in the PEAK on 4 May 2010 and the call was closed. One of the final entries states that day “Category 60 – Final – S/W Fix Released to Call Logger”. “S/W fix” means software fix. The call is closed the same day. This is 2 months after the incident. The PEAK also referred to other PEAKs. Mr Godeseth was asked about those too. PEAK 0195962 records that the transaction tool has “been used in live”. That led to an Operational Change Process or OCP 25882 and stated:
“Due to a system fault, the branch did a Transfer Out of £4000 and a corresponding Transfer In of £8000
Justification: Correct a loss of £4000 at the branch due to a system fault
When: Planned for 10/03/2010 16:00 with a duration of 30 minutes
Extra detail: The Transfer In details were incorrectly doubled up when they were written to the BRDB. This needs to be corrected using the Transaction Correction tool.”
(emphasis added)
355. The actual printed address of the OCP document is as follows:
http://deathstar/SSC2/SSC_OCP/viewocp.jsp?OCPRef=25882&SID=f630192836137
356. This may show that the folder in which the document was kept was named by someone with an interest in a particular series of well-known films, or it may not, and this does not matter. It does, however, show that it was plainly generated or kept by the SSC.
357. Mr Godeseth agreed that the OCP showed at least one use of the transaction correction tool for one balancing transaction. Part of Mr Godeseth’s explanation for what had occurred, even though there had been two duplicate JSN entries, was as follows:
“In this situation the symptoms, as I'm reading them, are that because there was a bug in Horizon and this is in the new system, it was pretty early days of the Horizon Online, because there was a problem, we had something coming through which got through the journal filter but then failed at the branch database and so therefore, as far as the branch database is concerned, it has not happened.
MR JUSTICE FRASER: I understand that. I think what Mr Green is putting to you is that this shouldn't have got past the journal filter.
A. It's a bug, so certainly the way that the system should have worked, a JSN -- the same JSN coming up would be just a simple repeat of the message for -- because there was some sort of glitch.”
(emphasis added)
358. His final evidence in cross-examination on this was the following:
“A. Somebody may have been trying to look for a duplicate JSN entry. I can't really comment on what the guy was doing at the time who was trying to investigate. I am simply asserting that barring an Oracle bug, which would have been huge, you cannot have two entries with the same JSN.”
359. In my judgment, the Horizon Issues are sufficiently wide and costly to resolve concerning the Horizon System, both Legacy and Online, without widening them to include some attempt at including specific bugs within Oracle too. The Oracle Corporation is not even a party, and has had no opportunity to provide evidence or submissions. I simply recount Mr Godeseth’s evidence as he attempted to explain the PEAK. He did, as can be seen, refer to potential bugs in Oracle as an explanation for something that he believed to be puzzling. He accepted it was a bug and/or “a glitch”. In any event, other than Mr Godeseth’s evidence expanded in re-examination that he was involved in 2010 when establishing himself in Fujitsu (and in this he referred to an Oracle bug, which involved nodes in the database going down) there is no evidence before me that there were any bugs in Oracle in respect of this doubling up, and I reject the suggestion that this can be laid at the door of Oracle. What this PEAK shows, in my judgment, is that Fujitsu itself, with a great deal of time to prepare for the Horizon Issues trial, simply could not explain then, and cannot explain now, what caused this event to occur. That is relevant to my consideration of the Horizon Issues, and in my judgment shows that there was in that respect a bug, error or defect within Horizon. It also shows that there were those in Fujitsu who wished to close the call with the error put down to the user, as shown by the “Defect cause” being “updated to 40 – General – User”. There is nothing in any of the material on this incident to suggest it was in any way caused by the user.
360. He was taken to another PEAK, PC0175821 which was dated 19 February 2009. This is during the period of Legacy Horizon.
361. PEAK 01320275 was in relation to an incident on 21 December 2005 which showed a SPM, at a five counter site with seven stock units, had rolled over a particular stock unit called BB. He rolled it over in what was called “an effectively empty state” but having declared the correct amount of cash, and also adjusting the stock levels to the correct volumes, his branch account showed “a gain of approximately £18,000”. This was an obvious impact upon his branch account, and it was investigated. One entry in the PEAK stated:
“We are unable to correct the system figures safely. We can however provide accurate figures for what should have been in the Final Balance for BB, to enable POL [ie the Post Office] to make the correction perhaps by using a Transaction Correction.
POL need to make a decision on whether they are able to correct the problem in this way, however we do not see any other alternative. Corrective action should be taken before 11th January when the branch is due to roll into TP10.
The cause of the problem is unknown and is under investigation.”
(emphasis added)
362. This accepts in terms that there is a problem, and accepts that its cause is unknown. The suggestion by Fujitsu later in the PEAK on 3 January 2006 was to generate a negative figure effectively by changing the relevant data in the messagestore to try and cancel out the £18,000 which the SPM had showing as a positive figure in his branch accounts. It is obvious to me that this would only remedy the effect of the problem, rather than identify what had caused the discrepancy in the branch accounts in the first place. However, the entry is illuminating in any event:
“If we get to the problem before the office is rolled we are able to change objects in the messagestore to reset the stockunit back to the CAP (TP) rollover trailer. The PM can then rollover. PM should get a large shortage which cancels out the large gain.
We don't want to be having to do this as making manual changes to the messagestore is open to error and each time we have to seek authorisation from POL to make the changes.
If we get to the problem after the office is rolled (as in this call) then we are unable to correct the system figures safely. Its not been decided how we get the PM sorted out.
All in all, we want this fixed asap.”
(emphasis added)
363. Mr Godeseth accepted that generating the artificial negative figure in the message store could be done, but said he was “not involved interfacing with Horizon” at the time and was not at the operational level shown in the PEAK. He was not involved in obtaining authority from the Post Office for such matters. In my judgment, the way that Fujitsu anticipated resolving this, by artificially seeking to engineer “a large shortage which cancels out the large gain” is not exactly grasping the nettle. It would have the effect of disguising why the large gain had occurred in the first place, in other words a bug, error or defect.
364. Another PEAK 0152014 dated 10 December 2007 showed a problem that was described in the PEAK as “POLFS Incomplete Summaries Report” which related to a foreign currency transaction. At this time, £484 sterling could buy $1,000 (the PEAK did not specify US or Canadian, but the associated OCP identified US$ specifically) and the PEAK recorded that:
“This is due to a single SC line written for $1000 (£484) with no settlement in the middle of two RISP transactions.
On call PC0151718 the harvester exception was corrected and now the transaction for the day don't zero, hence this issue with the incomplete summaries report.
Am currently retrieving the messagestore for this branch, we will then be inserting a new message on the counter to remove the effects of this. OCP 17510 has been raised.”
365. Again, this was being done to correct the effect of what had occurred, rather than discovering what had led to the issue in the first place. In two places in the PEAK, the following entry appeared:
“**Again, this may also have caused a receipts and payments error, can EDSC please confirm whether this is a gain or loss at the counter and the amount.**”
366. Mr Godeseth had been asked to look at this PEAK in advance (his evidence went over two days and this was agreed by the parties), and therefore he had been given a chance to familiarise himself with its contents, and those of an associated KEL. The PEAK stated "Worth noting that the branch did not have any issues with the mismatched transactions because this was fixed before they did the roll. The branch is not aware of this and it's best that the branch is not advised." (emphasis added)
367. When it was put to Mr Godeseth that this showed that on not all occasions were SPMs advised of impacts on their branch accounts of particular problems, he said that was “a fair inference”. An inference is a common sense conclusion, and I agree with Mr Godeseth that it is indeed fair to draw that inference. He did however then add “I think I would say that Post Office were well aware of this and I would argue that it's a Post Office decision whether or not to tell a subpostmaster.” This shows that Mr Godeseth was not immune from arguing the case or making suggestions not supported by the evidence. Nothing was produced by either side to show that the SPM in this case was told. Certainly the person at Fujitsu who wrote “….it’s best that the branch is not advised” cannot have shared Mr Godeseth’s view at the time. Mr Godeseth did however also say, in my judgment again importantly, that “in the background to this there was a dialogue with [the] Post Office” and he agreed with counsel who put to him that the “Post Office would have been aware of what was being done by Fujitsu”.
368. The associated OCP however, whose subject was “write corrective bureau message for FAD 183227” – bureau meaning foreign currency – gave a great amount more detail. It stated:
“A single SC message 183227-7-1101211 was written in error on 26th November at 12:43:17, selling 1000 US dollars, with no corresponding settlement line. To remove the effects of this message at both the branch and on POLFS, we will insert a new message to negate the effects of the original message.
Justification: If the change is not made in the counter messagestore (before the stock unit is balanced on Wednesday), the branch will have an unexpected gain of £484 (or thereabouts - depends on exchange rate), and a receipts and payments mismatch. This gain would have to be resolved at the branch. There would also be an inconsistency between the branch and POLFS to be resolved. By correcting the problem locally, the branch may not be aware of the problem, and there will be no inconsistency between the branch and POLFS.”
And:
“The message will include a comment to show it has been inserted to resolve this problem (this will not be visible to the branch).
This change will first be applied to a copy of the messagestore within the SSC environment, and the stock unit then rolled over to make sure there are no unexpected consequences.
Neither the new nor the old message will be included in data sent to POLFS.
Gary Blackburn (POL) is already aware of this issue.”
(emphasis added)
369. This shows that the Post Office did know, as Mr Godeseth had said. It also shows that the branch would not be able to see what was done. It also showed that the intention was to reverse, by means of a specific insert into the messagestore by Fujitsu, an entry or data in order to correct a discrepancy in the branch accounts caused by an error or problem within Horizon Legacy. This was approved by the Post Office as shown by a later entry. The OCP had been raised by Anne Chambers. The “extra detail” – effectively what the message needed to contain or achieve – was:
“Extra detail: The original message had ProductNo:5129, Qty:1, SaleValue:484, PQty:1000. The new message will have Qty:-1, SaleValue:-484, PQty:-1000, with other attributes (including exchange rate) as before.”
370. The specific message that was inserted into POL FS, as shown in the OCR, was as follows:
“Extra detail: This OCR is being raised so that EDSC is authorised to amend the txn details for branch 183227 and insert these into tps_pol_fs_summaries_incomp table on the host.
Comments
Andy Keil (POA SSC Support) wrote at 12/12/2007 15:07: Updated POLFS feed for branch 183227 product 5129 mode SC with SaleValue=1014.73 and PQty=2080”
Mr Godeseth agreed that this showed “just over $2,000 being inserted in the Post Office system”.
“The counter problem which caused the first issue has been corrected by inserting a message into the messagestore, for equal but opposite values/quantities, as agreed with POL (OCP 17510).
As a result of this corrective action, the net effect on POLFS is zero, and POLFS figures are in line with the branch. POLMIS received both the original message and the corrective message.
Once the problem was corrected, there should have been no impact on the branch. However it has been noted that the stock unit BDC had a loss of $1000, which was generated after the correction was made. We have already notified Gary Blackburn at POL (email attached). This appears to be a genuine loss at the branch, not a consequence of the problem or correction.”
(emphasis added)
372. On 17 December Anne Chambers also stated in the PEAK:
“Summary for development:
A single SC line was written for $1000 (£484), with no settlement, in the middle of two RISP transactions.
The line was missing some AdditionalData so it wasn't harvested properly, but the main problem was the lack of settlement. POL authorised us to insert an equal but opposite message, to prevent a discrepancy (in theory anyway) and to avoid problems on POLFS. Please note that this is exceptional and must not be seen as a convenient avoidance in place of a fix.
Subsequent investigations showed that a second packed pouch barcode had been scanned, before the receipt for the first had finished processing. See the audit log / messages on counter 7 26th Nov at 12:43 / 12:44.
PC0147357 is already with development for what I think is a similar problem - Mode reverts to what it was prior to RISP.
Please also see update on that call.
Routing to EPOSS-Dev via QFP.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under Investigation.”
(emphasis added)
373. The PEAK was finally closed on 17 April 2008, the same day that the “defect cause” was “updated to 14: Development – Code”.
374. There are two important points to make in relation to the entry in this PEAK which I have quoted at [371] above. Firstly, the discrepancy in the branch accounts, which was, after the “corrective action”, in deficit, was broadly in the amount of the difference between the two corrections that were made, namely the one by the OCP and the OCR. These corrections should, in my judgment, have been in the same amount, as one relates to “front end” (or the branch messagestore) and the other “back end” of the accounts so far as Horizon is concerned. The statement in the PEAK that “this appears to be a genuine loss at the branch, not a consequence of the problem or correction” is simply insupportable. That is a statement made by Anne Chambers, which in my judgment flies in the face of the documents. I am supported in my conclusion by two things. Firstly, Mr Godeseth’s evidence, when it was put to him that this was a fair possibility when he said “having read this PEAK in more detail overnight then yes, clearly that is what appears to have been the case.” I accept that evidence. It is supported by the documents and by the text of the two different corrections. Secondly, plain and obvious common sense.
375. The second important point is that the PEAK makes it clear this, or something very similar, has happened before. It states “PC0147357 is already with development for what I think is a similar problem - Mode reverts to what it was prior to RISP.” The reference to “development” is the department within Fujitsu where software is written to correct issues with the system. This is supported by the closure of the PEAK and defect cause stating “Development – Code”.
376. There is an associated issue that arises on this PEAK, the number of which I will again recite for clarity, PEAK PC0152014. I will deal with this here, rather than in the section dealing with Mr Coyne’s evidence or within the Technical Appendix (although the latter place could be a more suitable home for it). This is because it goes to the weight to be given to Mr Godeseth’s evidence.
377. A detailed explanation was put to Mr Coyne on this very matter, which occurred on the third day of his evidence on 6 June 2019 and went from page 103 of the transcript, ran until page 115 when the OCP was then put (which went to page 120) and then the OCR which was put between page 120 and 126. Leading counsel for the Post Office stated “I can tell you on instruction what that figure means”; “So on instructions I can tell you than the sale value of £1,0114.73 and the quantity of 2,080……” and “I appreciate that I’m telling you this on instructions because I have to say nobody knew that this suggestion was going to be made until it was put to Mr Godeseth….” It was made clear that there was what was being put to Mr Coyne as a detailed explanation of what had occurred, was “on instruction”, and it was nowhere in the evidence.
378. Mr Godeseth was cross-examined on 20 and 21 March 2019. When Mr Coyne was cross examined, about 2 ½ months later, a great deal was put to him “on instruction”, as I have identified. It was a detailed factual explanation that was not included in any evidence at all. Firstly, the OCR that went with it showed that “This OCR is being raised so that EDSC is authorised to amend the txn details for branch 183227 and insert these into tps_pol_fs_summaries_incomp table on the host.” This was required precisely because Mr Godeseth was correct, in my judgment, in his understanding of what had occurred. I accept his evidence on this. Secondly, the explanation put to Mr Coyne “on instruction” was (broadly) that there were two problems. A careful attempt was made to explain to Mr Coyne how (basically) Mr Godeseth had misunderstood, and to downplay the effect of what had occurred.
380. The explanation did not make sense on the face of the documents, and they did not support the points that were being put “on instruction”. Firstly, as Mr Coyne pointed out, the PEAK identified that a message was to be inserted into the messagestore. None of the documents put to Mr Coyne “on instruction” identified what that message was. This point was made by Mr Coyne during the cross-examination:
“Q. Yes. So what they are talking about here, what this change here is, is a change being made by using the TIP repair tool into the TPS, correct?
A. Right, so this doesn't relate to the creation of the message then.
Q. It doesn't relate to the branch accounts, Mr Coyne, does it? This is an OCR which involves an exercise -- well, the use of the TIP repair tool to change data that is in the TPS system, yes?
A. Yes, but the PEAK refers to the insertion of a message into the messagestore.”
381. The critical point, that the explanation skipped over, was what the message that was inserted in fact was, in actual terms. Had that message been produced, it could have been looked at, and deciphered. Understanding such a message is not an insuperable task beyond mortals. The second point is that the PEAK refers to a KEL, namely KEL obengc3120K. At the beginning of the trial, I asked for hard copies of all PEAKs and KELs put to witnesses to be prepared in hard copy too, so that I had a working file that I could use during the trial for PEAKs and KELs already referred to (in addition to the three OPUS trial bundle screens that I had to work from). This was done. That KEL is at tab 21 of volume 2 of the file entitled “PEAKs and KELs referred to in Days 5-8 and 12”. The KEL states the following, and I will reproduce the text in full, including the relevant table references.
“KEL obengc3120K
Title:
Not harvested: Could not update database: Updating table TMS_RX_BDC_ TRANSACTIONS, ORA-02290: check constraint (OPS$TPS.BT23B_BUREAU_REGION_CHK) violated.
Summary:
Harvesterd did not harvest message: Could not update database: Updating table TMS_RX_BDC_ TRANSACTIONS, ORA-02290: check constraint (OPS$TPS.BT23B_BUREAU_REGION_CHK) violated
Symptoms
The Harvester tried to insert into the BdC table TMS_RX_BDC_TRANSACTIONS a message with missing ‘Blackbox’ data. This caused Oracle [ORA-02290] to detect a check constraint (OPS$TPS.BT23B_BUREAU_REGION_CHK) violation, hence the appearance in the TPSC254.<br><br>Harvester exceptions do not normally cause entries on TPSC250, but this one seems to. <br><br>TPSC257 displays the contents of the TPS Incomplete Summary table where the summarisation program detects that the total harvested transactions for the day don’t NET to zero.
Problem
Due to some data in the ‘Blackbox’ or AdditionalData of the messagestore not being populated, the bureau product was missing key data including<br>BUREAU_REGION<br>MARGIN<br>MARGIN_PRODUCT<br>EFFECTIVE_EXCHANGE_RATE<br>and so when the Harvester attempted to insert the message into the BdC table TMS_RX_BDC_TRANSACTIONS, however Oracle [ORA-02290] detected a check constraint (OPS$TPS.BT23B_BUREAU_REGION_CHK) being violated.
Solution – ATOS
Warning: Check the AMOUNT (or sum of AMOUNTs) for the branch on the TPSC254 report.<br><br>If the total AMOUNT is NOT ZERO, and the branch is NOT on TPSC257 (same day), <br>or the total AMOUNT IS zero and the branch IS on TPSC257 (same day) then there is probably a problem with the messages written on the counter – a bureau SC message with no corresponding settlement. Use KEL <ahref=kel._view_kel.jsp?KELRef=acha3159Q>acha3159Q</a>.<br><br>Check the messages anyway, just to be on the safe side. If the session was settled properly….<br><br>MSU must raise an OCR so that SSC can use the TIP Repair Tool to populate the missing columns, using values from another transaction for the same currency, same branch, same day if possible.<br><br>NOTE: By repairing the txn, the TPS_POL_FS_SUMMARIES_INCOMP will be corrected automatically after that txn has been successfully harvested. <br><br>*** In the past, we have just repaired these, on the assumption that they are genuine SC transactions which have been corrupted. However I think it may not be a genuine sale, but is related to an attempted pouch reversal (this is certainly the case for the more extreme instances where the settlement is missing). If the SaleValue and the PQty have opposite signs, and there is no reversal (mode ER) for the transaction, this may have caused a loss or gain at the branch which they can’t resolve themselves. Let Anne know if there is another occurrence like this.
(emphasis added)
382. This KEL shows that there was a potential impact on branch accounts – it states “this may have caused a loss or gain at the branch which they can’t resolve themselves”, which is very clear. It shows that the explanation put to Mr Coyne “on instruction” is, in my judgment, completely wrong. It also shows that it has happened before, as “in the past we have just repaired these”. I would also note for completeness that this KEL is not dealt with at all by Mr Parker in the table accompanying his 1st witness statement. Two KELs from Ms Obeng are, namely CObeng1123 at number 24 in the table at E2/11/40; and obengc5933K at number 31 in the table at E2/11/43. However, this one is not.
383. I accept Mr Godeseth’s evidence on this point, which is wholly consistent with the documents, in particular the PEAK and the KEL. The text of those documents shows that a message was intended to be written into the messagestore; the actual message used is nowhere available; the effects of all this clearly being an impact on branch accounts.
384. A further PEAK was PC0175821, which was dated 22 February 2009. This also related to foreign currency transactions. This PEAK related to two problems, or a problem with two elements to it. It is at F/485/1 in the trial bundle, and it clearly and expressly refers to the same KEL, obengc3120K.
385. The two aspects were explained by Mr Godeseth as a change using the transaction repair tool to “get the feed into POLFS correct” and also “a change to the messagestore to get the branch aligned.” The repair “had to go two ways” because the branch and the POLFS had to be aligned. If they were not, there would be a discrepancy in branch accounts. This showed the two problems, or two sides to the same problem. It was stated as follows in the PEAK:
“The first is where all five SC transactions missing core data as described in the above-mentioned KEL." This means obengc3120K, referred to at the beginning of the PEAK.
And the “Second is absence of equal but opposite 2 (ie settlement) lines. See PC0152014 for a similar problem and how problem was resolved."
“There are two sides to the problem relating to these txns. The first is where all five SC txns missing core data as described in the above mentioned KEL. Second is absence of equal but opposite [i.e. settlement] lines. See PC0152014 for a similar problem and how problem was resolved.
For the first problem, I have used the TRT [the transaction repair tool] to insert the missing data i.e. Region, Margin, Margin Product and EffectiveExRate.”
The reference to “txns” means transactions. PC0152014 is the very PEAK referred to in [376] above.
386. Mr Godeseth agreed with both of those points in the PEAK as they were put to him by the claimants. Basically, half the data was missing. The transaction repair tool or TRT was used for the first problem.
387. Numerous other PEAKs and some KELs were put to him with very specific details. Both Gareth Jenkins and Anne Chambers were involved in raising, or investigating, a great many of them and they are referred to in the text of both classes of documents. In all of them, Mr Godeseth’s answers to questions were, in my judgment, consistent with the entries within the documents, and also consistent with the claimants’ case on the Horizon Issues. In particular, those in the PEAKs dealing with foreign currency which I have explained above was clear, and settlement lines were missing, which Fujitsu sought to correct by injecting lines into both the messagestore and into POLFS.
388. Finally, a series of questions were put to him demonstrating what are called APPSUP permissions. This is a very powerful permission (which really means level of access) and PEAK PC0208119, which runs from February 2011 onwards into 2012 and beyond (the final entry being in 2015), made the following relevant comment:
“As per the previous PEAK comments, the role 'APPSUP' is extremely powerful and should only be used under extreme circumstances and under MSC supervision. As such the Branch Database design was that 3rd line support users should be given the 'SSC' role, which is effectively read access, ie. 'select_any_table + select_catalogue'. SSC team members should only have to [access] BRSS for normal support investigations, unless the information has not replicated in time. SSC should only given the optional role 'APPSUP' temporarily (by Security Ops authorisation/emergency MSC) if required to make emergency data amendments in BRDB Live.
It is a security breach if any user write access is not audited on Branch Database, hence the emergency MSC for any APPSUP role activity must have session logs attached under the MSC. Host-Dev previously provided scripts, such as the Transaction Correction Tool, are written to run under the SSC role and also write to the audit logs.
SSC users created on BRDB should only have the SSC role, and the user creation script should be amended by Host-Dev to reflect this. A separate script giving/revoking emergency MSC access via APPSUP can be delivered, logging this to the hostaudit directory. In parallel Host-Dev should investigate any Host-Dev delivered script to ensure they are all executable by the SSC role. SSC should investigate any of their own scripts to ensure they have sufficient permissions under the SSC role, taking into account they should primarily perform their work on BRSS.
Any day to day scripts should not access BRDB directly. Any scripts needing more than the SSC role should be questioned, except those that would run under MSC APPSUP. Once the investigation is complete, all BRDB SSC users with APPSUP should have the role removed by ISD (via MSC) and ensure they do have the SSC role.
If anyone is in disagreement with the above course of action, then I'll set up a meeting with yourselves and Security when I'm down in BRA01 next week.”
(emphasis added)
389. This entry was from Andy Beardmore, Senior Software and Solution Design Architect Application Services. The experts are agreed that the APPSUP role would, effectively, permit anyone who had that permission to do almost anything on Horizon. It was available to 3rd line support at SSC, the level at which Mr Roll was employed by Fujitsu. This PEAK further substantiates the evidence of Mr Roll and is consistent with it. APPSUP was described by Mr Parker as “the more technically correct name for a type of privileged access to the BRDB”. It is a very powerful permission.
390. A later entry in the same PEAK states:
“The Business Impact has been updated:
1. Cost: There is currently no "cost" to this issue. The users affected have more access than is required.
2. Perceived Impact: The customer is not aware of this problem or change.
3. Scope: No actual impact/incidents of problems relating to this issue have been experienced yet (and not expected)”.
(emphasis added)
The customer was the Post Office. The “issue” referred to was Fujitsu SSC users having had far greater access, namely the powerful APPSUP permissions, on more than the intended basis, which was supposed to be in extreme circumstances only and temporarily. This is shown in an entry for 1 February 2011 by Mark Wright who stated “….SSC users have the APPSUP role” and “When we created SSC users for BDB/BRS etc. we used “appsup” as that is what SSC have always been and what they migrated as on Horizon databases.” (emphasis added)
It is clear that all the members SSC had always APPSUP, but were not supposed to have this powerful role all the time.
391. Anne Chambers said, in an entry on 1 February 2011:
“Unfortunately development write their scripts explicitly to use SSC. So I think we're stuck with it unless they deliver new scripts (which would not be a popular or quick option).
When we go off piste we use appsup. Can we have both??”
(emphasis added)
392. Mr Godeseth said that by “off piste” he was very confident that she meant “having to fix a problem that was not catered for by a script that is available to these people” by which he meant SSC. It was clear that SSC were using APPSUP and wished to continue to do so. It was equally clear that all the members of SSC were not supposed to have it.
“The main area we would encourage management focus on in the current year is improving the IT governance and control environment.
Within the IT environment our audit work has again identified weaknesses mainly relating to the control environment operated by POL’s third party IT suppliers.
Our key recommendations can be summarised into the following four areas:
Improve governance of outsourcing application management
Improve segregation of duties within the manage change process
Strengthen the change management process
Strengthen the review of privileged access”
(emphasis added)
394. The “third party IT suppliers” referred to were Fujitsu. Whether it was the work performed by E&Y that had led to the decision at Fujitsu that the APPSUP permissions were not being sufficiently restricted or controlled, or something else, does not much matter. A company such as E&Y would not lightly refer to “again identified weaknesses” in the Management Letter for a particular year unless that conclusion had been reached after a high degree of professional and in depth investigation, of a type that even a lengthy adversarial trial such as this one cannot hope to replicate.
“This is an Oracle supplied role for use by database administrators (DBAs). Lots of privileges are granted to this role so users have the ability to update/delete/insert into any of the Branch database tables.”
(emphasis added)
396. POA Unix users also are:
“Granted the ability to execute the following executable functions:
(a) OPS$SUPPORTTOOLUSER.PKG BRDB TXN_CORRECTION —framework to allow the user to insert fully audited balancing records into a BRDB transaction table (made against node ID 99).”
397. Both of these are powerful roles. Mr Godeseth was asked specifically about the DBA role explained in [395] above. His specific answers merit quotation verbatim:
“Q. You would agree that those people have the role which allows them privileges to update, delete, or insert into branch database tables whether they are using the correction tool or not?
A. Those people could log on to the database and do an awful lot of damage.
Q. And the only audit of that that we have prior to 2015 was log on and log off; that's correct, isn't it?
A. Correct.”
(emphasis added)
“I have not examined the privileged user logs, but based on my experience my expectation is that these uses of APPSUP, or at least the vast majority, are for support work that does not involve changes to transaction data. I cannot recall any cases in which it has been used to change transaction data, but I cannot state unequivocally that there are no circumstances in which it has ever happened.”
399. Mr Godeseth said in his written evidence that “Privileged users can, theoretically, inject, edit or delete transaction data in Horizon Online. As far as I am aware, this has never happened.” His evidence above about prior to 2015 makes it clear that he could not work out the number of times this may have happened in any event. Therefore the fact that he was not aware it had ever happened is neither here nor there, and does not assist the Post Office.
400. In his re-examination Mr Godeseth was asked about an entry in an OCP in relation to a fix that stated UNIX user brdb. He said he did not know the detail, it meant logging on to the database, and the “UNIX user” referred to “would be one of the guys in Ireland”. These were not members of the SSC. When asked to clarify “who are the ‘guys in Ireland’ exactly?” he said “the people who support the hardware, so UNIX is an operating system so they work at a pretty low level on the systems” and by “low level” he meant “they have pretty powerful user rights but they are very much driven by process as to how they use them”. He also said there was complete and utter control on the process they go through before they did any of this type of activity, which in that re-examination example was "Logon to BRDB Node 1 as UNIX user 'brdb'."
402. One of the KELs to which he was taken, with the reference JSimpkins338Q, related to problems with Riposte and what was sometimes called the Riposte lock or unlock problem. This was when an unexpected error occurred in Riposte while attempting to insert a message. The KEL shows it ran from 2002, with other events occurring in 2003, 2004 and 2005. The entry for September 2005 expressly states that “this problem is occurring every week, in one case at the same site on 2 consecutive weeks”. A PEAK was sent to development. Under “solution” in the KEL the following entry appears:
“SMC: If the event is seen at a multi-counter office during the working hours of the PO, or up to 18:00 on a weekday (in case they are balancing out-of-hours), RING THE OFFICE AND GET THEM TO REBOOT the eventing counter. If they are in the process of balancing, it is strongly advised that they reboot before continuing with balancing as they are at risk of producing an incorrect balance. Warn the PM that if transactions appear to be missing, they should not be re-entered - they will become visible after the counter has been rebooted.
If a reboot/Cleardesk does not resolve this problem, send the call over for further investigation - SSC can rebuild the messagestore on the affected counter.”
(Block capitals in original; emphasis by underlining added)
403. The risk of a branch producing an incorrect balance as a result of this problem with Riposte is clearly recorded. An incorrect balance is another way of referring to discrepancies in branch accounts, a central issue in this litigation.
“This office had severe problems balancing on Wk 25, resulting in a shortage of £6,414.46. After checking various reports I am satisfied that the error is made up of:
£3,489.69 – Transfers
£2,870 – Giro Deposits
£54.52 – unidentified (however due to all the coming and going with re- keying entries, then this could come back as an error).
The Spmr claims that there was a Horizon software problem on 14.09.05 from 15.30 onwards. This was picked up when a member of staff noticed that a transaction, which had been taken by another member of staff, had not been entered onto the system, so therefore she put the transaction through. She checked at the time with her colleague who said that she thought she had put it through already however she accepted that she could have made a mistake. Following on from that, it was picked up that other giro business deposits that had been entered had not come up on the system, so they were re-keyed.
There was also a problem with transfers from one stock to another, in that they had doubled up. The Spmr made several telephone calls to the NBSC, telling them about his problems and he was advised to carry on with balancing and produce his Cash Account. Whilst doing this a warning came up, however the NBSC told the staff to continue to roll over. The result was that the office balanced £6,414.46 short.
The Spmr spoke to the Horizon Support Centre (ref E0509150123) who investigated and agreed that there had been a navigation problem that had now been rectified. They told the Spmr that they would report to NBSC that they had identified and rectified the problem and that the amount could be held in the suspense account. However, as part of the shortage relates to transfers, and no error notice will be issued, then the Suspense Account Team are not prepared to authorise the entry.
I telephoned The Suspense Account Team (Ann Wilde), who told me that checks could be made with Girobank after next Wednesday, and if that shows that duplications have been made, then they will authorise the amount to be moved to the suspense account, until the office receives an error notice. However, Ann stands by what she said about the transfer problems, and that they would not move this amount to The Suspense Account.
I went back to the Horizon Support Centre and spoke to a supervisor (Ken). He said that the call had now been closed as the problem had been rectified. I asked what was to happen about the resulting shortage and he referred me back to NBSC, who they said would do various checks. I then contacted NBSC, spoke to Rob Hughes and told him the story – he said he would put a call through to Service Support. No follow up was received from Service Support regarding this call.”
(emphasis added)
405. The important points from this are as follows. This problem turned out to be an admitted bug. At the time, the SPM noticed the problem – and could only identify it as a “problem with Horizon” and reported it. The branch accounts were plainly affected by a sizeable amount, over £6,400. Horizon Support Centre, after investigation, agreed that there was a “navigation problem” – this is a problem with the Horizon system, and was caused by a bug. The Suspense Account Team would not authorise the shortfall in the branch accounts being moved to the Suspense Account; in other words, they were not prepared not to hold the SPM responsible. Mr Godeseth also accepted that “the genesis of it appears to go back to February 2003 and similar lock agent problems back in November 2000”, a point that he accepted as “that’s fair”. The history of the problem was therefore somewhat longer than something that just occurred in 2005 for the first time. During that five year period, branch accounts were usually being produced every four weeks.
406. Mr Godeseth admitted that it was “a horrible position” for the SPM to have been in. He said “Service Support” was not Fujitsu, it was the Post Office. The story at [404] and [405] read like a mini-summary of many of the factual issues in the litigation generally, with the exception that this is an admitted bug by the Post Office.
“1/ This problem is the root cause of the reconciliation error closed in PC0126042
2/ Presumably the root cause is deemed to be software not hardware
3/ The Postmaster has a workaround in place which is not to duplicate transactions (e.g. Transfer In) just because the original attempts were successful but not showing on all nodes
4/ POA CS MSU have a workaround in place which is that if 3/ above is not followed & PC0126042 reoccurs, a BIMS will be issued advising POL to issue a Transaction Correction
5/ There is no SLT for software fixes as they are delivered based upon the priority or severity of the issue and could remain open until both businesses decide a fix is necessary or the work around is adequate.”
This shows that the entry in the PEAK identifies this as a software problem; that is effectively now accepted, but this entry from 2005 did not lead to any public acceptance of a bug in Legacy Horizon either then, shortly afterwards, or for some years after that.
408. The acronym SLT means Service Level Target. What this entry means is that this problem had occurred before, and was known to have occurred before; there was something called “a workaround” in place, but that required the SPM physically deciding not to duplicate transactions if the original transaction did not show up as successful on all the nodes; if that were not done, TCs would be required; both Fujitsu and the Post Office had to decide a software fix was required in order for this to be corrected by fixing the software. No such decision had been taken at this point.
409. At Callendar Square itself, in January 2006 the SPM phoned one of his superiors. The record of that call in the log states the following:
“Issue Raised: PM WANTS TO SPEAK TO SANDRA MCKAY URGENTLY REGARDING SERIOUS PROBLEMS WITH HORIZON IT IS REPEAT PERFORMANCE AS TO WHAT WAS HAPPENING BEFORE AND HSH FAILING TO PICK IT UP
Response by SPM: Telephoned the office and Allan said that he was having problems again with transfers. He has contacted the Horizon helpdesk who have subsequently come back to him to say that there is no system problem and that he should contact NBSC. He did this and from what I can understand the NBSC have told him that he is trying to balance on two different terminals. Allan disputes this and is adamant that there is a system error.”
(Block capitals in original; emphasis by underlining added)
410. The statement that “there is no system problem” which is told to the SPM evidently conflicts with what the PEAK at [407] had recorded, that “presumably the root cause” was the software.
411. Emails in February 2006 to and from Anne Chambers stated expressly the following:
“…. I notice that in the early guise of this problem in the call it states the PM as Female:
Wed 12 October 2005 17:39 by UK956078 / HSH1 Saved: Wed 12 October 2005 17:39 Pm was trying to transfer £2490 from node 7 onto node 2. She states that she has accepted the transfer on node 2 but the system is not showing this. On node 7 it is showing pending transfer but it is not showing on node 2. It appears on her transfer sheet as completed.
At the bottom of this email re a magical £43k appearing and disappearing the PM is Male He reports:
You may recall that in September the above office had major problems with their Horizon system relating to transfers between stock units.
The Spmr has reported that he is again experiencing problems with transfers, (05.01.06) which resulted in a loss of around £43k which has subsequently rectified itself. I know that the Spmr has reported this to Horizon Support, who have come back to him stating that they cannot find any problem.
Clearly the Spmr is concerned as we have just spent a number of months trying to sort out the first instance and he doesn't want a repeat performance. He is convinced that there is something wrong with his Horizon kit. I would be grateful if you could investigate this and give him any support that you can. I'm due to visit the office tomorrow to have a look at his paperwork and discuss the situation with him.
So apologies for the long windedness but I have been given this by Liz as a problem so:
1. Is there a problem at this branch? is it Horizon kit or is there an issue with staff there?
2. If there is an issue is this S90 release the cure? how confident are you/we it will fix the problem?
3. S90 counter release due week 4th March. Getting Sarah to check if this site is in the pilot 24th or just part of the general release 4th March.
Appreciate your comments please.”
(emphasis added)
412. In the answer to this question posed in February 2006, Anne Chambers stated the following:
“Haven't looked at the recent evidence, but I know in the past this site had hit this Riposte lock problem 2 or 3 times within a few weeks. This problem has been around for years and affects a number of sites most weeks, and finally Escher say they have done something about it. I am interested in whether they really have fixed it which it why I left the call open - to remind me to check over the whole estate once S90 is live - call me cynical but I do not just accept a 3rd party's word that they have fixed something!
What I never got to the bottom of, having usually had more pressing things to do, was why this outlet was particularly prone to the problem. Possibly because they follow some particular procedure/sequence which makes it more likely to happen? This could still be worth investigating, especially if they have continuing problems, but I don't think it is worthwhile until we know the S90 position.
Please note that KELs tell SMC that they must contact sites and warn them of balancing problems if they notice the event storms caused by the held lock, and advise them to reboot the affected counter before continuing with the balance. Unfortunately in practice it seems to take SMC several hours to notice these storms by which time the damage may have been done.”
(emphasis added)
413. This shows the following important points. At least Anne Chambers in early 2006, and all those with whom she was corresponding, knew that this problem – now admitted to be a software bug – had been around “for years”. Horizon Support were telling the SPM, whose branch accounts were affected by discrepancies, that “they cannot find any problem”. The SMC – the part within Fujitsu responsible for providing corrective action for the “event storms” – would not always notice these had occurred in time and by then “the damage may have been done”. I find that by “the damage” this can only mean impact upon branch accounts.
414. Mr Godeseth gave evidence about S90, the software release intended to remedy this, which was distributed in March 2006. This was 6 years after Legacy Horizon became in use. His evidence was that this release was one year before what he called “the relevant period” for Ms Misra’s branch, one of the SPMs convicted of criminal offences, which he was asked about. He also gave evidence that the Callendar Square bug affected 30 branches, and had caused mismatches in the branch accounts at 19 (corrected in supplementary evidence in chief from 20) of them. He said in his witness statement that this information had come from Matthew Lenton, Fujitsu’s Post Office Account Document Manager and that “Fujitsu has established this for the purposes of this statement using the event logs described above.” He accepted that this part of his witness statement was not correct. The figures that he had provided had not come from an exercise specifically for the purposes of his evidence, or for the Horizon Issues trial. It had actually come from “the spreadsheet [which] is something that was done at the time, as I understand it by Anne Chambers, to actually investigate what had happened when we had had these locks.”
415. This was an important correction by Mr Godeseth, for reasons that I will now explain.
416. It is a matter of public record that Anne Chambers gave evidence before the High Court in the case of Post Office Ltd v Castleton [2007] EWHC 5 (QB) before HHJ Havery QC. Her evidence in that case is summarised at [23] of that judgment. Callendar Square is referred to, but as a single branch affected. The reason that I have referred to this case is because it was specifically referred to by Mr Jenkins in an email of 8 March 2010 which was referred to by Mr Godeseth in paragraph 14 of his witness statement. Mr Godeseth said that he agreed with that email, which stated:
“I’ve been asked about the issue at Callender Square, Falkirk (I’m not quite sure about the spelling) that came up at the Castleton Trial.
I thought I’d better keep you in the loop on this.
I’ve now dug back into the archives to provide the following summary:
1. The problem occurred when transferring Cash or Stock between Stock Units. Note that West Byfleet does operate multiple Stock Units so the issue could have occurred.
2. It manifests itself by the Receiving Stock Unit not being able to “see” the Transfer made by the “sending” Stock Unit and is compounded by attempting to make a further transfer. Note that such transactions usually reappear the next day.
3. It is clearly visible to the User as a “Receipts and Payments mismatch” at the time that one of the Stock Units is Balanced. This usually results in the Branch raising a call. There are no such calls in Andy Dunks’ Witness Statement which summarises the calls raised by West Byfleet. Also this can be checked on any Balance Reports or Branch Trading Statements that are available from the Branch which should show that Receipts and Payments do match and that the Trading Position is zero.
4. The problem is also visible when looking at system events associated with the Branch. The System events from 30/06/2005 to 31/12/2009 for West Byfleet have been checked and no such events have been found.
5. The problem was fixed in the S90 Release which went live in March 2006 and so would not have been relevant at the time of the detailed Transaction Logs obtained for West Byfleet between December 2006 and December 2007.
Therefore I can conclude that the problems identified in Calendar Square, Falkirk are not relevant to West Byfleet.”
417. This email does not provide anywhere near the same degree of information about the Callendar Square bug as was available at the time of the Horizon Issues trial, or as recorded in the PEAKs above. There has obviously been further investigation at Fujitsu into this specific issue since then, as made clear by Mr Godeseth’s evidence about the information coming from Mr Lenton and Ms Anne Chambers. The email does not use the term “software bug”. It does not refer to the fact that it is accepted by Fujitsu, following an investigation, that the admitted bug had affected at least 30 branches. Nor does it refer to the fact that the Callendar Square bug caused mismatches at 19 of those 30 branches.
418. The spreadsheet to which Mr Godeseth referred as having been “prepared at the time” by Anne Chambers was also put to him. It included textual notes and a list of call references and branches affected by approximate date, which were all in 2004 and 2005. The notes were:
“NB many other branches had multiple events, preventing replication, but these are the majority of those which came to Peak, having either reported a problem or it caused a reconciliation report entry.
From Sept 2005, cash accounts were replaced by branch trading statements and the TPSC256 report was no longer populated. I can't remember how we then knew about receipts and payments mismatches and if we would have picked up on further issues.”
(emphasis added)
419. These notes make clear that there were other branches potentially affected, and the list of 30 branches compiled in the spreadsheet were “the majority”. It does not state that it is an exhaustive list and the notes suggest it is not. The summary for each branch listed gives a variety of consequences or effects. They include:
1. Receipts and Payments Mismatch.
2. Receipts and Payments Error.
3. Host Generated Cash Account Line.
4. Unable to roll Stock Units.
5. Reconciliation Report.
6. Slow running counter.
7. Pension and Allowance Report printouts.
8. Screen freezes.
9. Problems with declarations.
10. Unable to create AA.
11. Loss due to double transfer.
12. Transfers not showing on nodes.
13. CAC Lines not matching.
420. The spreadsheet was disclosed to the claimants on 27 February 2019, or very shortly before the Horizon Issues trial, and the properties shown on the electronic document identify Anne Chambers as the author, with a date of 22 December 2015.
422. Mr Godeseth was taken to the contents of two letters from the Post Office’s solicitors to the claimants’ solicitors. The first was dated 28 July 2016. This was the Post Office’s solicitors’ substantive response to the claimants’ letter of claim, is very lengthy (as was the letter of claim itself) and ran to 99 pages. The second letter was dated 11 January 2019. He agreed that certain important elements of the paragraphs in those letters explaining Callendar Square were incorrect. They were:
1. That the bug was only discovered in 2005.
2. That the bug had only affected one branch. The second letter stated in very clear terms that “The Falkirk/Callendar Square issue was only known to have affected that one branch.”
423. Not only were both those statements in the Post Office’s solicitors’ correspondence completely wrong, but the numerous documents at [398] to [421] above demonstrate that there were a great number of references, including in both KELs and PEAKs (documents of a type that had been at one stage claimed to be wholly irrelevant to the issues in this litigation) to the direct contrary.
424. Mr Godeseth’s witness statement about this, prior to his cross-examination, presented a very different picture to the one that eventually emerged. Indeed, I would go further, and I find that his witness statements omitted some very important headline points in respect of the Callendar Square bug, presented a chronology very different to the real one, and had the effect (whether intended or accidental) of giving a misleading impression of the Callendar Square bug and its impact.
425. I find that the headline points omitted by Mr Godeseth are as follows:
1. The Callendar Square bug had existed “probably” (Mr Godeseth’s word in cross-examination) since the inception of Horizon in 2000. There were numerous incidents of it occurring prior to 2005.
2. By February 2006, Anne Chambers and others at Fujitsu knew that this bug “had been around for years”.
3. It had an impact “most weeks”. Preparing a software fix was not seen as a priority.
4. The bug had directly impacted the branch accounts of at least 19 branches, and possibly more, as the Anne Chambers’ spreadsheet listed the 30 branches as “these are the majority”.
5. There were numerous effects of this bug, and at least 13 different ones identified in the spreadsheet.
6. Even in 2006, Horizon Support was telling at least one SPM that they “could not find any problem” with Horizon when the bug’s impacts were reported to it, and this was recorded in documents linked by Fujitsu directly with the Callendar Square bug.
7. Fujitsu had not established the total number of branches affected by consulting the events logs for the purposes of preparing Mr Godeseth’s witness statement, or otherwise. To represent the spreadsheet prepared by Anne Chambers as being the totality of branches affected by the bug is to ignore the text of the notes.
426. Finally on this point, the Post Office’s solicitors’ letters were obviously not factually inaccurate. Nor can the Post Office’s own legal team have known about the Anne Chambers’ spreadsheet until very early 2019, otherwise disclosure of such an important document would surely not have been given only on 27 February 2019, only seven working days before the Horizon Issues trial began. I do not know why the Anne Chambers’ spreadsheet of 22 December 2015 was only disclosed to the claimants a mere 7 working days before the start of the Horizon Issues trial, and it is not necessary to speculate.
427. Although the Callendar Square bug in Legacy Horizon could, amongst other effects, cause a receipts and payments mismatch, there is another bug in Horizon Online that also had the same effect. This was called the Receipts and Payments Mismatch bug and is the first entry in the Bug Table, Bug 1.
428. Mr Godeseth gave evidence about the Receipts and Payments Mismatch bug. This was a problem at the cache. His information came both from Mr Jenkins and his own investigations. He did not know about it at the time in 2010, and he did not in his witness statement refer to what is in my judgment a very important document concerning this bug, namely the Receipts/Payments Mismatch issue notes dated in the trial bundle index 17 October 2012, although it is likely to be a 2010 document as an associated document from Mr Jenkins (trial bundle reference F/1000/1 and F/1001/1) is dated 29 September 2010. I will reproduce certain parts of the issue notes, as I consider them to be highly relevant to the Horizon Issues. There were 10 attendees, including Mr Jenkins. The document was marked Commercial in Confidence and is, according to the trial bundle index, a memo of a meeting. Another document with the same date is headed “Correcting Accounts for “lost” Discrepancies” and states “any branch encountering the problem will have corrupted accounts”. The issues notes document stated:
“What is the issue?
Discrepancies showing at the Horizon counter disappear when the branch follows certain process steps, but will still show within the back end branch account. This is currently impacting circa 40 Branches since migration onto Horizon Online, with an overall cash value of circa £20k loss. This issue will only occur if a branch cancels the completion of the trading period, but within the same session continues to roll into a new balance period.
At this time we have not communicated with branches affected and we do not believe they are exploiting this bug intentionally. (emphasis added)
The problem occurs as part of the process when moving discrepancies on the Horizon System into Local Suspense.
When Discrepancies are found during Stock Unit rollover into a new Transaction Period, then the User is asked if the discrepancy should be moved to Local Suspense. If the branch presses cancel at this point the Discrepancy is zeroed on the Horizon System.
Note at this point nothing into feeds POLSAP and Credence, so in effect the POLSAP and Credence shows the discrepancy whereas the Horizon system in the branch doesn't. So the branch will then believe they have balanced.
If at the next screen the rollover is completely cancelled, then no harm is done. However if the Rollover is re-attempted at this point, the rollover will continue without any discrepancy meaning Horizon doesn't match POLSAP or Credence.
This has the following consequences:
• There will be a Receipts and Payment mismatch corresponding to the value of Discrepancies that were "lost"
Note the Branch will not get a prompt from the system to say there is Receipts and Payment mismatch, therefore the branch will believe they have balanced correctly.
• When the Branch begins the new Branch Trading period the discrepancies will show at Zero, however the Receipts and Payment mismatch will carry over into the next period.
Impact
• The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.
• Our accounting systems will be out of sync with what is recorded at the branch
• If widely known could cause a loss of confident in the Horizon System by branches
• Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data
• It could provide branches ammunition to blame Horizon for future discrepancies.
Identifying the issue and forward resolution
The Receipts and Payment mismatch will result in an error code being generated which will allow Fujitsu to isolate branches affected this by this problem, although this is not seen by the branches. We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May. They will provide feedback in due course.
Fujitsu are writing a code fix which stop the discrepancy disappearing from Horizon in the future. They are aiming to deliver this into test week commencing 4th October. With live proving at the model office week commencing 11th October. With full roll out to the network completed by the 21st of October. We have explored moving this forward and this is the earliest it can be released into live.
The code fix will on stop the issue occurring in the future, but it will not fix any current mismatch at branch.
Proposal for affected Branches
There are three potential solutions to apply to the impacted branches, the groups recommendation is that solution two should be progressed.
SOLUTION ONE - Alter the Horizon Branch figure at the counter to show the discrepancy. Fujitsu would have to manually write an entry value to the local branch account.
IMPACT - When the branch comes to complete next Trading Period they would have a discrepancy, which they would have to bring to account.
RISK- This has significant data integrity concerns and could lead to questions of "tampering" with the branch system and could generate questions around how the discrepancy was caused. This solution could have moral implications of Post Office changing branch data without informing the branch.
SOLUTION TWO - P&BA will journal values from the discrepancy account into the Customer Account and recover/refund via normal processes. This will need to be supported by an approved POL communication. Unlike the branch "POLSAP" remains in balance albeit with an account (discrepancies) that should be cleared.
IMPACT - Post Office will be required to explain the reason for a debt recovery/ refund even though there is no discrepancy at the branch.
RISK - Could potentially highlight to branches that Horizon can lose data.
SOLUTION THREE - It is decided not to correct the data in the branches (ie Post Office would prefer to write off the "lost"
IMPACT - Post office must absorb circa £20K loss
RISK - Huge moral implications to the integrity of the business, as there are agents that were potentially due a cash gain on their system.”
(bold present in original)
429. I find this to be a most disturbing document in the context of this group litigation. It is a 2010 document and issues between the Post Office and many SPMs concerning the accuracy of Horizon had, for Legacy Horizon, gone on for a decade (2000 to 2010) and these continued under Horizon Online (introduced in 2010). Under “Impact”, some of the bullet points incorporate a summary of these issues.
“• The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.
• Our accounting systems [ie Horizon or the Post Office’s] will be out of sync with what is recorded at the branch
• If widely known could cause a loss of confident in the Horizon System by branches
• Potential impact upon ongoing legal cases where branches are disputing the integrity of Horizon Data
• It could provide branches ammunition to blame Horizon for future discrepancies.”
(emphasis added)
430. The attendees at this meeting included at least one member of Post Office (rather than Fujitsu) personnel, Andrew Winn of POL Finance. There were obviously legal cases going on at the time, hence the reference in the underlined bullet point to “ongoing legal cases”. If these were criminal cases, the Post Office would be the prosecuting authority, with certain important duties. If these were civil cases, the Post Office would be a party with disclosure obligations. An affected branch would believe it had balanced its accounts correctly; it would not have done so. There is an evident concern amongst those at the meeting which is recorded in this document that this issue should not become “widely known” in order to avoid causing “a loss of confidence in the Horizon System”. Fujitsu do not seem to have been particularly prompt in either identifying the problem or reacting to it.
431. Of the three solutions considered, the one that was adopted was issuing a debt recovery/refund even though there would be no discrepancy at the branch. Given one of the Common Issues was the status of a Branch Trading Statement, which the Post Office argued should legally have the status of a settled account with the SPM as agent, I note the suggestion adopted with more than passing interest. The solution explains “even though there is no discrepancy at the branch”. This means the Branch Trading Statement would be correct, or in balance. The discrepancy would not be in the Branch Trading Statement, it would be in POLSAP or Credence. That this is correct is shown in an associated document from Mr Jenkins (trial bundle reference F/777/2) where he stated that “the data used for the BTS will also have a zero value for Discrepancies at the end of the period.” BTS means Branch Trading Statement. A similar entry is in the document at F/1000/1 which states at /2 “Note that if the bug was not present, then the Discrepancy would have been transferred to Local Suspense and that would have been cleared, so there are a number of things wrong with the BTS.” (emphasis added). In other words, the Post Office itself was not considering the BTS as having the status of a settled account, based on these entries.
432. Further, an associated document entitled “Receipts and Payments Mismatch” states:
“1. The purpose of this note is to document a request that we have had from Post Office in terms of presenting details of what happened as a result of a bug in HNG-X in September 2010 which caused a Receipts and Payments mismatch and also resulted in Discrepancies being lost.
The background to this is the fact that there was a BBC documentary broadcast on Monday 7th February 2011 reporting on postmasters being unhappy about being pursued for losses by Postmasters on Horizon.
It should be noted that the issues described here relates to HNG-X (Horizon Online) and that the implementation of the accounting mechanisms in the two systems is totally different (but they do produce the same reports and support the same business process).”
(emphasis added)
433. The two references – one to “ongoing legal cases”, the second to a BBC documentary – show that there was a distinct sensitivity within both the Post Office and Fujitsu about keeping this information to themselves in order to avoid a “loss of confidence” in Horizon and the integrity of its data. A less complimentary (though accurate) way of putting it would be to enable the Post Office to continue to assert the integrity of Horizon, and avoid publicly acknowledging the presence of a software bug. The solution adopted was the issuing of TCs to the relevant branches. There was no publicity given to the SPMs at the time about the presence of this software bug in Horizon. An entry at F/1000/3 states:
“It should be noted that as Discrepancies are normally Losses, then a Lost Discrepancy would normally work in the Branches favour and so there is no incentive for the branch to report the problem. Also if we do amend the data to re-introduce the Discrepancy, this will need to be carefully communicated to the Branches to avoid questions about the system integrity.”
(emphasis added)
434. Mr Godeseth was asked about the lack of communication of this.
“Q. Are you surprised that this had not been communicated so people would just be warned about it?
A. There was obviously a fear that subpostmasters may be looking to exploit this because it gave -- there was a fear that people could see this as a way of defrauding the Post Office.
Q. So concealing it from SPMs who were honest was justified because of the expectation of dishonesty of subpostmasters in the network, in a nutshell?
A. In my view, this was a decision made by Post Office on how to manage this particular bug. You could interpret it the way that you have put it.
Q. Do you agree with the way that I have put it to you?
A. I think I'm agnostic. I can see the -- I can see a rationale for not broadcasting this, but equally, if the objective is to be totally open and honest and take the risk of causing more chaos in the network then yes, I would have to agree with that.”
435. Indeed, the approach of the Post Office to notification of the existence of bugs to SPMs is an illuminating one. The Post Office communicates with SPMs generally in a number of ways. Methods that have been referred to in both of the Common Issues trial and also the Horizon Issues trial include “Branch News”, a type of newspaper that was used, and also “Memo View”, a pop up message that appears on Horizon terminals. During the preparation of this judgment, in view of the fact that judgments are required to be factually accurate, because dates can sometimes become lost in the midst of huge trial bundles and very detailed submissions, and also because the simplest way to arrive at a date when something has been done is (sometimes) simply to ask the parties for a date or a reference, on 7 October 2019 the parties were jointly asked the following:
“For each of the Dalmellington and Callendar Square bugs, could the parties please indicate (by means of an agreed fact but only if that is possible, if not each party’s contended for answer) their answers to the following two questions:
1. The date when the existence of each of these was communicated by the Post Office to SPMs;
2. A trial bundle reference to a contemporaneous document, email or communication from the Post Office demonstrating question 1.”
436. The parties could not agree on a date when the existence of these two bugs – Callendar Square being an acknowledged bug in the Bug Table dealt with below (from the 2nd Joint Statement of the two experts) and Dalmellington, which is now accepted to be a bug – was communicated to SPMs. The claimants’ answer to the two questions was the same for each bug, namely that there was “no date in any contemporaneous document from the Post Office in the trial bundle showing that such communication in fact took place and identifying the actual content of any such communication.” The Post Office’s answer was as follows:
437. For Callendar Square: “It has never been Post Office’s case that, once it was discovered, the existence of the Callendar Square bug was communicated by Post Office to all SPMs or communicated to SPMs who had been affected. Post Office's case is that it was unnecessary and would have been inappropriate to do so.” A number of references and further explanation then followed, stating that when “an SPM was affected by the bug, this was detected and corrected in the ordinary course” and also that “In some cases, SPMs who were affected by the Callendar Square bug may have been made aware that there was a problem in Horizon which had caused an error in their accounts which required correction.”
438. For Dalmellington: “It has never been Post Office’s case that, once it was discovered, the existence of the Dalmellington bug was communicated by Post Office to all SPMs or communicated to the SPMs who had been affected. Post Office's position is that it was unnecessary and would have been inappropriate to do so.” The Post Office also stated that “when an SPM was affected by the bug this was detected and corrected in the ordinary course” and that the Dalmellington bug “was prevented, by effective counter-measures, from causing even a single impact on branch accounts”.
439. The Post Office also stated that there were real disadvantages to notifying SPMs of this. The Post Office relied upon passages in Dr Worden’s report that summarise reasons why users of an IT system should not be given information about parts of that system which they did not encounter in their daily work; that there was “no point in trying to educate all the users in details and terminology of the system which will never concern them”; that the best thing to do is try and fix a bug or defect, rather than “create some new error message to the users”; that “automated messages from the system are only of limited help to users”; and other points in similar vein. The Post Office also provided documentary references to instances where “an SPM might have been made aware”, although these documents were not deployed in the evidence.
440. I will reproduce two entries from one of these documents, a PEAK with reference PC0103864. This shows that on 3 June 2004 an SPM reported “that he had a problem with some transfers yesterday, he was transferring stock and cash between the aa main stock unit and the bb shared stock unit and although only one transaction shows for the transfer out the transactions were transferred into the bb stock unit twice giving the pm a discrepancy.”
441. The discrepancy in the cash account was £22,290 in total (shown in the entry for 8 June 2004) although the reconciliation error reported by the Host was £44,580. This was finally resolved on 5 August 2004 with the issue of an error notice. There is nothing to suggest that the SPM was told that this had been caused by a software bug.
442. The Post Office’s approach to this, in my judgment, entirely misses the point. In my judgment, the above passages are simply extraordinary. Two of the documents above are dated 2010, some 9 years ago. The PEAK is from 2004, 15 years ago. Their contents support the claimants’ case on the Horizon Issues. Fujitsu knew, to take Callendar Square as an example, that this bug existed in Horizon. They knew that it had affected branch accounts. It was not, as the Post Office puts it “unnecessary and inappropriate” to notify SPMs of this. I have listed the points on this bug at [425] above omitted by Mr Godeseth from his written evidence. Those same points all lead to the same conclusion in my judgment, namely that the Post Office ought to have notified, at the very least, all those SPMs whose branch accounts had been impacted by this bug that this had occurred, and that it had occurred as a result of a software bug. The fact that the integrity of Horizon data was a live issue at this time should not have influenced the decision to notify SPMs of a software bug. Further, the Post Office’s explanation in its submissions that SPMs had their accounts “corrected in the ordinary course” is not a suitable phrase, unless by “ordinary course” one means keeping the cause or reason for the correction secret and therefore hidden from the other party in the accounting transaction, namely the SPM. Also, one is not educating users in the details and terminology of the system (as suggested by Dr Worden) if one informs them that there is a software bug in the system and its symptoms are as follows. This is relevant to Horizon Issue 2, namely whether the Horizon IT system itself alerted SPMs of such bugs, errors or defects as described in Issue 1 above and if so how, and also Horizon Issue 9, which concerns transaction data and reporting functions available to SPMs. It is also relevant to Dr Worden’s consideration of countermeasures, which he considers includes vigilance by users (which means SPMs).
443. Mr Godeseth gave express evidence in his witness statement that the Receipts and Payments mismatch bug occurred in September 2010. That date too was factually incorrect. The issue notes refer to Fujitsu knowing about it far earlier, and Mr Godeseth accepted he had seen this document before his cross-examination. That document even records Fujitsu being taken to task by the Post Office about how long it had taken to react, as in “We have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May. They will provide feedback in due course.” Not just the impression, but the express text in Mr Godeseth’s witness statement, was to the effect that the bug was discovered in September 2010 and almost immediately dealt with. That was far from the case, and that written evidence was simply wrong.
444. Mr Godeseth gave evidence about another bug, called the Local Suspense Bug, also in Horizon Online. His witness statement quoted almost word for word from a contemporaneous document prepared by Mr Jenkins. Almost all his information on this bug had come from Mr Jenkins. The bug caused entries from local suspense accounts in 2010 to be reproduced in Horizon in two successive years. What was supposed to be only temporary data was retained, not deleted as it ought to have been, and the system then used it again. It affected certain tables in the branch database and the archiving strategy of deleted Stock Units. This affected 15 branches. The effects of the bug were brought to the attention of the Post Office by two SPMs “with the largest discrepancies” and Fujitsu became aware of it in January 2013. However, the note by Mr Jenkins of 15 May 2013 summarising it states that the problem was known about far earlier than that:
“In April 2011 a problem was found with the archiving strategy related to Stock Units that have been deleted in a Branch. A consequence of this is that some changes were made to the archiving strategy on 3rd July 2011. An unintended consequence of this change was that any Branch that deleted a Stock Unit at the end of 2010 which had local suspense transaction in that Stock Unit before it was deleted were left in the table used for constructing the BTS. This meant that as Trading Periods cycle around each year, these BTS records became visible in 2011 when the same Trading Period was reached.
The effect of these old records was that after the BTS was produced an incorrect figure was generated for the Opening Balance of the Local Suspense Account for the following period. This amount corresponded to the value of the historical record. These orphaned records were created between 16th November 2010 and 9th December 2010.
When the next Trading Period was balanced, then this incorrect Opening Figure would result in the total value for Local Suspense being calculated incorrectly and the SPMR being asked to make good an incorrect amount. It is at this point that transactions would be generated into the audit trail reflecting the fact that the SPMR had cleared the Local Suspense account for an incorrect amount. The audit trail operated correctly in the sense that it accurately recorded the transaction on the system.
This problem was not reported to Fujitsu in 2011/12 and only affected a small number of Branches and only for a single Trading Period. However the two branches with the largest discrepancies did report the issue to Post Office Ltd who could see the impact of the problem in their back end system and wrote off the loss or gain for the branch but did not ask Fujitsu to investigate further.
At the same Trading Period in 2012/13, the problem re-occurred and this time one of the affect Branches reported the problem to Fujitsu on 25th February 2013 (PEAK 223870) resulting in a detailed analysis of this issue and finding the orphaned BTS records. The root cause was determined by 28th February 2013 and a preliminary report was sent to Post Office Ltd. A further update was sent on 14th March 2013 with a full analysis of the issue and all the affected branches.”
(emphasis added)
446. The Dalmellington Bug related to Horizon Online and actually included two potentially separate issues, and the combination of these. One related to what is called Forced Log Out, when the Post Log On script was not correctly closed down and was left on the stack of incomplete processes. The other related to the way the Pouch Delivery script operated; because the Post Log On script had not correctly closed down meaning the stack was not empty, the Pouch Delivery script thought it had not finished and attempted to repeat the last part of the script. This had the effect of recording the remittance transactions and printing of receipts. This would lead to duplicate pouch IDs. Pouches have bar codes that are scanned and each is supposed to have a unique ID. The duplicate pouch ID would have a value attached to it – the sum in the original pouch. It would however be recorded twice as a result of this.
447. What happened at Dalmellington was the SPM scanned £8,000 from her core branch into her outreach branch. She was lucky in this sense, in my judgment, in that she “controlled” both ends of the transaction. The transfer replicated four times. She had Horizon receipts in her outreach branch of £32,000 (4 times £8,000) and therefore a discrepancy of £24,000, as she had only transferred £8,000 in cash out of her core branch into it. This was investigated by SSC and she was issued with a TC a few weeks after it had occurred. This was caused by a software bug, and the bug is now admitted by the Post Office, although it describes it as having “transient impact”. That simply means it was corrected.
448. When investigating this Fujitsu found 112 occurrences affecting 88 different branches in the previous 5 years. They were as follows, taken from an internal Fujitsu presentation dealing with this:
Feb 2010 to Jan 2011 65 incidents
2011 6 incidents
2012 9 incidents
2013 7 incidents
2014 9 incidents
2015 16 incidents
449. The same document showed fixes applied in April 2010, January 2011 and January 2016. It also showed only one call to SSC at Fujitsu in 2015, and none for any of the years 2010 to 2014. The presentation also showed that some branches that were affected were so-called “mediation branches” – in other words the SPMs would have been in the Second Sight mediation scheme that was brought to an end – but that none of the dates when those branches were affected matched the “mediation dates”, which means the periods for which ARQ data was to be used in the mediations to analyse exactly what had happened at each branch. There is nothing in the Fujitsu document used by Mr Godeseth for his evidence, to suggest that this analysis led to each branch being told of these software bug occurrences outside the mediation date range, and the items were usually corrected by means of TCs.
451. In his re-examination, some of which I have already referred to in the context of UNIX user and privileges above, Mr Godeseth also said that having seen the OCPs and OCRs in cross-examination, he accepted data had been deleted but did not consider it to be transaction data. Initially he described it as “operational data” but said that was too wide a term, and:
“There possibly ought to be a different definition for it because equally data in the database which is telling you which stock unit, you know, what -- which TP [ie trading period] you're in, whether a stock unit is in use, all of those are operational data which do not go down and impact the transaction data, so for me, the definition of transaction data -- I should be able to get from one opening -- one set of opening figures to the end opening figures by going back to the transaction data in its absolutely raw form.”
452. He said it was what one went through to bring different totals together “to get to your end position in a trading statement.” Regardless therefore of whether the data which he accepted was being deleted is, or is not, properly described as “transaction data”, it would feed into the end position in a branch trading statement. It would therefore, in my judgment, have an impact upon branch accounts, regardless of the term used to categorise it.
453. By the end of Mr Godeseth’s evidence, and by virtue of the way he answered questions, the reality of the development and operation of both Legacy Horizon and Horizon Online was far clearer than it had been before the Horizon Issues trial. I found Mr Godeseth to be a truthful, credible and helpful witness in his oral evidence. His evidence was of considerable assistance; he did not always answer questions the first time he was asked, but the final answer would be reached. Given his role throughout the whole of the life both of Legacy Horizon and Horizon Online, I found his evidence in cross-examination to be of central relevance to the Horizon Issues. His evidence in cross-examination was also, in my judgment, broadly supportive of the claimants’ general case.
454. There are, however, two aspects of Mr Godeseth’s evidence – though not his performance as a witness in the witness box - that were highly unsatisfactory and cannot pass without comment. The first aspect relates to his written evidence. The totality of his evidence, after his cross-examination, bore very little semblance to the picture that had been portrayed in his written witness statements. On some extremely important factual matters, such as the dates when Fujitsu had become aware of a particular bug, or the spreadsheet exercise by Anne Chambers prepared in 2015 (and disclosed in 2019), his written evidence was simply directly wrong. On others, such as the headline points omitted on the Callendar Square Bug that have been set out in [425] above, very important central elements detrimental to Fujitsu were simply omitted.
455. Legacy Horizon had, as has been explained, started life originally as something rather different to what it became, having initially been intended as a tri-partite project involving payment of benefits. It did not unfold in this way and became rather different. Horizon Online also did not have a happy birth. The pilot for it had to be stopped, and Fujitsu put it on what was called “red alert”. Mr Godeseth described this as “very serious”. The biggest issue was with Oracle, which was what Mr Godeseth was working on and hence knew the most about, but he explained that there were other problems going on at the same time. Some of these problems were put to him – and it must be remembered that this was a pilot scheme, with some problems to be expected – and they included cash being short on one day by £1,000 because a transaction for £1,000 did not show up on the online report facility; cash withdrawals being authorised on screen yet the printed receipt being declined (the customer very honestly brought the cash back next day having noticed the receipt wording); a similar problem with a cash deposit; and remming in figures all being doubled up. These are all somewhat – and indeed markedly - similar to some of the problems alleged by the different claimants’ witnesses in this litigation. These all occurred during the pilot scheme.
456. Not one of these different problems was referred to in Mr Godeseth’s witness statements, of which there were three. Witness statements are supposed to be accurate, and in a case such as this one with such centrally important issues, accuracy is clearly important. Quoting only selectively from, or wholly ignoring, contemporaneous documents prepared by (say) Mr Jenkins, who was the extensive source of much of the evidence, is not only unhelpful, it presents an entirely misleading evidential picture. It is not necessary to consider further how many personnel at Fujitsu may have assisted Mr Godeseth in producing such documents. Their content was wholly misleading in their original written form. Fortunately the cross-examination of Mr Godeseth led to a far clearer picture in so far as his evidence is concerned.
457. The second unsatisfactory aspect which arose from his evidence is the approach of Fujitsu as demonstrated in various documents, including the PEAKs and KELs, but also in particular in the Receipts/Payments Mismatch issue notes. To see a concern expressed that if a software bug in Horizon were to become widely known about it might have a potential impact upon “ongoing legal cases” where the integrity of Horizon Data was a central issue, is a very concerning entry to read in a contemporaneous document. Whether these were legal cases concerning civil claims, or criminal cases, there are obligations upon parties in terms of disclosure. So far as criminal cases are concerned, these concern the liberty of the person, and disclosure duties are rightly high. I do not understand the motivation in keeping this type of matter, recorded in these documents, hidden from view; regardless of the motivation, doing so was wholly wrong. There can be no proper explanation for keeping the existence of a software bug in Horizon secret in these circumstances.
458. The degree to which either, or both of, Fujitsu and/or the Post Office, expressly or constructively, knew exactly what and when, is for future trials in this litigation, and I make no findings in that respect in this judgment. They are not necessary in order to resolve the Horizon Issues and I do not speculate.
459. In my judgment, however, there are sufficient entries in the contemporaneous documents to demonstrate not only that Fujitsu has been less than forthcoming in identifying the problems that have been experienced over the years, but rather the opposite. The majority of problems and defects which counsel put to Mr Godeseth, and which were effectively admitted by him, simply would not have seen the light of day without this group litigation.
460. I do not know if this is because of concerns about the future of Horizon going forwards; or for Fujitsu to protect its corporate reputation; or for some other reason. It is unnecessary to speculate, and I do not do so. An internal Post Office IT risk management document from 2017 stated that “the HNG-X platform is end of life and is running on unsupported Windows software”, that it needs replacing, and also that the "Branch counter technology is aged and unreliable, with frequent hardware failures, resulting in branch disruptions." Mr Godeseth agreed with all of this. The unsupported platform is Windows NT4. As Mr Godeseth put it, “any technologist would tell you that was too old”. Windows NT4 is an operating system that was released in 1996, and Microsoft ended mainstream support for it in 2002 and extended support in 2004. Horizon now runs, not on NT4, but on Windows 10 and it is called HNG-A rather than HNG-X. This change to Windows 10 occurred in February 2017 and the roll out was administered by Computacentre, with Fujitsu providing the software. The Horizon system of 2019 is therefore a very different system than both Legacy Horizon and the earlier version of Horizon Online, HNG-X.
461. The original Horizon Online system was introduced in 2010, and was even then substantially based on the existing system, Legacy Horizon, itself by then at least 10 years old. That had been introduced in 2000, and therefore it should not be controversial to describe Horizon Online in its HNG-X form as old technology. There have of course been upgrades and other improvements since 2010, and other software additions and add-ons, but Horizon will by necessity require replacement at some point. I detected a degree of sensitivity in the litigation generally about any plans for replacing Horizon Online, as though the Post Office feared that any hint that it might be replaced would be seized upon by the claimants (and/or assumed by the court) as accepting that the criticisms of Horizon Online by the claimants would be corroborated by any Post Office plans for its replacement. Such fears, if there are any, are groundless, at least from the court’s point of view. It is inevitable that technology systems, particularly complex systems from as long ago (in technology terms) as 2010 will need replacing at some point. Technology moves at an increasingly rapid pace. There is reference in some of the documents in the trial bundle, but not in the evidence, to something called “Thin Client”, about which Dr Worden said he had no knowledge. He thought it was a cloud based system. It is unnecessary to speculate about this either. It is sufficient to state that, in my judgment, any plans there may be for replacement of Horizon Online in the ordinary course of the Post Office’s wishes to improve its business are not something that I need to consider in order to resolve the Horizon Issues.
462. The degree to which Mr Godeseth’s evidence affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Parker
465. He was originally a support consultant within the SSC, and stated that he was effectively the deputy manager of the SSC, providing 3rd line support. He was lead designer and part of development team for the internal website providing the support knowledge database (which is also called KEL), technical documentation management and operational change control. He assisted the SSC manager in the provision of the support service and operational management. He became the Manager of the SSC in March 2010. The SSC later expanded and provided support services to other Fujitsu customers, but the Post Office Horizon system is still the largest. He has between 25 and 40 staff in the SSC whom he manages. He said that he was deputy manager of the SSC when Mr Roll worked there, although he did not have that title formally, and that people in the SSC would know this because he would stand in for the manager in his absence, and make decisions on approving actions for him.
“We understand from Fujitsu that the SSC has been carrying out further work to identify any PEAKs that show transactions being injected at the counter in Legacy Horizon in addition to those referred to in paragraphs 29 and 30 of Mr Parker's second statement. On Monday [ie the 6th day of the Horizon Issues trial] we learned that an SSC technician has:-
• searched for all KELs that mentioned "RiposteMessageFile", "Ripostelmport" and "RiposteMessage';
• collated the responsive KEL references (AChambers2226M, CObeng1029162824, DSeddon822M, MYoung5043M, RColeman1250R, acha2340K, ballantj498J, dsed344J, AHolmes3343J, DSeddon1753N, GMaxwell46141, PCarroll12541, RKing5135L, 81111, pcar847S, wbra716s);
• re-searched the Peak system for any PEAKs which contained those KEL references; and
• identified the following PEAKs: PC0105560, PC0106885, PC0063599, PC0063871, PC0065796, PC0066061 and PC0083998 (which cross refers to PC0076029).
We understand that the PEAKs referred to above relate to either:-
• correcting configuration data after a PinPad change; or
• reference data.
We also learned that during this exercise the SSC technician has also identified three examples of the marooned transaction scenario described in paragraph 38.2 of Mr Parker's second statement: PC0068495, PC0099141 and PC0079196.”
467. The search terms identified in the letter were different to the ones that Mr Parker had used in his 2nd witness statement. These were "RiposteMessageFile", "RiposteMessage", "LPO Delete", "Marooned" and "RiposteObject put". The reason this is important is the command “RiposteImport”, used by the SSC as a search term during the week commencing 18 March 2019, but not used for the purposes of Mr Parker’s 2nd witness statement in January 2019, is actually one of the commands that was in fact used to inject transactions. In other words, the exercise carried out for the purposes of the witness statement was wholly deficient, in that it did not search for one of the specific terms used to inject transactions.
468. When asked to explain this in his cross-examination, and the corrections that he made, Mr Parker stated the following:
“A I notified the legal team that Mr Simpkins had done some more work and as a result of that we changed -- we -- this letter was actually generated.
Q And you didn't think it was important to correct paragraph 29 of your second witness statement accurately to reflect the situation as it would have been on the day you were due to give evidence, or indeed today?
A That wasn't a choice I made personally. I was advised that we generated this letter.”
(emphasis added)
He did not say who made the choice to which he referred, namely not to correct his witness statement.
469. It certainly seems to me highly curious – at best - that the exercise performed for Mr Parker’s 2nd statement, by personnel at SSC who are expert in such things, should have omitted using the search term “RiposteImport”. Given the purpose of paragraph 29 and 30 of Mr Parker’s 2nd witness statement was to identify how many times data had been in fact injected, a search exercise that did not use “RiposteImport” was never going to give the full picture. That initial search exercise was plainly inadequate. It is also rather curious that, having done that, the exercise that was described in the letter of 20 March 2019 from the Post Office’s solicitors that correctly used “RiposteImport” as a search term should have been done so very closely before Mr Parker was originally due to be called as a witness, namely 21 March 2019. In re-examination, Mr Parker said that “the thought occurred to him that he could add some other search terms into the work that he had done previously” and “he came to me with the data”. Why this thought should have suddenly occurred to the person it occurred to, who was not a witness, and at the time that it did, could not be pursued. Mr Parker denied that he was trying to conceal anything, and said he was just trying “to get the right data for the court which can be difficult sometimes when you are going back fifteen years.” This explanation overlooks that the decision not to use “RiposteImport” was one taken for his witness statements. His first witness statement was dated 16 November 2018; his second was dated 29 January 2019, and his third was 28 February 2019. None of these are “going back fifteen years”.
470. Mr Parker had also given evidence in his 1st witness statement in respect of Mr Roll’s written evidence that Fujitsu could inject and affect branch transaction data without the knowledge of the SPM. He had originally said the following:
"As I explain below, those suggestions are incorrect and Mr Roll's account of Fujitsu's actions and powers is inaccurate and misleading".
471. He explained this paragraph as follows in his cross-examination.
“A. In that paragraph I am trying to make the point that the suggestion that we frequently changed branch transaction data without informing the branches that such actions were being actually taken is not correct. "Frequently" is a subjective term but I would not have described the rate at which we were changing branch transaction data as "frequently".”
472. One has only to read the original paragraphs as drafted to see that Mr Parker was not originally taking issue with frequency, or how often such actions were taken; he was stating in express terms that this could not be done by Fujitsu at all. For someone who has effectively spent the best part of 20 years in SSC, rising to a very senior post, such evidence must have been – and I find that it was - quite obviously incorrect. His explanation of this was as follows:
“Q Now, given the change of your evidence in your later statements about the ability remotely to access and inject transaction data, can you explain to the court what you were saying you believed about that sentence? What should the court now read for those words in the light of the three statements that it has before it?
A If I take the last sentence in isolation, which is what I think you are asking me to do, then I don't understand how I apply it, because I am -- I have been simply trying to say there that the frequency was not high, and that we would always involve the Subpostmaster wherever possible if that sort of action was actually being taken. That's what I'm trying to say by that.”
473. This was a very poor attempt by Mr Parker at explaining away quite directly incorrect factual evidence in a witness statement on a very important area in the litigation, namely Fujitsu’s ability to inject transactions remotely. Mr Parker’s only recourse was to claim he had not been saying what he plainly had been saying in his 1st witness statement – that Fujitsu did not have the power to do it, and that Mr Roll’s evidence to the contrary was inaccurate and misleading. The power to do something cannot be equated with frequency.
474. Mr Parker also accepted that at the time he signed his 1st witness statement he had known that Fujitsu had the power to insert transactions into individual branch counters by using the correspondence server, a process Mr Roll described as “to piggy back through the gateway”. It was put to him that he had been giving his evidence initially from a position of not being very well informed about what could or could not be done. He denied that; he said “that would be wrong” and that in general terms he was confident of the information he gave. The obvious conclusion that I draw from this is that Mr Parker chose specifically to give the impression in his 1st witness statement that Fujitsu did not have the power (the word Mr Parker expressly chose) to inject transactions into the counter at branches, even though he knew that it did. This paints him in a very poor light as a credible witness. There is also no adequate explanation for why Mr Parker’s colleague should have decided, in the middle of the trial, to use the “RiposteImport” search term; equally, there is no adequate explanation for why this search term was not used for the exercise for Mr Parker’s witness statement. Injection of messages into the counter has been an extremely central issue in the long running dispute between the Post Office and its SPMs for a long time, and Mr Parker’s reliance in his re-examination upon the passage of time and the difficulty of “getting the right data” for the court is, in my judgment, a poor excuse. In reality, he had no credible explanation.
475. A series of propositions was put to him about the degree of expertise expected of all the members of SSC 3rd line support. He did not agree with all of the points put, and in particular did not agree that all those in the 3rd line support role would be expected to have detailed knowledge of the system, based on both documentation and the inspection of source code; or that they would be trained in the coding languages used within the application. He was, however, left rather exposed in terms of the accuracy of this evidence when it was shown that the points that he had been addressing in this part of the cross-examination – some of which he would not accept - were taken from a Fujitsu internal document, entitled “End to End Application Support Strategy” marked Fujitsu Restricted and Commercial in Confidence which he had himself drafted. Indeed, it is page 1 of that document that specifically identifies Mr Parker himself as the author of the document, and it was approved by the Head of Application Services at Fujitsu, an obviously very senior role. This refusal by Mr Parker to accept his own previously drafted points in what I consider to be an important contemporaneous document also paints him in a very poor light as a credible witness.
476. He accepted that Mr Roll was conscientious, skilled and experienced, and that he had given Mr Roll a reference when he left Fujitsu.
477. He, and others not identified in his statement (although he said who they were when he was asked, Mr John Simpkins and Mr Mark Wright), had prepared a spreadsheet that had been put to Mr Roll showing the total number of calls to SSC between 2001 and 2004. Mr Simpkins, for what it is worth, appears to have been the person behind the exercise referred to in the letter at [466] above, so he was obviously involved both in preparing evidence and doing searches. Mr Parker agreed that SSC during Mr Roll’s time needed to address coding issues, even if they did not necessarily have an estate wide impact, and he also agreed that the list of PEAKs that had been identified as involving Mr Roll was only those where he had been the person who put the final response on the PEAK, rather than where he had worked on the PEAK at all.
478. He also accepted that in categorising “software errors” he had excluded from the count those that were categorised as Avoidance Action, Administrative Response and Solicited Known Error.
479. He was taken to a PEAK in 1999 where an initial balance was multiplied twice “due to a known software error” in the amount of a discrepancy of over one million pounds - £1,082,544,32 to be exact. The PEAK ran to 14 pages. 10 people worked on it including those in Development as well as the SSC. He accepted that this would have involved people looking into code. It was, however, categorised as “Administrative Response”. That PEAK alone demonstrates to my satisfaction the lack of inaccuracy in the Fujitsu closure codes, but there were many others. Other PEAKs to which he was taken showed reconciliation errors, and a note by the developer identifying that what were “up to 10 unnecessary reconciliation errors each week” and that the PEAK was a regression of another PEAK; the investigation into it led to the conclusion that a subsequent software release had not caught a fix. It stated "Risks (of releasing and of not releasing proposed fix): Without this fix, there will be possibilities of system errors at counter and while doing reversal transaction". This software fix was “released to live”, which means the fix was actually written and introduced into the system. Mr Parker had not included this in his software category either, because the final person working on it had categorised this as Administrative Response.
480. In another PEAK, it was recorded that “the system is still playing up in that the screen is hanging in the middle of transactions -- PM did transaction ... but left office for 1 hour -- when he came back the monitor had 141 first-class stamps on screen totalling £38.07". Mr Parker accepted that this was not how the system was supposed to work, but the same PEAK also recorded the SPM becoming upset because the SSC were advising it was user error. The SPM wished to make a complaint about the person at SSC and the PEAK records “we both feel that this PM is complaining unjustly”.
481. The same PEAK recorded that ROMEC engineers had been to the site, and had actually seen the problem themselves, which were phantom transactions. It recorded “it’s not just the PM’s word now”. Indeed, this PEAK is particularly concerning, in terms of the accuracy of PEAK closures by Fujitsu. It records in one of the Fujitsu entries the following:
“I now have pressing evidence to suggest that unwanted peripheral input is occurring, the likely source being the screen. This has been seen at Old Isleworth ...When the PM has been asked to leave the screen on overnight I have observed system activity corresponding to screen presses happening with no corresponding evidence of either routine system activity or human interference. The way forward now is to correlate this with Microtouch complied monitoring software and to this ends Wendy is arranging for installation of the kit on Friday..."
However, notwithstanding this, the subsequent entries include “Phantom transactions have not been proven in circumstances which preclude user error" and “In all cases where these have occurred a user error related cause can be attributed to the phenomenon?”.
482. The use of the phrase “in all cases” in respect of attributing user error wholly ignores that the ROMEC engineer is said to have seen the problem themselves, and it is not simply the word of the SPM in this instance. It ignores the activity observed overnight by the Fujitsu employee who made the entry in the PEAK, when the branch was not even open. Also, there is no reason, given the lengthy entries and studying the text throughout the PEAK in detail, why user error should be reached as a conclusion at all. This appears to be been used simply as an excuse to avoid an adverse conclusion to Fujitsu.
483. Notwithstanding the detailed entries in the PEAK itself that in my judgment plainly point to the contrary, the PEAK was still categorised as No Fault in Product.
484. In another, dealing with phantom transactions, one branch was described as “a problem office” that was accordingly being “monitored”. The PEAK recorded that “there have been incidents which showed a possible correlation between system activity and phantom transactions. These pointed to a touch screen problem and as a result the screen was replaced with a resistive model. As this produced no measurable improvement it has to be assumed that the problems were user related". This was also closed as No Fault in Product.
485. A KEL on phantom transactions stated "There have been several calls over the last few months where Postmasters have reported phantom sales. Items appear by themselves for which the PM has not pressed an Icon. These may be individual items or several of the same item. Sometimes when no one has been near the screen items may appear". These are similar problems being reported by a number of different SPMs. That alone should have suggested, not as a matter of computer expert evidence but simply basic common sense, that something was amiss. If a number of different and unconnected people all report the same symptom, then that could well suggest a common problem to them all. However, the KEL, which ran from 2000 to 2004, suggested that the cable between the screen and the base unit was the root cause. The dates on this KEL show that, even on the face of the document, the problem was manifested for some years.
486. Another PEAK that showed a problem described as “non-zero trading position on office rollover” was put to Mr Parker. This had been given a priority C status by Fujitsu, a status he accepted meant it was not taken account in the Service Level Agreement with the Post Office in liquidated damages thresholds (a type of financial penalty), which was a feature of those software PEAKs given higher priority status A and B. Status C also was non-critical. The description of the problem was described as “PM states that he has rolled over but the system is telling him that he hasn't -- PM states that he is in balance period 7 and he states he is getting the message 'wrong trading period MSG 31318 office balancing error'" and "Non-zero trading position ... on rollover of branch by user WMC002 to trading period 8".
487. Mr Parker described himself as “hesitant” to describe this as a payments mismatch issue, and said it was an “office balancing error”, even though later in the PEAK senior personnel at SSC recorded “NBSC have ruled out user error” and also that "The problem occurred on 15/09/10 when stock unit 02 rolled over. This was originally reported, as per KEL, BALLANTJL759Q, in call PC0204537 ... but for some reason the call was closed without being investigated. There is a known problem with the use of the Cancel button during the stock unit rollover. This is fully described in KEL WRIGHTM33145J".
(emphasis added)
489. Another PEAK also identified as category C, non-critical, related to pouches being remmed in twice, the duplicate pouch being £25,000 in value. The PEAK stated “this might be a software issue” but also “PM is unsure if this is user error if it is a software fault. NBSC have not confirmed if it was user error or if they can reverse it.” One entry stated "POL have been informed of the error. Hopefully they'll issue a TC to correct loss at the branch. The underlying problem caused by using previous button during or just after scanning pouch barcodes, is still under investigation". This was closed as a Solicited Known Error.
490. One PEAK which I found most illuminating, in terms of Mr Parker’s response to it when he was asked questions, is PEAK PC0229446. This is dated 17 November 2013, hence is Horizon Online, not in the period of the pilot project and sufficiently after the introduction of it, in 2010, for it not to be put down as the sort of issue that might occur in very early days of a system whilst minor issues were ironed out. The problem was described in the PEAK as “PM doing cash declarations every now and again has major loss.” Entries in the PEAK included "PM has had cash declaration problem throughout the year and is losing a lot every now and again"; "He phoned up helpline told him can't [have] declared properly. He states that he loses £2,000 then jumps suddenly to £4,000, one point they lost £8,000 and is always losing money. PM stated that he has three post offices, only happens on this site"; and then "Done a declaration this morning and had a £6,000 also. It shows no error message when doing it. No report prints out only print-out of cash declarations". This passage of his cross-examination followed this:
“Q. Pausing there, if the PM is correctly reporting that, then that would be very serious for the Postmistress or Postmaster, wouldn't it?
A. If it is being correctly reported, yes.
Q And it would not be the system working as it should.
A If we attribute it as a system fault, yes.”
491. This again was categorised by Fujitsu as Category C, non-critical. In a sense, those two questions and answers sum up a large part of the case. The reason that this plainly, in my judgment, does demonstrate a system fault is Fujitsu’s own PEAK recorded:
“Voiced NBSC quoted ref H18174172 to see what checks they have done themselves before transferring call to Horizon.
They stated they had trainers come into the office and ruled out user error.”
And
“NBSC states the user error checks were carried out by Auditors at the site and not over the phone.”
(emphasis added)
492. This matter was assigned to Anne Chambers, whose name appears very often in many of the documents. The final part of her conclusion was (having said that “there were no known issues that would result in the variance being incorrect”):
“I can't tell why the declared cash doesn't match the expected cash figure, the branch need to make sure that what they have recorded on the system is correct, and investigate the anomalies.”
(emphasis added)
This is simply passing the buck back to a SPM in a branch. How an SPM could “investigate the anomalies” given they appeared to be generated by the Horizon system is not explained, nor could it sensibly be.
493. She also closed the response with the category “Final – no fault in product” and closed the call by defect cause “General – User”. I consider this PEAK to be an ideal illustration of what I consider to be the most extraordinary situation at SSC, and one which on the face of it is difficult to explain, given the function of SSC was to investigate faults:
1. Fujitsu routinely assigned non-critical Category C to matters that were really very important in their own right in any event, but of extreme importance to SPMs whose branch accounts were being directly affected. Mr Parker accepted that only Category A or B attracted financial penalties. It is not possible, on the evidence before the court, to conclude one way or the other whether it was this that affected the categorisation adopted by Fujitsu personnel.
2. Fujitsu would ignore information directly from the Post Office itself that demonstrated that a SPM was not at fault. ROMEC engineers observing specific matters occurring, or in this case the Post Office’s own auditors ruling out user error, were simply ignored. This effectively amounted to the Fujitsu SSC personnel positively over-ruling the Post Office’s own non-SPM personnel, who were giving them essential information about failures in the Horizon system.
3. Fujitsu would also close PEAKs in a variety of ways that entirely mischaracterised the issue. This particular one was “no fault in product” and defect cause “general – user”, in other words, it was caused by the SPM. This had been specifically ruled out by the Post Office’s own auditors who had visited the branch.
4. On the PEAKs that were used in the Horizon Issues trial, Fujitsu would routinely assign lower categories of importance to reported matters that were directly impacting SPMs’ branch accounts. They would also routinely assign user error to reported matters, not because they had uncovered user error, but because they could not explain what had occurred. It seems to have been used as a default setting. Given my findings in the Technical Appendix, this approach by Fujitsu simply cannot be justified in technical terms.
494. Fujitsu do not, on the face of these documents shown to Mr Parker, appear to me to have properly and fully investigated these myriad problems, nor did Fujitsu categorise such incidents correctly. They also seem to have moved away, in their investigations, from concluding that there were any issues with the software wherever it was possible for them to do so, regardless of evidence to the contrary, an approach that has been carried into the Fujitsu evidence for the Horizon Issues trial. In re-examination Mr Parker was asked about the way he had done his exercise, and also the point was put to him that the PEAKs to which he was taken were outwith the period when Mr Roll was employed at SSC. The implication of this was that Fujitsu may, or would, have been more accurate in the period in question rather than in later periods. Mr Parker said he could not possibly read all the PEAKs, that he considered his data to be accurate, and although “you may find a few PEAKs where the response code does not tally with the document” given the number of PEAKs involved “human beings will make those errors”.
495. I reject that explanation, such as it is. I do not consider that Mr Parker was interested in accuracy in any of his evidential exercises, and I do not consider that he was objective in the way he presented his evidence, although he sought to give the impression that he was. I consider that Mr Parker, and the team who assisted him, sought to portray the Horizon system – Legacy Horizon and Horizon Online – in a light as favourable as possible to Fujitsu, regardless of its own internal evidence to the contrary, and regardless of the facts. Mr Parker’s spreadsheet plainly did not calculate all the categories that were used by Fujitsu for problems with the system, and sought to give the impression that Mr Roll had been involved in far fewer incidents of this type than were the case.
496. My conclusion about Mr Parker’s lack of accuracy, is supported by the changing story of Fujitsu about remote injection of messages; the need for correcting supplementary statements, including from Mr Parker himself; his admission that he knew Fujitsu had the power to do this, even though his 1st statement expressly said otherwise; the attempt to shift the blame for this on to Mr Roll; the assertions in his subsequent statement that Mr Roll had been unclear; and also by his attempt to portray his original evidence as dealing with frequency, not power or ability. My conclusions are also supported by his “deliberate decision” to depart from Fujitsu’s definition in its own internal document, which I have explained at [488] above. I also consider that the fact that someone such as Mr Parker had been on the Post Office Account for such a very long time, in his case throughout the whole life of Horizon, means he was hardly best placed to be objective in this type of evidence. But even making allowance for the natural reaction of an employee to wish to protect his employer’s interests, which many people may have sub-consciously, I find that Mr Parker’s evidence to the court was inaccurate to a significant degree.
497. In my judgment, the exercise done by Mr Parker to demonstrate the number of PEAKs that Mr Roll worked on involving software, included in the spreadsheet of categorisation he and others prepared, is of no evidential value whatsoever. It relies upon the categorisation of the PEAKs performed by the Fujitsu personnel themselves, which in the vast majority of documents used in the trial were simply wrong and/or misleading; and it also used a different definition to the one included in Fujitsu’s own documents such as the PEAKs above. He accepted that he had made a deliberate decision to use a definition other than the one in the Fujitsu documents, which he justified by saying was based on his experience. That deliberate decision alone is remarkable. I find that he was, by means of the exercise he was involved in, trying to downplay the extent of Mr Roll’s involvement in software issues, and also trying to downplay the extent of software problems that were experienced on Legacy Horizon during Mr Roll’s employment at SSC.
499. The degree to which the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Membury
501. Mr Membury has worked for Fujitsu since 1998 and became the Quality Risk and Compliance Manager for the Post Office account in 2011. He has overseen multiple audits of the Horizon system. Given the timing of this, these would have all been on Horizon Online. In 2014 he became a Central Quality Partner specifically focusing on the Post Office as well as the Payment Card Industry (PCI) Security Standards Subject Matter Expert (SME) for Fujitsu UK and Ireland. In 2015 he became the PCI SME for the Europe, Middle East, India and Africa regions. He returned to the Fujitsu Post Office Account as Head of Quality and Compliance in October 2018.
502. He described his statement as an overview of the auditing and development regimes in place in relation to Horizon to the extent that those points were not covered by the disclosed technical documents. The statement was said to provide a high-level overview of the audits to which Horizon had been subject since 2000; Fujitsu’s audit methodology and his role in the audits; and to describe the development of changes to Horizon and how that tied in with or into the audit process.
503. Mr Membury’s statement was very brief; it ran to only 5 pages of text. It also omitted some highly material matters. For example, he stated that the Post Office and Fujitsu agreed in 2010 that Horizon Online would be audited against the International Standard on Assurance Engagements (ISAE) No. 3402 assurance standard (Horizon ISAE 3402 Audit), entitled "Assurance Reports on Controls at a Service Organization", which was issued in December 2009. This was developed to provide an international assurance standard for allowing public accountants to issue a report for use by user organisations and their auditors (user auditors) on the controls at a service organisation that are likely to impact or be a part of the user organisation's system of internal control over financial reporting. It provides assurance that the solutions in place to manage financial transactions are appropriate.
504. However, he completely omitted any reference to the Ernst & Young Management Letter for the year ending 27 March 2011 which has been referred to in [264], [393] and [524], which included concerns about privileges and other critical matters in relation to IT. Indeed, he made only two references to Ernst & Young. One was to state that:
“Ernst & Young have carried out the Horizon ISAE 3402 Audit since the 2012/2013 financial year (preparations for the audit began with Post Office in 2011)”
505. The second was a reference to how Ernst & Young performed the ISAE 3402 audit for the period 1 April 2014 to 31 December 2014. His failure to mention the 2011 Management Letter is, in my judgment, a serious omission.
506. He could not be asked any questions about his evidence; although that was not his fault, this goes to the weight which I attach to his evidence in any event.
The absence of Mr Gareth Jenkins
508. It is entirely a decision of the parties which witnesses they choose to call in any proceedings in respect of any evidence. The position of one person, however, who did not appear in the Horizon Issues trial, must be considered in more detail than would be usual, as the claimants make considerable complaint about this. The person in question is Gareth Jenkins, a senior Fujitsu employee who, although he retired recently, was obviously widely available to the Post Office and the source of a great amount of information both to the Post Office’s witnesses of fact, and also to Dr Worden (although he was not separately identified in Dr Worden’s “sources of information” paragraphs in his 1st Report). The fact that he provided information to Dr Worden emerged during the latter’s cross-examination. Mr Jenkins had previously given expert evidence for the Post Office in some of the criminal prosecutions of SPMs, in particular that of Ms Misra, to whom I have referred above, who was convicted of criminal offences in Guildford Crown Court in 2010.
“A. No, no, I didn't do a calculation to come up with the 60. I was quoting from other people.
Mr Justice Fraser: Someone just gave you the 60, did they?
A. I thought I was quoting from other people. I -- Gareth even said to me that in my statement I had said "approximately 60", so I was not -- clearly I didn't because the statement here doesn't contain that word. I had rather hoped it had when this was first brought to my attention, but no, I certainly did not do any specific calculation to come up with the 60 that I put into my original statement.
Mr Green: Did Gareth explain the change to you from 60 to 62?
A. No.
Q. But how did he come to tell you what was in your original statement? What was that conversation?
A. I picked up the number -- to be -- my objective in this was to explain to the court the symptoms of the bug and how -- the technical aspect of it. I did not pay particular attention to getting the detail on how many branches were affected, correct.
Q. Okay. So you have spoken to Gareth since your statement?
A. I don't think I have spoken to him about this in particular. I was -- as I say, when Gareth had said I had originally said "approximately 60" I was thinking that was quite neat, but that's not the case.
Q. Well, you said "Gareth even said to me that in my statement I had said 'approximately 60'", so he must have said that to you after your statement had been filed?
A. It was a comment in a document that we were exchanging.
Q. But you hadn't spoken to him about remote access since your first statement?
A. No.
Q. Why have you stayed off that topic with him?
A. Oh, sorry, this was just a comment. We have been exchanging documents, we have been commenting on documents, so it was not a particular conversation. It is merely a case of Gareth had commented on this when it was pushed back to us that I had originally said 60 and actually the answer was different.”
510. There is nothing wrong with Mr Godeseth correcting the number of affected branches from 60 to 62. There is also nothing wrong with him reproducing a number given to him by someone else, as long as it was properly identified that the information came from someone else, who ought to be identified (which it wasn’t in his statement, but was, in at least outline terms, in his cross-examination). The reason for reproducing these passages is simply to identify the extent to which Mr Jenkins was so closely involved in the litigation, a point relied upon by the claimants. There were a great many references throughout Mr Godeseth’s evidence, written and oral, about information he had obtained from Mr Jenkins.
511. When the Post Office served their evidence of fact, the claimants had asked the Post Office why there was no statement from Jenkins, whether Mr Jenkins was available to give evidence, and also whether he was involved as one of a team of what the claimants referred to as the “shadow experts”. This description was challenged by the Post Office, and the question of shadow experts is addressed further at [556] below. No explanation was given for Mr Jenkins’ absence in response to these requests, or in evidence in the trial, although it was confirmed that Mr Jenkins was not one of the team of so-called “shadow experts”.
512. There the matter might have rested. However, in the Post Office’s written closing submissions, an explanation of sorts was for the first time provided. This was in the context of two matters: firstly, by way of explanation of Mr Godeseth’s evidence, and potentially to downplay its impact; secondly, in relation to the claimants’ complaints about the second hand nature of some of the Post Office’s factual evidence because in large part this had emanated from Mr Jenkins. This explanation by the Post Office included the following passages in its written submissions:
“144. [The claimants] understandably complain that Mr Jenkins and the other source of Mr Godeseth’s information could have given some of this evidence first hand. However:
144.1 Taking into account that Mr McLachlan’s evidence specifically addressed things said or done by Mr Jenkins in relation to the Misra trial, Post Office was concerned that the Horizon Issues trial could become an investigation of his role in this and other criminal cases.
144.2 Moreover, Post Office was conscious that if it only adduced first hand evidence in the trial, it would end up having to call more witnesses than could be accommodated within the trial timetable.
144.3 Furthermore, so far as Post Office was aware, the relevant parts of Godeseth 2 were most unlikely to be controversial. For example, the Misra trial was a matter of public record, the four bugs were covered by contemporaneous documentation and Post Office had no reason to doubt Fujitsu’s account of the documents it held.”
514. The following are relevant in my judgment:
1. Of primary importance is the principle that it is for each party to decide whom to call as a witness, and what evidence they may seek to obtain from any particular witness.
2. Mr Jenkins is an important and central person so far as the operation, efficacy and robustness of Horizon is concerned, and also in respect of the number of incidents over the years that have led to PEAKs, KELs, problems and fixes. That is reinforced by the Post Office’s own closing submissions that stated, at paragraph 138, that there were two possible candidates for the witness giving the overview of Horizon, namely Mr Godeseth and Mr Jenkins. It also stated that taking into account Mr Jenkins’ involvement in cases before the CCRC (including the Misra case) the decision was taken to use Mr Godeseth. For that decision to have been taken, it is implicit that Mr Godeseth must know enough about the system to give the evidence that he did.
3. The point in the footnote about a potential further 34 witnesses is not relevant to whether Mr Jenkins was, or was not, called by the Post Office. The Post Office did not at any stage apply to the court and explain that further time was required for the length of the Horizon Issues trial for the specific reason that relevant evidence of fact from the Post Office would require a particular minimum number of witnesses. However, given the primacy of point (1) above, it was entirely for the Post Office to decide whether to call Mr Jenkins or not, how many witnesses to call, whom those witnesses were to be and what the content of their evidence should include.
4. The Post Office chose to proffer a reason for Mr Jenkins’ absence in closing submissions. They were not obliged to explain. However, the reason in paragraph 144.1 quoted above is not a valid reason for his absence. The claimants would have been entitled, in cross-examination, to put to Mr Jenkins any previous inconsistent statements he had made on the same subject, but obviously only if there were any. These could, potentially, have included previous statements he may have made in earlier proceedings, but in order to be allowed to do that, such statements would have had to be inconsistent with his evidence in the Horizon Issues trial. Putting a previously inconsistent statement on a particular fact is permitted as part of cross-examination if a witness’ evidence on that fact is in issue. However, the Horizon Issues trial would not have become “an investigation of his role in this and other criminal cases” – the Horizon trial was about the Horizon Issues. Also, this type of cross-examination would only have arisen if any statement(s) in the Horizon Issues trial was or were not consistent with those he had made in previous cases in which he had been involved.
5. Regardless of the validity of the explanation given by the Post Office, Mr Jenkins was not called as a witness, and it merits repetition that it is entirely a matter for the Post Office which witnesses it called, and which it did not. No speculation is permitted as to what evidence he might have given in the Horizon Issues trial. It ought to be recorded however that no limit on the number of witnesses was imposed by the court on any party, and the Post Office could (had it wished) have called both Mr Godeseth and Mr Jenkins. Nobody forced the Post Office to choose one or the other.
6. There is established authority that in certain circumstances the court can draw adverse inferences from the absence of a witness. The claimants do not invite me to draw any adverse inferences from the absence of Mr Jenkins, and I do not do so.
7. The fact that the Post Office chose to advance certain evidence in the Horizon trial, that emanated from Mr Jenkins, by means of another witness saying “Mr Jenkins told me that…..” means that the claimants were deprived of the opportunity properly to test that evidence by asking Mr Jenkins about it. It also emerged in cross-examination, although not in his written report, that Dr Worden has also obtained certain information directly from Mr Jenkins. By dealing with the material in this way, and having deprived the claimants of the opportunity to cross-examine Mr Jenkins, the weight to be given to evidence emanating from Mr Jenkins is less than it would be otherwise.
515. There are two consequences of the Post Office choosing to adduce such evidence in this way. The first is that Mr Jenkins did not have to undergo any cross-examination on such evidence; the second is that such evidence would not be given the same weight by the court as though Mr Jenkins had given it himself. The Post Office would have known this in advance, as this is wholly conventional. How these different factors were weighed by the Post Office and its legal advisers in deciding whether or not to call him, is solely a matter for the Post Office. The end result is that Mr Jenkins did not appear and was not cross-examined.
516. However, I reject the suggestion that Mr Godeseth was insufficiently knowledgeable of the matters upon which he was cross-examined for his evidence to be of significant evidential weight. The Post Office’s Closing Submissions sought to dilute the effect of Mr Godeseth’s cross-examination by, at the same time, implicitly suggesting that Mr Jenkins (or someone else, but not Mr Godeseth) would have been a far better person to have answered those questions; yet explaining why Mr Jenkins had not been called. These submissions, though understandable from the Post Office’s point of view in forensic terms only, have little force. A clear choice was made by the Post Office not to call Mr Jenkins as a witness; that was their right. A clear choice was also made to call Mr Godeseth. The Post Office was are entitled to call such evidence as deemed necessary by both the Post Office and its advisers. They cannot, however, have matters both ways, and try to downplay Mr Godeseth’s evidence because he was called to deal with certain matters, and Mr Jenkins was not. The Post Office has to abide by the consequences of their choice of witnesses, in terms of the evidence now before the court to resolve the Horizon Issues trial. I have already explained my view of Mr Godeseth’s evidence in Part E above, and it need not be repeated.
Inaccurate Statements by the Post Office
517. Litigation in this jurisdiction is adversarial. What that means is each side advances their own case, and challenges or rebuts the case of the other side, and the court assesses the evidence before it adduced by the parties, makes necessary findings on the facts and applies the law. Parties in civil litigation will usually advance their own evidence of fact, and although mechanisms are available for compelling other witnesses to attend and answer questions, there can be some pitfalls in doing so and issuing a witness summons (what used to be called a subpoena) is a relatively rare step. The court can only therefore resolve the case on the evidence before it, although it can draw inferences, that is to say common sense conclusions, on the evidence it does have.
518. The Post Office has no obligation to assist the claimants in advancing their case against it. The Post Office has, however, maintained publicly that it was seeking to be “transparent” about Horizon, and prior to the litigation it made certain public statements in relation to the increasing disquiet on the part both of SPMs, and others who became involved either on their behalf (such as some Members of Parliament) or in an investigative way (such as the BBC Panorama programme and other journalists). These statements by the Post Office routinely and strongly insisted that there was nothing in the criticisms being levelled at the accuracy of Horizon, and that losses that were shown in SPMs’ branch accounts were caused either by carelessness or dishonesty on the part of the different SPMs who experienced what they considered to be unexplained discrepancies and losses. In 2015 a Parliamentary Select Committee held hearings into the mediation scheme that had been set up to attempt to address the claims by SPMs in respect of Horizon. I have not considered any evidence submitted to the Select Committee, or any details of the mediation scheme (which was being conducted by Second Sight) and refer to them here simply as a matter of chronological record.
519. The claimants rely upon a public statement released by the Post Office in 2015 after the BCC Panorama programme. Part of the statement was headed “The Horizon System” and stated:
“The Horizon Computer System
Horizon is robust and effective in dealing with the six million transactions put through the system every day by our postmasters and employees at 11,500 Post Office branches. It is independently audited and meets or exceeds industry accreditations. There have been 500,000 users of the system since it was introduced.
Nevertheless, rigorous re-investigations were undertaken into claims made by 136 mainly former postmasters that the system caused losses in their branches.
There is overwhelming evidence that the losses complained of were caused by user actions, including in some cases deliberate dishonest conduct. The investigations have not identified any transaction caused by a technical fault in Horizon which resulted in a postmaster wrongly being held responsible for a loss of money.
There is also no evidence of transactions recorded by branches being altered through ‘remote access’ to the system. Transactions as they are recorded by branches cannot be edited and the Panorama programme did not show anything that contradicts this.”
(emphasis added)
520. Another document from 2015 upon which the claimants rely is an internal email chain, which originated from Paula Vennells, then the Chief Executive of the Post Office, on 30 January 2015. This was prior to her appearance before the House of Commons Select Committee in February 2015. She posed the following question in an email sent internally to Mark Davies and Lesley Sewell, both of the Post Office:
“Dear both, your help please in answers and in phrasing those answers, in prep for the SC:
1) "is it possible to access the system remotely? We are told it is."
What is the true answer? I hope it is that we know this is not possible and that we are able to explain why that is. I need to say no it is not possible and that we are sure of this because of xxx and that we know this because we have had the system assured.”
(emphasis added)
521. Ms Vennells obviously needed to know whether the answer matched her understanding, which was – as she put it, both “I hope” and “I need” -- that it was not possible to access the system remotely. This query was passed on through various people, including at one stage from James Davidson who has both a Fujitsu and Post Office email address, who answered to Mark Underwood:
“As discussed, can you hook up with Kevin to review what answers have already been provided to second sight as this should form the Post Office response.”
“Can Post Office or Fujitsu edit transaction data without the knowledge of a Subpostmaster?”
Post Office confirms that neither it nor Fujitsu can edit transaction data without the knowledge of a Subpostmaster.
There is no functionality in Horizon for either a branch, Post Office or Fujitsu to edit, manipulate or remove a transaction once it has been recorded in a branch's accounts.
The following safeguards are in place to prevent such occurrences:”
(various matters are then listed in the remainder of the email)
(bold present in original)
“Mark / Mel,
James has had a look at your answer to Q1. And thinks there’s too much detail for Paula – this was written for a different type of audience. He has captured the same points but in a more appropriate format :-
He states:-
Having looked again at the request from Paula, it appears that the fundamentals around this question (remote access) are not understood. I suggest that Paula is briefed along the lines of the following.
1) No transaction data is held locally in any branch. Transactions are completed and stored in a central database and copies of all data is sent to a secure audit database.
2) Sub-postmasters directly manage user access and password setting locally so system access (to create transactions) are limited to approved local personnel only who are responsible for setting their own passwords. Users are only created following an approval process which requires authorisation by the sub-postmaster. All subsequent transactions are recorded against the id used to log on to the system.
3) Once a transaction has been completed, there is no functionality (by design) for transactions to be edited or amended. Each transaction is given a unique number and ‘wrapped’ in a digital encryption seal to protect its integrity. All transactions are then posted to a secure and segregated audit server.
4) On approval, there is the functionality to add additional transactions which will be visible and have a unique identifier in the audit trail. This is extremely rare and only been used once since go live of the system in 2010 (March 2010)
5) Support staff have the ability to review event logs and monitor, in real time, the availability of the system infrastructure as part of standard service management processes.
6) Overall system access is tightly controlled via industry standard ‘role based access’ protocols and assured independently in annual audits for ISO 27001, Ernst and Young for IAS 3402 and as part of PCI audits.”
(emphasis added)
1. “Is it possible to access the system remotely? We are told it is.”
2. “What is the true answer?”
525. Both of these are really very simple questions. Question 1. means can it be done, because Ms Vennells is being told it can be done. Question 2, “what is the true answer?” means she is seeking the true answer to question 1. This is not complicated, either in linguistic, computing or even business terms, nor is it difficult to understand. She then expressed her aspirations in terms of what the true answer might be. “I hope it is that we know this is not possible and that we are able to explain why that is. I need to say no it is not possible.” (emphasis added)
526. This trial has shown that the true answer to the enquiry she made in early 2015 was “yes, it is possible.” It has taken some years, and many tens of millions of pounds in costs, to reach that answer. Kevin Lenihan, who wrote that he did not understand “the fundamentals around this question (remote access)”, seems to me to have avoided providing a simple answer to a simple question. It is not necessary to consider why that was, whether from a lack of understanding on his part, or otherwise.
527. I am unaware of whether there is any other final answer to Ms Vennells internally to her very straightforward question, or of what she in fact told the Select Committee. None of the people through whom this email chain passed were called as witnesses in the Horizon Issues trial. However, the point in bold in the email quoted at [522] above is now known, as a result of this litigation, simply to be incorrect. Such editing can take place without the knowledge of the SPM. All and any previous statements by the Post Office stating that this cannot be done are simply factually wrong. The more detailed points at (1) to (6) of the email quoted at [523] provide only selected information and, in my judgment, are entirely off the point. They have the effect of obscuring what ought to be a simple answer, with a level of detail which makes the actual answer highly unclear. The answer at (1) applies only to Horizon Online. For Legacy Horizon, transaction data was held locally in the first instance, on what was called the counter. This was explained very clearly by the Post Office in its opening, including in its oral opening submissions. The first sentence of (3) in the list in the email is not correct. The statement in (4) – which may refer to the Transaction Correction tool – is correct (or at least, not incorrect) but only so far as it goes, and in my judgment crucially omits the ability to edit transactions without visibility to the SPM. It also fails to consider the existence of APPSUP permissions at all, which every member of SSC had. The experts are agreed that the APPSUP permission was very powerful and enabled an SSC employee to do pretty much whatever they wanted, to paraphrase the Post Office’s own expert. It can also be seen, particularly by the evidence that emerged in the cross-examination of Mr Godeseth, that in a great many cases the PEAKs show that user error was attributed to problems that were not user error at all, and that others within the Post Office (for example its own auditors and ROMEC engineers) had either witnessed for themselves and/or satisfied themselves that user error was not the cause of specific issues, but Fujitsu decided they were user error in any event.
528. The extent of permissions enjoyed by SSC was the subject of some express concern by Ernst and Young (the Post Office’s own auditors) in 2011, and although the position may have been regularised by 2015, certainly prior to 2011 (which covers the Legacy Horizon years) the situation was very different. The email also states “he has captured the same points” yet the most important point to which an answer is given, or certainly the answer given in bold in the answer at [522] above, is not corrected, even though it is plainly wrong. The overall tenor of what Ms Vennells received, if it reflected the full content of this email chain, would have been that the Post Office and Fujitsu could not edit transaction data without the knowledge of the SPM. The evidence in this trial is directly to the contrary, and Fujitsu could do so. It is important to bear in mind the distinction between the way Legacy Horizon, and Horizon Online, work. Referring to keeping data on the “counters” is a Legacy Horizon expression. This refers to the way the data was held at the branch; these were discs (including mirror discs) at the branch which contained the data. The discs were contained in terminals on the counter. In Horizon Online, the data was held at a central database or server (the phrase used by the Post Office in its oral opening).
529. This tenor of these public statements made by the Post Office was maintained into, and as part of, the group litigation. For example, certain statements were made in the Generic Defence in these proceedings. A party’s Defence is the formal pleading document which sets out a party’s legal position in the litigation. Since the reforms in civil litigation adopted after what are called the Woolf Reforms, which led to the introduction of the Civil Procedure Rules in 1998 which govern all civil litigation in the jurisdiction, such pleadings must be accompanied by a Statement of Truth. These are, for larger companies, usually signed by directors or very senior personnel, which occurred in this case. That statement attests to the truth of the matters stated in the pleading.
530. This issue was dealt with in the pleadings in the following way. By this point the Post Office had accepted that there was some limited ability to do that which it had previously stated was not possible. In the Generic Statement of Case, the claimants set out the following part of their case:
“25. Further, the Defendant was, by itself and/or via its agent Fujitsu, able to alter branch transaction data directly and carry out changes to Horizon and/or transaction data which could affect branch accounts.
26. However, the Defendant has made public statements in the following terms:
26.1. "Horizon does not have functionality that allows Post Office or Fujitsu to edit or delete the transactions as recorded by branches" (Defendant's published reply to Second Sight's Briefing Report Part Two, concerning a review of the Horizon system); and
26.2. "Transactions as they are recorded by branches cannot be edited" (Defendant's published reply to the BBC Panorama documentary in relation to Horizon).
27. These statements were untrue, as the Defendant now admits.”
531. These paragraphs were pleaded to by the Post Office in its Generic Defence in the following terms:
“57. Paragraph 25 appears to be concerned with the editing or deletion of transaction data input by or on behalf of a Subpostmaster without his or her consent. Accordingly, Post Office assumes that it is not concerned with transactions such as Transaction Corrections which are sent to branches but must be accepted by or on behalf of the Subpostmaster before forming part of his or her branch account. As to the circumstances in which such transaction data can be edited or deleted without the consent of the Subpostmaster:
(1) Neither Post Office nor Fujitsu has the ability to log on remotely to a Horizon terminal in a branch so as to conduct transactions.
(2) A Post Office employee with "global user" authorisation can, when physically present at a branch, use a terminal within the branch to add a transaction into the branch's accounts. The purpose of "Global User" authorization is to allow access to the systems for during training and/ or audits. Any transactions effected by a Global User are recorded against a Global User ID and are readily identifiable as such.
(3) Fujitsu (and not Post Office) has the ability to inject transactions into branch accounts (since the introduction of Horizon Online in 2010, transactions of this sort have been called "Balancing Transactions"). These transactions do not involve any removal or amendment of the transactions entered at the branch. Their intended purpose is to allow Fujitsu to correct errors or bugs in Horizon by cancelling the effect of an error or bug on a branch's data. They may be conducted only by a small number of specialists at Fujitsu and only in accordance with specific authorisation requirements. They are rarely used. To the best of Post Office's information and belief, only one Balancing Transaction has ever been made so as to affect a branch's transaction data, and this was not in a branch operated by a Claimant. A Balancing Transaction is readily identifiable as such.
(4) There are a small number of Fujitsu specialists who have certain privileged user access rights which they could in theory use to amend or delete the transaction data for a branch. The intended purpose of privileged user rights is system support, not the alteration of branch transaction data. To have abused those rights so as to alter branch transaction data and conceal that this has happened would be an extraordinarily difficult thing to do, involving complex steps (including the writing of sophisticated computer programmes and circumvention of sophisticated control measures) which would require months of planning and an exceptional level of technical expertise. Post Office has never consented to the use of privileged user rights to alter branch data and, to the best of its information and belief, these rights have never been used for this purpose.
(5) Post Office cannot conceive of a reason why any Fujitsu personnel would have sought to add, inject, amend or delete any transactions in any branch accounts so as to create a false shortfall. It would for all practical purposes be impossible for any of them to generate significant shortfalls without detection and, even if they were able to do so, they would be unable to take the benefit of such shortfalls for themselves.”
(emphasis added)
532. This stance was maintained by the Post Office in the evidence served on its behalf for the Horizon Issues trial, until service of Mr Roll’s 2nd witness statement. To be fair to the Post Office, its origin was the witness statements served by Fujitsu employees, rather than Post Office employees. The position within the Fujitsu witness evidence, prior to its correction by the later statements from Mr Parker, was that what Mr Roll said was possible on Legacy Horizon, and what he had himself done, was simply not possible. Indeed, Dr Worden considered it sufficiently clear that as an IT expert he felt able confidently to assert in his 1st Expert Report that he, Dr Worden, had “established” that Mr Roll’s evidence of fact in this respect was wrong. After service of Mr Roll’s 2nd witness statement, Fujitsu finally came clean and confirmed (via Mr Parker) that what Mr Roll said was correct. Data could be altered by Fujitsu on Horizon as if at the branch; under Legacy Horizon, transactions could be inserted at the counter in the way Mr Roll described. This could be done without the SPM knowing about this. Mr Godeseth also confirmed that it would appear as though the SPM themselves had performed the transaction. This is directly contrary to what the Post Office had been saying publicly for many years.
533. Therefore some of the earlier public statements made by the Post Office, and the important one contained in the Post Office’s pleaded Defence to which I have referred above, were factually untrue in at least one highly important respect. This concerns the ability of Fujitsu to insert transactions into a branch’s accounts remotely, without the SPM being aware of this, and without the transaction being identifiable in the transaction data as having been inserted remotely in this way. The term “remotely” means from a location elsewhere than within the branch itself.
534. Although the phrase “remote access” was used during the Horizon Issues trial, the parties did not always use the expression in the same way as one another. The Post Office’s definition was explained in oral opening in the following way:
“When I talk about remote access I'm talking about action taken remotely to either inject new transactions or to edit existing transactions or to delete existing transactions in a way that could change the accounting position of the relevant branch.”
“Master, first of all, could I just deal with the remote access point? The letter to which my learned friend took you was, as you might expect, written by people who thought it was correct. The Horizon system is a very complicated system. It involves lots of departments in ... both in Fujitsu and in the Post Office. And the people who are responsible for the correspondence didn't know that, in fact, there were these two other routes. Very few people at Post Office knew that there were these two other routes. They were ... they were routes that are under ... essentially under the control of Fujitsu who's the expert independent contractor that is involved in the operation of the system. And it is a matter of enormous regret that the people who wrote that correspondence and made those submissions weren't aware of that but, you know, we are where we are; the point is that, the point having been discovered, the Post Office wasted no time in ... in bringing the truth ... the accurate ... and accurate set of facts to the knowledge of the claimants.”
(emphasis added)
536. The claimants rely upon the way this sequence unfolded, and the admission by the Post Office that its previously public statements were untrue, as making it particularly important that the true and accurate position was then provided by the Post Office. Indeed, the submissions I have quoted at [535] above make it clear that the court and the claimants were being told that the Post Office had wasted no time in bringing the truth, “the accurate set of facts”, to the knowledge of the claimants. As the analysis of the Defence and the evolution of the Fujitsu evidence of fact on remote access for the Horizon Issues trial demonstrates, the accurate set of facts did not emerge at that time either. The truth, the accurate set of facts, has only emerged as a result of the final supplementary statements for the Horizon Issues trial that were served, and only finally (so far as Fujitsu is concerned) after their witnesses had initially stated clearly in their witness statements that Mr Roll was wrong. Eventually these witnesses accepted that he was right. The supplementary witness statements that accepted this were dated 29 January 2019. Prior to that date, the opposite position had been maintained by the Post Office.
537. The Post Office in its opening submissions stated the following:
“Taking remote access as an example, the need for remote intervention affecting branch accounts will obviously be rare. On any view, the occasions on which privileged users at Fujitsu have exercised their ability to remotely inject, edit or delete branch transactions or accounting entries will represent a tiny percentage of the relevant transactions/accounting entries. And the occasions on which they have done so negligently or dishonestly will, in turn, represent a very small percentage of those occasions. So, compared with the volume of business recorded in branch accounts, the number of cases in which false data will have been remotely introduced will be extremely small (multiplying a small chance by a small chance). This is a “second order effect” (a small proportion of a small proportion) which is, by definition, extremely unlikely to have any significant impact on the robustness of Horizon.”
(emphasis added)
538. That submission misses the point, in my judgment. It elides two matters, namely whether something is technically possible, and the number of times that it has in fact been done. The former, whether it was possible, had been expressly denied, and that denial is now shown to be wrong. The latter, the number of times it was done and with what effect, is a different matter. There is also very little, if any, evidence that is relied upon by the Post Office to justify the assertion that it was a “tiny percentage” of times. That is a subjective assertion of very limited weight.
539. I consider the significance of the previously factually untrue statements to be considerable. The statement was made publicly by the Post Office, turned out not to be factually correct, and the Post Office gave an explanation and said the full set of facts was now available. The situation was pleaded to by the Post Office in its Generic Defence, with a statement of truth. That too turned out not to be correct, and the true position has only emerged in the Horizon Issues stage of the litigation as a result of the evidence of Mr Roll, which I have dealt with above. It was only following his written evidence that Mr Parker, and Mr Godeseth – both senior Fujitsu employees – prepared their supplementary witness statements correcting their first statements. These first statements, as I have explained above, were simply untrue in that important respect. These witnesses had previously stated that this was not possible. Mr Parker said Fujitsu did not have the power to do this.
540. I find as a fact that Fujitsu do have the ability to insert transactions into branch accounts on a remote basis (in other words, remote access as the Post Office defined it in opening submissions existed) and this could be done without these being visible to the SPM in that branch in question, either at the time or subsequently. It also follows that this could be done without an SPM in this position having knowledge of this, and therefore without consenting to it. Someone who does not know something is happening cannot give permission for it to happen. Mr Godeseth also gave important evidence about how such transactions would appear. It is not possible to say, in blanket terms, that permission was not sought and/or given in some cases. Indeed, Dr Worden accepted that the APPSUP permissions meant that – as he put it – Fujitsu personnel could do “pretty much anything”. On Day 18 he said in cross-examination the following:
“Q. Dr Worden, you knew that a central issue, not only a central issue legally but a very high-profile issue in the case, was the extent to which Post Office had remote access to the counters, didn't you? You knew that?
A. Yes, and what I'm talking about, what I was talking about was the extent to which this could happen without the knowledge of the subpostmaster.
Q. And that's the –
A. And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything.”
541. He was asked about this answer two days later.
“Q. It is Day 18, page 67. I think it is lines 5 and 6 actually. It says: "Answer: And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything."
A. Yes. We can see precisely what we agreed in a minute, obviously.
Q. In relation to the –
A. We agreed the experts couldn't demonstrate that they couldn't do everything. I mean, that's sloppy wording by me there. I think the joint expert statement says it better.”
542. The exact wording in the joint statement in respect of repairing transactions is as follows at 10.2 of the 4th Joint Statement:
“Certain facilities and procedures used by Fujitsu to repair the more common issues which arose in Horizon were standardised, and evidence of them persists. However, to repair less common issues which arose from time to time, standard tools and procedures might not have been sufficient, and evidence might not persist of what was done at the time. Even when evidence does persist, it may be extremely difficult for the experts to interpret it today, because of the scale and complexity of Horizon.
Therefore, it is usually difficult for the experts to make categorical negative statements of the form: ‘X or Y never happened’.”
543. This litigation is between the claimants and the Post Office. Fujitsu is not a party. Although one of the contracts between the Post Office and Fujitsu was uploaded to the electronic trial bundle on the penultimate day of evidence (by the Post Office’s solicitors) the litigation is not concerned with the details of that contractual relationship, other than (tangentially only) the cost to the Post Office of making ARQ requests, as the ARQ data is maintained by Fujitsu and charges are raised to the Post Office in this respect. However, Fujitsu’s involvement does cast something of a shadow. That contract has been addressed in terms of whether certain documents were, or were not, in the control of the Post Office. I have already dealt with the issue of PEAKs and KELs, and how the Post Office maintained initially it was not obliged to disclose these. Fujitsu personnel who were not witnesses were involved in providing information to the Post Office’s expert, Dr Worden. It was the Fujitsu witnesses who originally, and in my judgment unequivocally and directly, flatly contradicted what Mr Roll said could be done on the subject of remote access. I have also identified that in my judgment, the majority of the Fujitsu witnesses were more interested in following some sort of Fujitsu party line than they were in answering questions in cross-examination wholly frankly, although I exempt Mr Godeseth from that criticism. From what was said by the Post Office’s leading counsel to Master Fontaine at the very first case management hearing, those within the Post Office had relied upon Fujitsu in obtaining relevant information for its earlier public statements, and it plainly relied upon Fujitsu in terms of evidence of fact in the Horizon Issues trial, because it called a number of Fujitsu witnesses. I am unaware of the degree to which the Post Office has depended upon Fujitsu in the background to the Horizon Issues trial, and there is no reason why I should be, and I make it clear that I do not speculate on this. This information must have originated from Fujitsu at some point, but the Post Office must bear some responsibility for such incorrect statements having been made before, both publicly and in its pleadings.
544. Nor could it be said, in the Post Office’s and/or Fujitsu’s defence, that remote access was a side issue. It has been a very important central element of the whole dispute between SPMs and the Post Office over the Horizon system for many years. Mr Green for the claimants adopted a non-computing analogy, namely that of “a back door”. The origin of this was a phrase of Mr Roll, who referred to the back door to the system. The branch accounts can be likened to a locked room, to which the SPM has the key. This development in the case is comparable, after many years of the Post Office and Fujitsu maintaining that only the SPM has the key, that only the SPM could perform anything within the locked room, and if the Post Office and/or Fujitsu needed access to the room the SPM would know because they would have to borrow the key, to now being an admission by the Post Office (and Fujitsu) (and a finding of fact by the court) that there is a back door which they could, and did, use without the SPM even knowing such a door existed. This analogy is not entirely apt, because software and remote access to edit transactions is not completely the same as physical access through a door, but it is an attempt to capture the general idea in lay terms.
545. The truth concerning remote access has now emerged in 2019, in group litigation that started in 2017. I find it notable that the truth did not emerge in the first Fujitsu witness statements that were originally served for the Horizon Issues trial. Such statements stand as the evidence in chief of witnesses of fact. They are supposed to be accurate. Minor corrections are not unusual and indeed are almost expected, as a trial approaches, as witnesses either research further or remember (when preparing for trial) other minor details. This topic, however, did not undergo that type of correction, and is a subject far more central and important than that. The truth only finally emerged in later statements, which were required to correct what I find were directly inaccurate statements in the first witness statements of Mr Godeseth and Mr Parker. There has been no adequate explanation for the contents of those first statements, which not only omitted this important fact, but contained evidence directly to the contrary. Those first witness statements were misleading. The statement in the Defence was misleading too. It ought also to be noted that the truth did not emerge internally within the Post Office in the email answers provided to internal inquiries in 2015 by senior Post Office personnel, such as the Chief Executive, who posed the specific question in preparation for providing evidence to a Select Committee and asked: “What is the true answer?”
546. She also said in the same email “I hope it is that we know this is not possible and that we are able to explain why that is”. The true answer is that, contrary to her aspiration, it was possible.
547. She also stated “I need to say no it is not possible and that we are sure of this because of xxx and that we know this because we have had the system assured.” The true answer to that was also “yes, it is possible”.
548. It is also difficult to believe that the signatory of the Statement of Truth in the Generic Defence would have signed that statement if she did not believe the contents of that pleading to be true. There is another relevant feature in this case, which is that the Generic Defence also includes a counterclaim seeking damages against the SPM claimants, including damages for fraud. Fraud is the most serious allegation that can be brought in civil litigation and there are special rules in relation to pleading it, which means that a pleading containing a fraud allegation should be subject to particular scrutiny before it is served.
549. It may therefore be that the Post Office itself fell into error as a result of information provided to it by Fujitsu on this important matter. It may be that some within the Post Office were themselves surprised by these revelations prior to, and during, the Horizon Issues trial. There is no need for me to speculate on this, and I do not do so. Certainly Mr Godeseth did not appear to have known about this for very long. Whatever the origin of this, and whether it came from Fujitsu, internally, being less than frank with the Post Office or not, the effect is that the Post Office has made specific and factually incorrect statements about what could be done with, or to, branch accounts in terms of remote access without the knowledge of the SPM. The evidence in this trial has made it clear that such remote access to branch accounts does exist; such remote access is possible by employees within Fujitsu; it does exist specifically by design; and it has been used in the past.
550. It follows that the previously stated public position of the Post Office to the contrary, in the statements to which I have referred above, is specifically wrong in fact.
551. Further, and as part of the Post Office serving supplementary witness statements from both Mr Godeseth and Mr Parker in respect of Mr Roll’s evidence, the Fujitsu witnesses adopted a position in the litigation, the rationale of which is somewhat difficult to understand. Responsibility for the incorrect and potentially misleading evidence of Mr Godeseth and Mr Parker (in their first statements) was effectively laid at the door of Mr Roll for giving what was said to be vague evidence, or evidence that, it was said, had not been fully understood by Fujitsu. Infallibility is a rare commodity, and everyone is capable of making mistakes. However, it is how one reacts to mistakes that is telling. In this instance, the initial reaction of the Fujitsu witnesses was to seek to shift the blame for their own misleading written evidence upon someone else. In this case, that “someone else” was their former Fujitsu colleague whose very evidence was responsible for exposing the full picture. This is not something that could have been arrived at lightly. To be fair to Mr de Garr Robinson, it was not a point he adopted on the Post Office’s behalf during the Horizon Issues trial. In my judgment, he was entirely sensible in not doing so. That it was done at all in the written Fujitsu statements speaks volumes, in my judgment, about Fujitsu’s reaction to being caught out.
552. Mr Parker’s supplementary statement blaming Mr Roll said the following:
“In paragraph 20 of Roll 2, Mr Roll describes a process by which transactions could be inserted via individual branch counters by using the correspondence server to piggy back through the gateway. He has not previously made this point clear. Now that he has, following a discussion with colleagues who performed such actions I can confirm that this was possible. I did not mention it in my first witness statement because, when faced with a less clear account in Mr Roll's first statement, my recollection was that if it was necessary for the SSC to inject a transaction data into a branch's accounts, it would have been injected into the correspondence server (injecting via the server was the default option which was followed in the vast majority of cases).”
553. The recusal application was issued the day after Mr Godeseth’s cross-examination had made it clear, not only that this remote access existed, but after he was taken in careful cross-examination through specific examples of Fujitsu personnel manipulating branch accounts, and leading to discrepancies in branch accounts. I am aware that criticism of the Post Office and Fujitsu in this respect may prove to be controversial, as earlier criticism of certain aspects of the Post Office’s case in Judgment (No.3) was not well received by it. However, if criticism is justified, I consider it would be detrimental to proper resolution of the group litigation if that criticism were to be withheld simply because it might lead to a further negative reaction by the Post Office. It is also an inherent part of the judicial function in any litigation to make findings, which may include criticisms where justified, that may be contrary to a litigant’s own view of the merits of their case. Some litigants are so convinced of the righteousness of their own position that they consistently refuse to accept any possible view of the litigation other than their own. Such a blinkered view is rarely helpful, and would be particularly unhelpful from a publicly owned institution.
554. I consider that criticism is justified of the Post Office’s incorrect previous statements – which included public and high profile statements such as the response to the BBC Panorama programme, whatever may have been said on the subject to the Select Committee, as well as in its own pleadings - about remote access without the knowledge of SPMs. The Post Office should have done its best to discern whether such remote access was possible when this subject first arose, and whether it had occurred, before these statements were made. Fujitsu should have been frank and unequivocal, internally, with the Post Office, so that there could be no possibility of incorrect statements on this important point being made publicly by the Post Office. The Fujitsu witnesses should not, in their first witness statements, have made the incorrect statements that they did. Had those initial statements been factually accurate, there would have been no need for the supplementary statements from them that eventually led to the true position being accepted by Fujitsu, and therefore by the Post Office. It is highly regrettable that such a situation as this should have developed. I am not making any findings on any future issues, whether of fact or law, yet to be tried.
555. There is however considerably more to resolving the Horizon Issues than simply making findings about remote access to branch accounts.
So-called “shadow experts” instructed by the Post Office
556. I return to the issue of so-called “shadow experts.” The parties in this case agreed that costs management would apply, and the Post Office in its costs budget for the Horizon Issues trial included, for the costs management hearing on 5 June 2018, an item for incurred expert costs in its costs budget in the sizeable amount of approximately £800,000. This was in addition to the amount incurred by that point in terms of the fees of Dr Worden. His costs at that stage were only £58,000. The sum of £800,000 broke down to about £300,000 paid to Fujitsu, and £500,000 paid to other experts, who were not being instructed to give expert evidence. Dr Worden was the expert who would be giving evidence to the court, which meant he would owe the relevant duties of independence to the court under the CPR. The description of these other experts was given in a skeleton argument for the Post Office in the following terms:
“The Defendant has spent around £500,000 on investigations by internally appointed experts for the purposes of determining its litigation strategy. The resulting material - which is privileged - has not been provided to the Defendant's expert for the purposes of this litigation.”
557. The claimants adopted the term “shadow experts” for these, a term with which the Post Office did not agree. At the cost management hearings, the claimants pointed out that by “internally instructed” this meant these experts were not instructed by the Post Office’s solicitors. It was also pointed out that there was no corresponding entry in the budget for any conference with any of their counsel, and this meant that this was “an entirely ring-fenced operation”. The Post Office did, however, reserve the right to recover their costs as costs in the litigation, and therefore this item was included in its costs budget. All of the material produced by these other experts was said to be privileged.
558. It is a highly unusual situation that entirely separate experts, instructed directly by a party, without the involvement either of that party’s solicitors or their counsel (all the more so when those experts are not even identified), are instructed on such a task, whatever that task might actually be. I recorded an adverse comment in the costs management order of 23 July 2018 stating that the Post Office’s incurred costs for experts were extraordinarily high, unreasonable and disproportionate; and that Fujitsu’s costs of assisting with the litigation, and the costs of these internally appointed experts did not, on the face of it, appear to be properly recoverable sums in the litigation. I should clarify that this adverse comment should not be taken as applying to Dr Worden’s costs that had been incurred at that stage, which were modest. It is not necessary to deal with this matter any further to resolve the Horizon Issues.
F. Documents and Available Information
559. There are certain categories or descriptions of classes of documents that have featured heavily in the evidence at the Horizon Issues trial. The path to disclosing them has not always been smooth. The majority, if not all, of the technical documents that relate to how Horizon was actually operating in fact in IT terms are in the possession of either the Post Office or (more usually) Fujitsu. The two most important categories, in my judgment, are Known Error Logs (also known as “KELs”) and PEAKs. The first of these records or logs known errors, which means errors with the Horizon system. The latter is a browser-based software incident and problem management system used by Fujitsu for the Post Office account, in other words for incidents and problems associated with Horizon that occur.
560. Disclosure had previously been a troublesome topic between these parties. I had in Judgment (No.3) on the Common Issues, been somewhat critical of certain aspects of the Post Office’s approach to the disclosure of some documents. These observations were not well received by the Post Office and some had been categorised, incorrectly, by Mr Parsons in his 14th witness statement supporting the recusal application as “critical invective”. Whether the contents of Judgment (No.3) encouraged the claimants to mount, or continue to advance, criticisms of the Post Office’s disclosure in the Horizon Issues trial (in the sense that they may have felt they had a “following wind”) is not clear. The Post Office, in the Horizon Issues trial (predominantly in solicitors’ correspondence, but also in some submissions), also criticised the claimants’ disclosure. For the sake of clarity, I wish to make it clear that the claimants had no such following wind.
561. The Horizon Issues are decided in this judgment based on all the evidence and the submissions. I deal with disclosure in the subsequent paragraphs of this judgment in order to assist and guide the parties concerning the future conduct of the group litigation. Some issues, such as whether certain Fujitsu documents were within the Post Office’s control, are likely to crop up again in respect of other types of documents in the future. The court’s interest in disclosure is to promote compliance with the rules, and also to ensure cost effective progress of the litigation. The answers to the substantive Horizon Issues are no different because, for example, the experts were not given particular documents relating to the specific bugs in good time. For the most part, large numbers of KELs and PEAKs were provided to them, and the experts had an abundance of material. Some 5,000 KELs emerged from Fujitsu well after the trial ended, in October 2019. The experts did not consider these as they were not given them. Mr Parker had given evidence (at paragraph 61.9 of his 1st witness statement) that 1,491 KELs had been deleted at the time of writing that statement, but there was no reference to the existence of this far greater number that emerged after the trial was over.
562. The Post Office’s written closing submissions also included a separate section on disclosure as follows, which I will quote verbatim:
“I6. The Court’s interventions on disclosure
1145. Post Office is concerned that the Court may have lost sight of the nature of the Orders made and of the approach which the Court ordered. It is striking that Cs have not themselves made any applications for specific disclosure nor have they advanced any complaint that particular disclosure orders have not been complied with.
1146. The recent judgment of the Court of Appeal in Serafin v Malkiewicz & ors [2019] EWCA Civ 852 @ para 118 is relevant:
“We are also highly troubled by the repeated demands and criticisms by the Judge regarding the Claimant’s disclosure, in circumstances where pre-trial disclosure had been completed by both sides at a time when both the Claimant and Defendants had been represented by solicitors and counsel, and no application for further disclosure had been made by the Defendants…” ”
563. These submissions identified what were said to have been “the Court’s interventions on disclosure” in the Horizon Issues trial. The authority identified by the Post Office, and stated to be relevant to that submission, concerned a defamation case in which the judge had demanded disclosure of a litigant in person, and made criticisms of him, both during the trial of that litigant’s libel claim, and in the judgment itself. In some instances the court in that case had, of its own volition, demanded production of certain documents from the litigant in person, and within a very short timescale. This had been part of findings made by the Court of Appeal in respect of procedural irregularity and the successful appeal against the judgment.
564. There were no “interventions on disclosure” by the court during the Horizon Issues trial, nor were there “demands and criticisms”. When asked during oral closing submissions about this passage in the Post Office’s written submissions citing Serafin, Mr de Garr Robinson did not demonstrate a great deal, if any, enthusiasm in advancing any claims of procedural irregularity having occurred during the Horizon Issues trial. He submitted that “this is not that case” – which does raise the question of why the passages from Serafin were included in the Post Office’s submissions in the first place.
565. There were, however the following matters that arose in this litigation during the Horizon Issues trial, which given paragraphs 1145 and 1146 of the Post Office’s closing submissions, should be recorded:
1. Redactions. In Judgment (No.3) I had identified the issue of unjustified redactions made by the Post Office of some contemporaneous documents that were used in the Common Issues trial. During the early part of the Horizon Issues trial some documents were referred to which again had redactions, which due to their non-redacted parts did not seem to me to be of a type that would necessarily attract privilege. In view of the previous history of this subject, I asked leading counsel for the Post Office to perform his own review of the redactions that had at that point been made of documents deployed in the Horizon Issues trial. This review resulted in some of the redactions in some of the documents being removed, and unredacted versions of those documents becoming available to the claimants, some of which were then used in cross-examination. Redactions of 2 documents were maintained on the grounds of legal professional privilege, which is entirely conventional. He also provided a helpful two page note explaining the approach that had been adopted. Asking counsel to do a review of this nature in these circumstances is entirely conventional. There was no short timescale imposed for this to be done. I asked for a summary of the position on 4 June 2019, which was after the interval which occurred as a result of the recusal application, and about 2 ½ months after I had asked him to perform the review. It was an outstanding matter, of a type sometimes referred to in a trial as “housekeeping”, which had not been entirely forgotten, but which had been overtaken in the latter weeks of March when the trial was temporarily halted.
2. A witness statement was ordered from the Post Office to explain the express, and factually incorrect, submissions made to the court by the Post Office about the Royal Mail’s refusal to produce, at the Post Office’s request, audit documents. The Royal Mail had, contrary to what the court had expressly been told by the Post Office, not even been asked by the Post Office for these. The Post Office’s leading counsel accepted that he had, entirely unwittingly, and on instruction, provided misleading information to the court and explained and apologised as soon as he discovered this. A witness statement from the Post Office, which was provided by Mr Parsons, was entirely justified in these circumstances.
3. A witness statement was also ordered from the Post Office in respect of the production of over 2,000 documents in disclosure by the Post Office in the interval in the trial period between 11 April 2019 (completion of the factual evidence), and the experts commencing their oral evidence on 4 June 2019. Given the timing and quantity of these, in the circumstances a witness statement (again, being provided by Mr Parsons) was justified. The first explanation given to the court, again on instruction, again proved to be wrong. These documents were originally said to have all been of some age (ten years or so) and discovered at Fujitsu on a “long-forgotten” server. One deployed the day after this explanation in cross-examination turned out to be dated 21 August 2018. A witness statement was again called for.
566. None of the above, in my judgment, fall into the category of the court losing sight of the nature of the orders that had been made concerning disclosure, or of demands or criticisms by the court, unjustified or otherwise. Further, the claimants made numerous complaints of the Post Office about disclosure. Indeed, the factually incorrect submissions made to the court about the audit documents and the Royal Mail came at the end of a court day when the claimants, by their leading counsel, expressly attempted to make a specific disclosure application. I was not prepared to hear that application at that point. Royal Mail was the correct respondent to such an application, not the Post Office. Royal Mail was not represented, nor even present in court, nor had any notice been given by the claimants to the Royal Mail of such an application. Each of these features is a far from promising start. I did however make directions for the making of a third party disclosure application against the Royal Mail, but this did not ultimately prove to be required. Once those at the Royal Mail were actually asked for these documents, the Royal Mail readily produced them. The claimants also made regular complaints about disclosure and would often point out that documents had been produced during the hearing. This is because documents were often produced during the hearing. This is something that had also occurred during the Common Issues trial.
567. The Post Office is, however, correct to point out that there have been no specific disclosure applications mounted by the claimants, with the exception of the one to which I refer in [566] above in respect of the Royal Mail. General complaints about disclosure only take a party so far, and that is not very far, if deficiencies are known about in advance of a trial, and no applications are made.
568. The parties had also voluntarily adopted Model C of the Disclosure Pilot, which I approved. This was done prior to the coming into force of the Pilot, the full title of which is Disclosure Pilot for the Business and Property Courts (“BPC”) which became mandatory in all BPC cases where the first CMC is held after 1 January 2019. This group litigation is not a BPC case and the first CMC was held before that date in any event. The Pilot is designed to run for a two year period. The Pilot is found at Practice Direction 51U, which is made under CPR Part 51.2. The features of the Pilot that are particularly relevant here are the Principles included in paragraph 2 of the Pilot. These state:
2. Principles, “document”, “adverse” and “known adverse documents”
2.7 Disclosure extends to “adverse” documents. A document is “adverse” if it or any information it contains contradicts or materially damages the disclosing party’s contention or version of events on an issue in dispute, or supports the contention or version of events of an opposing party on an issue in dispute.
2.8 “Known adverse documents” are documents (other than privileged documents) that a party is actually aware (without undertaking any further search for documents than it has already undertaken or caused to be undertaken) both
(a) are or were previously within its control and (b) are adverse.”
3.1 A person who knows that it is or may become a party to proceedings that have been commenced or who knows that it may become a party to proceedings that may be commenced is under the following duties (“the Disclosure Duties”) to the court—
(1) to take reasonable steps to preserve documents in its control that may be relevant to any issue in the proceedings;
(2) once proceedings have commenced against it or by it and in accordance with the provisions of this pilot scheme, to disclose, regardless of any order for disclosure made, known adverse documents, unless they are privileged;”
(emphasis added)
569. The fact that the parties chose Model C does not go to answer the complaints that the claimants have made about the Post Office’s disclosure. The Post Office had an obligation to disclose known adverse documents, as do the claimants as well. “Adverse” as defined plainly includes any documents that refer to bugs, errors or defects, or the operation of the Horizon system that led to potential impact upon branch accounts. This obligation is “regardless of any order for disclosure made”, in other words it exists whether any specific classes of documents are, or are not, already the subject of any order. The rational for this is obvious. Reliance upon the adoption of Model C is not therefore the answer to the claimants’ complaints in this respect. However, the fact that the claimants did not make applications for disclosure is relevant to the criticisms made at the Horizon Issues trial.
570. The Post Office has, essentially, submitted that the breadth of disclosure requests has been too wide. Particular criticism is that “Freeths’ [the claimants’ solicitors] did not control Mr Coyne’s requests”. I am not persuaded that it is the correct approach to matters of independent expert evidence to seek to “control” the documents that an independent expert wishes to see. The claimants did not issue any opposed applications for disclosure of documents which their expert wished to see, so a filter of some sort was applied in that sense to the documents Mr Coyne wished to see, and was not given.
571. Finally, disclosure is a particular feature of litigation in this jurisdiction. However, there was a clear perception in commercial litigation generally that the then-current disclosure regime had become inadequate and also that standard disclosure was often excessive in scale, cost and complexity. This is why the Disclosure Working Group was created in May 2016 at the initiative of Sir Terence Etherton, then the Chancellor of the High Court and now the Master of the Rolls. This led to the Disclosure Pilot.
572. It would be highly regrettable if disclosure in this case were to become yet another battleground between the parties. There are more than enough points in issue between the parties without adding endless disagreements about disclosure to that list. There are the following areas, however, of disclosure which should be recorded. In my judgment, the background to the evolution of the following categories does throw light on the parties’ different approaches to the litigation.
The Known Error Log or KELs
573. KELs are sometimes referred to in the singular – for example, in some places reference is made to the Known Error Log, as though it were a single Log. In practice, the Log as a whole includes multiple KELs, as they were referred to during the trial. If an incident or problem occurs, a PEAK is raised, and if the problem or incident refers to an existing issue which is known to have occurred before, reference will be made to that KEL entry, or more usually, it is said that the PEAK will refer to the KEL. Both PEAKs and KELs have reference numbers; the former are usually all numbers after the prefix PC, whereas KELs have letters (taken from the identity of the employee who raised the KEL followed by four digits and a letter). Therefore, when a problem (which I shall refer to generally as an example as problem x) occurs for the first time, a KEL will be raised which will be given a reference. Subsequent PEAKs which identify the same problem x again, over time, afterwards will all refer to the same KEL. Sometimes a PEAK will be identified as including a problem which is similar to problems identified in more than one KEL, in which case both the KELs will be referred to within that PEAK. Any creation of a new KEL, or updating of an existing KEL, must be authorised by SSC before it can be seen by all users.
574. The experts agreed the following in the 2nd Joint Statement about both PEAKs and KELs: “KELs and PEAKs together form a useful source of information about bugs in Horizon but are a limited window on what happened. It is sometimes necessary to use evidence from both to try to understand, but even so they are not a comprehensive picture. It is to be expected that both KELs and PEAKs are incomplete in various respects.” It is also agreed that KELs will often give information about the impact of a bug or user error and they may also provide information about causes. There are other documents referred to as OCPs and OCRs. They are Operational Change Proposals and Operational Change Requests.
575. The subject matter of this litigation spans many years, starting from the first implementation of the Horizon system (what is now being called Legacy Horizon) in 2000. In the Letter of Claim from the claimants dated 28 April 2016, the Known Error Log was sought from the Post Office, that letter stating:
“We understand that Fujitsu maintained a 'Known Error Log' for Horizon and that such reports will have been provided to Post Office. Please see the list of the categories of documents relating to Fujitsu referred to below, that we request disclosure of.” Item 22 in the list of documents sought was “The 'known error log' kept by Fujitsu and provided to Post Office as referred to above, and all correspondence relating to the same.”
576. The answer in a letter from the Post Office’s solicitors against the specific item 22 was:
“In circumstances where you have not particularised any factual basis on which Horizon is defective, disclosure of these documents (if they exist) is not relevant, reasonable or proportionate.”
577. The suggestion in that letter that the Known Error Log was not relevant, is simply wrong, and in my judgment, entirely without any rational basis. The further suggestion, viewed with the hindsight now available, that the “known error log” may not exist, is disturbing. The claimants’ request used the precise title – “known error log” – and this clearly did exist. To suggest in an answer “if they exist” is somewhat misleading.
578. Item 23 in the same list of documents sought was “Internal memoranda from Fujitsu and POL referred to by Second Sight as identifying a `Horizon bug' within Horizon Online.”
579. The answer against that item was:
“We do not recognise the document to which you refer. Please provide further details.”
580. In my judgment, the documents sought in that entry must clearly include any PEAKs that identified a bug within Horizon Online. “Internal memoranda” is a plural reference, yet it was interpreted by the Post Office’s solicitors as though it were singular, and the request was for a single document, or a document with the title “internal memoranda”. This is, in my judgment, obstructive.
581. The claimants were not to be dissuaded, and sought the Known Error Log or KELs again. A reply from the Post Office’s solicitors on 13 October 2016 is relied upon by the claimants as showing that the Post Office was denying the relevance of the Known Error Log. This reply stated:
“The claims which you have particularised concern errors with the Core Audit Log. Following a review of the Known Error Log, Fujitsu have confirmed that there have been no logs in respect of Core Audit Log. The remainder of the Known Error Log does not relate to the claim which you have particularised and as such disclosure of this document is not relevant.”
(emphasis added)
582. Existence of the Known Error Log was at that stage accepted, but its relevance to the proceedings was now challenged. The Post Office’s solicitors stated that its contents “did not relate to the claim” and that “disclosure of this document is not relevant”. Disclosure of it was plainly resisted. The claimants did not therefore have it when the Generic Particulars of Claim was pleaded on 6 July 2017. In the Generic Defence, which is dated 18 July 2017, the Post Office changed its position, and now pleaded that the Known Error Log was not in its control. At paragraph 50(4) of the Generic Defence, the Post Office stated:
“It is admitted that Fujitsu maintain a "Known Error Log". This is not used by Post Office and nor is it in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a knowledge base document used by Fujitsu which explains how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data.”
(emphasis added)
583. This was expressly challenged in paragraphs 3 and 4 of the Generic Reply.
584. The two main points to consider within that passage in the Defence are that the Post Office stated that it did not have control of the Known Error Log, and that entries within it related to minor/trivial matters and could not affect the accuracy of a branch’s accounts. This position was maintained, notwithstanding the terms of the Post Office’s skeleton argument for the first case management hearing on 19 October 2017. That had stated:
“Cs’ response to the criticism of their case on Horizon is to argue that they do not have the access to information and documents that would allow them to plead any properly articulated case as to the “bugs” that they wish to blame for the shortfalls in their branches. Without prejudice to its position that the case is not properly pleadable, even at a generic level, Post Office is prepared to take steps to help Cs investigate Horizon, take an informed view as to whether they really wish to maintain their claims and, if so, decide how to proceed with them. Post Office wishes to put an end to the speculative attacks on Horizon and the related allegations as to its own conduct. It puts forward its proposals in paragraphs 2 to 8 of the Draft Order as a pragmatic compromise between the parties’ competing interests and concerns.”
(italics present in original, emphasis by underlining added)
585. At the first case management hearing itself, upon specific enquiry by the court, this position was further maintained.
586. Disclosure of the Known Error Log was resisted by the Post Office on the grounds both of control, and content. So far as control is concerned, Mr Parsons had provided his 4th witness statement for that hearing, the first before me as the managing judge, which expressly stated at paragraph 35 “Despite Post Office explaining the irrelevance of the Known Error Log and that it was not within Post Office’s control…” and also at paragraph 37 “Due to the large amount of information involved and the fact that the Known Error Log is not in Post Office's control….”. This position was confirmed by the Post Office through its leading counsel in the following exchange with the court at that hearing, the transcript of that showing the following exchange.
“Mr Justice Fraser: Do you still maintain it is not in your control?
A: My Lord, yes. It’s a Fujitsu document.
Mr Justice Fraser: No. Just because it’s a Fujitsu document doesn’t mean it’s not in your control.
A: That’s right. At no point has Post Office ever suggested that it’s in its control so far as I’m aware in any of the letters that I –
Mr Justice Fraser: Okay.
A: (Reading from pleading)
"To the best of Post Office's information and belief the known error log is a knowledge based document used by Fujitsu which explains how to deal with or workaround minor issues that can sometimes arise in Horizon for which often because of their triviality system-wide fixes have not been developed and implemented. It is not a record of software coding errors or bugs for which system-wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief there is no issue in the known error log that could affect the accuracy of branch accounts or the secure transmission and storage of transaction data."
587. This was the relevant passage in the Generic Defence that was read out and Post Office’s leading counsel then continued:
“It will be clear from what I've just read to your Lordship that Post Office thinks that this is a complete red herring.”
588. The position was then, again, expressly confirmed as follows.
“Mr Justice Fraser: I think it's going to be useful for me, certainly, for you to define all the different grounds upon which you currently resist disclosure of the known error log. Number (1) is you say it's not in your control.
A: Control.
Mr Justice Fraser: Number (2) you say the subject matter is not relevant. Is that right?
A: My Lord, yes. An order that Post Office disclose documents relating to bugs and errors causing the branch account errors, if Post Office had control of the document and it dutifully complied with that order it would not be disclosing the known error log because, as far as it is aware, the known error log is not that kind of document.”
“It contains things like there's a problem with printers. There's a printer. You have to kick it on the left-hand side to make the printer work. I mean there's a vast range of hardware problems of that sort and maybe some software problems …. but not the kind of bugs, errors and defects that the claimants are wishing to pursue in their particulars of claim so far as Post Office is aware.”
590. This exchange with the court then continued.
“Mr Justice Fraser: Well, that's the rider which slightly might concern---
A: Well, my Lord, unfortunately, that's all that Post Office can say because it's not Post Office's document. It's Fujitsu's document. Fujitsu are the experts.”
591. Given there was no witness statement from anyone at Fujitsu available on the subject at that hearing, that was as far as the matter could be taken on that occasion in terms of making any specific order relating to the Known Error Log. The Post Office had made clear that any order for disclosure that did not specify the Known Error Log, but was more general requiring disclosure of documents in relation to bugs and errors causing branch account errors, would not lead to disclosure of the Known Error Log in any event. A practical way forward, without requiring a fully argued separate application which would have included Fujitsu, was adopted whereby the IT experts were permitted to inspect the Known Error Log. The status of the Known Error Log as a disclosable document was resisted both on the grounds of control and relevance. It has to be acknowledged that this practical solution was one suggested by Mr de Garr Robinson.
592. It must therefore have come as a surprise to the Post Office, given the contents of its pleadings and the express submissions that it had made to the court, that both experts considered the Known Error Log to be highly relevant. This should also have led to the Post Office beginning to doubt what it was being told by Fujitsu, given the source of what the court was told about this was what Fujitsu had told the Post Office, as set out above. The explanation of what the Known Error Log was, what it contained, and its lack of relevance, was not remotely accurate.
593. Although a certain number of entries in the Known Error Log, which led to a significant number of different entries which are each called KELs being deployed in the trial and examined closely, were thereafter available, the issue of control of the Known Error Log and the Post Office’s earlier position resisting disclosure of this, did not go away. An entire appendix to the claimants’ opening submissions was devoted to criticism of the Post Office on disclosure, including its shifting position on the Known Error Log. In closing submissions, at the very end of the final day of the Horizon Issues trial, the claimants handed up a document entitled “Claimants’ Points in Reply”. Part F of that was a short document headed “KELs said not to be in the Post Office’s control” together with a variety of references. It ought also to be recorded here that some 5,000 KELs were later disclosed by the Post Office in October 2019, well after the trial ended, once the Post Office was told by Fujitsu that previous entries, which Fujitsu had previously told the Post Office were not retained, were in fact retained.
594. The subject of whether the Post Office had control of the Known Error Log therefore remained somewhat live. It simply was not, therefore, going to go away. The Post Office’s legal team had loaded version 12 of its contract with Fujitsu (“the Fujitsu contract”) onto the electronic bundle database on the last day of evidence, Day 20, for re-examination of its expert witness, Dr Worden. The contractual situation between the Post Office and Fujitsu was therefore before the court, which it had not been for the first case management hearing on 19 October 2017. It is also the case that “control” of documents (other than the KELs) held by Fujitsu could potentially crop up again in this litigation going forwards. It is cost effective and efficient to resolve that now, for the assistance of the parties.
595. I therefore gave both parties the opportunity to lodge further written post-hearing submissions restricted to whether the Known Error Log was in the Post Office’s control. It was a point relied upon by the claimants and it has to be resolved. There were two reasons for this. One is that in group litigation of this type, the parties’ approach to disclosure goes beyond a single document, or type of document. Guidance as to the court’s general approach will be of assistance in the litigation going forwards. There may well be other documents held by Fujitsu that need to be disclosed later, and the issue of control needs resolving. Secondly, the court is entitled to expect accurate evidence from parties on interlocutory matters, and accurate submissions.
596. The Fujitsu contract that was loaded onto the electronic bundle was a recent one. This document has a version history at the front which shows that it runs, with amendments therein identified, from 31 August 2006 to version 12 on 3 July 2017. Version 12 pre-dates the date of the Generic Defence, but in any event, clause 25 (which appears below at [602]) does not appear against any of the later version numbers in the version history of the Fujitsu contract. It is likely therefore that clause 25 in substantially the same (if not identical) form appeared in all the earlier versions of the Fujitsu contract prior to Version 12.
597. The claimants made the requested post-hearing written submissions based on the meaning of ‘control’, for the purposes of CPR Part 31.8, which is uncontroversial.
598. That rule provides as follows:
“(1) A party’s duty to disclose documents is limited to documents which are or have been in his control.
(2) For this purpose a party has or has had a document in his control if –
(a) it is or was in his physical possession;
(b) he has or has had a right to possession of it; or
(c) he has or has had a right to inspect or take copies of it.”
599. Sub-paragraphs (a) to (c) are not exhaustive, but collectively those two sub-paragraphs mean, in this case concerning this document or documents, that if the Post Office has, or had, a right to possession of the Known Error Log, and/or a right to inspect or take or be provided with copies of it, it is within the Post Office’s control. Control will be established if a party has a contractual right to inspect or take copies of the document or documents in question. Given the terms of the Fujitsu contract, the claimants submitted that the Known Error Log is in the Post Office’s control and has been at all material times.
600. The Post Office’s written submissions on this were also received. These conceded, as it was put in paragraph 3, that the KEL was in its control. That paragraph stated at sub-paragraph (1):
“As at July 2017, Post Office understood the KEL to be a relatively unimportant internal working document produced and used by Fujitsu to assist in the performance of some services under the contract. On that basis, the KEL would not properly have been characterised as a “record…relating to the performance” of Fujitsu’s services under the Fujitsu Contract and would thus not have been within Post Office’s control. But, on the facts now known to Post Office, it would not contend the KEL to be outside its control.”
(emphasis added)
601. The title of the log – the Known Error Log, or known error log – includes rather obviously the two words “known” and “errors”. The Horizon Issues use the expression “bugs, errors and defects”. The presence of the word “error” in both might give an early indication of likely relevance of the log. Further, as shown in [603] below, in the Fujitsu contract the term records includes the phrase “full and accurate records” (emphasis added). Although records is used in the definition of Records – a point correctly identified by the Post Office, which it says is largely circular – it is clear that “full and accurate records” are included.
602. Clause 25 in version 12 of the Fujitsu contract states the following:
“25.8 In addition to its obligations under Clauses 25.2 and 25.3, Fujitsu Services shall provide the Court Case Support Services to Post Office in relation to prosecutions and other disputes (whether civil or criminal) with any third party including but not limited to any fraud, theft, breach of contract or impropriety (the “Court Case Support Services”). The Court Case Support Services shall include any matters whether they relate to Horizon, HNG-X or any other system provided by or on behalf of Fujitsu Services to Post Office, its agents or its subcontractors (including Post Office Service Integrator and any Tower Contractor). Fujitsu Services shall provide the Court Case Support Services within the timeframes required by Post Office or the relevant court or other authority.
25.9 Without prejudice to Clause 25.3, the Court Case Support Services shall comprise:
25.9.1 the provision of copy reports;
25.9.2 the provision of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data) where such data is held by or in the control of Fujitsu Services;
25.9.3 the compilation of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data);
25.9.4 the interpretation of data (including transaction data, event logs, helpdesk call logs, non-polled data and remuneration data);
25.9.5 the provision of technical reports regarding technical aspects of any system (whether Horizon, HNG-X or otherwise);
25.9.6 live witness evidence at Court if any of the information provided (including without limitation that provided pursuant to Clauses 25.9.1 to 25.9.5) is challenged to the extent to which Fujitsu Services provided said information; and
25.9.7 the right of access to Records, including but not limited to information, reports and data, held by or in the control of Fujitsu Services, and the assistance of Fujitsu Services’ personnel with appropriate knowledge of the applicable Records (to the extent any such personnel remain employed or contracted to Fujitsu Services) for any independent experts and/or legal advisors instructed by Post Office and/or any other claimant(s) or defendant(s) and the Prosecution in any mediation, arbitration tribunal, court case or dispute in which Post Office is involved in relation to the Horizon and HNG-X or any other system provided by or on behalf of Fujitsu Services to Post Office.”
(emphasis added)
“Records: means the full and accurate records relating to the performance of the Services.”
I consider that this term plainly includes the Known Error Log, as that relates to the performance of the Services, and in any event this contains data and hence also falls within clauses 25.9.2 and 3. I reject the lengthy analysis in the Post Office’s submissions on this point that the KEL is not properly characterised as being a record. It plainly is.
604. In my judgment the Post Office clearly has, and had, a contractual right to be provided with the Known Error Log by Fujitsu, given these are civil proceedings for (inter alia) damages sought by the claimants for breach of contract, and also fraud being alleged against the claimants by the Post Office by way of counterclaim. I do not accept that the KEL is a type of document covered by the authority cited by the Post Office at paragraph 11 of its submissions on this point, namely working papers prepared by professionals for their own assistance in carrying out expert work for their clients; Hanley v JC & A Solicitors [2018] EWHC 2592 (QB) at [42]. That point is , in my judgment, plainly wrong. Given the terms of the contract between the Post Office and Fujitsu, it is not necessary to consider that point further.
605. I consider it verging on entirely unarguable, given the express terms of the Fujitsu contract which is now available to the court, that the Known Error Log was not in the control of the Post Office. Mr Parson’s witness statement had merely stated “it was not within Post Office’s control” and I simply cannot understand the basis for that statement, given the express terms of the contract that the Post Office had with Fujitsu. It plainly is in the Post Office’s control, given the terms I have reproduced above, and that point is now conceded.
606. The fact that the Post Office has submitted that in July 2017, on its understanding then, the KEL was not a “record” that “related to the performance” of the Fujitsu services demonstrates a worrying lack of knowledge on the part of the Post Office, about both Horizon, and Fujitsu’s record keeping. It also means, when this is put together with what the Post Office, by its leading counsel, submitted to the court at the 1st case management conference, that Fujitsu were extraordinarily inaccurate about the information it provided to the Post Office at that stage of these proceedings. As at 2017, the Horizon system (both Legacy Horizon and Horizon Online) had been in use for about 17 years. KEL appears in a table of Abbreviations/Definitions at F/87/3 in a document entitled “CS Support Services Operations Manual” that is dated 29 January 2001. That document is referred to in footnote 34 of Mr Coyne’s 1st report, and three passages within it state the following:
“4.5.1 Maintaining the Known Error Log on the SSC intranet site
The SSC generates and maintains a Known Error Log (KEL) system that uses searchable documents in HTML format. The mechanism for searching is a query entry in an intranet site. The KEL system is available to first, second, third and fourth line support units as well as SSC staff.
4.5.2 Transferring knowledge between support units
The SSC intranet site has KEL search facilities and other useful diagnostic data, documents and tools.
SSC and SMC staff raise KELs based on customer-observed symptoms.
KELs are further maintained once the fault has been resolved.”
“4.7.1 Known Error Logs (KELs)
The intranet site holds known error details in Microsoft Word format, the contents of which may be searched for, in full text form. Documents are created to a defined template wherever possible. An application has been generated which limits the properties of the document to a subset of possible values, for clarity and ease of search. This application is made available to all support units.
The process for creating KEL entries outside of the SSC has not yet been formulated, but it is expected that no KEL will be allowed onto the system before it has been authorised by SSC staff.”
(bold present in original)
607. The Post Office plainly did not know this when making the submissions that were made in 2017. The Services Operation Manual had been in existence for 16 years by then, and this ignorance about the Known Error Log, which is clearly explained within that document, is most surprising.
608. Even without such an express obligation in the Fujitsu contract itself, I find it very difficult to believe that Fujitsu could mount any sort of coherent or legitimate objection to producing it at the request of the Post Office. The assertion that the Log was not in the control of the Post Office is, in my judgment, not a valid one.
609. Similarly unsustainable is the submission that was made to the court in 2017 that its contents were not relevant to the Horizon Issues, and that it did not contain references to bugs and errors having an impact on branch accounts. On that latter point, the Post Office’s submissions were based upon what the Post Office had been told by Fujitsu. Fujitsu were, again, being clearly inaccurate about what they had told the Post Office was actually contained in the Known Error Log.
610. Nor is the passage quoted above in the Generic Defence an accurate description of the Known Error Log. The Known Error Log does far more than “explain how to deal with, or work around, minor issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide fixes have not been developed and implemented”. The Known Error Log appears to be a comprehensive record of all the errors and defects of which Fujitsu have become aware over the life of Horizon. On the basis of the evidence that has been heard in the Horizon Issues trial, there are many issues in the Known Error Log that could affect the accuracy of a branch's accounts or the secure transmission and storage of transaction data, contrary to what is pleaded by the Post Office. Some KELs led to software fixes too. Indeed, Mr Coyne has used it in order to discover the existence of, and then investigate, the numerous different bugs from what was called “the Bug Table” in the 2nd Joint Statement, 11 of which Dr Worden agrees in his written evidence. Dr Worden himself has discovered some of these bugs, from the sample of the entries in the KELs that he had inspected. It is the KELs that are the origin of these expert conclusions, due to the text in the entries that they contain.
611. This discovery of these different bugs by both experts has directly flowed from the entries in the Known Error Log by Fujitsu, together with the associated PEAKs.
612. It is only fair to record that after contested disclosure had been considered at the first hearing before me on 19 October 2017 (dealt with in detail above), the Post Office’s leading counsel suggested the “pragmatic solution” that was giving Mr Coyne the opportunity to inspect it. It was because of this that the matter of control was not resolved at that point, as disclosure was permitted. Mr Coyne inspected the Known Error Log at Fujitsu’s premises. He found it highly relevant. The claimants received most of the KELs they had for the trial on 10 May 2018, with further KELs in respect of particular bugs disclosed on 16 November 2018, one month after the first expert reports of Mr Coyne and Dr Worden had been served. Further KELs were provided in 2019, and a very substantial quantity of 5,000, well after the trial in October 2019. There has been no detailed explanation for that very late post-trial disclosure in any form of any witness statement from anyone at the Post Office or Fujitsu.
613. The KELs provided to the experts prior to their reports featured centrally not only in Mr Coyne’s two reports, but those of Dr Worden too. They were also the subject of a great deal of cross-examination by both sides. I consider that disclosure of the KELs to the experts – the pragmatic solution suggested by leading counsel for the Post Office - has been central in the discovery and investigation of the bugs, errors and defects that the experts agree were, or are, present in the Horizon system, and also of the bugs, errors and defects that are not agreed, but upon which both experts opine and in respect of which I make findings in the Technical Appendix. Dr Worden has also analysed them extensively, and devoted a separate appendix of his report to them. The best description of the KELs is taken from Dr Worden, who stated that they were “a rich source of evidence”. The notion that they were not relevant, or did not contain relevant material, is extremely difficult to fathom, and I do not understand why, or how, Fujitsu would or could sensibly choose to inform its own customer of many years, the Post Office, directly to the contrary. If Fujitsu’s description of the contents of the KELs had been taken at face value and not challenged, then the knowledge now available to the two experts and to the court about the extent of bugs, errors and defects would not have been available, with the obvious detrimental impact upon the fair resolution of the Horizon Issues.
614. The further developments, after the trial, that led to disclosure of another 5,000 KELs by the Post Office after the trial in October 2019 are dealt with at [627] below and following under the heading “Further post trial disclosure”.
PEAKs
615. Turning to PEAKs, upon inspecting the KELs at Fujitsu’s premises in 2018, Mr Coyne discovered the existence of a different category of documents called PEAKs, as these were referred to within different KELs. These documents were requested from the Post Office in July 2018. The claimants maintain that the Post Office were obstructive over this, but whether that is correct or not, over 218,000 different PEAKs were disclosed by the Post Office’s solicitors on 27 September 2018. This was two weeks before the date for exchange of experts’ reports. A further 3,866 were disclosed on 25 October 2018 – two weeks after the experts’ first reports. OCP documents (which are mentioned in PEAKs) were disclosed on 24 January 2019, and a number of PEAKs were identified by Mr Coyne in his supplementary report which he had identified but which had not been disclosed. These were provided (or extracts of them) on 18 February 2019, which is only three weeks before the Horizon Issues trial started.
617. However, the day after this explanation was given, and during cross-examination, a PEAK dated 21 August 2018 was put to Dr Worden and the court was told this was one of the documents that had been disclosed on 31 May 2019. This plainly could not have come from “an old database” in excess of ten years of age which Fujitsu had only recently discovered. A further and more accurate explanation was plainly called for.
“20. In the course of investigations into the bugs referred to by Mr Coyne and/or Dr Worden, including in the Second Joint Statement, on 29 March 2019 Mr Lenton identified PCO273234 to Ms Simmonds as being a document relating to the Drop and Go Bug. For the next couple of months my firm continued to investigate these bugs and a number of further documents relating to these bugs came to light. PCO273234, along with 17 other documents (excluding redundant image files) that had come to my firm's attention when preparing for the Horizon Issues trial were disclosed in a single batch on 31 May 2019.”
(emphasis added)
619. Mr Lenton is a member of Fujitsu personnel. The 2nd Joint Statement was signed by the two experts on 25 February 2019. This passage means that one month after that document, Mr Lenton identified the specific document relating to the Drop and Go Bug. This is one week after Mr Godeseth had completed his evidence. I find the rather glossed-over reference in the witness statement to “continuing investigations” that went on “for the next couple of months” extraordinarily opaque, in the context of a trial that commenced on 11 March 2019, and a document identified on 29 March 2019. I also find the wholesale lack of any explanation of when the “further documents relating to these bugs came to light” – and indeed how – puzzling, again in a witness statement ordered specifically to explain what was already a highly unusual situation. Further, the PEAK relating to the Drop and Go bug was plainly identified by Mr Lenton, a Fujitsu employee, on 29 March 2019. It is in my judgment verging on the extraordinary, in the context of any trial that was actually underway, that it took two months more for this PEAK to be disclosed to the claimants. This is particularly concerning in the context of this trial between these parties on these issues. It is also obvious from the witness statement that the PEAK in question dealing with the Drop and Go bug was not disclosed immediately, until “for the next couple of months my firm [ie the Post Office’s solicitors] continued to investigate these bugs.” There is no reason that investigations, whether ongoing or not, should delay disclosing a document of this nature to the other side, particularly in the middle of such a trial as this.
620. The conclusion to be drawn from this is that disclosure was given in a manner that could only have disrupted and delayed proper investigation of the issues contained in the documents. In this specific case, that includes the Drop and Go bug. That document should have been provided to the claimants very rapidly once it came to the attention of the Post Office’s solicitors, not kept “for the next couple of months”, as Mr Parsons puts it. In subsequent trials, if relevant documents of this nature are discovered by either side or their legal advisers, it is a matter of some importance that disclosure is given very rapidly. This is particularly important whilst the trial is underway. A failure to do so risks disruption, delay and increased costs.
621. The experts agreed the following about PEAKs and their content. “PEAKs record a timeline of activities to fix a bug or a problem. They sometimes contain information not found in KELs about specific impact on branches or root causes – what needs to be fixed. They are written, by people who know Horizon very well. They do not contain design detail for any change. They are generally about development activities and timeline rather than about potential impact. PEAKs typically stop when development has done its job, so they are not likely to contain information about follow-on activities, such as compensating branches for any losses.” It is also agreed, and indeed can be seen from the actual PEAKs themselves, that some of them record observations of financial impact. PEAKs do not however always use the symbol “£”, which was a search item used by Dr Worden to create a set of 259 KELs that contained that symbol. Even where they referred to a financial discrepancy, PEAKs sometimes just recorded the figure, or sometimes used “pds”, and did not use the symbol “£”.
Release Notes
622. A spreadsheet listing the Release Notes was produced which shows that there have been a great number of changes and updates both to Legacy Horizon and Horizon Online over the years. The total number of Release Notes is 19,842 at final version. Each expert provided an estimate of how many changes per week there have been to Horizon. Mr Coyne’s estimate is approximately 19 changes per week. Dr Worden’s estimate is 5 changes per working day. Given the number of working days per week, these figures are broadly the same. No details of the actual Release Notes were provided. The spreadsheet was not, in my judgment, complete, nor was it in useful usable form. No Release Notes detrimental to the Post Office’s case have been produced. The Post Office adopted an unhelpful approach to Release Notes, requiring the claimants to identify from the very difficult spreadsheet which Release Notes they would like to see, with an offer to provide those. On 14 February 2019 the claimants’ solicitors indicated which ones they did want to see. On 20 February 2019, the Post Office’s solicitors stated “we are currently taking instructions in relation to the requests”. Both the request from the claimants, and the response, are wholly unhelpful and far too late. To be fair to the Post Office, a late request will undoubtedly lead to a late response. Further, the spreadsheet itself is almost wholly unusable.
623. I have commented upon the approach of the Post Office to disclosure in this litigation before, and not in favourable terms. At the first CMC in October 2017, I said in a short ruling on disclosure that the Post Office had been “obstructive” in its pre-action behaviour in this respect, and that its attitude to disclosure up to that point had been somewhat less than ideal. In Judgment (No.3) I made some further criticisms, which it is not necessary to repeat. It is regrettable that the Post Office’s approach to disclosure, admittedly this time in conjunction with Fujitsu, has continued in different respects to fall short of what is required. However, it does seem to be improving and it is to be hoped that such improvement will continue. Particularly in litigation such as this, with enormous distrust and suspicion on both sides, it is simply counter-productive to have that atmosphere added to by an unnecessarily combative stance on disclosure.
624. The claimants submitted in their opening that the Post Office’s approach to disclosure had the effect of impeding the claimants from obtaining a full view of the documents and the totality of the Horizon system. I agree with that submission. However, balanced against this, it does have to be remembered that the claimants did not issue any applications for disclosure, as I have already explained.
625. The Post Office’s case in this litigation will be subject to exactly the same degree of measured and independent assessment, and weighed in the same objective way, as the claimants’ case, and as it would be were these failures not present. However, there must be a change in approach to this group litigation. Disclosing documents late, and in the middle of trials, in such sizeable numbers, and failing (as clearly identified from the passage of Mr Parsons’ own witness statement referred to in [618] above) to act with expedition when clearly relevant documents are unearthed actually during a trial, has two distinct effects. It is disruptive to the proceedings, and it leads to an increase in the costs and/or causes delay. It is the antithesis of co-operation, which the Civil Procedure Rules expressly require. I consider that these passages within this judgment should provide sufficient encouragement to a more constructive approach to prompt disclosure of obviously relevant documents when (or if) they are discovered shortly before, or even during the course of, forthcoming trials.
626. A PEAK discovered on 29 March 2019 relating to the Drop and Go bug ought to have been disclosed to the claimants within a period of about 48 to 72 hours, not two months. With modern electronic communication, that is a readily achievable timescale.
Further post-trial disclosure
“We write further to the email sent by the Claimants' solicitors to the Managing Judge's clerk on 27 September 2019 at 12:29, which brought to the Managing Judge's attention that 32 documents relating to the Horizon Issues Trial were disclosed by Post Office on 25 September 2019.
During the course of responding to a number of queries that the Claimants' solicitors had raised in relation to the disclosure of these documents (which can be located at [H/358] of the Horizon Issues Trial Bundle), it has come to Post Office's attention that a statement within its EDQ is incorrect and that there may be previous versions of the Known Error Log that have not been disclosed to the Claimants. Please find enclosed a letter which was sent to the Claimants' solicitors today explaining the background to these documents coming to light and the proposals which have been made in respect of providing any disclosure which may be sought by the Claimants.”
(emphasis added)
628. The actual statement that was wrong was further explained in the accompanying letter to the claimants, which stated:
“In respect of your query regarding the fact that intervening changes to KELs are not captured on the face of the KELs disclosed, we have made further enquires with Fujitsu to confirm our understanding that previous versions of the KELs are no longer held. We regret to say that these enquiries have revealed that our understanding was wrong.
As you will be aware, Post Office's EDQ stated that "[t]he KEL only contains the current database entries and is constantly updated and so the current version will not necessarily reflect the version that was in place at the relevant time. The previous entries /versions of the current entries are no longer available".
This statement was based on info provided by Fujitsu. In response to our recent enquiries, however, we were informed by Fujitsu on 30 September 2019 that this is incorrect and that previous versions of KELs are available.
This takes Post Office greatly by surprise. It relied on the information provided by Fujitsu at the time of filing its EDQ that such documents were not available. It is extremely sorry that this information was incorrect.
Regardless of whether these documents may or may not be "adverse", Post Office has taken immediate steps towards arranging for the previous versions of these KELs to be extracted and has instructed Fujitsu to begin producing a script to extract the documents into a readable HTML format. Unless you disagree, once this script has been produced we will ask Fujitsu to extract all of these versions.
When this has been done, again unless you disagree, we propose to take immediate steps to disclose previous versions of the KELs which were referred to by either Dr Worden or Mr Coyne in their expert reports or the joint statements or the bug table, or were for any other reason included in the trial bundle.
Disclosure of the previous versions of these KELs (where there are previous versions) will be given as a matter of urgency.
We invite you to tell us whether the Claimants also wish to be provided with disclosure of previous versions of the KELs which were not referred to by the experts or included in the trial bundle.
Given the seriousness of this matter, we propose to notify the Managing Judge of this development immediately and will send him a copy of this letter under cover of an explanatory email.”
(emphasis added)
629. This was correctly characterised by the Post Office’s solicitors as a serious matter. Further, when these KELs were produced and disclosed, it turned out that there were approximately 5,000 of them. This is an extraordinary number, and far more than were available to the experts for the trial.
630. There was no explanation of the way that what was obviously now incorrect information – that previous versions/entries of KELs were not retained by Fujitsu – had been communicated to the Post Office’s solicitors, or by whom, and there was no explanatory witness statement provided on a voluntary basis. I did not order one, nor did this disclosure interrupt preparation of the draft judgment, or lead to any application for further evidence or cross-examination.
631. I include it therefore for completeness. There are two points that arise from this.
1. It would have been a matter for argument, and an exercise in discretion, as to whether any further submissions or evidence (including cross-examination) would be permitted, had such an application been made. The Post Office were pragmatic and sensible, and essentially left this decision to the claimants. The claimants made no such application. However, had they done so, my preliminary view (without hearing any argument) is that I would not have permitted the Horizon Issues trial to be resumed and further evidence adduced. This is for three reasons. Firstly, there has to come a point when the parties are given a decision on the Horizon Issues. The trial had already been interrupted to a substantial degree by the recusal application. This trial was supposed to be completed in April 2019. Resuming it would have led to that date becoming sometime in early 2020. The over-riding objective requires consideration of other court users in other cases. Secondly, subsequent stages of the group litigation would, once again, become consequentially delayed. It is with a sense of wistfulness that I recall a statement at [16] in Judgment No.3 that this litigation would be speedily progressed. Thirdly, the parties have told me that they wish to mediate before Christmas 2019 and would like to know the outcome of the Horizon Issues trial to assist in that process. This would not be possible if the trial were resumed.
2. This extensive further disclosure does, however, rather reinforce the criticism which the claimants level at the Post Office and Fujitsu. The Post Office are the party in the litigation, and they are the party with the disclosure obligations. They obviously rely upon Fujitsu in this respect. Quite how Fujitsu, a reputable company with a worldwide reputation who have been involved in Horizon for nearly 20 years, could be so wrong on yet another important point going to Horizon is not explained, and will remain unclear. Given the statements made by Fujitsu, relied upon by leading counsel for the Post Office at an earlier hearing and repeated to the court, that the KELs did not contain any material relevant to bugs, errors and defects, a pattern has emerged on the part of Fujitsu. This is that its statements about KELs have been shown to be clearly wrong on more than one occasion, and in important respects.
Criticisms of the disclosure of the claimants
632. I turn therefore to the criticism that the Post Office has levelled at the claimants. It should be borne in mind that the bulk of the disclosure burden fell upon the Post Office, simply because of the nature of the Horizon Issues. However, that does not mean that the claimants were not themselves obliged to give relevant disclosure to the Post Office. This disparity is obvious in the number of documents disclosed by each party, which the Post Office in its Closing Submissions stated was some 517,965 documents by the Post Office, and only 1,525 by the claimants. Obviously that previous number does not include the 5,000 KELs disclosed by the Post Office in September and October 2019, after the submissions. The comparison in the figures demonstrates the different burden upon the claimants and the Post Office. It is also correct to identify that for many categories of very important documents, the Post Office is reliant upon Fujitsu. The claimants are not.
633. The Post Office submitted that despite it making appropriate and narrow requests for Model C disclosure of documents held by those claimants who were witnesses in the Horizon Issues Trial, the claimants have not engaged in discussions concerning the scope of the claimants’ disclosure nor provided disclosure of the documents sought by the Post Office.
634. The Post Office, in the context of its complaints about the claimants’ disclosure, submitted that “it was therefore not appropriate for claimants to serve claimant-specific evidence from current or former SPMs” because of the nature of the Horizon Issues and the terms of the Order that ordered witness statements, and that the Post Office sought to avoid making disclosure requests nearer to the trial. However, once the claimants did so, and served their nine witness statements in evidence in September 2018, the Post Office’s solicitors wrote to the claimants on 22 October 2018, 7 November 2018, 30 November 2018 and 20 December 2018 and did not receive any response. Even though the Common Issues trial was underway for part of this time, I do not see why such letters should have gone unanswered by the claimants’ solicitors. When the response did come, it was to the effect that there would be no further disclosure as this was not envisaged by existing orders.
635. The Post Office contrast this with the disclosure it was prepared to give in respect of specific witnesses, and also stated that “given that claimants had flouted that Order, Post Office sought to understand what searches (if any) had been conducted by claimants so that it could understand whether further disclosure would be required in light of claimants’ evidence. A further request for details by claimants was made by Post Office on 17 January 2019.”
636. I am invited to contrast this lack of response by the claimants, which it is said should be viewed in light of the approach adopted by the Post Office. One example of the further disclosure that Post Office had agreed to give which was outside that ordered by the Court was disclosure made in respect of the “operation of branches by Angela Burke, Akash Patny, Anup Patny, Jayesh Tank, Setpal Singh and Adrees Latif….”
637. The Post Office complains that the claimants “clearly intended disclosure to be a one-sided exercise”, seeking disclosure of this nature but refusing to provide their own claimant-specific exercise. Mr Tank did search, in February 2019, within an archived Subpostmasters “Yahoo Groups” forum where he used to post (which means to make comments on the site) and after doing so, Mr Tank identified a post he made on 13 December 2011 relating to the £195.04 shortfall referred to by Ms Van Den Bogerd, and also a post he made on 29 September 2014 relating to the £600 shortfall that he refers to in paragraphs 6-11 of his witness statement.
638. Mr Tank has also found a letter he received from Post Office’s Agents Accounting Team in Chesterfield dated 13 October 2014 which related to this shortfall (and which Mr Tank referred to in his forum post).
639. Mr Tank was cross-examined about this, and explained that at the time of his first witness statement he “hadn’t fully researched the whole background” and he did not realise at that time that the post he had made previously was as important as it had subsequently become, by which he meant in the trial. He also said he had only become aware of the £195 loss to which it related once he had read Mrs Van Den Bogerd’s witness statement. He also explained that he kept a box file and that was where he found the documents. Complaint is made of the fact that the documents that were disclosed should have been disclosed on 17 July 2018, the date for disclosure. I accept that, and even if one makes the assumption in the claimants’ favour that they did not know at that date that Mr Tank was to be a witness, certainly no later than the date of his first statement (which was 28 September 2018) disclosure ought to have been given. Mr Tank’s lack of understanding of relevance and disclosure should not have arisen, and the existence of his box file should have become known to the claimants’ solicitors far earlier than it was.
640. Further disclosure was also necessary in relation to Mr Latif. This came about because he served an amended witness statement on 1 March 2019, which came about following enquires with Mr Latif's former branch manager, who consulted the documents within the branch to confirm the date on which particular transaction corrections had been issued.
641. He explained this in his cross-examination:
" Q. First question about that: by amendment to your statement you now say that this was in around January 2018 rather than March 2018.
A. Correct.
Q. What caused you to make that correction?
A. I had a look at the -- we hold the records for the information in the office, so I had my assistants look at the records, transaction logs and that's when I confirmed that it was January rather than March. (Inaudible) was logs in March as well. We made calls effectively every month to Horizon help desk concerning this issue.
Q. When do you say you asked your assistants to check about the date?
A. Yes.
Q. Sorry, when do you say that happened?
A. It was after I made the initial statement, I was checking.
Q. Roughly when, Mr Latif?
A. It would have been a few weeks ago, sir.
Q. So is it right that you didn't check those records from the branch before making your witness statement?
A. No, I thought I was correct but I double checked and made sure that actually in fact they were correct, those (inaudible), so I was right but initial incident happened in January when TA (inaudible) transaction acknowledgement, the TC, the corrections, they came in March.
Q. Okay. So you say, do you, that the TCs in relation to these transaction acknowledgements came in March 2018?
A. Yes.
Q. And you say that's something you have checked from your records?
A. Yes."
642. The Post Office made requests for disclosure during the trial, on 15 March 2019 (which was the end of the first hearing week) requesting disclosure of various documents relating to Mr Latif and Mr Tank and their answers in cross-examination. This also included requests of CCTV recordings which Mr Tank said he had looked at. These requests were refused on 27 March 2019, and the claimants stated that this refusal was on the basis that they were either outside their control, no longer existed or were not relevant to the claimants’ evidence and were not relied upon at trial.
643. Documents that were held in Mr Latif's former branch were plainly outside of his control, even though his former branch manager had accessed them previously. However, that does not mean that the former branch manager could not at least have been asked for them. Neither the claimants did that, nor indeed did the Post Office (so far as I can tell). I consider the claimants’ attitude in this respect unhelpful, although it should be noted that for at least the documents in respect of Mr Latif, the Post Office could have taken its own steps in this regard.
644. There are some unsatisfactory aspects to the Post Office’s submissions about the CCTV footage. Mr Latif clearly said in his first witness statement at paragraph 7 “I also checked the CCTV to make sure that I had performed the transaction correctly.” Secondly, the Post Office’s closing submissions refer to requests for “the CCTV recordings which were consulted by Mr Tank for the purposes of drafting his witness statement”. He was asked about the CCTV during his evidence and did not say that he had consulted it for the drafting of his witness statement, he said that he had checked it at the time. Had the Post Office wished, it could have asked for that footage shortly after receiving that statement. It is also the case that the event in respect of which he referred to the CCTV occurred in July 2015. It may be that the CCTV images do not still survive, so long after the event in question, but this point was not pursued in cross-examination.
645. There is, however, one major difference between the claimants and the Post Office in this respect. The Post Office were entitled to, and did, explore with specific witnesses certain issues in respect of disclosure by asking the relevant person (in the above example, Mr Latif) questions about it. The claimants did not have this opportunity for the following reason. The majority of complaints made about the Post Office’s disclosure arise as a result of Fujitsu. Save for rare circumstances, such as Mr Parson’s specific witness statement about the Royal Mail issue, all that the claimants and the court are usually told is “Fujitsu say X” or “Fujitsu told the Post Office” such and such. Specific individuals are not identified, and they are certainly not called as witnesses to be asked questions about their approach, or (for example in respect of KELs) how they could have been so factually wrong in their explanation of the KELs’ contents. I have formed a dim view of the accuracy of information of this nature coming from Fujitsu, regardless of the person in that organisation responsible for communicating it.
646. Notwithstanding this, I wish to make one matter completely clear. If a witness refers to a particular document in their oral evidence, and the other party in this litigation wishes to see a copy of that document, then (absent issues of privilege) that ought to be disclosed, even more so if a specific request is made. This is by far the most sensible and proportionate way forward. It is in no party’s interests to engage in squabbling about relevance of a specific document in those circumstances. It costs far more to argue back and forth in solicitors’ letters than it does simply to give disclosure. In any event, the relevance of the document is made clear by the fact that it has been referred to orally in evidence.
647. It is also correct to say that notwithstanding the different burden upon the different sides in terms of disclosure in the Horizon Issues trial, which has undoubtedly been far heavier for the Post Office, this burden will not always be borne in these proportions. The Post Office also has the benefit of a far higher number of legal advisers. As at the date of this draft judgment, simply in terms of the total number of leading counsel engaged in the group litigation, the Post Office outnumbers the claimants by 4:1. In terms of the number of solicitors’ firms, the Post Office currently outnumbers the claimants by 2:1; although only one firm was acting for the Post Office on the Horizon Issues, and the second firm acted for the Post Office on its unsuccessful attempt to appeal the Common Issues.
Summary on disclosure
648. The Post Office is, in my judgment, somewhat at risk of downplaying certain unsatisfactory aspects of its own disclosure that have emerged during the Horizon Issues trial. For example, the incident concerning the incorrect statements made to the court about the Royal Mail refusing disclosure without an order of the court was described in the Post Office’s closing submissions as a “mix up” which was “regrettable but should not be the subject of criticism of Post Office.” I do not consider telling the court something in express terms which is factually wrong can properly or correctly be identified as merely a “mix up”. Two points that were expressly explained by its leading counsel were simply not factually accurate. The Royal Mail was not refusing disclosure, and it did not require an order of the court before it would provide certain documents. It had not even been asked for them.
649. Similarly, in its closing submissions the Post Office referred to the interlocutory discussions about the Known Error Log as a “debate about relevance”. That again downplays what had happened, including the positive submissions made, that the Log was specifically not relevant and even if an order were made in respect of documents showing bugs, errors and defects, the Log would not be disclosed (even if it were in the Post Office’s control, which it was stoutly maintained it was not). The submissions made by the Post Office on the adequacy of its own disclosure have been rather undermined, since the trial ended, by the production (emanating from Fujitsu, once again) of over 5,000 KELs, something rightly described by the Post Office’s solicitors as “a serious matter”.
650. The claimants have not been entirely co-operative on their own disclosure, however. I consider that the criticisms of the claimants for serving what was described as “claimant specific” evidence are unjustified. At least one of the experts, Mr Coyne, adopted a specific approach and investigated specific issues or occurrences, and there is no reason that both of the experts could not have done so. Dr Worden did not, but that does not mean that the claimants were wrong to do so. Some of the criticisms made by the Post Office regarding the claimants are unjustified, therefore, but not all of them are. Although I do harbour the suspicion that the Post Office may have decided that, on disclosure at least, attack may be the best form of defence, some of the criticisms made in the Post Office’s closing submissions are justified. For example, for the claimants’ solicitors simply to state that “at present, it is not the Claimants’ intention to make another round of disclosure in relation to the Horizon Issues trial” as was done in July 2018, is not co-operative. Further, the reason that documents which Mr Tank had in his box, that were plainly relevant, were not disclosed was because Mr Tank had not been asked for them. Had he been asked – which is something that the claimants’ solicitors should have done, once it was known he would be a witness – these could and would have been disclosed. They certainly should have been.
651. The following points are of general application to both parties in this litigation. At some point a party who seeks to complain to the court about disclosure ought either to shelve its general disquiet on this subject, or make an application.
652. It is important to identify, so that the parties in this group litigation are aware, the court’s approach to disclosure. This is given for the purposes of guidance, and cost-effective and efficient progress of the group litigation going forwards.
1. Care must be taken when explanations or submissions are made to the court. If further time is needed in order for accuracy to be achieved, that time should be sought. Those giving instructions to counsel must be astute to ensure that they are giving correct information.
2. All the substantive issues will be resolved on the evidence, the applicable law, and taking account of submissions. A party does not obtain a preliminary advantage or “head start” on the substantive issues because of any criticisms about disclosure.
3. Criticisms about the other party’s disclosure should be proportionate. Depending upon the circumstances and the nature of the document, the fact that disclosure is given late might be relevant, but simply because continuing disclosure is given by one party does not mean that there is anything sinister lurking in the background.
4. The court can always be asked to resolve disputes about documents. Mr Coyne sought a large number of different types of documents, but no applications were made by the claimants even though these were not provided. Criticisms of the Post Office’s disclosure by the claimants has to be seen in the light that the claimants did not consider it sufficiently important at the time to make any applications to the court. This is not to encourage applications. Sensible and well-advised parties who co-operate will usually deal with almost all disclosure issues consensually.
5. The parties are expected to comply with their disclosure obligations. If a document comes to light that should have been disclosed but has not been, it should be provided promptly. If this is discovered actually during a trial, it must be disclosed very rapidly indeed. As happened with the document referred to at [618] above, a document concerning the Drop and Go bug was discovered actually during the Horizon Issues trial, then (as Mr Parsons put it in his explanatory witness statement) “for the next couple of months” kept by the Post Office’s solicitors while “research” was done. It was then disclosed just before resumption of the trial “in a batch”. This is not working to the time scale either required, or expected, by the court, and it is not the correct way to deal with disclosure during a trial. This is a laissez-faire approach to disclosure of an important document during the trial itself, and the document should have been disclosed very promptly.
6. Both parties will be held to the same standards. The Post Office has, so far, and given the issues that have been tried, borne the bulk of the disclosure burden. The claimants are expected to observe their disclosure obligations in the same way, and will be expected to do so in the future when further issues and the claimants’ different cases are tried in 2020.
7. Witness statements will be ordered from a party when the court requires an explanation. Nothing else should be read into the ordering of a witness statement, other than it means precisely that -- an explanation is required.
8. It is obvious that the Post Office has had to rely upon Fujitsu to a large degree. However, given it was Fujitsu who told the Post Office what the Known Error Log contained – see further [586] above – Fujitsu has, so far, shown itself not to be entirely reliable in this respect. Fujitsu are also responsible for the Post Office making a directly incorrect important statement in its EDQ about retention of KELs, which led to the disclosure of about 5,000 of these some months after the trial closed.
9. Adoption of the Model C procedure does not mean that parties can keep adverse documents to themselves and not disclose them until a specific request is made. Such adverse documents should be disclosed promptly.
10. Finally, disclosure is very expensive. The court will be astute to guard against it becoming either satellite litigation or a weapon in the interlocutory arsenal.
653. I have exhorted the parties to co-operation many times before in this group litigation. I will continue to do so. Eventually, it is to be hoped that they will emerge into the broad sunlit uplands of the over-riding objective on disclosure matters.
G. The Experts’ Agreements
654. There were four experts’ agreements in this case. Experts’ agreements are very important. Permission to rely upon IT expert evidence was granted in the First CMC Order dated 25 October 2017. The two experts met on a number of occasions and agreed certain points.
655. As will be seen from Part H of this judgment, the experts approached their respective exercises in very different ways. However, they did reach some areas of agreement that were very helpful. Their joint statements or experts’ agreements did however also set out a considerable number of areas of disagreement. As the trial approached, I requested through the parties that the joint statements of the experts focus more on areas of actual agreement, rather than recitations of position and disagreements.
656. The 1st Joint Statement was dated 4 September 2018 and set out the following, inter alia, at the very beginning of the statement:
“Each expert's approach to writing his report, and to this joint memorandum which foreshadows their reports, could broadly be one of three possible approaches:
a) To focus mainly on negative points found in the disclosed documents about where Horizon may have fallen short.
b) To focus mainly on those aspects of Horizon which were intended to achieve robustness and reliability, and the evidence implying that they succeeded.
c) To provide the court with a clear foundation for understanding the design and operation of Horizon; then, building on that foundation, to provide a balanced assessment of the ways in which Horizon succeeded, whilst addressing any disclosed issues where Horizon may have fallen short.”
657. This was the first area of disagreement. Mr Coyne stated in the section immediately following, entitled Areas of Disagreement:
“In my opinion the technical issues can be answered without reference to the Claimants high level allegations document. The issues are about how Horizon and its interactive components operated and the processes employed by Post Office and Fujitsu in supporting these systems and the data within.
Whilst my report will take a balanced approach, it is the case that many of the issues require a deep focus on the occurrences of bugs, errors and defects as well as the potential for modification of transactional data. Whilst context will be provided as to how Horizon should work in typical circumstances, it is the non-typical operation where focus will be placed.”
658. This approach was not agreed by Dr Worden. He stated:
“I intend to take the balanced approach (c).
To provide a joint statement at this stage which will be useful to the court, under each Horizon issue the experts need first to agree what is worth agreeing having regard to the case which is being alleged. For instance, it would not help the court for the experts to list a number of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case alleged; or if collectively the points imply a view which, in the opinion of either expert, is not a balanced view. Therefore, the areas of agreement may at this stage be limited and more high level.
My approach will also focus on the Horizon system. It will consider Horizon within its wider business context, but I do not believe that the Horizon issues extend to offering any opinion on the wider business practices of Post Office.”
659. I consider that the balanced approach, which both experts stated they would be adopting, was the correct one. However, I do not find the following statement by Dr Worden in this 1st Joint Statement as helpful:
“under each Horizon issue the experts need first to agree what is worth agreeing having regard to the case which is being alleged. For instance, it would not help the court for the experts to list a number of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case alleged.”
660. It is a matter for the court to consider how helpful a particular point of detail, agreed by the experts, is or is not to the issues to be decided. Obviously the parties can make submissions about that, but it is not for an expert to decide that a point of detail, upon which he may be agreed with his opposite number, should however not be agreed because that point will not be helpful to the court; or that it should not be included in a joint statement. It is also not for experts to direct the court in any particular direction. The Horizon Issues were carefully drafted by the parties, approved by the court, and specifically included in an Order. They were the issues that the experts were to agree, and detailed points of an IT nature that arose within those Horizon Issues should, where possible, have been agreed.
661. Because Mr Coyne and Dr Worden approached their tasks so differently, the former inspecting a number of PEAKs and KELs about 10 times greater in number than the latter, this meant that a great amount of very detailed matters shown within those documents were not agreed by Dr Worden. He approached his expert exercise from what he described as “a top down” approach, a methodology that found its most stark manifestation in his statistical analysis of likelihood in his Section 8.
662. I will include within this judgment, not the full text of all four of the different Joint Statements, but some lengthy passages. This is to identify the large degree of agreement between the experts. I will however omit (with some isolated exceptions) the passages where the experts set out their separate, not-agreed, views.
The 1st Joint Statement
663. I also quote at this stage from the 1st Joint Statement, signed on 4 September 2018, and the points of agreement and disagreement in respect of Horizon Issue 1. This is because in my judgment it is an important point of agreement, in respect of an important Horizon Issue, but also it is useful to set out the approach within the Joint Statements.
“2. Issues and Position
2.1 Issue 1
Issue 1 – To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?
Areas of Agreement:
Evidence exists that bugs/errors/defects have caused actual discrepancies or shortfalls relating to Subpostmasters’ branch accounts/transactions.
Each time any IT system (including Horizon) is changed there is the potential to introduce new bugs/errors/defects.
Once bugs/errors/defects are discovered, they take time to resolve and therefore systems such as Horizon often continue to operate with bugs/errors/defects with or without workarounds in place.
Theoretically, bugs/errors/defects that existed within Horizon have the potential to cause apparent or alleged discrepancies or shortfalls in relating to Subpostmasters’ branch accounts/transactions.
In the event that any specific bug/error/defect had such an effect, the experts have differing views as to the ‘extent’ of the impact that such bugs/errors/defects may have had on branch accounts.”
(Emphasis added)
The statement continues:
“Areas of Disagreement:
Jason Coyne: As identified by select Known Error Logs (KELs), many thousands of bugs/errors and defects occurred and many of these known errors had impacts that recurred in different circumstances, therefore the reliability of Horizon to accurately process and record transactions is questionable.
Such facts undermine the reliability of Horizon to accurately process and record transactions.
Examples:
(i) the Calendar Square/Falkirk bug – which led to Horizon failing to recognise transfers between different stock units, thereby affecting Branch accounts;
(ii) the Payments Mismatch defect – which affected at least 62 branches and was related to the process of moving discrepancies into the local suspense account, thereby affecting Branch accounts. This defect was not capable of being identified by Subpostmasters, who would have believed from Horizon that their account was balanced when it was not; and
(iii) the Suspense Account bug – which erroneously replicated suspense account items for 14 branches from 2010 in the same monthly trading periods in 2011 and 2012, thereby affecting Branch accounts.
In my opinion there is possibility of bugs/errors/defects existing in Horizon that have not yet been discovered. Further, most bugs/errors/defects that exist in Horizon have existed for extended periods of time before they were discovered.
I understand that Horizon had at least 19,842 ‘releases’ of new software, each one of these could have introduced new bugs/errors/defects.
Dr Worden:
My current preliminary view is that undetected bugs/errors/defects in Horizon were very unlikely to be the cause of permanent shortfalls or discrepancies in a branch's accounts.
It is necessary to define measures for the 'extent' of issue 1. At least two measures are possible: the number of such bugs and errors, and their net expected financial impact on claimants' branch accounts, compared to the total shortfall experienced by all claimants, which is of the order of £18 million. I intend to assess both measures. The latter measure may be more useful, because if any set of bugs has expected financial impact much less than £18M, it can do little to account for the claimants' shortfalls, either collectively or individually.
I propose to assess these measures from a number of different sources of information: (a) the defects of Horizon cited by the claimants in para 1.3 of the outline; (b) the KELs identified by the claimants in para 1.4 of the outline; (c) other KELs; (d) data on their shortfalls provided by the claimants in their schedules of information; and (e) data from other sources (if available).
I am still scoping the data needed for (e). If it is available, I will provide it to Mr Coyne as soon as I have it.
My preliminary analyses of the data on (a) - (d) imply that:
• For the majority of KELs, (which record known bugs), it can be shown, from an understanding of the robustness measures built into Horizon or otherwise, that they would have no permanent impact on branch accounts.
• The net financial impact on claimants' branch accounts of any other bugs, including undiscovered bugs, is very small compared to the claimants' total shortfalls. I am not yet able to quantify this, but I intend to do so.”
664. In my judgment, this presages Dr Worden’s statistical analysis of likelihood, which I deal with as part of Section I. The following text used by Dr Worden should be considered:
“….if any set of bugs has expected financial impact much less than £18M, it can do little to account for the claimants' shortfalls, either collectively or individually”
The phrase “much less than £18M” could, potentially, direct expert consideration of Horizon Issue 1 away from what it is in fact asking, towards a different issue entirely. The experts were not tasked only with investigating bugs, errors and defects that had a proven financial impact of (say) £18M, £10M, or any financial figure at all, and certainly not one in the millions of pounds. This was for the following reasons. It was never part of the Post Office’s position at the case management stage when the Horizon Issues were discussed and considered, that there should be any sort of threshold for financial impact, above which the experts should investigate, and below which they should not. The wording of Horizon Issue 1 has already been addressed above, and this uses the terms “possible or likely”. This notion of imposing a financial threshold must have come from Dr Worden.
665. Nor is the litigation intended to be a wide-ranging enquiry into all of the Post Office’s accounts with all of its SPMs over the 15 or 16 year period in question. The Post Office’s position is that Horizon is sufficiently robust such that it cannot be the cause of the different shortfalls complained of by the hundreds of different claimants. The counter-position to that by the claimants is not that particular discrete sets of bugs had, individually, an impact of millions of pounds upon their branch accounts. If that were the claimants’ case, then Dr Worden’s approach as set out in this joint statement might be understandable. Rather, the claimants’ case is that there are a large number of different bugs, errors and defects that each affected each of their branch accounts over time, with the end result being that each of them was (in accounting terms) identified as owing the Post Office sums in the tens of thousands, or thousands of pounds (or less) with the consequences which are well known. Some discrepancies were only a few pounds, but are said to have occurred many times. Some were said to be tens of thousands of pounds cumulatively. Some occurred many times; others less often, but with greater individual effect. One bug – of which more later – is said to “double up”, which would mean an exponential effect as a loss doubles, then quadruples and so on. If one imposes, at the beginning of the expert investigation process, a filter or threshold that looks for a “set of bugs” causing losses of “millions of pounds” that will have the effect of restricting the field of investigation. There is no reason for such a restriction, in my judgment.
666. On Horizon Issue 2, the experts were agreed, so far as that issue was concerned, that the extent to which any IT system can automatically alert its users to bugs within the system itself is necessarily limited, and while Horizon has automated checks which would detect certain bugs, there are types of bugs which would not be detected by such checks. The experts were also agreed, in respect of Horizon Issue 6, that whilst Horizon contains measures and controls for detecting system integrity concerns, the automatic mechanisms have failed in the past. The experts did not agree as to the ‘extent’ of prevention, detection, identification, reporting or risk reduction that the automatic and manual control measures delivered; however, the agreement that automatic mechanisms have failed in the past, which was contained in the 1st Joint Statement, is in my judgment important. It means that the Post Office cannot rely upon the presence of automatic mechanisms or processes as being an answer to the consequences of failures in system integrity.
667. It was also agreed by the experts that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to the SPMs, but that other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible to identify from reports or transaction data available to the SPMs in question, because of their limited knowledge of the complex back-end systems. Identification required the cooperation of PO staff and the SPMs. Again, I consider this expert agreement to be important. This was, essentially, the essence of the complaints raised by Mr Bates himself in the very earliest days of Legacy Horizon, namely that there was or were insufficient information or reports available to him to enable him to identify the causes of apparent or alleged discrepancies. The experts are agreed that, for some causes of apparent or alleged discrepancies this is indeed the case, due to the complexity of the back-end systems (back-end means not at the branch) and the limited knowledge a SPM would have.
668. The experts agreed that Horizon had evolved, and its robustness may have varied throughout its lifetime. They also agreed that the level of robustness may have increased or decreased as the system was changed. The existence of branch shortfalls was agreed, but they were not agreed at the stage of the first joint statement that this indicated any lack of robustness. They were also agreed, as at the date of their 3rd Joint Statement, that Horizon as it is in 2019 is relatively robust. However, given the span of the years under consideration is 2000 to 2010 for Legacy Horizon, and 2010 to date for Horizon Online, and given the agreement to which I have just referred that robustness may have increased or decreased, this only takes one so far.
669. On Horizon Issue 4, the experts agreed that there are a number of actual reported errors in data recorded within Horizon arising from (a) data entry, (b) transfer or (c) processing of data. Therefore, the potential exists. The experts do not agree as to “its “extent”.” Dr Worden saw this issue as a subset of Horizon Issue 3. He felt that because of the measures “built into Horizon”, errors of data entry, transfer and processing were very unlikely to affect branch accounts. The experts agreed on Horizon Issue 6 that the automatic measures and controls for detecting system integrity within Horizon had failed in the past; they could not agree as to the extent that the automatic and manual control measures delivered. Dr Worden also made the point that bugs, when discovered, could not be fixed instantly and what he called “trade offs” were required when dealing with practical release management.
670. So far as remote access and Horizon Issue 7 was concerned, at the time of the 1st Joint Statement Dr Worden was unclear about this and the technical document provided to him by Mr Coyne had not been received in sufficiently timely manner such that he could consider it. By the time the Horizon Issues trial had started, the Fujitsu witnesses such as Mr Parker had corrected their earlier incorrect witness statements and accepted that this could be done. However, those statements were not served until November 2018 so would not have been in existence as at the time of the 1st Joint Statement.
671. On Horizon Issue 9, the experts agreed that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to SPMs. Other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible to identify from reports or transaction data available to SPMs, because of their limited knowledge of the complex back-end systems. Identification required the cooperation of PO staff and SPMs.
672. On Horizon Issue 10, it was agreed that by the very nature of rolling out fixes in any IT system, including those implemented by Fujitsu, these had the potential to affect transaction data or data in branch accounts. This could be seen as stating the obvious, but at least the experts agreed it. The experts were also agreed that the use of tools and facilities to do this should be auditable, however the maintenance of logs recording this would be dependent upon retention periods and the size of the logs in question. Again, this could be seen as stating the obvious. The experts were not agreed on Issue 12, how many times the ability to do this was used, if at all. Nor was there any agreement on Issue 13, the extent to which such facilities had the potential to affect the reliability of branch accounts; Issue 14, a detailed issue with 7 sub-issues concerning functionality; or Issue 15, which concerned Transaction Corrections or TCs.
673. The experts agreed, so far as the existence and use of permission controls is concerned, which is the essence of Horizon Issue 11, that the usage of tools and platforms which existed specifically for the purpose of accessing and modifying transaction data, should be auditable. It was agreed however that the maintenance of logs would be dependent upon retention periods and size. Such tools which were identified initially in the 1st Joint Statement under Horizon Issue 10 areas of disagreement by Mr Coyne, were at that stage split into the following:
1. Global Branches which would enable the input of transactions within Horizon as though it had come from an actual Branch;
2. the Branch Transaction Correctional tool;
3. the Transaction Information Processing repair tool.
674. The best description of “Global Branch” or “Global User” is contained in a Post Office document headed “Global User Process Management” which said that it allowed people such as auditors “to be able to go out to any branch and use the same log on to access any Horizon kit in Branch without having to request a single use password at each branch. The individual requiring a Global Use account has to be approved by a Manager showing on the Global User requester and approver list”. However, the document also identified that PWC had discovered that “the current Global User list inherited from Fujitsu contained employees who had left the Business. This raised concerns over the security of access to Horizon. Although this is a low risk in terms of Horizon access (the individual would have to gain access into a Branch first) it is still a risk and needed to be mitigated.” There were 32 such former employees. It was in respect of the permission control procedures concerning these tools that the documented authorisation process was identified, using documents called the “Master Service Change” or MSC documents and “Operational Control Procedures” or OCPs. These documents were requested by Mr Coyne but no actual disclosure application was issued. The document to which I have referred explaining Global User was in the trial bundle.
The 2nd Joint Statement
675. The 2nd Joint Statement is dated 25 February 2019 and identified and collated the different bugs, most of which were given actual names, which had either been identified and agreed, or which Mr Coyne maintained were bugs. This statement was in landscape form and lengthy, and included what became to be referred to as “the bug table”. I adopt the terms used by the experts for the descriptions of the different bugs, or alleged bugs. The number of different bugs that were agreed by Dr Worden, by the time the trial ended, was 11. The description of this in the joint statement itself was
“The structure of the document captures i) A table of bugs/errors/defects containing evidence of financial impact upon branch accounts that both experts agree (or indicate if they do not), ii) all expert agreements grouped by Horizon issue, and iii) additional comments and observations input by the respective expert.”
676. Prior to the commencement of this litigation, the Post Office had acknowledged the existence of only two bugs in Horizon. Mr Coyne’s evidence was that there were 29 relevant to Horizon Issues 1 and 4, and he clarified in his cross-examination that the number that he considered had lasting impact upon branch accounts was 21. I deal with that cross-examination at [794] to [798] below.
677. The 2nd Joint Statement also said in its Introduction that:
“Because of time pressures and the complexity of the issues, we have not been able to address all the Horizon issues in this joint statement. We will issue a subsequent joint statement addressing those issues we have not yet addressed. For those issues we have addressed in this statement (issues 1, 2, 9, 14 and 15), the layout and references are not as polished as we would have wished.”
678. It is extremely useful for the court to have Joint Statements from experts, and I am grateful to the experts for producing these statements, in particular what was called “the bug table” in the 2nd Joint Statement, in what is a very complex case, and when they were under considerable time pressure. The bug table was very widely used in both cross-examination by the parties and indeed in their submissions.
The so-called “Bug Table”
679. The bug table, and my findings upon the bugs within it, are in my judgment central to the Horizon Issues. Evidence concerning bugs in the Bug Table formed a large part of the expert evidence that was tested in cross-examination. The different bugs, or alleged bugs, identified in the Bug Table were as follows. I have added to the following the summary of the position of the Post Office in Appendix 2 of its Closing Submissions, which in some cases accepted there was such a bug, and qualified that acceptance in most instances by references to what was said to be the impact of the bug. I deal with detailed findings on those bugs that are in issue in the Technical Appendix to this judgment. For the detailed submissions by the Post Office for each bug, the paragraph numbering in Appendix 2 started again at 1 for each bug. This did not cause particular difficulty, and because the Post Office explained that Appendix 2 had been put together by different parts having been drafted by different counsel/solicitor teams, this is understandable. I refer to it here, because otherwise a reader of the passages below might become confused as to why there seem to be so many paragraphs all (for example) numbered 2 in Appendix 2. The following summary of the Bug Table is intended to be a useful summary for this judgment, and is not restricted solely to the entries within the 2nd Joint Statement.
1. Receipts and Payments Mis-match bug.
This was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts. Dr Worden added in his comments that “Therefore, the extent of this bug is well established, in the GJ analysis.” GJ is Gareth Jenkins. The identified year of its effect was 2010. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In the summary in Appendix 2, the Post Office stated that “In the event, however, this bug resulted in transient impact only.”
2. Callendar Square/Falkirk bug.
This too was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts. Its identified years of effect were 2000 – 2006. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Dr Worden added different comments, including that “the bug arose from a fault in the underlying Riposte software, so it is not surprising that it took Fujitsu some time to understand it, or that they had to rely on the suppliers to fix it. It does not show poor system design or support by Fujitsu”. That latter sentence is a little surprising, and seemingly very defensive on Fujitsu’s behalf. This is because the Horizon Issues are all about the operability and functionality of Horizon, not who is to blame for the presence of any bugs, errors or defects. Riposte software was part of Legacy Horizon. The fact that it is a product designed and supplied by Escher, and not Fujitsu, is not relevant to the dispute between the Post Office and the claimants. In any event, and as accepted and recited in paragraph 2 of the Summary of Appendix 2 dealing with this bug, “Mr Coyne asserts that Bug 2: Callendar Square is a bug with lasting financial impact and in JS2, Dr Worden appears to agree that there is strong evidence of this….”
3. Suspense Account bug.
This was agreed in the bug table as an “acknowledged bug” and its identified years are 2010 to 2013. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. Dr Worden’s comments do not include that it had an agreed impact upon branch accounts. This is because of a concept he introduced (expanded in his reports) of “transient effect”. What this meant was that if there was an impact on branch accounts by something within Horizon, but that was then corrected by a TC, he concluded it had a “transient effect” upon branch accounts and dealt with it differently. I address this concept below. Other comments of his in relation to this acknowledged bug are:
“It was a transient effect arising not from a fault in the software, but from a change in database archiving policy in 2010. The delay in correcting it arose from a failure of communication between PO and Fujitsu. Because the bug would only manifest itself annually for any affected branch, the effects of this delay were not widespread.
Peak PC0223870 shows that Fujitsu were able to identify the branches affected, even when Subpostmasters did not report it. There is evidence that the branches were compensated, as I would expect from the normal error correction processes.”
4. Dalmellington bug/Branch Outreach Issue.
This was not acknowledged as a bug in the bug table, but is effectively accepted by the Post Office as a bug. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient impact”. Indeed, for some time during the trial the Post Office would refer to this as “an issue”. I find that it is a bug – that much does not seem to be controversial, as paragraph 2 of the detailed part of Appendix 2 relating to this bug states the following:
“This is a bug which Mr Coyne states has lasting impact on branch accounts. Post Office submits that there was no lasting impact on branch accounts.”
The years of its effect were 2010 to 2015. It is named in respect of the branch where it was “discovered” in 2015, although the investigation undertaken by Fujitsu in that year identified previous occurrences. There were over 110 occurrences of it over this 5 year period (the actual number is probably 116, but at least 114) with 88 branches affected. Some were impacted multiple times. It is what is called a “cash remming error” and only happened in certain circumstances. It was explained further by Mr Godeseth in his evidence at [445] onwards above. It would, in my judgment, have an impact upon branch accounts – it is the extent of that impact that is in issue, whether it was lasting or transient. Dr Worden relied upon what he described as “a well-tested process of reconciliation and TCs to detect and correct errors in cash remming (used 20,000 times per year)” and said it was “straightforward for Horizon to detect any discrepancy between a “rem out” and the corresponding “rem in” (a mismatch arising either from a miscount, or a multiple count of a pouch) and then a TC can be issued.” He also added that “this process catches and corrects remming errors, whatever their cause - including if they arise from, or are provoked by, software faults.” He therefore implicitly accepted that the Dalmellington bug was a software fault, although he did not say so in terms. I find that it was plainly was. He was also reliant upon the process of TCs to correct it. Further, the bug was there for 5 years and was not discovered, although its effects doubtless were. I find that this was a software bug that impacted upon branch accounts and I will come to the nature of its effect in my findings on the bugs that are not agreed.
5. Remming In bug.
This was not acknowledged as a bug in the bug table. However, Dr Worden accepted it was a bug or defect in his cross examination and it is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is again said to have had transient impact. The years it was said to have been present in terms of effect were “March – August 2010 and recorded as fixed approx. 2011”. Mr Coyne said there were 14 branches affected. Dr Worden’s entry in the Joint Statement relied upon TCs, and stated “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts.” In paragraph 2 of the detailed part of Appendix 2 relating to this bug, the Post Office submitted that “any discrepancy would be transient as instances of this bug are caught by automatic reporting.”
(emphasis added)
The answer to this is therefore dependent upon my answer to the point concerning lasting and transient impact.
6. Remming Out bug.
This was split into two in the bug table, 6(i) which was identified in KEL acha508S and 6(ii) which was identified in KEL GMaxwell3853P. The former was identified in February/April 2007, recorded as fixed approx. 2007, with Mr Coyne identified 57 branches affected; and the latter was in May 2005 with one branch being affected. They were both remming out issues, hence the grouping by the experts in the bug table. It was not acknowledged as a bug in the bug table. However, it is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Again, for both of these iterations, 6(i) and 6(ii), Dr Worden used the same wording as he had in the entry for bug 5, the Remming In bug, namely “As for the Dalmellington bug, above – PO had a robust process for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on branch accounts” (emphasis added). In paragraph 2 of the detailed part of Appendix 2 dealing with this bug, the Post Office stated that this “comprises two separate issues, only one of which was a bug. Any discrepancy caused by either issue would be transient as instances of both issues were caught by automatic reporting.” It was also said that Mr Coyne had conflated two unrelated issues under the heading “Remming Out”. Given this was an entry in the 2nd Joint Statement agreed by the experts, it is a little unfair to state that Mr Coyne had done this, as the entry was obviously agreed by Dr Worden. However, it was split into 6(i) and 6(ii) and I will consider each separately.
7. Local Suspense Account issue, not the same as 3. Suspense Account bug.
This was reported in 2010 and recorded as fixed in September 2010. It was not acknowledged in the bug table, but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, although it is one of those bugs said to have had transient impact. Mr Parker had identified 33 branches affected. Mr Coyne recorded what he said were four associated KELs, namely acha5259Q (for which there were 6 PEAKs)(this KEL is mistakenly recorded twice in column 3, with acha5838T only mentioned in the text in that column); cardc2043L (10 PEAKs); PorterS199P (3 PEAKs); and acha5838T (which states there are “two different but similar problems” and appears in the text, but not in the list of KELs at the end of the text by Mr Coyne). Dr Worden’s comments were that
“The KEL acha5259Q implies that PO and Fujitsu were able to identify all occurrences of the problem, without being notified by any Subpostmaster. I would therefore expect them to have corrected any impact on branch accounts as part of normal error correction processes.
I would not expect evidence of all corrections to accounts to have survived to the present day. PEAKs and KELs are not used to record corrections of financial impact.”
He also relied upon a statement by Mr Parker that there was ‘Temporary financial impact which would have been cancelled out in the following period by a corresponding discrepancy’.
This introduces another concept, similar if not identical to Dr Worden’s “transient impact”, and that is “temporary financial impact”. Dr Worden is correct that PEAKs and KELs do not record corrections of financial impact; they do however record financial impact, because when an SPM reaches SSC (which raises the PEAK) they often start by recording “SPM says he/she has a problem in that……” and financial impact is often then recorded. The Post Office in paragraph 2 of the detailed part of Appendix 2 dealing with this bug states that “any discrepancy would not be lasting”. This does therefore accept, albeit implicitly, that it would have an impact upon branch accounts. It is also submitted that it was a “teething problem” from the early days of the Horizon Online pilot scheme.
8. Recovery Issues.
The Post Office does not accept these are bugs at all. These are not agreed as bugs by the experts, and there are four different types included in the table, with years of effect from 2010 to 2018. Mr Coyne’s entry stated that “The text within the PEAKs and KELs suggests that in each case a branch account discrepancy would be evident and would require correction by the Post Office.” Dr Worden stated that:
“The KELs and PEAKs cited by Mr Coyne are not indicative of errors in Horizon. They provide guidance on how to correct discrepancies caused by human errors or other errors in transaction recovery ('recoverable transactions')
Because there were many such errors, there were many calls to the help desk and many PEAKs and KELs. Normally, correction of errors involved back office reconciliation and issuing TCs. This was accurate and effective; I have derived an upper limit of £2 per branch per month on the mean impact of erroneous TCs.
One important KEL acha959T was guidance to the back office MSU, not for Subpostmasters”
There was a minor typographic error in the submissions in paragraph 4 of the detailed part of Appendix 2 which referred to bug 9. However, the four different types of recovery issues are addressed in the subsequent paragraphs and the conclusion paragraphs deal with bug 8. This was later corrected in a sheet of corrections, which clarified that paragraph 4 of the detailed part of Appendix 2 should have stated "Mr Coyne states that Bug 8: Recovery Issues is a bug with lasting financial impact. Post Office submits that it is not a bug at all". The relevant findings on these issues are dealt with in the Technical Appendix.
9. Reversals.
This was not acknowledged as a bug in the bug table but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Indeed, Dr Worden’s entry in the 2nd Joint Statement in relation to this stated “Transaction reversals are a complex area which, like recoverable transactions, are less familiar to Subpostmasters and are more prone to human error. They lead to many calls to the help line and to many KELs and PEAKs - not necessarily related to any fault in Horizon.” This can be seen as yet another example of attributing fault to the SPMs where possible, but given it is now accepted as a bug it is not necessary to consider that any further. Mr Coyne’s entry in the 2nd Joint Statement stated “In April 2003 due to a failure in regression testing, Horizon version S30 was released by Fujitsu and this introduced a bug where the value of transactions reversed by Subpostmasters was shown twice in the amount of the reversal in branch accounts.”
10. Data Tree Build Failure discrepancies.
This is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The PEAK dealing with this reads “Data trees have been failing to build fully, and the system has not been detecting this.” Data trees were part of Legacy Horizon, and is used to build a summary (or picture, the word used by the Post Office in its Appendix 2) of the accounts. The building of data trees is a software function, and given “the system” is Horizon and is supposed to detect failures of this nature, it is difficult to see how it could ever have been in issue that this was a bug. Mr Coyne’s entry in the bug table recites that “Dugannon branch suffered a £43,000 discrepancy but the cause was not immediately known. £52,814.29 at the Yate Sodbury Branch. £9,368.40 at the Appleby Westmoreland branch.” These are sizeable sums and arose in the branch accounts. That this is a bug was de facto accepted by Dr Worden in the bug table due to the text of his entry which states:
“There was a bug which has potential impact on branch accounts, early in the lifetime of Horizon. Soon after it arose, the error was trapped and detected by DEP and was then soon fixed.
The fault was easily noticeable at branches before the error trapping which was soon introduced and would be even more noticeable after that. Only three branches appear to have been affected, as described by Mr Coyne.
Because it was so noticeable at the branch, and the Peak is concerned with a software error rather than any other cause, I would expect any discrepancies in branch accounts to have been corrected.”
(emphasis added)
This is therefore undoubtedly a bug, but findings in respect of this therefore depend upon the nature of its impact on branch accounts, which are made in the Technical Appendix.
11. Girobank discrepancies.
Mr Coyne considered this was a bug. Dr Worden did not and stated that “the first fault concerns reports. A fault in a report is not a discrepancy in branch accounts, and only causes one if it causes a person to make a mistake.” This is now essentially accepted as a bug by the Post Office, but it is submitted it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.” The detailed part of Appendix 2 dealing with this submits that there is no evidence of any financial impact upon branch accounts, let alone a lasting impact. This was in the early days of Legacy Horizon. There are said to be six distinct issues arising under this heading, and further detail on this is included within the Technical Appendix. I do not know if what are called “giros” are still in use in 2019.
12. Counter-replacement issues.
There were two KELs associated with this dealt within the bug table. It is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Mr Coyne considered that when replacing a counter within a branch, the process could result in “the total loss of a transaction”. Dr Worden stated that the cause, recorded in the first KEL (which was created in December 2000 and last updated in July 2007) was that Riposte was coming online from the Recovery mode too early, and causing messages to be overwritten. The nature of the correction was stated as being “to find the overwritten transactions for reconciliation we need to look at the Ripostemirror messagestore' followed by detailed instructions”. Riposte, as has been explained, was part of Legacy Horizon and was a product provided by another company, Escher, but it is plainly part of Horizon. Dr Worden also stated that “the incident arose from a hardware replacement (probably from a hardware fault) not from a fault in Horizon. It is a different kind of recovery issue.” In my judgment the hardware is part of Horizon. Findings in respect of this are made in the Technical Appendix.
13. Withdrawn stock discrepancies.
The Post Office does not accept these as a bug. Mr Coyne maintains that it is, and in the bug table he extracted part of a PEAK that stated these “Can cause confusion and unexpected (though hopefully temporary) discrepancies at branches by allowing them to declare stock which has already been withdrawn.” Dr Worden stated that “some impact on branch accounts cannot be ruled out, although it is small”. The Post Office detailed submissions in Appendix 2 of its Closing Submissions concentrated on the fact that withdrawn stock – which of course is part of the way that the Post Office manages its business, adding and removing types of products from time to time – this removal of products is done by means of an update to reference data, “and not a change to the core code in the system.” This again is concentrating on the code, rather than the way the system operates. Findings on this are made in the Technical Appendix.
14. Bureau discrepancies.
These relate to foreign exchange, hence the name. It must be differentiated from bug 23 (also foreign currency, but entitled Bureau de Change). This one arose in 2017. Mr Coyne considered it to be a bug, and Dr Worden effectively agreed in that his entry stated “This appears to be a system error with impact on branch accounts. Although it is possible that a subsequent discrepancy between branch accounting and POLSAP would reveal the problem, leading to a correction (e.g. see Peak PC0265443, and Mr Coyne's para 3.146), I cannot be certain of this.” It is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The detailed part of Appendix 2 states that Mr Coyne has drawn together two distinct issues. Paragraph 4 of the detailed part also states the following:
“Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact. There are two distinct issues which fall under this heading. With regards to the first issue the branch was made good and a fix was implemented. The second issue was not a bug in Horizon nor an issue which could have impacted branch accounts; it created what was essentially a cash flow problem for the branch.”
Findings on this are made in the Technical Appendix, but so far as the first issue, given a software fix was required, it is undoubtedly a bug, and it is correct to record (as the Post Office do) that there was the potential for lasting financial impact, as without that, there would be no need for the “branch to be made good” because that means the Post Office corrected the branch account discrepancy caused by the bug.
15. Phantom Transactions.
There are three different issues grouped under this heading. The Post Office does not accept these as a bug. Mr Coyne in his cross-examination stated that this was referable to Horizon Issue 4. He also accepted that some of the issues were not bugs in Horizon. Dr Worden relied upon the fact that what he called “the master PEAK” was closed as “no fault in product”; however, the evidence of fact on this is dealt with above at [209] to [213] when the PEAK was put to Mrs Van Den Bogerd. Dr Worden stated that “There is no evidence for bugs in Horizon with impact on branch accounts.” That entry was obviously before the cross-examination, which in my judgment provided greater factual information. I do not consider reliance can be placed upon the Fujitsu conclusion in the PEAK. Findings on this are dealt with in the Technical Appendix.
16. Reconciliation issues.
The Post Office does not accept these as a bug. There are a number of different issues grouped together in this heading. The issue was that the SPM was shown a discrepancy on his or her screen. Mr Coyne accepted that the discrepancy would not be shown in the branch accounts; Dr Worden stated that “as it concerns an issue in reporting, the software fault (which was fixed after 5 months) had no direct impact on branch accounts. The only effect of an error in this report would be to mislead or confuse the Subpostmaster - probably leading him to check his figures more carefully and costing him some time.” That accepts a software fault; the fault was in fact a miscounting of the number of files by the system. However, there are six different issues grouped under this heading and the process of reconciliation, which is effectively the comparison by Horizon of two different sets of data, is part of its design function. Mr Coyne in his cross-examination also stated that this was referable to Horizon Issue 4, rather than Horizon Issue 1. The recovery messages were held in the branch messagestore. Findings on this are dealt with in the Technical Appendix.
17. Branch Customer discrepancies.
This was an entry in respect of Horizon Issue 4 and recorded as such in the bug table. The Post Office does not accept these as a bug. Mr Coyne accepted in his cross-examination that the entry in the PEAK did not suggest any impact upon branch accounts. The entry in the bug table originated in a single PEAK, although there were two PEAKs that relate to the same issue, which was a discrepancy between the financial records held by the Post Office (from Horizon) and a bank’s records, after a counter crashed whilst a transaction was being processed. Findings on this are dealt with in the Technical Appendix.
18. Concurrent logins.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. In paragraph 2 of the detailed part of Appendix 2 dealing with this bug, the following is stated by the Post Office: “Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial impact. There is no evidence of any discrepancy in the PEAKs referred to by the experts.” The problem was in the early days of Legacy Horizon when it was possible for users to log in to two terminals at once. Dr Worden in the bug table made the following useful summary statement:
“discrepancies could occur - manifesting themselves as a receipts/payments mismatch. This had the potential to affect branch accounts. The mismatch would bring it to the attention of the Subpostmaster, who would require it to be investigated, except possibly in the case of small mismatches, which he might pass off as an error in the branch (e.g. of counting stock).
“….Fujitsu believed it was a problem with the underlying Riposte software, and passed it to Escher. In September 2000, the problem was 'Now formally fixed in Build 223 update 19 which was released overnight.' However, the new release from Escher did not, as it was expected to, fix the problem. Escher denied that is was a bug in Riposte, but Fujitsu believed in July 2001 that 'This is clearly a bug in the Supplier's code'.”
Dr Worden considered that “these faults had to the potential to produce discrepancies in branch accounts, of small amounts, for a short period of time”. How the bug is therefore characterised depends upon my findings in relation to lasting and transient impact, which is dealt with below. There is no doubt, in my judgment, that this was a bug, and Fujitsu clearly recognised and recorded that in 2001. The fact that the bug was in the code supplied by Escher, as part of Riposte, and not that of Fujitsu is wholly irrelevant for the purposes of the Horizon Issues.
19. Post & Go/TA discrepancies in POLSAP.
It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. The entry relates to Horizon Issue 4 rather than 1. “Post & Go” are self-service terminals, that are no longer available in branch Post Offices, and are only available in Crown and WH Smith main branches. They are basically self-service kiosks, designed to avoid queuing, and a customer can weigh a letter or parcel, the terminal will print the relevant stamps/labels, and the item is then posted. The Post Office submitted that this issue was irrelevant to the Horizon Issues trial as ”this issue does not relate to branches that are the focus of this trial”. Findings on this are dealt with in the Technical Appendix.
20. Recovery Failures.
The entry in the bug table identifies this as relating to Horizon Issue 4. Post Office does not accept these as a bug and states that there is no evidence of a bug in Horizon. There are three different PEAKs. Mr Coyne accepted in his cross-examination that this (or these) should be removed from the table that he used as the originator for his list of bugs (which was Mr Coyne’s Table 1). Dr Worden stated in the bug table that “there was some implication of hardware faults, with a replacement of a base unit, but the PEAK has no evidence of software faults in Horizon.” In one of the PEAKs, he concluded that there was no evidence of any fault in Horizon. Findings on this are dealt with in the Technical Appendix.
21. Transaction Correction Issues.
This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. Mr Coyne’s entry was that “Transaction Correction bugs/errors and defects do not cause discrepancies with branch accounts but” and he then listed various consequences. In his report he referred to “technical flaws”. Dr Worden, in respect of some PEAKs, accepted they were a bug, such as PEAK PC0129587 where he stated “In my opinion this bug would result in an inconvenience to the Subpostmaster (inability to rollover to the next TP) but would not result in inaccurate processing of any TC, or any impact on branch accounts”. The bug is included in the list of paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.”
These are submitted by the Post Office as there being “no evidence of any financial impact upon branch accounts”. Findings on this are dealt with in the Technical Appendix.
22. Bugs/errors/defects introduced by previously applied PEAK fixes.
This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact”. It is also submitted by the Post Office that “a number of PEAKs arose in testing. The PEAKs that arose in the live environment do not indicate evidence of branch impact.” It is listed in the bug table as concerning Horizon Issue 4. Mr Coyne stated in the Joint Statement that “Branch accounts would be affected by this bug which would cause a discrepancy when handling cheques where the value of the cheque would be doubled” although he accepted that the SPM in question was processing a cheque in a different manner to that recommended. Dr Worden used the term “a fault” in his entries, maintained that there was no impact on branch accounts if the SPM followed correct procedures, and also in respect of another branch relied upon the fact that the effect was an error of 2p only. Findings on this are dealt with in the Technical Appendix.
23. Bureau de change.
These arose in 2005, 2006 and 2010, and should be differentiated from bug 14, which arose in 2017, was during Horizon Online and is referred to as Bureau Discrepancies. The Post Office does not accept this as a bug. The Post Office also states that Mr Coyne has identified three issues with the same heading, and submits that they all relate to user error. Dr Worden’s entry in the bug table in relation to one of them states “Analysing the second KEL (2010) I noted: ‘Impact small until bug fixed - rounding errors 10-5 in exchange rates.’ The Post Office submitting that this is not a bug is a somewhat bold submission, given the KEL to which Dr Worden refers expressly states that the problem is a different exchange rate appearing on the HNG-X rateboard and the Horizon rateboard. It states “Problem – there is a bug in the code” (emphasis added). Findings on this are dealt with in the Technical Appendix.
24. Wrong branch customer change displayed.
Mr Coyne concluded this was a bug and part of his entry in the 2nd Joint Statement was “the KEL explains that “the cash amount entered is multiplied by the Qty and hence the new stack total is wrong”, this Horizon bug was due to incorrect reference data and led to an incorrect amount of change being displayed on the branch screen leading to the operator to provide the branch customer with the wrong amount of money thereby leaving a discrepancy in Branch Accounts. It is possible that the amount of change shown on screen is more than the actual money tendered by the customer.” Dr Worden’s entry was “When analysing this KEL I noted ‘Sounds like a genuine problem which may have led to giving the customer the wrong amount - i.e. not recoverable.’ It is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. It is also said that it is a reference data bug, and that therefore once discovered could be quickly fixed by changing the relevant reference data. Findings on this are dealt with in the Technical Appendix.
25. Lyca top up.
It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. It is also a reference data bug, and paragraph 2 of the detailed part of the submissions on this in Appendix 2 states “Lyca Top Up is a bug with the potential for lasting financial impact. This is also [a] reference data bug. As set out above, the experts have agreed that that while reference data bugs may be a significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly identified.” It is also submitted that it was identified through Fujitsu’s automatic reporting. Findings on this are dealt with in the Technical Appendix.
26. TPSC250 Report.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. The origin of the experts discovering this bug is a KEL raised by Anne Chambers in February 2005, and last updated in April 2008. Findings on this are dealt with in the Technical Appendix. I do however find it notable that although Dr Worden says the amounts involved are what he calls “small”, he accepts it is as a “back end reporting problem” – which is nothing to do with the SPM, by definition – and also the KEL states that “the accounting tree has not handled this properly when calculating the daily recon figures and it has resulted in a mismatch…” The mismatch in that case was 66p. The accounting tree is part of Horizon. The KEL also appears to be incomplete because only the first page is present. The section that follows the heading “Evidence” is entirely missing. There is also nothing in terms of the text available that demonstrates what occurred in 2008, even though it is plain for the page that is available that the KEL was updated then. The Post Office submits that there are five separate issues drawn together within this bug. Findings on this are dealt with in the Technical Appendix.
27. TPS.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. One of the Fujitsu documents referenced in the 2nd Joint Statement states that the Transaction Repair Tool or TRT is being used “to repair 1 harvester exceptions for” a particular branch and “There is no correction to be performed and hence no call for confirmtiprepair - this is just an oddity performed by that very flaky mails code.” (emphasis added) There were 40 associated PEAKs, and Mr Coyne observed that both the credit and debit sides of a transaction were doubled, so the net impact of the bug was zero, although he drew attention to the entry in two PEAKs that suggested that SSC requested confirmation of any gain or loss at the counter. Dr Worden believed it was a back-end reporting problem, although the chances of impact upon branch accounts were small. Findings on this are dealt with in the Technical Appendix.
28. Drop and Go.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were resolved)”. This occurred in July 2017 and related to a duplicate “Drop and Go” transaction for £100. This was performed twice; the branch was debited with £200, but the customer credited with only £100 (which had been the amount “topped up” by the customer). Mr Coyne considered it a bug with impact upon branch accounts; Dr Worden stated in his entry “My analysis of this KEL was ‘Possible financial impact. Seems very visible on the counter. Script = reference data - therefore fixed easily’. Paragraph 2 of the detailed part of Appendix 2 on this bug states “Peak PC0260269 relates to an issue involving a Drop and Go transaction (a £100 mobile phone top up) that timed out on Horizon” (emphasis added). This does not appear to be correct, and I do not believe that the transaction relates to a mobile phone. However, I deal with this further in the findings in the Technical Appendix.
29. Network Banking Bug.
This related to a KEL which was raised in 2004 and updated in 2005, with 12 associated PEAKs in the range 2004 to 2010. The Post Office does not accept this as a bug. Mr Coyne stated that “Horizon appears to mis-handle communications, leading to errors within network banking and in turn causing the potential for branch account discrepancies.” One of these was pension transactions being declined yet the customer’s bank account being debited; when the customer complained, the SPM refunded the £50 in question. Dr Worden considered it was “mainly about a communication problem from BT, outside Horizon” but also wanted to investigate further due to an entry that referred to a “CNIM own goal”. The Post Office maintained there were two separate issues and the £50 was paid out due to “user error”. Findings on this are dealt with in the Technical Appendix.
680. The names given to the different bugs above came, sometimes, from the names applied to the bugs by the Fujitsu personnel who knew about these bugs and worked upon them over the years. These names would sometimes appear in the PEAKs themselves that identified the bugs or which represented the views reached by different personnel within Fujitsu who investigated the background, sometimes identifying that previous incidents had similar, or sometimes even identical, characteristics to earlier occurrences of the bug, and who worked to provide fixes or solutions to some of them, or came to conclusions that there was nothing wrong with Horizon and attributed blame, fault or responsibility to the SPM. Some of the names are self-explanatory. Some, such as Callendar Square or Dalmellington, are the names of places where that bug was first identified by Fujitsu.
681. Certain terms were used, dependent upon whether a particular bug was agreed as being a bug (such as the Callendar Square bug), or something else, such as “issues” or “discrepancies.” However, as at the end of the trial, it can be seen from the above entries that the experts’ agreements, the clarification provided by each during cross-examination, and the other developments in the trial such as (one assumes) some reflection by the Post Office on the actual text of PEAKs and KELs, lead to the following situation. Of the 29 different entries in the bug table, eight were alleged by the Post Office not to be bugs at all. They are those numbered 8, 13, 15, 16, 17, 20, 23 and 29. This is set out in paragraph 4 on the first page of Appendix 2 to the Post Office’s Closing Submissions. This careful terminology therefore became a little clearer. The remaining number can properly therefore be described as bugs, even on the Post Office’s own case, although their effect was challenged.
682. This means that the remainder of what is, in my judgment, a lengthy list, were accepted to be bugs. In my judgment, that number of accepted bugs, even on the Post Office’s own case about their effect, and the non-acceptance of the eight others, is a sizeable number. It is approximately a ten-fold increase in the number admitted by the Post Office prior to the Horizon Issues. It is correct that the Post Office did not accept lasting impact on a number – a point I deal with below under “Lasting and transient impact” – and maintained that only 9 “had the potential to cause lasting impact, but were resolved by the Post Office and Fujitsu” but, in my judgment, that is rather to miss the point. The wording of Horizon Issue 1, agreed by the parties, and (one assumes) a major issue that both parties wished to have resolved in order sensibly to progress this group litigation (and omitting the references to the pleadings) is:
“To what extent was it possible or likely for bugs, errors or defects of the nature alleged… to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions.”
683. The use of the words “possible” and “potential” in this issue makes it clear, in my judgment, that the number of “bugs, errors or defects” that fall to be considered is, even on the Post Office’s own submissions, 21 different bugs. I find that to be a significant total.
684. Part only of Appendix 2 to the Post Office’s Closing Submissions dealing with bug 16, Reconciliation Issues, was, by administrative error, not submitted by the date for closing submissions, or until some time after oral submissions were delivered. The Post Office’s solicitors realised this in September 2019. There were three missing pages. I am satisfied that these were drafted prior to the date for service, and I allowed these to be submitted late, together with a clarification document, and short responsive written submissions by the claimants (as the claimants had been deprived of the opportunity to address these three missing pages orally in July). I consider that I am permitted to do this under the court’s general case management powers, and that to do this was to adopt a reasonable and proportionate course. The Post Office’s Closing Submissions was a very substantial document, and it was in my judgment sensible to admit the 3 pages only that had been omitted from one of the appendices.
685. Other elements of the 2nd Joint Statement were also agreed. These included that Horizon had produced over 3 million sets of monthly branch accounts (a point relevant to Dr Worden’s Section 8 statistical exercise), and also that the experts had different views on “branch impact”. The relevant passages in the Joint Statement were that:
“Mr Coyne refers to any discrepancy that caused a loss (or gain) within branch accounts that needed corrective action as an “impact to branch accounts”. Dr Worden only considers an effect or impact on branch accounts where a discrepancy loss (or gain) was not rectified by a correction such as a Transaction Correction.”
686. This in particular demonstrates an important difference in the approach of the experts. Given, as I have explained above, that the Horizon Issues do not refer to Transaction Corrections, which even the Post Office’s own witnesses explained had not been addressed because the Post Office’s solicitors said these were outside the scope of the Horizon Issues trial, I consider Mr Coyne’s approach to “impact on branch accounts” to be the correct one. Dr Worden’s approach minimises or avoids impact on branch accounts that have been caused by bugs, errors or defects, even where those that are (or have been) present in the Horizon system have in fact affected the branch accounts of SPMs. It is a feature that I take into account when I consider which of the two experts’ approaches and evidence I prefer. I also explain this further in “Lasting and Transient Impact” below.
687. The 3rd Joint Statement continued the process of setting out the experts’ agreed position by working through the remaining Horizon Issues. This statement dealt with Horizon Issues 3 to 8 and was dated 1 March 2019. The experts agreed the following points in respect of each. The numbering is mine. One entry below relates to one passage of Dr Worden’s separate and not agreed view under Horizon Issue 3. It was not the only such entry – in different places the experts would record their own separate views.
The 3rd Joint Statement
Horizon Issue 3
“1. Irrespective of how you define the detail of robustness, in line with most other large-scale computer systems, Horizon's robustness has generally improved. From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events. This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.
2. Computer systems are considered more robust if access to the back-end databases is restricted tightly.
3. It was possible for some of the Horizon support staff working at Fujitsu to modify the Horizon back-end branch database. In 2012, Post Office's auditors observed that there were inappropriate system privileges assigned to the APPSUP role (which allowed amendments to the BRDB).
4. Post Office does not consult the full audit data (unfiltered ARQ Data) before deciding how to handle discrepancies and issuing Transaction Corrections.
5. PEAKs show that some defects have lain undetected in Horizon for extended periods without being diagnosed and fixed.
6. During the life of Horizon there have been 19,842 changes made to it via the Fujitsu/Post Office release mechanism.
7. It is common modern IT development practice to make frequent incremental builds and releases of software.
8. Specific release note detail has not been provided. Of the 19,842 changes, we would expect that many were minor changes. It is likely that others contained changes to improve the system or to fix bugs and defects.
9. The effectiveness of various countermeasures changed throughout the life of Horizon.
10. Countermeasures are basic elements of practical IT system design
11. Countermeasures work by limiting the impact of Horizon bugs/error and defects on branch accounts. Countermeasures do not always eliminate the effects of adverse events (they are not perfect) but they are often effective in the area where they are deployed; that is why they have become basic elements of practical IT systems design.
12. It is difficult to measure the extent of the robustness of Horizon, apart from how it might limit the extent of impact on branch accounts, as in Issue 1.
13. There are indications that in its first year of operation, and in the first year after the introduction of Horizon Online, the system suffered from more problems than in other years. One might expect a higher level of problems in these early periods. The extent to which these problems were serious, or evaded countermeasures, or caused discrepancies in branch accounts, is not agreed.”
14. Dr Worden stated the following, which was not agreed, following the entry immediately above. “The main way in which the experts have assessed the extent of robustness of Horizon is to ask to what extent failures of its robustness impacted branch accounts. I have addressed this in Horizon Issue 1. In answering this question, fluctuations over the years are of less importance than the sum over all years, if the sum over all years is small. I have found the sum of impacts on branch accounts over all years to be very small”. (emphasis added). This approach is considered further by me when I address Dr Worden’s Section 8 statistical analysis.
15. The agreed passages continued. “The users of any IT system play a role in assuring its robustness. The designers of a system should not make unrealistic assumptions about the users of the system. Unrealistic assumptions would lead to inappropriate design, making the system less usable.
16. As Horizon has changed throughout its lifetime, the existence and effectiveness of any countermeasures has too. To have considered the time dependence of all robustness countermeasures over 20 years, would have made the expert reports impossibly lengthy. There was not the time to do so.
17. Many software bugs can have the same effects as a user error (as illustrated, for instance, by the Dalmellington bug, which produced a remming error)”.
(emphasis added)
“1. Bugs, errors and defects identified in relation to Horizon Issue 1 are often relevant to Issue 4 in that they are ultimately errors arising from the processing of data in Horizon.
2. There is evidence within the PEAKs and KELs of bugs/errors/defects within Horizon arising from parts (a), (b) and (c) of this issue that occurred without causing financial discrepancies as well as some that occurred causing financial discrepancies.
3. Reference data is critical to the operation of Horizon and errors in reference data have led to discrepancies in branch accounts.
(emphasis added)
4. Of the bugs which in the experts' opinion had the potential to produce discrepancies in branch accounts there may be some involvement of reference data in Bureau Discrepancies, Bureau de Change, Wrong branch customer change displayed, Lyca top-up, and Drop and Go. (Rows 14, 23, 24, 25, and 28 of the bugs table in the second expert joint statement). It is notable that these bugs all concerned specific products (arising from the reference data defining those products).
So, while reference data bugs may be a significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly identified.
5. Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's clients is largely automated. Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/Error Notices to the Subpostmasters.
6. The adequacy of Post Office back office processes to prevent discrepancies in branch accounts can be measured by the quality of the TC process. This quality includes:
• The processes of consideration of available data
• The level of errors observed in the process
• The level of complaints or disputes raised following a TC
• The level of upheld complaints following a TC
• The level of financial impact of erroneous TCs
7. Errors in third-party data have led to discrepancies in branch accounts, through erroneous TCs being issued on Subpostmasters.
8. PO does not control the level of errors made by its third-party client organisations (which may lead to errors in TCs), or the delays in their processes (which may lead to delays in TCs).
(emphasis added)
9. PO can and should ensure, by careful investigation of disputed TCs, that only a small proportion of errors by PO clients lead to losses for Subpostmasters, provided that the Subpostmasters are in good control of their branches and have the required information available.”
Horizon Issue 6
“1. It is agreed that there are many measures and controls within Horizon that existed to prevent, detect, identify report or reduce the risk of varying errors.”
Horizon Issue 7
“1. Fujitsu could access all transaction data recorded by Horizon.
2. Both Post Office and Fujitsu can read data remotely, and FJ needs remote access for support purposes.”
Horizon Issue 8
“1. Post Office had access to data which would not have been available to Subpostmasters.
2. The descriptions of facilities for PO in the two expert reports are consistent and can be taken together as a description of those facilities.
3. Post Office were reliant upon Fujitsu for diagnosis of whether the occurrence of shortfalls was caused by bugs/errors or defects within the Horizon system.”
(emphasis added)
688. It can therefore be seen from the passages of agreement reached by the two IT experts in the 3rd Joint Statement, that a great amount of relevant material directly relevant to the Horizon Issues was not in issue. It was agreed that errors in reference data have led to discrepancies in branch accounts, and also that errors in third-party data have led to discrepancies in branch accounts too. Further, the Post Office has no control over the level of errors made by the third-party clients of the Post Office, and this is agreed. The point also ought to be made that SPMs have no control over that either. Both errors in reference data, and also errors in third party data, being agreed to have led to discrepancies in branch accounts, are points consistent with the claimants’ case in respect of the Horizon system generally.
The 4th Joint Statement
689. This dealt with Horizon Issues 10 to 12, and was the last joint statement agreed by the experts. It was dated 4 March 2019 and was agreed just before the trial commenced. Again, the numbering is mine, and I also reproduce one passage from each expert that was not agreed. One entry below in relation to Horizon Issue 11 was agreed with the word “whenever”. Both experts agreed that this should be worded “usually when” and this latter agreement on that wording was reached during the expert evidence phase of the trial.
1. Dr Worden stated the following, which was not agreed, in relation to privileged user access, and as one of a number of passages at the beginning of the Joint Statement that were not agreed concerning remote and privileged access. “The Privileged User Access logs are not a useful source of evidence about remote access, including Balancing Transactions.”
The following were agreed:
“2. Within Horizon Online, a Balancing Transaction can be created using the BRDB Transaction Correction Tool (called BRDBX015).
3. In the one acknowledged Balancing Transaction which has been disclosed, there were SQL INSERTS into only two tables of the BRDB. These tables were ops$brdb.brdbb_rx_eposs_transactions
and
ops$brdb.brdb_rx_rep_session_data”
4. Mr Coyne stated the following, as one of a number of passages that were not agreed. “Outside of the Transaction Correction Tool other methods were available in Horizon Online.
The APPSUP role used by Fujitsu has elevated privilege and could be used “for un-envisaged ad-hoc live amendment not covered by BRDBX015” as outlined in the referenced document.
Fujitsu also has the ability to perform balancing transactions via direct SQL operations (using a command line interface) to perform corrective transactions in Horizon Online (including deletions of operational data).
Such modifications may or may not be visible to the Subpostmaster and they could be done without their consent.”
The following were agreed:
5. “‘Fixes’ implemented by Fujitsu had the potential to affect transaction data for branch accounts, for transactions which occurred after the fix was deployed.
6. We have not been provided with logs or audits from the Transaction Repair Tool (TRT).
7. Certain facilities and procedures used by Fujitsu to repair the more common issues which arose in Horizon were standardised, and evidence of them persists. However, to repair less common issues which arose from time to time, standard tools and procedures might not have been sufficient, and evidence might not persist of what was done at the time. Even when evidence does persist, it may be extremely difficult for the experts to interpret it today, because of the scale and complexity of Horizon.
Therefore, it is usually difficult for the experts to make categorical negative statements of the form: ‘X or Y never happened’”.
All of the following are agreed entries:
“1. Evidence from several PEAKs indicates that usually when [corrected from whenever] Fujitsu needed to make any change to data which impacted branch accounts, they were concerned to seek permission from PO to do so, and to ensure that PO took responsibility for the resulting change.
2. For any large commercial IT system, it is necessary for some technical users to have privileged access to databases, with wide-ranging capabilities, for system maintenance and problem-solving purposes rather than application-related purposes.
It is important to keep the number of these users down to a minimum possible and for each action that they take whilst logged in to be recorded and audited.”
(emphasis added)
“3. The logging of Privileged User Access (in PAU logs) commenced October 2009.
Between 2009 and 2015 these logs only displayed the fact that a Privileged User had logged on or off but not what actions they had taken whilst the Privileged User was logged in.
The use of the Transaction Correction Tool cannot be seen in these logs.
4. At all times, any privileged user access log only shows what tables of BRDB were accessed for a very small minority of accesses.
There are no privileged user access logs which show access to the two tables of BRDB used to insert the one acknowledged balancing transaction.
5. The authorisation for the use of the Transaction Correction Tool for the acknowledged Balancing Transaction can be seen in one OCP within the OCP disclosure of 25th January 2019.
The record of the uses of the Transaction Correction Tool, including the one acknowledged Balancing Transaction, can be seen in the .aud files, disclosed 22nd February 2019.”
Horizon Issue 12
The following are all agreed entries:
“1. Post Office report that there have been 2,297 uses of the Transaction Correction Tool. For each of these, the usage was logged and the SQL used to alter BRDB has been disclosed.
Only one of these uses (POL-0512131) changed transaction data in two transaction tables, in March 2010.
This Balancing Transaction consists of about 500 lines of SQL, in a pre-defined template.
None of the other disclosed uses of the Transaction Correction tool contain the relevant SQL INSERT commands such as: ‘INSERT INTO ops$brdb.brdb rx_rep_session_data ...’
2. Usage of the Transaction Correction Tool is recorded in the BRDB_TXN_CORR_TOOL_JOURNAL table within the BRDB.
The tools Low Level Design sets out that it has the capability to insert entries into the following tables:
BRDB_RX_BUREAU_TRANSACTIONS
BRDB_RX_EPOSS_TRANSACTIONS
BRDB_RX_APS_TRANSACTIONS
BRDB_RX_EPOSS_EVENTS
BRDB_RX_NWB_TRANSACTIONS
BRDB_RX_REP_SESSION_DATA
BRDB_RX_DCS_TRANSACTIONS
BRDB_RX_REP_EVENT_DATA
BRDB_RX_CUT_OFF_SUMMARIES"
690. Of the agreed entries in relation to Horizon Issue 11, I find the following of very great relevance. The experts are agreed that for any large commercial IT system, which Horizon obviously is, it is necessary for some technical users to have privileged access to databases with wide-ranging capabilities. This however, is not for application related purposes, but is for system maintenance and problem-solving purposes. It is also agreed by both IT experts that it is important that the number of users with these abilities is kept down to the minimum possible, AND for each action that they take whilst logged in to be recorded and audited (my use of block capitals).
691. I consider that to be entirely conventional, obviously agreed by both the experts, and also to be technically justified, particularly in a system such as Horizon which deals with accounting. It is also not requiring an unrealistic level of perfection; it is simply a matter of having properly controlled access to a system such as this one.
692. However, when one then considers the subsequent passages of the 4th experts statement, it can be seen how far from this joint agreed (and technically justified) position the Horizon system was. The logging of Privileged User Access (in PUA logs) commenced in October 2009. For the period 2009 to 2015 – obviously a 6 year period - these logs only displayed the fact that a Privileged User had logged on or off, “but not what actions they had taken whilst the Privileged User was logged in”. Therefore the actions they were taking when logged in were being neither recorded nor audited. All that could be seen is they were logged in. Further, it has already been seen that the number of users with the relevant privileges was not, in my judgment, restricted to a minimum. Further, the use of the Transaction Correction Tool cannot be seen in these logs. Yet further, the experts are agreed that at all times, any privileged user access log only shows what tables of BRDB were accessed for a very small minority of accesses.
693. Yet further, it is agreed by the experts that there are no privileged user access logs which show access to the two tables of BRDB used to insert the one acknowledged balancing transaction. Indeed, one of Dr Worden’s entries in the experts’ joint statements was
“The Privileged User Access logs are not a useful source of evidence about remote access, including Balancing Transactions.”
694. What this amounts to, in my judgment, is a serious deficiency both in the required level of controls in Horizon, in the recording of what privileged users were actually doing (other than that they were simply logged on) and also a corresponding absence of recording and auditing of those activities.
695. In my judgment, and based upon the experts’ agreement in this respect, this serious deficiency lasted up to 2015. It also means, because of the extent (or rather the power) of the APPSUP privilege, the Transaction Correction Tool is but a small subset of the way that records in the BRDB can be affected or changed by Fujitsu. It also means that the tool’s Low Level Design document that was available in the trial bundle, and considered by the experts, has been overtaken by use of the Transaction Correction Tool, because the SQL INSERT command is not present in 2,296 acknowledged uses of the tool. Both the number of uses, and the non-use of the SQL INSERT command, are in agreed passages by the experts in the 4th Joint Statement. This is also a point that was put to Mr Godeseth in his cross-examination (and accepted) and is dealt with at [339] to [344] above.
Lasting and transient impact
696. Of the total number of bugs accepted by the Post Office, the Post Office submits that nine of them had, or potentially had, only transient impact. These are the following:
Bug 2: Callendar Square;
Bug 4: Dalmellington;
Bug 5: Remming In;
Bug 6: Remming Out;
Bug 7: Local Suspense (not to be confused with Bug 3. Suspense Account bug)
Bug 9: Reversals;
Bug 12: Counter Replacement;
Bug 19: Post & Go/TA Discrepancies in POLSAP; and
Bug 25: Lyca top up.
697. Transient impact is a concept introduced by Dr Worden. It does not form any part of the Horizon Issues as formulated, agreed and ordered. Dr Worden agreed, in his cross examination, that he had added in the word “lasting” to Horizon Issue 1. He also said:
“…for better or worse I took my role to be concerned with lasting effects on branch accounts rather than transient ones.”
698. This is a plain and obvious departure from the wording of Horizon Issue 1, and indeed in my judgment represents a narrowing of it. I consider this departure unjustified. It is not the issue that was drafted and agreed by the parties, approved by the court and ordered to be tried.
699. Dr Worden also said in the 2nd Joint Statement (in one of the passages that was not agreed) that “transient inaccuracies in branch accounts, which needed some form of correction, have arisen so frequently and from so many causes that to list them is not useful; and that evidence of each correction being carried out is unlikely to persist to this day.”
700. He stated that “not rectified” means lasting, whereas “rectified” means transient. He also stated that if a bug caused something that required correcting with a Transaction Correction or TC, then that would mean (if the TC were issued) that it were a transient effect; but if the TC was never issued, that would be a lasting effect. The passages were:
“Q. So if the TC process takes a long time, then you might say something was lasting, even notwithstanding years later it might be corrected?
A. That is not how I understood lasting –
Q. How did you understand lasting?
A. There were delays in the TC process which might be due to client organisations or might be due to all sorts of things, and they could be at the outside, I believe, several months. My definition of lasting did not depend on TCs coming in within a certain timeframe. If TC never came in, that would be lasting, but if TC took several months to come in that is not what I would call a lasting effect. A lasting effect is permanent; it is at the end of the day, you know, he has lost money forever.”
701. The use of the phrase “lost money forever” demonstrates the artificiality, in my judgment, of this distinction. Whether this was the intended purpose or not, applying such a concept is to minimise as much as possible the impact on branch accounts of bugs, errors and defects that otherwise exist. He also stated that “the branch accounts would look wrong” for the period prior a TC being issued. That is undoubtedly correct; the accounts would not only “look wrong”, they would be wrong. This makes it even more clear that Dr Worden’s approach is at odds with the wording of the Horizon Issues, as well as with the way which the Horizon System was operated.
702. His approach is, in my judgment, wholly artificial, for at least five reasons.
1. It ignores the express wording of Horizon Issue 1 itself.
2. It ignores the very function of Horizon, which was used as the branch accounting mechanism between SPMs and the Post Office. A SPM would “roll over” their branch each trading period, usually a 4 week period (sometimes a 5 week period). At the end of each trading period, the SPM would produce a Branch Trading Statement (or “BTS”) using Horizon, which was the accounting statement of the branch for that period. Any shortfall or discrepancy in the accounts as demonstrated by the BTS would have to be made good by the SPM (which means the amount of the loss would have to be paid) or “settled centrally” (which means the SPM would seek time to pay). Therefore, any occurrence of any bug, error or defect within that trading period would have an impact on the branch accounts (by reason of the regular production of the BTS) at the end of that particular 4 week trading period.
3. The issuing of TCs is done by the Post Office outside of Horizon in any event.
4. It pays no attention to whether a TC is issued within a branch trading period (and hence before the creation of the Branch Trading Statement for that period) or afterwards, in the next or any subsequent trading period. The fact that (for example) a TC might be issued some months later does not mean that the branch accounts have not been impacted. The branch accounts specifically have been impacted by the occurrence of the bug, error or defect.
5. The correct characterisation of a bug, error or defect within Horizon, and its impact, cannot sensibly depend upon an act (the issuing of a TC) that is done outside Horizon. If a SPM were to find themselves reporting, for the first time, an incident to SSC in relation to a hypothetical bug, bug H, and upon investigation SSC eventually were to accept that there was a bug and a software fix was required, then the nature of bug H should not depend upon how the PO chooses to correct it. If, say, a SPM had retired, closed their branch, been suspended or in any other way ceased to be a SPM, then a TC could not sensibly be used to them. One assumes that in this hypothetical scenario, the PO would refund the SPM (or their estate, if they had died) but that step should not affect the characterisation of bug H.
703. The point at (2) above is consistent with the way that branch accounting works. This was the subject of a considerable amount of evidence at the Common Issues trial which is dealt with in Judgment (No.3). I do not know if Dr Worden had read this judgment, and he was not asked about it. However, I do not consider that any aspect of the outline accounting process operated by the SPM using Horizon at the end of each Branch Trading Period is controversial.
704. In my judgment, if any bug, error or defect within Horizon could have an impact upon branch accounts in any particular trading period, then unless a TC were issued within the same trading period, it would have had an actual impact upon branch accounts. Similarly, if any bug, error or defect within Horizon could have an impact upon branch accounts in any particular trading period, then – regardless of the issuing of any TC in respect of that occurrence – it would have the potential to impact upon branch accounts.
705. This can be tested by considering the Callendar Square bug, number 2 in the bug table list. The evidence I have identified at [401] to [414] above makes it clear that this bug was discovered in 2005, and Mr Godeseth accepted that it had probably been present in Horizon since its inception in 2000. It affected at least 30 branches, and was recorded in one of the contemporaneous documents as “this problem has been around for years and affects a number of sites most weeks.” Yet on Dr Worden’s somewhat artificial constraining of impact on branch accounts, it would be categorised as only having a transient impact. The dictionary meaning of transient is fleeting, passing, brief, temporary, momentary, transitory, short-lived or ephemeral, and plainly does not apply to the effect of this admitted bug. In my judgment the use of this word by Dr Worden and the Post Office is unjustified. The use of it by Dr Worden was unnecessarily to restrict his analysis.
H. The Period 21 March 2019 to 4 June 2019
706. During this period two matters of note occurred in the group litigation (in addition to hearing the recusal application and handing down of Judgment (No.4), which it is not necessary to address in this judgment).
707. The first was the further disclosure of documentation by the Post Office, which is explained further at [616] above.
708. The second was service of a further supplementary expert’s report by Dr Worden directly upon the court. This occurred on 22 May 2019. It came under cover of an email from Dr Worden, copied only to the Post Office’s solicitors. It is extremely unusual, if not verging upon unheard of, for an expert witness to communicate directly with the trial judge, and directly serve material upon him or her. It is also the case that no party (whether by itself or its solicitors) in ongoing litigation should communicate with the court without copying in the other parties to the case. The only very narrow exception to this is ex parte proceedings, in respect of which there are special rules, and which is not the case here.
709. In the interests of transparency, I reproduce the whole email from Dr Worden here. It was accompanied by some attachments: “Third Report Open”; and three appendices. The email stated:
“Dear Judge,
As was explained by Mr de Garr Robinson in court on 11 April, I have made some further analyses of Horizon Issues 1, 12, and 13. This work was done at my own instigation, and was not prompted by the Post Office or its lawyers. In my opinion this work has led to a material change to my opinions, and I am therefore obliged to inform the court of those changes in a report. With this email I am sending you the report and its Appendices, in completion of the action. I am doing this on the basis that the trial will re-convene on 4th June, and so the court may need to be aware of it then. The report is 13 pages long.
As the court is aware from my first expert report, I have interpreted ‘extent’ in Horizon issue 1 as requiring a number - the maximum proportion of the claimed losses which is attributable to bugs in Horizon. My first estimate of that number, in my first report and revised in the second expert joint statement, is 0.4%. This estimate was adjusted in a direction to favour the claimants, so that the court may better rely on it.
In my opinion, that estimate is still sound. The enclosed report contains an independent estimate of the same number, which is 0.6%. Because that estimate is based on a simpler analysis, and relies on evidence that in my view can be checked over a weekend, I believe it can be of assistance to the court. That is why I started this analysis, and why I am now sending it to the court.
Similarly, for Horizon Issues 12 and 13, the analysis in the attached report is in my opinion simple, easy to check, and leads to results which I was not able to derive before. Therefore I believe it can be of assistance to the court.
A draft version of this report was first provided to Mr Coyne on 25th April, and a final version was sent to him on 16 May. I have not yet been able to have a substantive discussion of this report with him, other than my explaining points in the report to him.
I understand that when cross examined on the Horizon issues, this report is now a part of my opinion and I may need to refer to it. It is also my understanding that Post Office does not intend to make any application to the court in relation to this report.
Yours Sincerely,
Robert Worden”
(emphasis added)
710. The reference to what had occurred on 11 April 2019 is as follows. Following dismissal of the recusal application on 9 April 2019, the trial resumed on 11 April 2019 with the resumption of the Post Office’s final evidence of fact. This involved the Post Office calling Mr Parker to be cross-examined, and explaining that Mr Membury (another of the Fujitsu witnesses) was unable to be called, and the relevant Civil Evidence Act notice being given under CPR Part 33.2. The position of the claimants on that latter point was explained as their consenting to the necessary extension of time (rather than a waiver by them under section 2(3) of the Act, a difference that is immaterial in the circumstances). Mr Membury’s witness statement was therefore not cross-examined upon, but Mr Parker was called and was cross-examined.
711. Upon conclusion of Mr Parker’s evidence, Mr de Garr Robinson, leading counsel for the Post Office, explained that, as he put it, “a matter of some awkwardness” had arisen. He submitted the following:
“Dr Worden has recently realised that there is a new way of looking at the evidence in this case which, in his view, could greatly assist your Lordship, assist the court, in deciding Horizon issues 1, 12 and 13.
This approach involves focusing on those PEAKs, OCRs, OCPs and MSCs which actually mention the FAD codes of one or more of the claimant branches. Just to explain, when Fujitsu did any authorised remote handling of data, to put it neutrally, which might affect branch accounts, they raised an OCP, OCR or MSC whose text was likely to include the six digit FAD code of the relevant branch. So it's therefore possible to search all the OCPs, OCRs and MSCs with a view to finding all of those which mention the claimant branches during the relevant claimant's period of tenure. This search yields a limited number of OCPs, OCRs and MSCs, and it's therefore possible to assess expert issues 12 and 13, which is how often was remote access facility exercised and what effect did it have. It is possible to assess those questions as they affect the claimants by examining that much smaller document set. My Lord, that is the first exercise that he would like to undertake, and indeed he has embarked work on -- I think this week he has embarked work on that.
Second, if a detected bug affected the accounts of any branch the Peak relating to that bug was likely to mention that branch's FAD code. Typically, it will also -- it may also mention a sum of money. It's therefore possible to search all the PEAKs in the same way that I have just outlined, looking for PEAKs which mention any claimant's FAD code during the relevant claimant's period of tenure, and again, this document could shed some light on Horizon Issue 1 to which extent is it likely that bugs have affected the relevant branches.
Now, Dr Worden has specifically asked me to offer his apologies to the parties and to the court that he didn't think of this before. In fact, frankly, he is kicking himself that he didn't do so. He believes that he and Mr Coyne would only need a short time to consider the relevant documents and to consider how it affects their views on those issues. He wishes to discuss the documents with Mr Coyne with a view to agreeing what they do or do not show.
My Lord, in the days since the recusal application was issued he started to consider how the new approach affects his views. He believes on Issue 1 it allows the parties to make a much simpler analysis of the point, and he takes a similar view in relation to the remote access issues. It makes, in his view, it possible for the experts to form a view as to how often remote access was exercised and what its likely effect was.
It is Dr Worden's view that it is his duty under CPR Part 35 to inform the court of this change of view and to allow the court to consider whether or not it wishes to see it considered. That belief is based, as your Lordship will be aware, on CPR35.3 which imposes a duty on experts to help the court on the matters within their expertise, whether or not they are instructed so to do. My Lord, it's also based on CPR35 -- I should say the practice direction CPR35, paragraph 2.5, which requires experts to inform the court of any change of views.
I should emphasise this -- none of this comes at the request or instigation of my client. This has come from Dr Worden. This is his idea. My Lord, he wishes to discuss it with Mr Coyne in a further meeting between the experts, but of course it's -- it's only right that your Lordship should be aware of that. I'm not making any application for permission to put in supplemental expert reports –
Mr Justice Fraser: I don't think you have any supplemental experts' reports to apply for permission for, do you?
Mr de Garr Robinson: I'm not making any kind of application, I'm simply sharing with your Lordship the view that has been expressed to me by Dr Worden.”
712. The Post Office, in continuing this explanation of what was happening, referred to another case which had concerned late expert evidence, the judgment in which had been referred to in the opening submissions for the Horizon Issues trial. That case is Imperial Chemical Industries Ltd v Merit Merrell Technology Ltd (No.2 Quantum) [2018] EWHC 1577 (TCC).
713. The exchange with the court continued:
“Mr Justice Fraser: …In order to apply for permission to adduce extra expert evidence you would have to have a draft of the report for which you would be seeking permission, wouldn't you?
Mr de Garr Robinson: My Lord not necessarily.
Mr Justice Fraser: You don't think so?
Mr de Garr Robinson: My Lord, I would submit not. It would depend on the circumstances. Often one would have such a report. I'm conscious that in the ICI case your Lordship cited as a reason for not giving the relevant party permission to put in a report which they had prepared, that the experts hadn't gone through that collaborative process and I'm quite anxious to ensure that my expert doesn't fall into the same trap, if I can put it that way.
Mr Justice Fraser: Well, depending on whichever point one reaches in terms of your actually make an application to put in a supplementary expert's report, that application will be dealt with as and when it's made, so I'm not dealing with that either positively or negatively at the moment.
Mr de Garr Robinson: Thank you.”
714. It is not necessary to dwell upon the point, but any party seeking to obtain permission to rely upon a very late additional expert’s report should ordinarily expect to have at least a draft of such a report available. It is difficult to see how permission could be given in general terms for late expert evidence, the content of which was not available. However, no such application was made by the Post Office in any event. The trial had commenced on 11 March 2019. Dr Worden’s first expert report was dated 7 December 2018 and his supplementary report was dated 1 February 2019. This notification to the court on 11 April 2019 of Dr Worden having “recently realised that there is a new way of looking at the evidence in this case” came over four months after the date of his first report, over two months after the date of his supplementary report, and exactly one month after the Horizon Issues trial itself had actually started. I requested a witness statement from the solicitors for the Post Office setting out the chronology of this realisation by Dr Worden.
715. The case to which reference was made on behalf of the Post Office, ICI v Merit Merrell, concerned a supplementary report by a quantum expert, instructed on behalf of ICI, which was produced on the first day of trial without warning. This was based on an analysis of documents the expert had only recently been provided with by ICI, which it turned out had been in ICI’s possession for over three years, but which the expert had not previously been given. The application for permission to adduce that supplementary report in that case was refused. Points relevant to the application in that different case on those facts were set out in [158] of the judgment, which stated:
716. Giving permission for late expert evidence is a matter of discretion, and each case will be different. At [237] in ICI v Merit Merrell I had stated the following, as one of a number of principles that relate to expert evidence:
717. Here, the following distinct matters fall to be considered. Firstly, just because in the ICI case that expert performed a new exercise without telling his opposite number he was doing so (which led to criticism), does not mean that all an expert has to do is to inform his opposite number during a trial that he has decided to undertake a new exercise, or produce a new report, and that by giving such advance notice this means the new report will be admitted. The whole ethos of exchange of expert evidence well in advance of trial is so that each party (and their respective experts) have advance notice of the opposing point of view, so that evidence can be properly addressed and surprise avoided. That is undermined by service of very late reports containing new analyses. Sometimes this is unavoidable and late material emerges. It is not possible to be entirely proscriptive, as each case is different. That is why permission of the court is required. Secondly, the stage of the trial at which this 3rd Report was produced by Dr Worden was very late indeed. It was weeks after the trial had started. Thirdly, the extent to which the court needs to consider all the circumstances depends upon whether the 3rd Report was sought to be relied upon by the Post Office. Finally, and of wider application, is the way in which the Third Report was dealt with, namely by service by the expert himself directly upon the court.
718. Dealing with those matters in turn, the first two are effectively bound up with one another. I do not know the actual date when Dr Worden realised that this new way of looking at things appeared to him to be useful. However, the court was told he had started thinking about this on 11 April 2019, and the claimants’ solicitors had already been told of his intention by the Post Office’s solicitors the previous day on 10 April 2019 (one day after the recusal application was dismissed, and the day before resumption of the trial). Thereafter, a draft report was provided on a without prejudice basis to Mr Coyne on 25 April 2019 by Dr Worden, his “current workings” having been provided on the same basis on 11 April 2019 and the appendices, again on a without prejudice basis on 25 April 2019. There was then a degree of to-ing and fro-ing between the solicitors, which led to the actual report itself being sent on an open basis to Mr Coyne on 16 May 2019 (after another experts’ meeting on 15 May 2019).
719. Regardless of the exact date when Dr Worden realised there was a different way of looking at the evidence in the case, this 3rd Report emerged at an extraordinarily late stage in the proceedings. As can be seen in [721] below, it was in any case based on material which Dr Worden accepted he had had for some time. It was not based on material he had only received late, for whatever reason.
720. Turning to the third of the matters identified in [717] above, this can be readily dealt with. The Post Office made it clear, both on 11 April 2019 in the passages which I have quoted above, and subsequently, that it did not seek permission to adduce the contents of Dr Worden’s 3rd Report in evidence. It issued no application in this respect. Therefore, it was not necessary for the court to consider the balancing exercise inherent in resolving any application in this respect (on the basis that such an application was resisted by the claimants, which was likely). It is inherent within this decision by the Post Office not to issue an application to adduce it therefore, that there was no explanation of when or how Dr Worden had experienced this “new way of looking at the evidence”, other than what has been identified above. It also meant that this report was not adduced as any part of his evidence-in-chief, and he was not cross-examined about it. He referred to its existence only once, very much in passing, and given that the contents of it had not been adduced in evidence, that was as far as it went.
721. Had I been faced with a contested application by the Post Office to be permitted to rely upon the contents of the 3rd Report in evidence, the matter would have been considered in accordance with the principles in the CPR generally, CPR Part 35, the Practice Direction that goes with it, as well as the Guidance for the Instruction of Experts to Give Evidence in Civil Claims 2014 (“Guidance of Experts in Civil Claims”) which is in the White Book with the Practice Direction. Doubtless such an application, had it been made, would have been supported by a detailed explanation of all the relevant matters, together with argument on both sides. I do not have the benefit of either a detailed explanation or of any argument. On the material of which I am aware, but without hearing any submissions on it for obvious reasons, a substantial hurdle to the success of such an application would have been its lateness. This is because of the enormous disruption and yet further delay that would have been the inevitable consequence of permitting reliance upon such a late expert’s report. Severe disruption and delay had already been caused to the Horizon Issues trial by the time this report emerged, and by late May, the trial was shortly about to resume after an enforced interval, in very unusual circumstances, of over two months. Had this new report been admitted into the evidence, Mr Coyne would have required time to study it, and then potentially produce a further expert’s report of his own, and none of that could have been accommodated between late May (whenever such an application could have been listed) and 4 June 2019. Calling him would either have been delayed, or he would have had to be recalled later to deal with it. Further delay to the Horizon Issues trial would have been undoubtedly caused. Delay is not always fatal to such applications, but the later an application of this kind is made, the stronger the explanation for that delay has to be.
722. The claimants had also made the point on 23 May 2019, the day after Dr Worden had served his 3rd Report upon the court, that it was based upon an analysis of documents, namely OCPs, OCRs and MSCs, that Mr Coyne himself had expressly requested in July 2018 and which the Post Office’s solicitors had then said were irrelevant. In any event, when I asked him at the end of his cross-examination, Dr Worden confirmed that he had had the documents upon which he had based this 3rd Report “for some months” and that he had had the full set of PEAKs since, as he put it, “way back in 2018”. As the Post Office’s own counsel explained on 11 April 2019, Dr Worden was “frankly, kicking himself” that he “didn’t think of this before”. The lateness seems therefore to have been the consequence of Dr Worden failing to consider this approach until very late in the day, rather than only very recently being provided with the material. That is not promising ground for the making of such an application. It is also the case, in my judgment, that there were no exceptional circumstances which made it unavoidable that this new report was produced during the actual trial. Indeed, rather to the contrary. This was entirely avoidable. Finally, the covering email from Dr Worden made it clear that his estimate, included in his first report and revised in the 2nd Joint Statement of 0.4%, was “still sound”. This new exercise simply sought to bolster his conclusions. It was not an explanation of a change of opinion by him.
723. I turn therefore to the fourth matter, the issue of direct communication with, and service upon, the trial judge of an expert’s report by the actual expert rather than through solicitors.
724. When the email was received to which I have referred at [708] above, my clerk immediately sent a copy of the email and attachments to all the legal advisers for both parties in the litigation, stating that witnesses should not communicate directly with the court, and also requesting that the claimants should not read any of the attachments until the matter was addressed in court the next day (when the parties were attending in any event to argue costs orders and other consequential matters following Judgment (No.3)).
725. The legal advisers for the Post Office did not seem concerned at the hearing of 12 April 2019 that this direct service upon the court by the expert himself had occurred. An analogy was drawn with an expert seeking directions under CPR 35.14, and it was dealt with by the Post Office as though the direct service upon the court was simply the actions of an expert seeking in the normal way to fulfil his duties to the court, and was entirely usual. It is, however, not usual; it is highly unusual. Further, the covering email from Dr Worden makes it clear that the new report was not a change of opinion; in particular, the sentence that makes this entirely clear is “in my opinion, that estimate is still sound” concerning the calculation in his earlier reports. His exercise was simply a different way of reaching the same conclusion. Therefore, the provisions of paragraphs 64 to 66 in the Guidance of Experts in Civil Claims do not apply.
726. I do not consider that any witness, lay or expert, should communicate directly with the trial judge about the content of their evidence, or indeed about any subject at all. There is a substantial risk that the efficient management of proceedings will be compromised if this is done. Communications with the court should always be done by solicitors, if they are on the record. Further, communications between an expert and those instructing him, or her, are privileged. If, as here, an expert takes it upon himself to inform the court of applications that his client may, or may not, be intending to make, there is a real danger that privileged information might be divulged, both to the court and also to the other side in the litigation. This is far less likely if communications are performed by solicitors, who have specific qualifications in legal matters and will be experienced in litigation. Finally, there is specific provision in the Guidance of Experts in Civil Claims dealing with the situation where an expert wishes to exercise their right to ask the court for directions. This is contained in paragraphs 28 and 29. These proposed directions have to be sent to the party instructing the expert at least seven days before the request is made, and to the other parties in the litigation at least four days before. They are to be made by letter in the form prescribed at paragraph 29(a) to (e). That gives the other parties the ability, firstly to have advance notice of what is proposed, as well as providing advance notice that the court will be asked for directions and what those directions are, and why; and also give the expert’s instructing solicitors the ability to consider the communication in advance so that privileged information is not included.
727. In this case, that process was not followed. Mr Parson’s 17th witness statement explained the outline chronology, and made it clear that Dr Worden’s opposite number Mr Coyne had a copy of the 3rd Report, initially on a without prejudice basis, and then on an open basis, and that Dr Worden told Mr Coyne on 16 May 2019 (when he sent it on an open basis) that he would be sending the report to the court but not before 22 May 2019.
728. Finally on this subject, when I asked Dr Worden at the end of his evidence whether he had ever preformed this type of direct service himself on the court before, he told me he had not. He also said that he had sent the email direct to my clerk as this was what he had been advised to do by the “Post Office’s lawyers”. He said that he had not given his instructing solicitors the opportunity to check or approve the covering email, but “had been given some advice as to what it should cover”. I assume that advice did not include telling Dr Worden that he should inform the court of the Post Office’s potential intentions on what applications it may, or may not, be minded to make in the litigation. This is a highly unusual situation. In closing submissions the Post Office invited me to provide guidance in this judgment as to what should be done for future cases, as though the position was unclear. I do not know why it should be thought necessary to tell witnesses – lay or expert - that they should not unilaterally communicate with the court. However, if it is necessary, no witnesses should communicate with the court directly in this way. Any communications from legal advisers should always be copied to the other parties in litigation. No expert should inform the court directly what applications his or her solicitors might be contemplating, or not contemplating, in the future conduct of the litigation.
729. The points that I draw from this episode are as follows:
1. Experts should not serve their own material, particularly new material during a trial, and whether or not accompanied by detailed covering emails, directly upon the court themselves. Service should always be done by a party’s solicitors if solicitors are on the record.
2. A party’s solicitors will know that all communications to the court must be copied to the other parties in the litigation.
3. Experts should not notify the court directly of the intentions of their instructing solicitors and/or ultimate client in the litigation in which they are instructed. Such matters are privileged.
4. It is difficult to be critical of Dr Worden in this respect, given he told the court that he was advised by the Post Office’s legal team to send his report directly to the court. Advice of that type should not be given to experts.
5. Dr Worden must have considered that the documents upon which he relied were, at least, relevant. These were documents of a type sought by Mr Coyne in July 2018, whose relevance was disputed at that stage by the Post Office. Mr Coyne’s request ought to have led to these documents being provided to both the experts in the middle of last year, a reasonable period of time after Mr Coyne specifically asked for them.
730. I will now turn to each of the expert’s evidence, their reports and their approach to the Horizon Issues.
731. The subject matter of this group litigation spans a long period. Mr Bates was a SPM before Horizon was introduced in 2000, and it is from Horizon’s introduction that year that, on the claimants’ case, the problems and difficulties arose, and arose as a result of the functionality of the computing system introduced by the Post Office, which in this judgment is referred to as Legacy Horizon. As the Post Office pointed out regularly during these proceedings, there are thousands of branches and millions of transactions each year in all of those branches, which means that over such a lengthy period, there is a vast amount of computing activity that had to be considered by the experts.
732. The expert evidence was therefore required to be substantial. The aspects of the litigation that require expert evidence are not, however, of unheard of proportions, so far as the High Court is concerned. The High Court often deals with matters that span a lengthy period; it also often deals with matters of great complexity that require expert evidence. By comparison with many other cases, one expert discipline alone, with only one IT expert per party, is not a particularly expert-heavy case. The experts’ reports, though lengthy, were not of an unusual size. There were two reports from each expert that were adduced in evidence, and each of them was of a type that would be expected in a trial of medium complexity, which I consider the Horizon Issues trial to have been. Nor was it necessary for the experts to examine every claimant SPM’s separate experience; the whole purpose of having generic Horizon Issues was to avoid precisely that. The Post Office would often concentrate on how vast the panorama was before the court, but this may have been done simply to emphasise how few, in relative terms, the claimants are; which also emphasised at the same time how well (so far as the Post Office generally is concerned) the Post Office business operates day by day, week by week, across so many branches. The experts agreed that Horizon has produced over three million sets of monthly branch accounts.
733. At times, however, this focus on scale did seem to come close to an assertion that the IT experts had an impossibly vast task; or that the court faced a significant challenge even in understanding all the expert evidence; or that the parties could not practically deal with the other side’s expert evidence in the time allotted.
734. As has been set out in Part G of this judgment, the experts reached a great deal of agreement in their different statements. Notwithstanding that, there remained large areas of disagreement. Given the modern approach to trying cases generally, it was neither possible, proportionate nor desirable to give the Post Office as much time as their advisers maintained was necessary to conduct the Horizon Issues trial. As an interesting comparison, in the case of The Ikarian Reefer dealt with further below, which was before the implementation of the CPR and concerned the US$3 million loss of a vessel, that trial took no fewer than 87 trial days. Such a lengthy trial, post the introduction of the CPR, would be highly unusual in a case of the financial scale of this one.
Time limited trials and the approach to detailed evidence
735. Prior to the introduction of the Civil Procedure Rules, fewer trials were time-limited and some trials dealing with a large number of complex issues could run for a great many months, if not far longer. Since the introduction of the overriding objective, which is contained in CPR Part 1.1(1), this states “These Rules are a new procedural code with the overriding objective of enabling the court to deal with cases justly and at proportionate cost”. That is explained further in Part 1.1(2), which states that:
“Dealing with a case justly and at proportionate cost includes, so far as is practicable:
(a) ensuring that the parties are on an equal footing;
(b) saving expense;
(c) dealing with the case in ways which are proportionate –
(i) to the amount of money involved;
(ii) to the importance of the case;
(iii) to the complexity of the issues; and
(iv) to the financial position of each party;
(d) ensuring that it is dealt with expeditiously and fairly;
(e) allotting to it an appropriate share of the court’s resources, while taking into account the need to allot resources to other cases; and
(f) enforcing compliance with rules, practice directions and orders.”
736. Since the Civil Procedure Rules, trials are usually set down for a fixed time estimate and are expected to be time-tabled to finish within that period. The court system does not have the luxury (if luxury it is) of giving endless periods of time to litigants to argue their cases to the nth degree. This case, as with all litigation, is very important to those involved in it – to all of the claimants, but also to the Post Office, as an institution generally, and also to its other directors, employees, SPMs who are not claimants and its clients too. The issues and different causes of action in both claims and counterclaims are of considerable importance, and the amount of money at stake is not small. However, the amount of money is not large in terms of specialist litigation, for example. It is not measured in the billions, or even many hundreds of millions. Ignoring claims for general damages, the amount of damages that are specified in liquidated sums are said to be about £18 million (although that is only an approximate figure currently). There are, of course, also reputational issues at stake as well as money.
737. Judgment (No.5) [2019] EWHC 1373 (QB) made it clear at [78] that the total of the parties’ reported costs as at the date of that judgment was over £25 million (£12.6 million for the claimants, and £12.8 million for the Post Office). Since then, on 25 June 2019 the Post Office notified the court that its costs had exceeded £13.9 million.
738. The reason for explaining this in detail is to put into context the following points about the scope of expert evidence, and the number of trial days available for the experts to be cross-examined. At the Pre-Trial Review, the Post Office submitted that at least four days was needed for the cross-examination of Mr Coyne. It was realistically accepted that longer than that was not likely to be permitted, although leading counsel for the Post Office did submit that he would prefer longer. The claimants submitted that they also wanted four days, but had trimmed that period to three days, for the cross-examination of Dr Worden. Given there were further experts’ meetings to come, and given my view of the scope of the expert material and considering the overriding objective, I set a trial timetable that permitted each side two and a half days to cross-examine the other side’s expert. That would have given a total number of five days of cross-examination for all the IT expert evidence. At that time, I considered that would be sufficient, and also proportionate, taking account of the overriding objective.
739. However, the subsequent Joint Statements following the PTR did not reach as much agreement as I had (perhaps naively) anticipated. I therefore adjusted the trial timetable and set a new one, with the Post Office being given four days to cross-examine Mr Coyne, and the claimants being given three days to cross-examine Dr Worden. That trial timetable also allowed an interval of one non-sitting week, to follow the evidence of fact, for each side properly to prepare their time-limited expert cross-examination in detail.
740. In my judgment, the period of time I had allotted at the PTR of 2½ days each was sufficient in itself; but the increased period of time allowed was undoubtedly sufficient for each party properly to cross-examine and test the expert evidence adduced by the other side in the litigation. Seven court days for the total cross-examination of one expert discipline in a case such as this is, given the Horizon Issues in this trial, generous. Counsel might not have had time to put each and every single individual point to the expert on the other side, but they had more than enough time to put the main ones (and a considerable number of subsidiary ones too, for that matter). Each side had leading counsel, well-versed in all of the issues, who have been involved in the case from at least the making of the GLO itself in early 2017, as well as a team of other counsel assisting too. Considerable pre-reading had been done by the court. As it turned out, the issuing of the recusal application on 21 March 2019 led to an even longer period of the trial not sitting, with only a half day (for Mr Parker) of the trial taking place between 21 March 2019 and 4 June 2019, a period of 75 days, or over ten weeks. The Post Office had a different counsel team acting on the recusal application than the team acting on the Horizon Issues trial. That 75 day period was a substantial period of time for each side to prepare detailed cross-examination on all, and any, points that they considered important, prior to the resumption of the trial and the commencement of the expert evidence on 4 June 2019.
741. The amount of the costs burden explained in [737] above has been incurred on the basis of the number of trial days made available. Obviously, further time for cross-examination would have led to even higher cost figures. There was a sufficient and proportionate amount of time at trial made available for the expert evidence to be properly challenged and tested, and there was more than sufficient time available for preparation of the time-limited cross-examination. I have set this out in detail in order to make it clear that the time limited nature of the Horizon trial, and the number of days allotted for cross-examination, has in my judgment had no effect on my findings on the Horizon Issues, or restricted either party’s opportunity to put its own case and test the case of the other party.
742. The other feature of time-limited cross-examination, given the depth of experience of counsel, is that I permitted considerable latitude to the cross-examiners. If counsel wanted to put the same point to an expert again and again; or spend an hour on one specific sentence or detail, when the same point could have been accomplished in five minutes; or devote significant time to high-level issues and avoid descending to specifics; then that was a judgment call made by that counsel. With more junior counsel, in another case, the court might discourage excessive time on less important or agreed points, to encourage progress; but here both sides had highly experienced leading counsel. I therefore intervened rarely. The amount of time made available in the trial timetable for each side to cross-examine the other’s expert was set down in advance, and if one side wished to devote (say) half a day to having an expert state, and then restate, an agreed position on a particular point, then that was very much a matter for them. Such an approach would lead to that period not being available to allocate to other subjects, but in my judgment this case in particular benefitted from allowing the parties to conduct their cross-examination, more or less, in the direction they wished.
743. I did, from time to time, seek either to have clarified what particular Horizon Issue was the subject of the cross-examination, or to reference it to particular other subjects. However, this did not occur very often, and the parties were permitted to use their allocated time as they wished.
744. Each side, both in their cross-examination and closing submissions, criticised the other side’s expert for not being sufficiently independent. This was done in different terms and in different ways, but the upshot of it was broadly the same. It was that their expert’s evidence should be preferred to that of the opposing side, for reasons of methodology and approach, as well as for reasons of detail.
745. The court, when faced with opposing experts, might dismiss the views of one expert entirely on the basis that they had not complied with their duties to the court. However, it is not necessary to do that, in order to prefer the views of one expert over another. There may, depending upon the subject matter, be room for opposing expert opinion, and it is simply a case that the court accepts one opinion for a variety of reasons. Also, in some cases, it might be that one expert’s view is accepted on some points, and the opposing expert’s views on others.
746. It is also not necessary to resolve every single point of detail between experts to arrive at the answers to the Horizon Issues. However, given these criticisms of the experts have been made, it is necessary to address them. This is because it is only fair both to the experts, and to the parties, to decide whether each expert has been sufficiently impartial in the exercise upon which they have been engaged such that the court can reliably base its answers to the Horizon Issues upon the conclusions reached by the experts. Impartiality and independence of experts is also of considerable, if not central, importance to the administration of justice generally.
747. The case that is most often stated in the context of the importance of independence on the part of experts is National Justice Compania Naviera SA v Prudential Assurance Co Ltd [1993] 2 Lloyd’s Rep 68 (Comm Ct), but it is almost always referred to by the name of the ship involved in the case, The Ikarian Reefer, which ran aground off Sierra Leone and then caught fire, becoming a total loss. Lord Woolf MR, the author of the Civil Procedure Rules, made clear in Stevens v Gullis [2000] 1 All ER 527, 533 that CPR Part 35.1 and its accompanying practice direction “did no more than reflect the position as it had been well enunciated in the authorities prior to the CPR coming into force”. The Ikarian Reefer remains central to the way that experts should perform their role, including the separate duty owed to the court.
748. In Imperial Chemicals Ltd v Merit Merrell Technology Ltd [2018] EWHC 1577 (TCC), a more recent case in which a number of experts all called for one party were found to be partisan, at [237] the following was stated:
“The principles that govern expert evidence must be carefully adhered to, both by the experts themselves, and the legal advisers who instruct them. If experts are unaware of these principles, they must have them explained to them by their instructing solicitors. This applies regardless of the amounts at stake in any particular case, and is a foundation stone of expert evidence. There is a lengthy practice direction to CPR Part 35, Practice Direction 35. Every expert should read it. In order to emphasise this point to experts in future cases, the following points ought to be borne in mind. These do not dilute, or change, the approach in The Ikarian Reefer. They are examples of the application of those principles in practice.
1. Experts of like discipline should have access to the same material. No party should provide its own independent expert with material which is not made available to his or her opposite number.
2. Where there is an issue, or are issues, of fact which are relevant to the opinion of an independent expert on any particular matter upon which they will be giving their opinion, it is not the place of an independent expert to identify which version of the facts they prefer. That is a matter for the court.
3…..
4…..
5. Where late material emerges close to a trial, and if any expert considers that is going to lead to further analysis, consideration or testing, notice of this should be given to that expert's opposite number as soon as possible. Save in exceptional circumstances where it is unavoidable, no expert should produce a further report actually during a trial that takes the opposing party completely by surprise.
6. No expert should allow the necessary adherence to the principles in The Ikarian Reefer to be loosened.
It is to be hoped that expert evidence such as that called by ICI in this case, and also in Bank of Ireland v Watts Group plc, does not become part of a worrying trend in this respect. There are some jurisdictions where partisan expert evidence is the norm. For the avoidance of any doubt, this jurisdiction is not one of them. Not only experts, but the legal advisers who instruct them, should take very careful note of the principles which govern expert evidence.”
749. The Ikarian Reefer, and the requirements of CPR Part 35, are the touchstones of expert evidence in this jurisdiction. It is of the utmost importance that they are adhered to both by experts themselves, and also by those who instruct such experts.
Mr Coyne
750. Mr Coyne is a Partner of IT Group UK Ltd, and has 30 years of experience in IT. He has designed, programmed and supported Electronic Point of Sale (or EPOS) systems for use in payment handling, stock control and distribution in addition to full business accounting systems. He has also assisted public sector councils with revenue and benefits processing systems, investigated failures within stock market trading systems, European gambling and gaming networks and dealt with fraud in retail banking, point of sale, cash in transit and electronic funds transfer systems.
751. He is a member of the Academy of Experts, the British Computer Society and the Society of Computers and Law. He has an Expert Witness Diploma from Cardiff University Law School, achieving this in 2008. He has been instructed as an expert witness in both civil and criminal cases over the last 19 years, including by both UK and International firms of solicitors, in connection with technology project fault analysis, software development and digital forensic investigations. He has appeared before different tribunals both in the UK as well as overseas. He served two detailed reports with numerous appendices. He also signed four Joint Statements with Dr Worden.
752. Both the experts agreed in the 3rd Joint Statement that Horizon as it is now is relatively robust. The passages in that Joint Statement are that:
“in line with most other large-scale computer systems, Horizon's robustness has generally improved. From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other adverse events. This increase in robustness has, in part, developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.”
753. I consider that this is an important passage of the experts’ agreement, and indeed robustness is an important aspect of the Horizon Issues. It makes clear that, in part, it is the discovery by the Post Office of bugs, errors and defects, the application of fixes (which means changes to the software by means of a software fix) and the improvement of monitoring that has led to the increase in robustness that has been achieved in the system as it is at 2019. This inevitably means that in earlier times, Horizon was less robust, or not as robust as it is now. This applies both to Legacy Horizon and also to Horizon Online. There are undoubtedly more modern or recent bugs in Horizon, as can be seen from the Technical Appendix. Drop and Go, for example, was not even a product until about 2013, and the relevant bug in the Bug Table which is bug 28 occurred in July 2017. That caused an impact to the branch account in question of £100; modest or tiny in comparison to the total Drop and Go business for that year, or even that month, but still £100 which appeared as a deficit in the branch accounts, and still £100 which the SPM had to deal with. Until it was discovered the bug could not have been fixed. It would not have been present prior to 2013 (because Drop and Go did not exist) and any fix cannot have occurred prior to July 2017, if not a month or two later. The agreement that Horizon is relatively robust as at 2018 and 2019 is something that is doubtless of great importance to the Post Office, as it affects how the Horizon system as it is at the date of this judgment is both used by the Post Office, and perceived by the SPMs who use it every day. The Horizon Issues however deal with Horizon from the year 2000 onwards.
754. The other aspect of this part of the experts’ agreements that bears consideration is that the Post Office have to know about bugs, defects and errors in order that they can agree with Fujitsu that fixes have to be written and applied. This makes it particularly important, in my judgment, that Fujitsu were open and transparent with the Post Office about the presence of software bugs, errors and defects. Given Horizon was being used by the Post Office as the accounting procedure between it and its SPMs, there is every reason that Fujitsu would (or should) have to make the Post Office aware of bugs, errors and defects.
755. A point that was put to Mr Coyne (it was taken from the Post Office’s pleadings, but it is a theme that runs through both its submissions and also the evidence of Dr Worden) is that there are both what are called technical controls, and also operational procedures and practices conducted both by the Post Office and by SPMs, to increase the reliability of data. The passage of the Defence that was put was as follows:
“Q. So robustness -- we can move on to paragraph 54 on page {C3/3/23}:
" ... in addition to the technical controls referred to above, there are several operational procedures and practices conducted by Post Office and subpostmasters that serve to increase the reliability of the data stored in the central data centre as an accurate record of the transactions entered on branch terminals.
Then there is a list of those. So you will see that what's being said is that the robustness of the system is said to be affected both by technical measures and by human measures. Do you see that?
A. Yes, I can see that. That is what is said, yes.”
756. In my judgment it is correct to describe the second part of these, as the Post Office did in its pleading, as “human measures”. They rely upon humans actually performing or being aware of certain occurrences. Two of these rely upon SPMs, namely those at paragraph 54(3) and 55(4) of the Defence. The second of these is that “the accounting and record-keeping obligations placed on Subpostmasters reduce the risk of any errors going undetected.” In my judgment, for SPMs to be an effective – or indeed any – part of a “human measure” and to assist or contribute to the process of the Post Office discovering bugs, errors and defects in live use (so that, where necessary, software fixes can be initiated, developed and applied) the Post Office ought sensibly to be transparent with SPMs too.
757. It does however need to be recognised that human measures are not part of the Horizon system. They are not included in the definition of the Horizon system that forms part of the Horizon Issues themselves.
758. It was also, in substance, put to Mr Coyne that Horizon Issue 1 should have been addressed in the way that Dr Worden had approached it, by effectively looking at actual (rather than potential) impact, although it was not put in exactly those terms.
“Q. Given it was admitted that there have been occasions when these things have happened, to answer the question: is it possible or likely that these things have happened by saying "yes" isn't a very helpful answer, is it?
A. If the answer was just simply "yes" and with no further context then I agree that that would ...
Q. The critical issue that's raised by the four issues that I have read to you is the word "extent" and the word "likelihood" and the word "risk", isn't it?
A. They certainly are critical elements that would need to be reviewed, yes.
Q. What you were being asked to do by those issues is to give your opinion to the court on the extent to which it was and is likely that bugs caused shortfalls in any particular set of accounts?
A. Well, across the accounts of subpostmasters, yes.
Q. The extent to which various measures in place reduce the risk faced by any given subpostmaster when doing a transaction or when producing a particular set of accounts, yes?
A. Yes.
Q. That's the critical question on which the court needs your assistance, would you agree?
A. It is one of the critical questions, yes.
Q. Could you identify a question which you say is more critical than that in the context of these proceedings?
A. I haven't taken a view on -- I have taken each of the Horizon Issues that's put to me with equal priority. I haven't set out to decide a prioritisation for them.”
(emphasis added)
759. This approach by the Post Office was, in my judgment, an attempt effectively to re-draft Horizon Issue 1 in terms more favourable to its case. I have already identified above that the Horizon issues were drafted and agreed by the parties and approved by the court. Horizon Issue 1 plainly states “to what extent was it possible or likely for bugs, errors or defects of the nature alleged….to have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to” SPMs’ accounts. The approach taken by the Post Office (and also Dr Worden) was entirely to ignore the word “potential”.
760. I also consider that it is a matter for the court to decide which issues are more important than others. An expert might decide a particular issue is important in that it represents a starting point for their opinion, which will then inform or guide their views on other issues. But it is not for an expert to decide one issue in a list is more important to the court than others, and concentrate upon that to the potential exclusion of consideration of some or any of the others. Mr Coyne was therefore correct to treat all of the Horizon Issues with equal priority.
761. Mr Coyne had not performed a similar statistical analysis to Dr Worden seeking to demonstrate either the likelihood, or unlikelihood, of a bug impacting the claimants’ branch accounts. In my judgment such an exercise was not necessary in order properly to give expert evidence to the court on the Horizon Issues. Dr Worden’s evidence, and his alternative approach, was properly put to Mr Coyne. Mr Coyne accepted that he could have approached his evidence in that way, but explained why he had not. He described the approach adopted by Dr Worden as “a fundamentally flawed approach”.
“Q: You could, using your skills as an IT risk analyst, you could have, by reference to the KELs and the PEAKs that you had identified, formed an assessment as to the likely financial impact in each of the instances that you had identified?
A. Yes, I could. If I could please answer that by way of illustration to the problem with that approach.
Q. Please do.
A. We will often see a bug, error or defect with a very wide range of impacts and the impact is typically whatever the counter was doing at that point in time. So if the counter was doing a foreign currency transaction for just £50 and something goes wrong, the discrepancy may be £50.
By knowing that there is a bug, error or defect in the Horizon system that leads to problems with foreign currencies, you can't then say it is only a £50 defect because that isn't incorrect. If another person was to be subject to that defect and they were doing a £10,000 foreign currency transaction, they would likely have that same level of defect.
Now that's an illustration to say you can't value bugs and their potential impact by looking at what has happened historically. You can't value it in that way.
Q. But what you can do is form an estimate having regard to the totality of the PEAKs that you have seen, can't you?
A. But it is a fundamentally flawed approach. If you have seen three branches that have had an impact because they were doing three foreign currency transactions, a £20, a £50 and a £100, but then there is a fourth person that believes that they have been subjected to that but that isn't recorded, you can't simply look at the three where it has been recorded and say the fourth couldn't possibly have occurred because we know there is a defect there.
Q. I'm not suggesting that you could arrive at a certain conclusion of an absolute cast iron number, but I repeat my question. Can't you form an estimate having regard to the totality of the PEAKs that you have seen?
A. Yes, but your estimate would have to be based on the three people where it has been recorded to have occurred, so you would say £20, £30 and £50, and the best you could possibly do is come up with an average of that, and you would say that the value of that defect is whatever that is.
Q. There is another way that you could do it, isn't there, which is that you could look at the three bugs, the receipts and payments mismatch, the suspense account bug and Callendar Square, the ones that have been thoroughly investigated, and you could form inferences from the scale of those bugs, yes? Would that be reasonable?
A. For those types of bugs, potentially yes.
Q. Those are quite large bugs, aren't they? They are not small bugs in the scheme of things. That's why they were identified in the letter, because these were major bugs of which even Post Office was aware?
A. Yes. And carrying on from the one we were not told about until later, the Dalmellington one, there were some which were only pounds, just a few pounds, and I think there was at least one that was £25,000.
Q. But just fixing our attention for the moment on those three bugs, would you accept that the evidence indicates that altogether the total financial impact of those three bugs is no greater than £100,000. Do you agree with that?
A. No, I haven't done that so I would not know whether that was right or close to right”.
762. The documentary evidence showed that Dalmellington had over 100 branches with some being impacted more than once. Using the Post Office’s definition of “lasting impact”, which means the branch accounts were given TCs to correct the effect that the bug had upon them, 114 of these impacts were made good. Of the other four, two turned out not be the Dalmellington bug and the other two had only very small financial discrepancies.
763. However, this bug, which had taken 5 years to be discovered, was put to Mr Coyne as a good example of how countermeasures had to be considered.
“Q. In fact, Dalmellington is quite a good example of how countermeasures need to be brought into account and how it is important to look at financial impact to make an assessment of the extent question that's raised in Horizon Issue 1, isn't it? Because it is no good saying 118 branches were affected and some of them were affected by large amounts, when in fact we all know that even before the bug was detected the branches involved were in fact made whole anyway, yes?
A. It does suggest that, but it also suggests that there were deficiencies with the countermeasures because I believe it was five years after the first occurrence of the bug that the defect was finally discovered.
Q. Because, as you have already agreed, to an outside observer it looks exactly like a human error because it was a human error. The bug wasn't causing losses, it was causing errors to be made and those errors were picked up, would you agree with that?
A. No, it wasn't a human error, it was a defect of the system. It made it look like a human error.”
(emphasis added)
764. The Dalmellington bug was not accepted as a bug in the Bug Table at the time of the 3rd Joint Statement where it is Bug 4, but is now accepted as a bug by the Post Office but said to have “transient impact”. I find that it is a bug, and it cannot properly be described – as the Post Office did during the above cross examination– as human error. As Mr Coyne pointed out in his answers, it was a defect of the system. It also did take 5 years for it to be discovered. One of the branches affected by it had its branch accounts affected to the sum of approximately £25,000. The fact that a TC was issued does not mean that it did not have an impact upon branch accounts. That impact may have been later corrected by means of the issuing of a TC, but that cannot mask the fact that the branch account was affected by that amount for the period of time until that correction was made.
765. Mr Coyne accepted that Dr Worden’s arithmetic was correct, and also accepted that the sum Dr Worden had calculated in respect of Dalmellington (£100,000) was “quite small compared to the £19 million that’s claimed by the claimants in this case. It is less than 1%”. As Mr Coyne accepted, on the “pure numbers”, that was correct.
766. Mr Coyne did not accept Dr Worden’s basic logic as a starting point, nor did he accept the logic that there was “no evidence that” the claimants’ branches were “more or less likely to be hit by bugs than your average Post Office branch”. I deal with that point in further detail below at [821] and [822] below, but this amounts to an assumption by Dr Worden that a group of SPMs who specifically allege they have experienced the effects of bugs are to be treated, in statistical terms, as though they are a random group of SPMs of the same sample size drawn from the wider population of all SPMs. They plainly are not a randomly drawn sample of nearly 600 SPMs. They are a very specific group (or sample) of those who say their branch accounts have been impacted by, or have experienced, such incidents. In statistical terms, the correct term for the group is that they have a bias – they all allege that they have experienced the effects of bugs, errors and defects.
767. An illuminating series of questions and answers highlighted the entirely different approach of the two experts.
“Q. Are you aware of some special factor applying to the claimant branches which marks them out as very different from the rest of the branch network making their susceptibility to bugs very different from the wide range of branches in the rest of the network?
A. I'm not aware of it, no, but it would require -- in order to look at that it would require a detailed understanding of the particular processes of the branches. And I think that's illustrated by what we know of Dalmellington, that impacted -- I think one branch was impacted five times, where there was only a handful of branches -- I think, was it 88 that was impacted in total?
Q. Mr Coyne, it is quite important not to get confused between particular instances of things happening and the law of averages. What I'm asking you about is what's likely to happen over a large number of cases, do you understand the difference? So of course if people are all walking down the streets, some of them are going to get hit by lightning. Everybody won't get hit by lightning in the same way. It will be a very small number of people who will be hit by lightning and they'll usually be in particular situations when it happens. But one can make a generalisation about the likelihood of people being hit by lightning, do you understand, because of the law of averages?
A. I do understand, but in that illustration it would be very unusual for one person to be hit by lightning five times.”
768. This then continued with some discussion of instances where some people may have been hit by lightning more than once.
769. There are some important points that arise out of this approach. Firstly, the terms of the contracts between the Post Office and an SPM does not invoke the law of averages, either as it was explained in this cross-examination, or at all. The terms of the contracts, whether SPMC or NTC, have a very precise accounting process which is run by Horizon. Secondly, the existence of a bug such as Dalmellington, which was present in the Legacy Horizon system for 5 years prior to discovery, is of direct relevance to the Horizon Issues. It does not support the proposition that a SPM with an outreach branch in the 5 years before it was discovered was more likely (or equally likely) to be hit by lightning as having their branch accounts impacted by the Dalmellington bug. Thirdly, even if the “law of averages” approach were the correct one to adopt in terms of individual SPM complaints that their branch accounts had been impacted by a bug, error or defect in Horizon (and to be clear, I find that it is not) it is an exercise that could only sensibly even be attempted by firstly using the total number of bugs present in the system (which would require at least this judgment, if not more investigation too following the judgment), a definitive total for the number of instances of each bug occurring (again, which would require another judgment), a definitive total for the full amount of losses caused by each incident of each bug (information again not yet available, if ever) and some sort of probability calculation taking account of all those matters. That calculation would only, in any event, lead to a figure for the probability chances of any SPM in the period having their branch accounts impacted by a bug. So many of the essential ingredients or pieces of data for such a calculation are not known, that to try to attempt it in the way put by the Post Office, is to start from the end and work backwards, and work backwards in a way wholly lacking in mathematical precision. The “law of averages” approach cannot be used, in my judgment, in the way contended for by the Post Office.
770. So far as Dr Worden’s “scaling factor” is concerned, and his approach to the claimants’ branches, Mr Coyne was entirely reasonable, in my judgment, in how he would consider hypothetical situations put to him and also explain why they were not valid.
“Q. At this point in the equation what Dr Worden is trying to do is to work out how likely it is branches are going to be affected?
A. Yes.
Q. He reasons that branches with more transactions, doing more transactions, are statistically over a large number of occasions more likely to be hit by a bug than a branch doing fewer transactions and would you agree the principle underlying that observation?
A. I think that's probably a reasonable principle. If we look at the defects we found though, there is probably other factors that could be brought into that.
Q. But you are not aware of any, you are not in a position to suggest a single factor which you have any evidence to think actually applies in relation to the claimants as compared with the rest of the branch network?
A. I don't know the make up of the claimant, but one example might be, do they have an outreach branch?
Q. You are aware, aren't you, how small the number of outreach branches there is, how insignificant that is in the context of the Post Office network, aren't you?
A. Well, it wouldn't be insignificant to the people that are impacted by the defect and I don't know whether any of the -- because I haven't studied the claimants, I don't know whether any of the claimants have that or not.
Q. What you are doing is you are speculating that some of the claimants might have outreach branches?
A. I'm not. I'm illustrating the potential problems with the approach that you are taking just simply using the unit of number of transactions.”
(emphasis added)
771. Similar points can be made, in my judgment, to each of the bugs in the Bug Table, and not simply Dalmellington and outreach branches. Some affect foreign currency transactions. Bug 9 in the Bug Table, Reversals, only affected the value of transactions that were sought to be reversed by a SPM, and were impacted by the bug actually introduced into Horizon by version S30 which was released by Fujitsu. That this occurred at all was due to a failure in regression testing by Fujitsu prior to release of S30. This would, if the bug hit that branch, show the value of a reversal twice in branch accounts. The impact on the branch accounts would be the value of the reversal; if it were a large figure, the impact would be large, and vice versa. Something of that nature cannot sensibly be dealt with by the law of averages. Concurrent log ins, in the early days of Legacy Horizon, and numbered bug 18 in the Bug Table, would only occur if a user was logged in at the same time to two terminals. Again, that cannot sensibly be dealt with by use of the law of averages.
772. As Mr Coyne put it when asked about Dalmellington above and reminded that there are not many outreach branches in the Post Office network and how insignificant this was, “well, it wouldn't be insignificant to the people that are impacted by the defect”. This answer, which in my judgment is an entirely accurate one in the context of this group litigation, is one that could apply to each of the instances or incidents of each of the bugs, errors and defects impacting branch accounts that have now come to light in the Horizon Issues trial. Individually and when considered across the whole of the 3 million sets of branch accounts in the period from 2000 onwards, the amounts are individually small amounts (although sometimes they are several thousand pounds). The same point can be reached by simply taking the sum at stake in the group litigation in terms of accounting losses for all the claimants (as opposed to other heads of claim not quantified) which is £18.7 million, and comparing it to the total amount of business transacted in the Post Office branch network as a whole across the entire UK from the year 2000 to date. No one has calculated that latter figure so far as I know (and if they have, they have not brought it to my attention) but it will be an enormous sum. The fact that £18.7 million is but a small proportion of that enormous figure for total transacted business over 18 years is of no assistance whatsoever, and neither is the law of averages approach.
773. Finally on this, in one of the answers to Mr Coyne’s requests for information (on another point, discrepancies from external data sources), the solicitors acting for the Post Office pointed out in the document dated 4 June 2018, the number of transactions that Horizon deals with each day. This was done in order to give a sense of scale to the request posed by Mr Coyne. The exact answer provided was “To give this a sense of scale, it is noted that Horizon processes around 6,000,000 transactions per day.” The reason for identifying this is that this figure for transactions per day, together with the agreed figure for the number of monthly branch accounts over the period agreed by the experts, namely 3 million, gives a sense of scale of the Post Office’s business conducted by Horizon and the total number of transactions over the period. Given there are fewer than 600 claimants, the number of impacts on branch accounts complained of by the claimants is a very small fraction of this (whatever its precise total). The law of averages approach does not assist the Post Office.
774. So far as countermeasures are concerned, Mr Coyne stated that “typically if you were going to design a countermeasure you would want to have it finding the defect straightaway. If it was only going to find it sometime later it would be of limited value.” Not only do I agree with that in general IT terms, but also in terms of the Horizon system and its accounting function between the Post Office and each SPM’s branch trading period specifically. A “countermeasure” in this instance would be something that would correct an impact to the branch accounts. If that is only going to occur, as Mr Coyne put it “sometime later” then it would only be of limited utility, if any, depending upon the period of time this takes. Further, the Post Office used the term “countermeasure” as including “human measures and controls”, as made clear to Mr Coyne during his cross-examination:
“I'm using it as a term to refer to measures and controls in existence, both technical and human measures and controls, which are designed or have the effect of identifying bugs in Horizon or identifying the consequences of bugs in Horizon so that they can be looked at and fixed.”
775. This means therefore that the Post Office was relying upon matters outside the definition of the Horizon System at the beginning of the Horizon Issues, which stated “ “The Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed, as defined in GPOC, at §16 and as admitted by Post Office in its Defence at §37.”
776. As Mr Coyne explained, countermeasures in the system are addressed at the design stage:
“Typically countermeasures are a design aspiration. When you sit down to design a computer system and what controls are in place, then you would consider what controls are required, where to position them. Should it be the case that the controls fail do you have a human aspect doing a secondary check? It could go all the way to employing auditors to re-check figures. You could go on ad infinitum with –
Q. So you accept, then, that these controls or countermeasures, whatever you would like to call them, some of them have the effect of stopping the bug in its track. Well, some of them have the effect of stopping a bug arising in the first place?
A. The best way of dealing with defects is to ensure that they don't arise in the first place ……..so what you would do is throughout your testing regimes you would flush out all of the defects at that point in time, and you would put in controls to ensure that if a defect does trigger, there is a way that you are alerted to it and it is fixed as soon as possible.
Q. Yes. So one of the objectives is to have a testing regime which picks up as many bugs as possible?
A. Indeed.
Q. But I think you accept that it is impossible to have a testing regime that picks up all of them?
A. It is impossible to have a testing regime that picks up all of them and that is why you have a process of categorisation and you would allow a product to go live if -- one of the terms that's used -- all severity 1 and severity 2 defects have been dealt with. Severity 3 might be largely cosmetic and you would allow a system to go live with cosmetic defects.
Q. That's not quite what I'm asking you. You accept there are going to be bugs which arise in the operation of any IT system which would simply evade any test that any human being can devise?
A. There's always going to be bugs that are unknown to be within the system at the point that you go live and they are only discovered sometimes weeks afterwards, sometimes years afterwards.”
777. Both defensive programming and impact management are part of, or are descriptive terms concerning, countermeasures. In my judgment they are both more to be described as “technical controls” (to use the language of the Post Office’s pleading) rather than “human measures”. These matters were all effectively part of the 3rd Joint Statement. So far as countermeasures within the Horizon System itself are concerned, these plainly affect the robustness of the system. The 3rd Joint Statement agreed that (using different numbering):
1. The effectiveness of various countermeasures changed throughout the life of Horizon;
2. Countermeasures are basic elements of practical IT system design;
3. Countermeasures work by limiting the impact of Horizon bugs/error and defects on branch accounts. Countermeasures do not always eliminate the effects of adverse events (they are not perfect) but they are often effective in the area where they are deployed; that is why they have become basic elements of practical IT systems design;
4. It is difficult to measure the extent of the robustness of Horizon, apart from how it might limit the extent of impact on branch accounts, as in Issue 1.
(emphasis added)
This shows that IT countermeasures are designed into systems. That is entirely to be expected. However, it also means that the robustness of a system cannot be measured by other measures (such as TCs, or a human complaints department as another example) taken to correct the effect of the IT system. A ready way of illustrating this is as follows. Consider a hypothetical IT accounting system that has 10 countermeasures designed into it, as a matter of IT design. It also has a human operated complaints department. All 10 of the countermeasures fail so a user has to contact the complaints department, which upholds the complaint and issues a correction, that undoes the impact within the accounts the IT system has allowed to occur. The efficiency of that department, and the accuracy with which it deals with such matters, cannot be used to assess in IT terms the robustness of the IT system itself. It is separate.
778. The Post Office also has an inconsistent approach to TCs. On the one hand they are relied upon (through Dr Worden) as being an important countermeasure – even though they are not part of the Horizon system as defined – yet on the other hand questions about them were not dealt with as though they were relevant to the Horizon Issues. For example, one of the requests raised by Mr Coyne in respect of a particular PEAK was as follows:
“Regarding POL-0032919.pdf, the GoldenGate replication between Oracle 10g and 11g being aborted and resulting in a number of branches reporting cash declaration and stock reporting discrepancies, were any transaction corrections sent to the 247 affected branches as a result of the discrepancies and which branches were affected by the incident?”
779. The answer that was given by the Post Office’s solicitors was as follows:
“The relevance of this request is not understood. Transaction Corrections are issued by Post Office to correct transient discrepancies in branch accounts in order to restore the correct position.”
780. This answer to a request by the claimants’ IT expert therefore – correctly in my judgment – identifies that TCs are not part of the Horizon system. That is entirely at odds with them being part of a countermeasure within the Horizon system. This is effectively accepted in the Post Office’s opening submissions which expressly state at paragraph 114 that the “Post Office’s reconciliation processes are outside the scope of the Horizon Issues”. TCs are part of the Post Office’s reconciliation processes, and I find that they are outside the scope of the Horizon Issues. Dr Worden therefore cannot sensibly take them into account as a Horizon countermeasure. The Post Office cannot have it both ways.
781. One of the matters that Mr Coyne relied upon was what he described as “the sheer volume” of KELs and reconciliation reports (a statement, it should be noted, made prior to discovery of the 5,000 KELs disclosed by the Post Office in October 2019). His evidence was that this confirmed “the wide-ranging extent of the impact of such bugs/errors/defects. This evidence demonstrates that such bugs/errors/defects would undermine the reliability of the Horizon system to accurately process and record transactions”. He accepted that there was the need for reconciliation reports, and that this was part of the way that a system would check whether (say) the banking records at the Post Office transactions matched the corresponding records at (say) the Bank of Ireland for the same transactions. In my judgment KELs are indeed part of the records (or contemporaneous records) that can be considered in order to arrive at an overall conclusion for the existence and potential impact of bugs errors and defects, but the PEAKs need to be consulted too. This is because the PEAKs are the documents that often have greater information within them, added to which one single KEL could relate to a couple of short PEAKs, or alternatively to many a great many far lengthier PEAKs, and it is only from the PEAKs themselves that one can gain a better view of the impact of that particular bug. Indeed, some of the KELs simply have a start and end date, with very little information on the number of impacts during the intervening period. It is clear to me that if there is more than one PEAK referring to the same KEL, one would only realise that there had been a re-occurrence of the same defect if one considered the PEAKs; the KEL would not show this. Simply looking at the KEL could give the erroneous impression that such a defect had only occurred once. It is also telling, in my judgment, that some PEAKs that are clearly related to errors, for example the PEAK associated with the issue that the subject that “SSC Database users do not have correct permissions”, namely PC0208119 dated 10 March 2012, expressly states in one of the 25 October 2011 entries the following “Have relevant KELs been created or updated? No. None believed to be required”. This clearly demonstrates that Fujitsu would not always even create a KEL for known errors (as SSC users having the incorrect permissions, all having APPSUP when they were not supposed to have this, is clearly a known error). SSC users were (as set out in the PEAK) only supposed to be given “the optional role 'APPSUP' temporarily (by Security Ops authorisation) if required to make emergency amendments in BRDB Live”. I find that the KELs alone do not constitute a record of all bugs, errors and defects.
“Q. But I don't understand, Mr Coyne, why you think that the existence of reconciliation reports confirms the wide-ranging extent of the impact of bugs in Horizon which are having an impact on branch accounts?
A. I have worked and designed banking systems, stock broking systems. I have never seen the need for tens of thousands of transactions per week to have a human intervention. That suggests that something is going wrong. It is working outside of process on a larger scale than I would have expected.”
(emphasis added)
783. The origin of the number of reconciliation reports is as follows. Mr Coyne had asked, in requests for information prior to the trial, what was the purpose of setting an NB102 exception to F99 by Fujitsu. The Post Office in the answer explained that this explanation would have to come from Fujitsu, and the explanation was given:
“When there is an incident involving a reconciliation exception in Network Banking which has been fully processed, then the transaction needs to be set to F99 to indicate that processing is complete.
Therefore, this is done for any transaction that appears in a reconciliation report, once the resolution is complete.
The reconciliation process document is SVM/SDM/PRO/0012 RECONCILIATION AND INCIDENT MANAGEMENT PROCESS and describes how reconciliation is carried out.
SVM/SDM/SD/0020 END TO END RECONCILIATION REPORTING is the specifications of the reports that are produced for the reconciliation service.
The CCD that describes the contracted service:
SVM/SDM/SD/0015 RECONCILIATION SERVICE: SERVICE DESCRIPTION.”
784. The next request was “How often has [setting an NB102 exception to F99] occurred?”
785. Initially the Post Office would not answer this, stating inter alia that it was very unlikely that the information would have been pooled or collated. When Mr Coyne persisted, and pointed out it would be very different if the answer was “it was an isolated incident in 2003” or “it was 10,000 transactions each day for the last ten years”, an answer was provided by the Post Office through its solicitors. I shall reproduce it in full:
“Fujitsu currently "F99" 10,000+ transactions per week across all NB102 associated reports (DCP and NBS)”.
786. This was correctly understood by Mr de Garr Robinson for the Post Office who (understandably) corrected Mr Coyne when he initially said he thought it was hundreds of thousands per week.
“…I asked for the number of reconciliation problems that required manual intervention, and the answer I was given was that it was hundreds of thousands per week, I think was the –
Q. 10,000 a week that have to be F99ed, do you remember that?
A. Yes.”
787. So far as the next request, which was associated with this, is concerned, that paints the same picture of the Horizon system in my judgment. The request was:
“What is the cause of an ‘Uncleared Transaction Corruptions’ and how often do [Uncleared Transaction Corruptions] occur?”
The answer was, again from Fujitsu:
“The DCP report NB102 section 3 (Uncleared Corruptions) usually contains 100s if not 1000s of transactions per week (Kiosk transactions only).
This is where there is a discrepancy on banking and payment card transactions. To determine the exact cause of any individual Uncleared Transaction Corruption, that particular occurrence would have to be analysed. However, the most usual cause is for a banking request or authorisation failing to reach its destination, which is something that can have a variety of causes. An example may be that a bank may authorise a transaction, but that confirmation is not delivered to the counter so the SPM believes that the transaction in fact failed due to some issue.”
788. This is a great number of transactions per week that require manual intervention – over 10,000, on the numbers provided by Fujitsu. I consider that Mr Coyne is correct when he says “this suggests that something is going wrong”. I do not accept that on a properly functioning and robust system there should be such a high number as that every week. I accept Mr Coyne’s evidence in his answer at [782] above.
789. Mr Coyne also considered, in respect of the control environment, that the Ernst & Young management letter of 2011 had recommended changes to strengthen the change management process. E&Y had noted that the Post Office was not usually involved in testing fixes or maintenance changes to the in-scope applications (which includes Horizon Online) and they were unable to identify an internal control with the third-party service provider to authorise fixes and maintenance changes prior to development for applications. The risks were outlined by E&Y in the following terms:
“There is an increased risk that unauthorised and inappropriate changes are deployed if they are not adequately authorised, tested and approved prior to migration to the production environment”
Mr Coyne also drew attention to a PEAK some years earlier than this (in 2003) that provided evidence that Fujitsu did not even always follow the correct processes that were in place in any event. In this instance a branch terminal was replaced, which resulted in a number of bills which branch customers had paid (in terms of making payment at the branch Post Office) not in fact being paid (to the ultimate payee, the Post Office clients that had issued the bills). The PEAK stated that “something went wrong and the marooned transaction did not find their way back to the office”, something which had occurred because the mirror disc in the branch terminal was replaced (when it should not have been). The PEAK also stated:
“I believe that we shall have to confess to POCL that we have not followed the correct procedure and we should advise that POCL make manual payments…..”
POCL means the Post Office, under its earlier name of Post Office Counters Ltd.
790. Mr Coyne was of the opinion that the specific recommendations in the 2011 E&Y letter directly impacted upon Dr Worden’s view of countermeasures. He also drew attention to the fact that the non-acceptance of these recommendations by the Post Office was an opportunity to impose or improve countermeasures in Horizon which the Post Office chose not to take. The four areas, and the countermeasures they would impact, were:
“1. Improvements to outsourcing application management (Quality and Change Control; managing non-functional requirements; testing good practice);
2. Improvement to segregation of duties within the “manage change” process (Quality and Change Control, Security);
3. Strengthen the change management process (Quality and Change Control);
4. Strengthen the review of privileged access (Security).”
791. Mr Coyne also relied upon the fact that although both Legacy Horizon and Horizon Online did contain a number of measures and controls, these mechanisms had been shown to have failed, and this was agreed by the experts. The minutes of the Post Office’s own Risk Compliance Committee meeting of 18 September 2013 showed that not all of the risks identified in the then-recent E&Y audit had been addressed. All four pages of these minutes were heavily redacted, but one surviving entry shows that risk not addressed was “relating to the communication by Fujitsu of changes made to the Horizon system, was still outstanding. It was identified that it would cost over a £1m to implement the mitigation being suggested by the audit and that this was not proportionate to the risk being managed.” Mr Coyne accepted in his cross-examination that this did not relate to a bug being fixed, and I accept that evidence. In my judgment it does go to the robustness of Horizon.
792. There was one area which was concentrated upon towards the end of his evidence, and this was an entry in the 2nd Joint Statement. This was dated 25 February 2019, and under Horizon Issue 1, under the topic “the extent bugs were possible or likely” the experts had agreed the following entries relevant to this:
“1.1 We agree that bugs were possible, and the table above displays a number which in the opinion of either or both experts appeared to have impacted branch accounts.
1.2 Referring to the table of bugs above, the experts agree that the bugs in rows 1, 2, 3, 10, 13, 14, 18, 23, 24, 25, 27 and 28 may have had financial impact on branch accounts. Other rows, the impact is not agreed between the experts.”
And later on in the 2nd Joint Statement:
“1.15 The number of distinct bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in branch accounts, is between 12 and 29.”
793. Just before the short adjournment on the final day of his cross-examination, Mr Coyne was asked about the figures in that range. He was also asked to “forget Horizon Issue 4”. Given that Horizon Issue 4 asked “to what extent has there been potential for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?”, being asked to ignore it somewhat narrowed down the utility of the exercise, but it is without doubt the case that Mr Coyne, when asked for a number for the upper range in the Joint Statement, stated it should not be 29, it should be a different figure. He initially stated that it should be 13. The court then rose for the short adjournment. As soon as court resumed one hour later, he immediately sought to correct the number 13 that he had given. He said the number should be 21. A complication in this cross-examination exercise was that initially Mr Coyne was using what he called his bug table in his 2nd report (which had been dated 1 February 2019, and amended on 6 March 2019) and the cross-examination was being done by reference to something also referred to throughout the trial as “the Bug Table”, which was the one in the 2nd Joint Statement. The former in the report had a column headed “evidence of branch impact” with the entry “yes” or “no”; the entry in the Joint Statement about which Mr Coyne was being asked was item 1.15 and is as set out above. The questioning was done solely by reference to Horizon Issue 1. Table 1 in Mr Coyne’s 2nd report made it clear that Phantom Transactions, Reconciliation Issues, Concurrent log-ins, Post & Go/TA discrepancies in POLSAP, Recovery Failures and Transaction Correction Issues were all listed under Horizon Issue 4, and not Horizon Issue 1. An associated difficulty was the distinction that was being made by the Post Office between lasting and transient effect.
“Q. Are you suggesting that you have only ever suggested 21 and no one was suggesting anything above 21?
A. No. There are 29 bugs, errors and defects in the list and in the table starting at number 1, but some of those go to other Horizon Issues apart from Horizon Issue 1, and they do say that clearly in the heading.”
795. There is no doubt that Mr Coyne made matters a little hard for himself during this passage of questioning, and the Post Office seized upon his change of number (which was, after all, an agreed number in an agreed statement by both experts) and made much of it:
“Q. Mr Coyne, everybody on this side of the court, and I mean everybody on this side of the court, believed that your view was that the evidence showed that there were 29 bugs with lasting financial impact. You are now saying that we made a mistake, are you?
A. Yes.
Q. Could I ask you to explain where you made it clear what you were actually saying?
A. Well, as set out in my second report, there is a table there that quite clearly says whether there is evidence of branch impact or not. And there is also a list of which Horizon Issue it specifically relates to. The only addition to that is when you come to the very last column, there's other bugs, errors and defects that are re-introduced and they are only covered in the joint statements.”
796. Mr Coyne also said:
“A. Throughout the report I have set out next to every single defect whether it is relating to the test of Horizon Issue 1 or a later Horizon Issue, and I have also set out whether I believe there was impact on branch accounts all the way through report number 2.
Q. Let's not talk about report number 2 because the statement -- the most up to date statement of the position, I think you agreed with me at the beginning of today, the culmination of all your work, including work you did after your second report, is the bug table in joint statement 2?
A. Yes, but when you put that to me I said that that can't be read in isolation, it must be read with the second report.”
797. So far as the number is concerned, Mr Coyne settled on 22 different bugs, his explanation of his earlier different numbers was as follows:
“Q. And I would like to ask you why is it you thought there were 21 bugs at 2 o'clock, whereas you thought -- was it 14 bugs at 1 o'clock? 13 bugs at 1 o'clock.
A. Because I was just counting down the table at number 1 in my report. In the pressure of the situation I just tallied that column down, I didn't have enough time to consider it fully.
Q. To go back to a question I asked you earlier, where do you make this clear in the joint statement?
A. In the heading next to the -- in the bug, error or defect.”
799. It is the experts’ responsibility to agree the content of their Joint Statements, and this is part of their duty to the court as independent experts. It is the basis on which the court gives permission for expert evidence. No authority is required for these propositions as this is wholly conventional, but to the extent that authority may assist, Males J (as he then was) fully explained this in [3] of Mayr and others v CMS Cameron McKenna Nabarro Olswang LLP [2018] EWHC 3669 (Comm). The agreed text of the agreed parts of a joint statement therefore cannot be approached as though it were a unilateral passage in only one of the expert’s reports. It is an agreed passage. I do not consider that Mr Coyne’s changes of the number he was asked about demonstrates any unreliability on his part, or discredits him or his evidence.
800. I consider Mr Coyne to have been a helpful and constructive witness, and I find the suggestions made to him that he was biased to the claimants and not independent are criticisms that are not justified. He, and his small number of assistants, had done a great amount of investigation into the very numerous PEAKs, and the smaller number of KELs, and he had embarked upon a careful and sensible exercise necessary for him to reach conclusions on the Horizon Issues, as drafted and agreed by the parties and as approved by the court.
801. There were some fundamental methodological differences between him and Dr Worden, but the agreements that the two of them reached were detailed notwithstanding this. I found the contents of all of those agreements of great assistance.
Dr Worden
802. Dr Worden was educated at the Université de Grenoble, Cambridge University and the California Institute of Technology, better known as Cal Tech. He has a Ph.D in Theoretical Particle Physics. He is currently a consultant with a practice called Charteris Associates, having started there in 1997, but he has been a consultant since August 2010. Starting with the first entry on his CV in terms of his work experience, between 1975 and 1986 he worked on the development of knowledge-based systems and developed Logica’s proprietary relational database management system, and between 1986 and 1990 he was the Managing Director of Logical Cambridge. This is an advanced technology subsidiary of Logica. Between 1991 and 1995 he acted as an independent reviewer of many of Logica’s largest projects, the key projects that constituted a large part of Logica’s commercial risks, and also a large part of its commercial reputation. Between 1995 and 1997 he was a Business and Technical Consultant in Logica, moving to Charteris in July 1997. He has given evidence in the High Court before on several occasions, including in what was at that time the biggest IT dispute tried, namely BSkyB Ltd v HP Enterprise Services UK Ltd (formerly EDS Ltd) [2010] EWHC 86 (TCC). This case is notable for a range of reasons, not least that the judgment (excluding quantum) runs to some 2350 paragraphs; and also that the educational qualifications considered in the judgment included those of one main factual witness who turned out to have fabricated the establishment from where he claimed to have been awarded a MBA. This was a point demonstrated by counsel enrolling his own dog (“Lulu”) in the same establishment, Lulu gaining an MBA with higher marks than the witness. The hearing ran for 109 hearing days and Dr Worden was cross-examined in that case for more than 35 hours, which is a very considerable period of time for anyone to be in the witness box. In his CV he identifies that the majority of his opinions referred to in the judgment of Ramsey J were accepted by him. At [277] in that judgment he was described as “more of a generalist in the field of IT with some experience dealing with CRM systems but with good general knowledge of the estimating, systems integration and the types of problems which can arise on projects such as” the project in that case.
803. He is a highly qualified person, both in IT generally and specifically, and in statistics. He has earlier experience with relational database management systems when he was at Logica in the late 1970s and early 1980s, and from the dates it is possible to tell that he has been involved in the IT field for many decades, and (in comparison with many IT personnel today) from its infancy. He has also, whilst at Charteris, developed and deployed software for data integration and migration. He described himself as having done databases “since the year dot” at one point. He describes himself in his CV as having strong practical experience of information technology and its business applications. He has several years’ line management experience, has successfully managed large IT projects, and acts as a trouble-shooter of major integration projects and consultant in information strategy. He has particular expertise in XML, in large-scale information architectures and tools, data integration and data migration, scientific data processing, and in system performance. He explained that as he is trained as a scientist and engineer, he has used mathematics all his life and part of that mathematics is probability theory and statistics. Given the subject of his Ph.D, it will be no surprise that he uses mathematics extensively. I doubt that there are any theoretical particle physicists who are not extremely skilled and expert at mathematics at a very high level indeed. He therefore has great experience in statistics, and at one point in his cross-examination was described as a statistician, something with which he did not disagree. He does not however, as he put it, have any formal qualification in statistics.
804. He is also a highly experienced expert witness, as demonstrated not simply from the BSkyB case I have referred to above but many others, and he has given expert evidence in different cases in the Technology and Construction Court, Chancery Division and Queen’s Bench Division too. He served two detailed reports with numerous appendices (adding schedules of corrections in the usual way in supplementary evidence in chief), and he also took part in the joint experts’ exercises and signed the four Joint Statements with Mr Coyne. He also served a third report directly upon the court, but no application to adduce this as part of his evidence in chief was made by the Post Office and it did not form part of his evidence.
805. In the instant case, he was cross-examined for three days. One of his corrections in chief was in respect of paragraph 761 of his 1st report where he had stated the following in respect of his summary of what was called a statistical analysis of the likelihood of the bugs that had been found by the experts being the cause of the claimants’ shortfalls and discrepancies.
“759. In section 8.5, 'Scaling of Financial Impact of Bugs', I gave what I think is the simplest analysis of why, in quantitative terms, bugs in Horizon cannot have accounted for a large part of the Claimants' shortfalls. I summarise it here.
760. Because Post Office has had an average of 13,560 branches over the lifetime of Horizon, the total number of monthly branch accounts has been about 3 million.
761. Therefore, if a bug like the Suspense Account bug has occurred 16 times in the lifetime of Horizon, the chance of it having occurred in any given branch in any given month is about 16 in 3 million. Because the Claimants tended to have smaller branches than the average, doing fewer monthly transactions (by a factor 0.37), the chances of the bug occurring in a Claimant's branch would be about 2 in 10 million.
762. I have considered a bug similar to the suspense account bug, which occurred about 10 times, and had a mean financial impact of about £1000 per occurrence. How many similar bugs would be needed, to give a one in ten chance of one such bug occurring, with an impact of £1000, on a particular Claimant's branch in a particular month?
763. The answer, given by elementary arithmetic which I describe in section 8.5, is that there would need to be 50,000 of these distinct bugs. If there were fewer than 50,000 similar bugs, if any Claimant were to assert that in a given month a shortfall of £1000 in his accounts was caused by bugs in Horizon, then the chances of his assertion being correct are less than one in ten.
764. So the Claimants cannot credibly assert that their shortfalls were caused by bugs in Horizon, unless there were something of the order of 50,000 such bugs.
765. Only three such bugs have been found. My own search of KELs has found only 8 other possible bugs. Mr Coyne's examination of over 5000 KELs has found no other bugs which definitely caused shortfalls.
766. Thus the Claimants' case requires 50,000 bugs in Horizon - but only a handful have been found by the experts. Neither expert can quantitatively support the Claimants' case.”
(emphasis added)
806. His correction was that in respect of paragraph 761 the sentence that reads "the chances of the bug occurring in a Claimant's branch would be about 2 in 10 million" should be "the chances of the bug occurring in a Claimant's branch would be about 2 in a million". He said that this was a typographical error. When he was asked if he had spotted any other errors, he said:
“A. No, I didn't because there were quite a few statements of this "in a million" nature in my report. I made careful checks of the main thread that led to my key result in section 8.7, and that was my focus. And similarly on the transaction corrections I had made checks to that spreadsheet and, you know, I included the spreadsheets with the reports so that you can see how the arithmetic is done from the spreadsheets.
Q. And some of it you point out is basic arithmetic?
A. I think all of it is basic arithmetic.”
807. He said, in relation to a question as to whether Mr Emery, his assistant, had “checked the maths” that “No, he didn't actually. The maths is there in the spreadsheets for anybody to check. It is not advanced statistics, it is basic multiplication and division, and it is laid out in the spreadsheets.”
(emphasis added)
808. His methodology in terms of his statistical analysis was explained by him in section 8.5 of his 1st report. This stated:
“617. I shall assess the financial impact of bugs in Horizon over three different scopes:
617.1. (a) Across all Post Office branches, during the lifetime of Horizon.
617.2. (b) Across all Claimant branches, while they held them.
617.3. (c) On a single Claimant branch in a single month.
618. It is possible to relate the financial impacts on these scopes, by numerical scaling factors. I calculate those scaling factors in this sub-section.
619. Over the period 2000 – 2018, (i.e. 19 years) the Post Office network has consisted of more than 11,000 branches. The mean number of branches in all years over the period has been about 13,560. This figure is derived from the spreadsheet referred to at paragraph 178 of Ms Van Den Bogerd's Second Witness Statement, assuming that the spreadsheet is accepted. If this evidence is accepted, the number of 'branch months' (a single branch, trading for a single month) has been 13,560 * 12 * 19 = 3,091,680. This is the number of monthly branch accounts that have been produced.
620. This means that for a typical Post Office branch, the scaling factor between scope (a) and scope (c) for the impact of bugs in Horizon is a factor of 3 million.
621. For Claimants' branches, rather than typical branches, the scaling factor of approximately 3 million may need to be adjusted for two possible effects:
621.1. It might be asserted that Claimants' branches are more or less likely than other branches to be hit by bugs in Horizon, because of some special property of Claimants.
621.2. Claimants' branches may, on average, be smaller or larger than typical branches across Post Office network. If they are smaller, they handle fewer transactions in a month, and so are less prone to Horizon bugs in those transactions.
622. It seems implausible to me that there is some special factor about Claimants' branches, which makes them much more prone to bugs in Horizon - bugs which one would expect to strike any branch at random. Nevertheless, I have considered the possibility carefully in Appendix F. I have shown there that there is no significant difference between Claimants' branches and other branches, in proneness to bugs in Horizon.
623. It appears, from the spreadsheet attached at paragraph 179 of Ms Van Den Bogerd's Second Witness Statement, that the Claimants' branches are, in terms of customer transactions carried out per day, smaller than the average across the whole Post Office branch network.
624. If this spreadsheet is accepted, it implies the following about Claimants' branches:
624.1. From summing rows of the spreadsheet, the 561 Claimants' branches carried out 558,000 customer transactions per week in 2007.
624.2. This is 558,000/6 = 93,000 transactions per day, assuming a Post Office branch is open for 6 days a week.
624.3. Across 561 Claimant branches, this is an average of 93,000/561 = 165 customer transactions per branch per day.
625. For comparison with this figure, I need to estimate the average size of branches across Post Office Network. I have done this using two pieces of evidence. Ms Van Den Bogerd's Second Witness Statement says that, across the whole Post Office network, there are approximately 48 million customer transactions per week, or 8 million per day in 2017 (assuming again that branches are open for 6 days in the week).
[626. I have omitted the table of EPOSS Transaction Rates, which are then
summarised in 627]
627. The best estimate I can make of the average daily volume in 2003 from this table is to divide the peak month by 26 working days. This gives an estimate of approximately 4 million transactions per day in 2003, compared with 8 million transaction per day in 2017.
628. I therefore estimate that the average volume of transactions over the period 2000-2018 has been approximately 6 million transactions per day, mid-way between the 2003 figure and the 2017 figure (assuming Mrs Van Den Bogerd's evidence is accepted. I note that Mr Coyne quotes the same figure from her).
629. To estimate how much smaller the typical Claimant's branch is than the average Post Office branch:
629.1. The average transaction rate for all Post Office branches is 6,000,000/13,650 = 439 customer transactions per branch per day.
629.2. So, in terms of customer transactions per day, the typical Claimant branch was smaller than the average Post Office branch by a factor 165/439 = 0.37.
630. So, Claimants' branches, being generally smaller than Post Office average, have fewer transactions per month and so are less likely to be hit by a Horizon bug in a given month. (The calculation of the factor 0.37 contains some minor approximations that can be improved with further effort. I intend to do so in my supplemental report.) The factor 0.37 increases the scaling factor above, between scopes (a) (see paragraph 617.1) and (c) (617.3) from about 3 million to about 8 million.
631. I illustrate what the factor of 8 million means using a hypothetical example of a bug which has occurred 16 times over the lifetime of Horizon, with mean financial impact on these occasions of £1000. Call this Bug A. The financial impact of Bug A is similar to that of the Suspense Account bug.
632. If I selected a Claimant's branch and a month at random, then the chances of Bug A occurring at that branch in that month are only 16 in 8 million, or 2 in a million - an extremely small probability.
633. Different types of bug occur independently of one another, so their probabilities of occurring are additive. If there were a second bug similar to the hypothetical bug above - call it Bug B - then the chances of either Bug A or Bug B happening to one branch on one month are twice the previous figure - 32 parts in 8 million.
634. If there were 100 similar bugs - called Bug A, Bug B, Bug C,... Bug Zz - the chances of any one of them happening to one branch on one month are still only 100 * 16 parts in 8 million, or one part in 5,000. This is still a very small probability.
635. It then follows that in order for one occurrence of a bug, of similar financial impact to the Suspense Account bug, to have even a one-in-ten chance of occurring to one branch on one month, there would need to be 50,000 such distinct bugs - because 50,000 * 16 / 8,000,000 = 800,000/8,000,000 = 1/10. There would have to be a Bug A, Bug B, Bug C, and so on, in a list with 50,000 distinct bugs.
636. Even if there were some 'super-bug' - with financial impact ten times larger than the suspense account bug - there would have to be approximately 5,000 such super-bugs to give a one in ten chance of affecting a Claimant's branch accounts in a given month. There is no evidence for even one such super-bug - let alone for 5,000 different ones.”
[637 and 638 then provided a further table and calculation that I omit]
639. The Claimants have never asserted that there are as many as 50,000 distinct bugs in Horizon, with each bug on the same scale of financial impact as the Suspense Account bug. Their case rests on two or three known bugs of this scale, and on the unproven assertion that there may be others.
640. If any Claimant were to assert that, for instance, a deficit of £1000 had occurred in his branch in a particular month, caused by a bug similar to the Suspense Account bug, the chances of that assertion being correct are extremely small, because Horizon bugs strike so rarely - unless Horizon contained of the order of 50,000 distinct bugs of that kind (and even then, the chances are only one in ten).
641. Mr Coyne has examined more than 5,000 KELs, and not found definite evidence for even one bug with impact similar to the Suspense Account bug - let alone 50,000 of them.
642. The implication of this result - the very small probability of any error in one months' accounts from a bug in Horizon - is that the accounts for any branch on any month are overwhelmingly likely to be correct (apart from effects such as delayed TCs, which are corrected after a variable delay).
643. In my experience, no commercial IT system could ever go live with as many as 50,000 serious bugs - and certainly could not have the good in-service record over 18 years that Horizon has had.”
(emphasis added)
809. I have reproduced extensive passages of Dr Worden’s explanation of his own methodology in order to allow his explanation to be available to any reader of this judgment. I consider this a wholly flawed methodology, and obviously so, and I reject it.
810. There is another aspect to this which needs addressing at this point. It needs some context in order properly to be understood. Sums recovered from SPMs for discrepancies in branch accounts would be put into something called the suspense account. The claimants pleaded in paragraph 38 of the Generic Particulars of Claim the following:
“38. The Defendant operated one or more suspense accounts in which it held unattributed surpluses including those generated from branch accounts. After a period of 3 years, such unattributed surpluses were credited to the Defendant's profits and reflected in its profit and loss accounts.
39. The Defendant thereby stood to benefit and/or did benefit from apparent shortfalls wrongly attributed to the Claimants which did not represent real losses to the Defendant.”
811. This was pleaded to by the Post Office in the Defence as follows:
“73. As to paragraph 38:
(1) Post Office does not have a clear understanding of what the Claimants mean by the term "suspense account" or by the phrase "unattributed surpluses including those generated from branch accounts".
(2) In relation to its dealing with its clients, Post Office is almost always able to reconcile its figures to those of its clients. However, as is inevitable given the nature and scale of those dealings, occasionally Post Office is not able to do so. For example:
(a) A client, such as a bank, makes a payment to Post Office in respect of a particular transaction undertaken in a branch but the payment exceeds the amount that Post Office considers to be due.
(b) Post Office seeks but does not reach agreement with the client as to the amount due and the client does not accept repayment of what Post Office considers to be the overpayment. In parallel, Post Office also seeks to determine whether the overpayment should be credited to a branch or to any other relevant part of its business.
(c) Where the proper beneficiary of a credit cannot be determined, Post Office temporarily records the overpayment in its accounts. If the overpayment is not resolved within 3 years, the overpayment may be credited to Post Office's profit and loss account.
(3) The existence of processes for recording and resolving credits of the type described above are an ordinary business practice.”
812. This was then pleaded to by the claimants in their Reply and Defence to Counterclaim:
“29. The Defendant's professed lack of clear understanding of the term 'suspense account' at paragraph 73(1), is noted at paragraph 2.1 above, and its response to the Claimant's RFI is similarly evasive.
30. The effect of the Defendant's example pleaded at paragraph 73(2), is that the Defendant admits the possibility that:
30.1. a sum of money may be received which the Defendant believes to be an 'overpayment' from a third party client (e.g. a bank) in respect of a transaction undertaken in a branch: paragraph 73(a);
30.2. the Defendant would seek to determine whether the overpayment should be credited to a branch, or to any other relevant part of its business: paragraph 73(b);
30.3 where the Defendant does not identify ‘the proper beneficiary’ of the credit, Post Office ‘temporarily’ records the overpayment in its accounts and credits that to its profits, if not resolved within 3 years.
31. This logic would apply where the Defendant’s internal records were wrong due to a Horizon error or in the case of any failure to identify the relevant branch or Subpostmaster”.
813. I find that there is no sensible basis for the professed lack of understanding by the Post Office in its Defence of the term “suspense account”. That term is widely used by witnesses for both sides in the evidence, and I will give two simple examples showing its use. One of the Post Office’s own witnesses, Mr Johnson used it in his evidence, in a passage quoted at [279] above. Further, in an internal document which was prepared after the Callendar Square bug was discovered, it was widely used. An extract from that document appears at [404] above. Not only is the term used in four separate places in that document, there is even reference to the Suspense Account Team. The notion that the Post Office did not understand the term used in the Particulars of Claim is wholly fanciful. Further, there is an agreed bug (number 3 in the Bug Table) which the experts call the Suspense Account bug.
814. Putting to one side the fact that the Post Office professed not to know what the term suspense account meant, notwithstanding its use in internal documents, the Defence makes it clear that unattributed sums within it, the correct destination for which the Post Office could not work out, were credited to its own profit and loss account. I accept the logic explained in the Reply in respect of the effect of this.
815. Dr Worden had wholly failed to consider this issue. The point was put to him, not by reference to the pleadings, but by reference to another document which he admitted he had read, namely a report by Second Sight. That report had stated what is basically the same point but in slightly different terms, namely as follows:
“2.11 We note that Post Office's control and reconciliation procedures rely on correct information being supplied by third party clients. It follows that, if incorrect information is provided by any client company, this can give rise to a loss being charged to a branch. We also note that, for most of the past five years, substantial credits have been made to Post Office’s Profit and Loss Account as a result of unreconciled balances held by Post Office in its Suspense Account.”
816. The question and answer are as follows, having read out the passage:
“Now I'm not asking you about that as evidence of itself, I'm just asking were you aware that that was a concern that Second Sight had raised about substantial credits being made to Post Office's profit and loss account as a result of unreconciled balances held by Post Office in its suspense accounts?
A. I read the Second Sight reports and the Post Office's reply.
Q. But did you spot that?
A. I didn't spot that, no.”
817. This point was put in the context of evidence Dr Worden had given, namely the effect of small bugs or what he called “micro bugs” being unlikely to have an effect.
“Q. Okay. Well, what about -- because elsewhere in your report, I don't want to stray too much at the moment, but you also mention micro bugs, small bugs, being unlikely because you would expect to see large sums of money somewhere --
A. Money that disappears somewhere has to pop up --
Q. It's got to come back somewhere else, it's got to pop up somewhere else.”
818. The above passage was then put to him. Dr Worden’s failure to spot what is, in my judgment, a point directly relevant to the likelihood of Horizon bugs, errors and defects having a potential impact upon branch accounts is a notable one. Whether the point is taken from the pleadings, or from the Second Sight report (which Dr Worden accepted he had read, together with the Post Office’s reply to that document) does not much matter, in my judgment. I consider that this one omission alone - which I consider to be a serious omission on his part – is sufficiently fundamental as to undermine his evidence on this issue on its own.
819. The following points are relevant in respect of Dr Worden’s statistical exercise:
1. Statistics is the branch of mathematics that concerns the collection, presentation and analysis of data. There was no permission granted by the court for expert statistical evidence in this case. The permission that was granted was for expert IT evidence.
2. Had either party sought permission to rely upon evidence from an expert statistician, I would not have granted it. Statistical analysis does not assist in resolution of the Horizon Issues, which required (in my judgment) detailed technical analysis and consideration by IT experts of the detailed occurrences of what could, potentially, be bugs, errors and defects. This was so that their expert opinion could be brought to bear, firstly, upon the important issue of whether there were any, and also if there were, their extent and the degree to which they potentially could impact branch accounts. This both experts did in any event, and that evidence – based on contemporaneous records and evidence about their occurrence - is of far greater assistance than this wholly theoretical argument about why Horizon bugs cannot have affected the claimants’ branch accounts.
3. Dr Worden uses statistics, as he uses mathematics, as part of his professional life. Any particle physicist would, and does, use mathematics every day. Dr Worden accepted that this was not advanced statistics, and he was right to do so. It is a combination of simple assumption together with basic multiplication and division. The assumptions are not, in my judgment, valid.
4. None of the Horizon Issues required the experts to perform a mathematical or statistical analysis of the likelihood of the claimants’ alleged shortfalls and discrepancies being caused by bugs, errors and defects in Horizon. Horizon Issue 3 – using as it does the term “extremely unlikely to be the cause of shortfalls in branches” – does make this exercise by Dr Worden relevant if that was how he chose to address that issue. That does not however make it either correct, or of any assistance.
5. The accepted correction to the figure of “2 in 10 million” – which he accepted before his cross-examination started should be “2 in a million” – shows, in my judgment, how very broad brush this exercise was. As he put it in his oral evidence, “…there were quite a few statements of this "in a million" nature in my report”.
6. It is plain that Dr Worden’s own evidence accepts that there are two fundamental assumptions that are central to this exercise. The first is that any bugs in Horizon strike branches randomly (which is spelt out in his report). The second is that the claimants’ branches are a random sample. This can be seen, implicitly, from his statement at paragraph 621.1 that “it might be asserted that Claimants' branches are more or less likely than other branches to be hit by bugs in Horizon, because of some special property of Claimants” and also at paragraph 632 “if I selected a Claimant's branch and a month at random….” Neither of these assumptions are valid, and the reasons that they are not valid are explained in this judgment.
7. His paragraph 636 makes it clear how fundamentally unsound this exercise it. He stated “Even if there were some 'super-bug' - with financial impact ten times larger than the suspense account bug - there would have to be approximately 5,000 such super-bugs to give a one in ten chance of affecting a Claimant's branch accounts in a given month.” However, the evidence shows, quite specifically, that even for bugs such as Dalmellington and Callendar Square, SPMs’ branch accounts were specifically affected, and some by quite significant amounts. This therefore demonstrates that some SPMs as individuals could, and did, have their branch accounts affected by bugs. If the evidence does not match a theory (which here it plainly does not) then that must mean that the theory is flawed.
8. Of his four different ways of addressing financial impact, set out in his paragraph 614, the second assesses the net impact of all bugs referred to in KELs. However, KELs do not invariably recite known financial impact. In some, known financial impact is recited, but a conclusion in many KELs is that user error cannot be ruled out and therefore a conclusion is reached that a bug is not responsible. I consider that Fujitsu would regularly record user error simply as a “catch all” and, effectively, require someone to rule it out to Fujitsu’s satisfaction, and unless this was done Fujitsu would remain wedded to that as an explanation. Given the technical investigations were being done by Fujitsu itself, Fujitsu rarely (if ever) ruled our user error by the SPM. This was wholly in Fujitsu’s interests.
9. Dr Worden’s reasoning is entirely circular. By layering assumption upon assumption (and in places routinely misdescribing these assumptions as estimates) he felt able to conclude that there would have to be as many as 50,000 serious bugs for the claimants to be correct (as set out in his paragraph 643) and that “no commercial IT system could ever go live with as many as 50,000 serious bugs - and certainly could not have the good in-service record over 18 years that Horizon has had”. The whole rationale of this litigation is to work out, more or less, exactly what the “in-service record” of Horizon in fact is. I do not consider reliance on an assertion that it has had “a good in-service record over 18 years” can be used to underpin an exercise, the whole purpose of which is to consider this very issue. I reject his exercise that led to his figure of 50,000 bugs – it is wholly flawed. I also reject the circularity of his reasoning, which lacks logic.
10. Dr Worden also implicitly assumes that all known bugs have been both reported and discovered, as he uses this (the total number of occurrences of the Suspense Account bug) in his table at paragraph 637. It is his label F and the source is “evidence on the Suspense Account bug”. Mr Coyne estimated that there may be as many as 40, based on an extrapolation of the number of KELs and PEAKs inspected.
11. Dr Worden also misidentified the approach of the claimants, or certainly the approach of Mr Coyne. He stated “Their [ie the claimants’] case rests on two or three known bugs of this scale, and on the unproven assertion that there may be others.” That is not correct. The approach of Mr Coyne has been to consider PEAKs and KELs; see what is reported contemporaneously and what the conclusions were (or appeared to be) within Fujitsu; and consider what that contemporaneous account shows. Mr Coyne then uses that to draw conclusions. A great many of the bugs in the Bug Table use contemporaneous entries by Fujitsu which records bugs, errors and defects, as part of the consideration of whether there were such bugs, errors and defects. This is not mere assertion.
12. However and in any event, by the end of the Horizon Issues trial, the Post Office admitted the existence of far more bugs than the “two or three known bugs” that were referred to by Dr Worden in the preceding sub-paragraph.
13. Finally, I consider that Dr Worden’s statistical exercise is nothing more than a superficial short cut which avoids a detailed technical analysis of the history of the Horizon system in the way undertaken by Mr Coyne. In this case, on these issues, with all the detailed evidence available, it can be shown to be an unjustified way to resolve any of the Horizon Issues.
820. Dr Worden also assumed that bugs in Horizon strike branches randomly. This too is not borne out by the evidence. For example, the Dalmellington bug (bug 4 in the Bug Table) appears to have affected only outreach branches. Relatively few of the Post Office’s branches are outreach branches. Bug 14 in the Bug Table, which Mr Coyne said was a bug but which Dr Worden described as “appears to be a system error with impact upon branch accounts” is entitled in the bug table “bureau discrepancies” and relates to foreign exchange transactions. Dr Worden’s exercise takes no account of what is, effectively, an admitted bug, and one that has an effect only upon foreign exchange transactions. If a branch conducted very few, if any, of these, it would not be affected by this bug. The chances of a branch being impacted by the bureau discrepancies bug depends upon whether it carries out foreign exchange transactions, and if so how many. That is not a bug striking all branches randomly. Indeed, rather tellingly for Dr Worden’s exercise, he assumes that any such bug would strike randomly across the whole branch network and have an equivalent random effect upon the branch accounts of branches that had performed foreign exchange transactions, and those that had not. This is plainly invalid.
823. The exercise also wholly ignores what branch accounts are, how they are supposed to be created from the full list of transactions carried out in that particular branch in that particular period, and how precise they are required to be. The analysis of the “lasting and transient impact” approach taken by Dr Worden has already identified that this ignores the effect on branch accounts of the impact of a bug if no transaction correction has been issued before the end of the next trading period.
824. There is also another fundamental problem with Dr Worden’s exercise, which is that it entirely ignores the specific purpose of Horizon, at least so far as the contractual relationship between the Post Office and SPMs is concerned. The Horizon system is, whether under Legacy Horizon or Horizon Online, the method by which the Branch Trading Statement or BTS is produced for an individual SPM’s branch Post Office. The BTS is the accounting mechanism under either the SPMC or NTC (the two different types of contracts used by the Post Office for SPMs during the period from 2000 onwards, explained in detail in Judgment (No.3) on the Common Issues). It was the common intention of the parties that the Horizon system would be used for the creation and recording of accurate information regarding all transactions performed in the branch which would be used for the production of the Branch Trading Statement for that branch. This means that at the end of each branch trading period, the BTS will (or should) provide a statement of the branch activity, which includes whether the accounts are in balance (which is where the cash and stock in the branch matches the Horizon system’s records of what should be there) or whether there is a discrepancy. If there is a discrepancy and it is a shortfall, the contractual terms of the SPMC and the NTC require the SPM to “make good” the shortfall, which means pay the money to the Post Office. The other alternative for an SPM is to “settle centrally”, which means seek time to pay. If the Post Office agrees, the shortfall can be paid over 12 months by deduction from the SPM’s remuneration. This purpose of the Horizon system is entirely ignored in Dr Worden’s statistical exercise across the whole branch network.
825. Consider a hypothetical bug, bug X. Also consider that bug X impacts upon branch accounts in a single branch upon a single occasion leading to a shortfall in the branch for that branch trading period. That shortfall would be treated as a debt by the Post Office and the SPM either has to pay, seek time to pay, or may even end up in a dispute with the Post Office and/or be suspended as a SPM and not pay. Analysis and resolution of the correct and true situation of the branch accounts between the Post Office and the SPM for the trading period in question does not depend upon whether, in all the other millions of branch accounts, there was no such incidence of bug X. The correct analytical approach in my judgment is to consider the branch activity for that branch for that period; consider the evidence both for and against (1) the existence of bug X and (2) the likely cause of the discrepancy, bearing in mind both the burden and standard of proof; make findings on the cause of the discrepancy; and then apply those findings. Expert IT evidence of most assistance in that exercise would be whether or not bug X exists or existed, and what were its effects. It is of no assistance to have an exercise that in effect says the statistical likelihood of any bug having an impact upon the branch accounts of that branch in that period is very low.
826. The section 8 analysis is, in my judgment, so riddled with plainly insupportable assumptions as to make it of no evidential value. It is the mathematical or arithmetic equivalent of stating that, given there are 3 million sets of branch accounts, and given there are so many sets of branch accounts of which no complaint is made, the Horizon system is mostly right, most of the time. It is a little more sophisticated than that, but not by very much.
827. The way that this can be further tested against the evidence is as follows. Mr Godeseth accepted that the Dalmellington bug had probably been present in the system from 2010 until it was discovered in 2015. Dr Worden’s analysis would lead to the conclusion that it was overwhelmingly unlikely that it had affected branch accounts during that period. But all the evidence is that this bug did affect branch accounts. It is correct that the Post Office relies upon the correction of known and identified discrepancies by means of the later issuing of TCs – a human measure which in my judgment is outside the definition of Horizon - but even putting that to one side, during this 5 year period branch accounts were actually affected. The existence of the bug was not known about for 5 years before it was even discovered. This round of the group litigation is to make findings on the Horizon system and whether there are any, or many, or no, other such bugs, errors and defects in the Horizon system similar to this one, or with similar detrimental effects. The conclusion of Dr Worden’s section 8 exercise is that it was overwhelmingly unlikely that any of the claimants’ branch accounts were affected by any Horizon bugs. This must include a conclusion that the Dalmellington bug cannot have affected any of the claimants’ accounts. This means that he must assume that Fujitsu were 100% successful in tracking down all incidents of the Dalmellington bug, and that any claimant SPMs who report symptoms the same as the Dalmellington bug and who had an outreach branch are either wrong, or making it up. These assumptions are ones that cannot sensibly be made, in my judgment.
828. The whole section 8 exercise effectively is entirely theoretical, and is made in a vacuum that must ignore real world evidence to the contrary from one of the Post Office’s own witnesses, Mr Godeseth, as set out at [445] to [450] above. Even if I am wrong about all of the above, and even if a statistical analysis of this nature is valid, then in my judgment this would not assist the Post Office in its case on the Horizon Issues in any event. There are agreed to have been over 3 million sets of branch accounts over the life of Horizon, and approximately 6 million transactions every day across the whole Post Office branch network. There are fewer than 600 individual claimants, with the total value of accounting discrepancies over a 17 year period, amounting to only £18 million. As a proportion of the value of the Post Office network business over that period, and as a proportion of the total number of transactions, the number of transactions which underpin the claimants’ case is exceedingly small in percentage terms.
829. Dr Worden’s probability calculation regarding two bugs, Bug A and Bug B, striking independently, was expressed as follows:
“632. If I selected a Claimant's branch and a month at random, then the chances of Bug A occurring at that branch in that month are only 16 in 8 million, or 2 in a million - an extremely small probability.
633. Different types of bug occur independently of one another, so their probabilities of occurring are additive. If there were a second bug similar to the hypothetical bug above - call it Bug B - then the chances of either Bug A or Bug B happening to one branch on one month are twice the previous figure - 32 parts in 8 million.”
(emphasis added)
830. He described the probability of these two unconnected events both occurring, each with a probability of 16 in 8 million (or 2:106), as being additive and he added 16 in 8 million, to 16 in 8 million, to arrive at their sum, 32 in 8 million. In terms of basic probability theory, the chances of two independent events occurring is usually seen as the product of the two probabilities, not their addition. This was recognised, as a SPM would not need both bugs to strike in order to experience an impact on branch accounts, only one would be sufficient. The principle of additivity is a fundamental principle that if the event whose probability is sought can be represented as the union of several other events that have no outcomes in common, then the probability of the union is the sum of the probabilities of the individual events making up the union. However, regardless of this, the point was neither put nor explored in any detail, and the outcome in any event depends upon how the events, and the union, are defined. The situation is further complicated by the fact that depending upon the nature of how “the bug” was defined, to impact upon branch accounts, this could be either Bug A or Bug B (in Dr Worden’s example) independently, or in conjunction. The PEAKs show that sometimes branch accounts would be impacted by a combination of rare circumstances. Such bugs may, but would not necessarily, have to occur at the same time. If independently able to impact branch accounts, they would simply have to occur at any time during the 4 week branch trading period. I refer to this matter for three reasons. Firstly, to demonstrate that probability theory only takes one so far in this case, and that is not very far; secondly, to explain that depending upon how one defines “event” and the intersection (or joint probability) of both or either occurrence, one can arrive at different results; and thirdly, to demonstrate that there were numerous flaws and/or difficulties with Dr Worden’s approach, not all of which were put, but which in my judgment undermine the validity of his exercise. This point about the probability calculation of two bugs striking a branch is not, however, not the main reason (or indeed, any reason) why I have rejected his analysis. The probability calculation for two bugs occurring in one month is not a central element of the statistical exercise he performed; only one bug would be needed in any one branch trading period for it to affect those monthly branch accounts. I have explained this in order to illustrate that it depends upon how one defines the impact of a bug, and chooses to assess its probability, that one can end up with an answer that rather suits the way the question is initially posed.
831. Dr Worden accepted so far as his evidence generally was concerned, that he had assumed that the Fujitsu evidence was true, and in particular that of Mr Jenkins (whom he had met, although Mr Jenkins was not listed as a source of information in his 1st expert report, and from whom no witness statement was available) and that “generally, if findings -- if things happen in oral evidence which go against witness statements then I would need to come back and say what's the impact.”
832. This therefore means that his evidence is of limited assistance in any event. I have found that the evidence of Mr Godeseth was somewhat different, at the end of his cross-examination, than how it had been presented in his three witness statements. Litigation cannot go on endlessly. For the Horizon Issues trial, evidence of fact and expert evidence were both ordered, and findings can only be made by the court in its judgments that resolve the disputes of fact. Disputes of fact in the evidence for the Horizon Issues will be resolved in the Horizon Issues judgment, namely this one. This is entirely conventional. If an expert chooses to prepare his or her evidence on the basis that one side’s evidence is true – which Dr Worden did in his reports, in respect of the party instructing him, namely the Post Office – they are not entitled to serve further expert evidence and wholly amend their opinions if that evidence is not accepted by the court. Dr Worden must have known this. There was going to be no opportunity for him “to come back and say what’s the impact”.
833. This was put to him in his cross-examination.
“Q. The content of this report in this respect, we will come to other examples later, focuses on what the consequences would be if the defendant's factual evidence is accepted?
A. Generally, if findings -- if things happen in oral evidence which go against witness statements then I would need to come back and say what's the impact.
Q. Yes. You understand, don't you, that the moment at which everyone finds out whether factual evidence is accepted or not is when the judgment is handed down?
A. Absolutely, yes.
Q. So you are proposing to revise your opinions after that, are you?
A. No. That's a good point. What I was saying there is if it becomes evident, all I can do is make assumptions. The court will find findings and all I can do is make assumptions and drive them through my opinions and try and assist the court that way. So all of this is assumptions I have made based on the evidence I have seen, and the court may find differently.
Q. Can I pause there. You were assuming there that Mr Godeseth's version was true?
A. I was assuming mainly Mr Jenkins' written analysis which Mr Godeseth's evidence confirmed.”
(emphasis added)
834. The “evidence I have seen” to which Dr Worden referred was the Post Office’s evidence, and what Mr Jenkins had said (which was not evidence in any event). He did not take any account of the claimants’ evidence. This was notwithstanding that in his expert’s declaration he had expressly stated “I have not assumed that any particular version of events is true…..” when, indeed, he plainly had. It was also based on Mr Godeseth’s written evidence, not on his oral evidence, which was markedly different.
835. He also said further on this same subject:
“A. Well, my analysis of the claimants' evidence is mainly contained in my supplemental report, and I explained there that individual claimants' evidence, particularly individual subpostmaster evidence, I did not feel able to make strong use of that and I gave the reasons in my supplemental report. So my opinions have little dependence on that. And the core of my opinions, the numerical estimates I make, those estimates have been designed -- or the process and the method has been designed so that if -- so that to make my assumptions evident where those assumptions come in, and if the court finds something different from my assumptions the court can go to the spreadsheets and re-do the method for itself.”
(emphasis added)
836. This can therefore clearly be seen as his accepting the Post Office/Fujitsu evidence as true, particularly that of Mr Jenkins (not even a witness, and in respect of whom no witness statement was served); and also failing to base his opinions upon, or take account of, any of the individual evidence of claimant SPMs served by the claimants.
837. Dr Worden is also under a misapprehension if he considers that the court’s function is to “re-do the method for itself” if his assumptions were not made out in the factual evidence. Firstly, he could have provided alternative expert evidence and opinion, were the written evidence of Mr Godeseth and the information coming from Mr Jenkins to be rejected. Secondly, he should have taken the claimants’ evidence into account, because there is no basis for an assumption that it was not relevant to the Horizon Issues which his expert evidence was the subject of. It is entirely normal for experts to provide alternatives of their opinion, depending upon which party’s evidence is accepted. Thirdly, the “method”, such as it is, demonstrated in the spreadsheets simply cannot be “re-done” to take account of findings that the assumptions were incorrect. The invalid and unjustified assumptions lead, in my judgment, to the inescapable conclusion that the whole methodology is flawed. No amount of “re-doing” of the calculations by the court can remedy that.
838. Dr Worden also accepted that although he had missed some days of the trial, he was “pretty sure” that he had heard all of the Post Office’s evidence; he was then asked the following:
“Q. Have you had any changes of heart about anything in your report having heard it?
A. Changes of heart?
Q. Anything you want to change about what you said?
A. Well, had there been some fundamental change I would have felt obliged to communicate it to the court.
Q. Of course, because you take your duty very seriously?
A. Yes, I do. I can't think of any major change of heart, but there may well be things that you bring me to and I say there's an adjustment here. But to come back to the point about precision, I am conscious of how precise my numerical results have to be in order to be of assistance to the court.”
839. This effectively must mean that Dr Worden did not consider that any of the changes, in the Fujitsu evidence in particular from the cross-examination, would impact upon his expert opinion. This is notwithstanding that he had assumed that the Post Office’s evidence would all be accepted, and anyone present for Mr Godeseth’s cross-examination can only have concluded that the factual position presented by Fujitsu in the written witness statements was very far from an accurate one. Indeed, the extent to which the cross-examination did not have any effect – either appreciable or otherwise – on Dr Worden’s expert opinions shows how very theoretical the whole exercise was.
840. Finally on this point, due to the disruption caused by the recusal application, and the unavoidable re-arrangement of the trial timetable, the experts came to give their evidence in June 2019, and all of the factual witnesses who were cross-examined (with the sole exception of Mr Parker) had completed their evidence by 21 March 2019, many weeks earlier. Even Mr Parker had completed his evidence by 11 April 2019. Dr Worden was called on 11 June 2019, two months after this. It is unusual for there to be such an interval in a trial, but circumstances in this case led to extra time being available. There was more than ample time for Dr Worden to have identified how, if at all, the Post Office’s evidence of fact that emerged earlier in the trial (in particular from Mr Godeseth, but also from Mr Parker too) impacted upon his opinions. It plainly did not.
841. I will provide only one example of Dr Worden’s unsatisfactory approach to the evidence, although there were many. In terms of whether transactions could be inserted at the counter in Legacy Horizon, what is called “remote access”, Dr Worden had in his first report concluded that Mr Roll (a witness of fact) was factually wrong. He expressly stated that he, Dr Worden, had “established” that Mr Roll was wrong. He accepted that use of the word was wrong but what he had done in his 2nd report was equally telling, in my judgment. This was after those witnesses who were to be called for the Post Office, still employed at Fujtisu, had corrected (or partially corrected) their erroneous first witness statements, and accepted that remote access without the knowledge of the SPM was possible. He said (at paragraph 83) that he required:
“further factual information before I can comment on this evidence. Which 'specific person'? Under what circumstances? How frequently? Until I have that information, it remains possible in my view that any transaction which 'would appear to the subpostmaster as though it had been carried out through the counter in branch' might only be a transaction that he had given his consent for, as the 'specific person' - and which had in effect been made on his behalf.”
842. Dr Worden also then stated:
“84. Therefore, Mr Roll's new evidence does not cause me to alter the opinion expressed at paragraph 1119 of my first report, when commenting on Mr Roll's first witness statement, that he could not alter branch accounts without the Subpostmaster knowing.”
“Q. Yes, the one that we agreed was incorrect in approach and conclusion.
A. We agree now, but at the time I wrote the second report, on the basis of Mr Roll's evidence and Mr Parker's evidence, I didn't see reason to change that opinion.
Q. I would suggest to you that you were inexplicably reticent to accept something that was contrary to Post Office's interests.
A. No, I was reticent. Not inexplicable, I was reticent because I had not seen sufficient evidence to convince me that these things could be done without the knowledge of the subpostmaster.
Q. But Dr Worden, would you accept that the approach you have taken there contrasts very, very strikingly in how you approach Mr Roll's evidence with the approach you took at 1119 in your first report when you accepted, effectively, Mr Godeseth's evidence?
A. I have accepted that my approach at 1119, that use of the word "established" was wrong and my approach was wrong, and we have established that -- you know we have done that before the interval –
Q. My question is: do you accept the contrast is very striking?
A. I think the court will have to -- I accept that my attempt to make my position clear that I'm not trying to find findings of fact, I'm not trying to find one witness or the other, I accept that on this occasion I fell short of that.
Q. Do you mean only in your first report or also in your second report?
A. Not in the second report. I believe the second report was -- you know, I believe that the evidence from Mr Parker and Mr Roll, and if we look again at Mr Roll's paragraph 20, I believe that the question that I asked in paragraph 83 that I want further evidence, I believe that was a valid approach.
Q. When you say in paragraph 84 -- because we have dealt with what you accept about your first report at paragraph 1119 several times, haven't we?
A. Yes.
Q. So in your second report, you say, notwithstanding what you have now accepted about 1119, in your second report that you were right not to change that view.
A. I hadn't seen sufficient evidence to change that view.
Q. But do you say you were right not to change that view when you revisited the entire piece in your second –
A. At the time I believe it was right not to change that view because I hadn't seen evidence that convinced me that this change could be made without the knowledge of the subpostmaster. That's where I am.
Q. Dr Worden, do you accept that that betrays a complete failure to appreciate the need to consider the situation both on the basis of whether the claimants' evidence is right as well as on the basis that the defendant's evidence is right?
A. No, I don't accept that.”
(emphasis added)
844. Dr Worden had a further witness statement from Mr Roll saying effectively that Fujitsu could perform, and had performed, remote access. Dr Worden wanted “further evidence” and also referred to needing “sufficient evidence”. He said he “hadn’t seen evidence that convinced” him. He accepted he was reticent to accept the claimants’ evidence. All of this plainly demonstrates that he was taking a partisan view on evidence of fact. I find that this approach to disputed evidence of fact is simply not the correct approach for any expert to adopt. By the time of Dr Worden’s 2nd report he had a responsive witness statement from Mr Roll saying that what Dr Worden had “established” could not be done (a conclusion that was in any event directly contrary to Mr Roll’s earlier witness statement), very specifically, could be done. Even then Dr Worden would not accept it. As he stated in his cross-examination, “I hadn’t seen evidence that convinced me” that this could be done. The evidence was, very plainly, the witness statement saying that it could be done, and he chose to disregard it. This is just a more sophisticated way of stating the witness statement was not true. He described this as his being “balanced and fair”. I disagree with that description: it was not balanced, and it was not fair. He also said that there were many, many occasions when he had “tried not only to be fair to the claimants but to be biased in my numerical estimates in favour of the claimants”. I disagree with that too. Firstly, it is no answer to a criticism that an expert has taken a partisan view of evidence of fact and ignored one side’s factual account, to say that this has been effectively cancelled out by numerical assumptions elsewhere in different parts of the report. Secondly, I do not accept that Dr Worden was performing numerical calculations assuming in the claimants’ favour in any event, when one assesses the Section 8 exercise as a whole. The assumptions he made were very much not in the claimants’ favour at all. Minor adjustments to numerical figures in an exercise based on assumptions specifically unfavourable to one group does not either make the result “biased” in favour of that group, or cancel out the unfavourable assumptions. Further, the court requires an independent approach from an expert at all stages of their analysis, not a raft of slanted analyses, either one way and/or the other, with the hope (or here, an assertion) that they will eventually cancel each other out. For the avoidance of doubt, here they do not.
845. Another area of difference between Dr Worden and Mr Coyne is the issue of what Dr Worden called “countermeasures”. Mr Coyne described these as “self-defined countermeasures” and stated that many of them – Dr Worden had identified 18 different ones – were basic elements of practical system design, and that such design aspirations did not, in themselves, demonstrate that Horizon was particularly robust.
846. The 18 different ones are as follows, together with the acronyms adopted by Dr Worden:
1. Reliable and redundant hardware – RHW;
2. Robust data communication and replication – ROC;
3. Transactional Integrity and database recovery – TIN;
4. Defensive programming – DEP;
5. Generic, data driven software – DDS;
6. Secure kernel hardware and software – SEK;
7. Redundant data storage and computing, with cross-checks – RDS;
8. Double entry accounting – DEA;
9. Early detection of user errors – DUE;
10. Later correction of user errors – UEC;
11. Manual workarounds – WOR;
12. Testing good practice – TGP;
13. Manual Inspection of Data – MID;
14. Bug finding and correction – BFC;
15. Large scale IT architecture – ARC;
16. Quality and change control – QCC;
17. Managing non-functional requirements – NFR;
18. Security – SEC.
847. These acronyms are, for the most part, not used in the IT industry, and were adopted by Dr Worden in this case. Some of them are, such as NFR and SEC. I use them in this judgment for consistency, but their use should not be taken to be a suggestion or acceptance that the acronyms are industry norms; the majority are not. The actual term countermeasures suggests that these consist of measures present in the Horizon system which have the effect of countering something else (bugs, errors or defects, issues that would have an adverse impact on robustness) and thereby improving the robustness of the Horizon system as defined. The Horizon system had an agreed definition for the purpose of the list of Horizon Issues which was as follows:
“the Horizon System” shall for the purposes of this list of issues mean the Horizon computer system hardware and software, communications equipment in branch and central data centres where records of transactions made in branch were processed”.
848. However, it can be seen simply from the descriptions above, which are Dr Worden’s own descriptions, that not all of the 18 are even part of the Horizon system. Some are simply manual checks. Some genuinely are countermeasures, but Dr Worden groups both types together.
849. This can be seen from the following. Transactional Integrity and database recovery or TIN is undoubtedly a countermeasure within the Horizon system. The explanation given by Dr Worden in his summary table for this is “database management systems provide many facilities so that numerous kinds of failure cannot leave the data in an inconsistent, unusable state, or lose any data that have been previously stored.” This means if there is a failure, which may happen in a variety of different ways, the data should not be left in an inconsistent or unstable state, or be lost. The database management system accomplishes this in a variety of ways. Indeed, Dr Worden himself, with his extensive experience of database management, has particular familiarity with this. He designed and developed one of the first relational database management systems; it is an essential requirement in such a system that data should not be capable of being left in an inconsistent or unstable state. The reliability of any database relies upon reliability of the data within that database.
850. Transactional integrity is a fundamental facility built into all database management software and this is made clear in, amongst others, published material such as An Introduction to Database Systems 8th edition by CJ Pearson published in 2003 which was one of Dr Worden’s appendices. As stated at 15.1 of that edition:
“Recovery in a database system means, primarily, recovering the database itself: that is, restoring the database to a correct state after some failure has rendered the current state incorrect, or at least suspect. (We will elaborate on what we mean by "a correct state of the database" in the next section.) And the underlying principle on which such recovery is based is quite simple, and can be summed up in one word: redundancy. (Redundancy, that is, at the physical level; we do not want such redundancy to show through to the logical level, naturally, for reasons discussed in detail elsewhere in this book.) In other words, the way to make sure the database is recoverable is to make sure that any piece of information it contains can be reconstructed from some other information stored, redundantly, somewhere else in the system.
Before going any further, we should make it clear that the ideas of recovery (indeed, the ideas of transaction processing in general) are largely independent of whether the underlying system is relational or otherwise—though it is true that most of the theoretical work on transaction processing has historically been done, and continues to be done, in a relational context specifically.”
851. This is a countermeasure. Transactional integrity is intended to give protection against a wide variety of conditions, such as hardware errors or communication failures, software errors (which can lead to failures and cancellations) and operations being cancelled when they have not been completed. Such cancellations can be made by what Dr Worden calls the operator (the SPM or their assistant) but cancellations can – and the factual evidence shows this occurred – be caused by outages, power cuts and other incidents. A lay person might say that a system has “crashed", or one of the terms used in the factual evidence in this trial was “system outage”.
852. Other measures are not strictly speaking countermeasures, they are sensible design intentions (Mr Coyne used the term design aspirations). An example of this is defensive programming or DEP. Since programming is at the heart of any software – indeed, programming creates the software itself – the way that programs are originally written (what could be called the programming of the system) constitutes the very system itself. DEP was described by Dr Worden in the following way in his table summary: “software is divided into small self-contained modules, which do not assume that other modules are correct, but defend themselves by checking their inputs and raising alerts early.” A better description was given elsewhere in his report where he stated that DEP was “where small parts of a program are written to assume that other parts of the program may be in error and are written to always check their inputs for the presence of errors.” In other words, part of a program does not start from the point of assuming that all inputs are without error; they check for the presence of errors first (in a variety of ways) before moving to other aspects of the program.
853. This could be described as a countermeasure, but a more accurate description of it is a design feature that ensures subsequent steps are taken without repeating or carrying through errors. It is a function of design.
854. However, Dr Worden included within his 18 separate countermeasures matters that are not, in my judgment, countermeasures at all and include remedies. For instance, early detection of user errors, later correction of user errors, manual workarounds, manual inspection of data and bug finding and correction are not all countermeasures built into the system. Just to use that latter item as an example, the many hundreds of PEAKs and associated KELs show that 3rd level of support in the SSC at Fujitsu would investigate certain instances when SPMs reported certain matters. Often this would take many months. Depending upon the outcome of the investigations, matters would sometimes be reported onwards to Development for a software fix. The investigation by the personnel at 3rd level support (of whom, almost invariably, there would be more than one) and also the software writer in the Development section, were done by personnel who are plainly not part of the Horizon system as designed. They are human beings who are undertaking certain tasks. In fact, a software fix designed to prevent a problem re-occurring would change the Horizon system when that fix was introduced in live, a term meaning added to the software within the system. It is, in my judgment, illogical to treat such a process in the same way as either defensive programming (DEP) or transactional integrity (TIN).
855. This approach is at its most extreme in Dr Worden’s view of transaction corrections, which he considered to be a Horizon countermeasure. The issuing of transaction corrections are plainly a remedy, or something done by the Post Office outside of Horizon to correct issues that arise because of Horizon. As Dr Worden accepted, this is not even done in a consistent way, depending upon the client in question. He stated:
“There are many differences of detail in how reconciliation is carried out for different client organisations, or where it is carried out; sometimes the client organisation does it from a file sent to it by Post Office, and sometimes Post Office does it. However it is done, and wherever it is done, the result of reconciliation is always in principle the same.”
856. The way that Dr Worden explained the process was as follows:
“285. When reconciliation finds a transaction for which Post Office's record and the client record do not match, it is passed to a department in Post Office accounts which handles reconciliation discrepancies. Each such discrepancy is, until it has been dealt with, an error in the accounts - and so it must be dealt with. The task of this department is to determine how each discrepancy arose, which therefore determines how it needs to be handled.
286. When the appropriate department in Post Office decides that responsibility for a discrepancy lies with the branch, a request for a TC is issued and is passed from POL FS to the BRDB in Horizon Online. At this point, there is no impact on branch accounts. The request is passed on from the BRDB to the branch Horizon system, so that it will show on the Subpostmaster's screen when he starts the Horizon system the next morning. At this point, the Subpostmaster may either accept the correction or may question it and ask for further investigation.”
(emphasis added)
857. This description makes it clear that there is a substantial link, or series of links, in that chain that are entirely outside of Horizon. The discrepancy, which Dr Worden correctly identifies as “an error in the accounts”, is passed to a Post Office department. That department has to consider it, and determine how it arose. Then that department has to decide responsibility, potentially issue a request for a TC, and that is then passed to the BRDB (which is part of Horizon) and the request goes to the branch. A substantial element of the whole process does not involve Horizon at all. To treat the issuing of TCs as a countermeasure in the way that Dr Worden does is not logical. Even the Post Office’s own witnesses of fact did not deal with the process of issuing transaction corrections, because the Post Office’s solicitor correctly took the view that transaction corrections were not part of the Horizon Issues and were outside the scope of the Horizon Issues trial.
858. Further, the scope of the experts’ agreements in this respect under Horizon Issue 5 in the 3rd Joint Statement is relevant. That issue was “how, if at all, does the Horizon system itself compare transaction data recorded by Horizon against transaction data from sources outside of Horizon?” (emphasis added). Agreed entries include the following:
1. Under sub-topic Reconciliation:
“5.1 Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's clients is largely automated.
Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/Error Notices to the Subpostmasters.”
and:
“5.3 The adequacy of Post Office back office processes to prevent discrepancies in branch accounts can be measured by the quality of the TC process. This quality includes:
• The processes of consideration of available data
• The level of errors observed in the process
• The level of complaints or disputes raised following a TC
• The level of upheld complaints following a TC
• The level of financial impact of erroneous TCs.”
3. Under sub-topic Third Party Data:
“5.4 Errors in third-party data have led to discrepancies in branch accounts, through erroneous TCs being issued on Subpostmasters.
5.5a PO does not control the level of errors made by its third-party client organisations (which may lead to errors in TCs), or the delays in their processes (which may lead to delays in TCs).
5.5b PO can and should ensure, by careful investigation of disputed TCs, that only a small proportion of errors by PO clients lead to losses for Subpostmasters, provided that the Subpostmasters are in good control of their branches and have the required information available.”
“The figure quoted by Mr Coyne (77% of disputed Santander TCs upheld) illustrates that this process worked well.”
861. As put to Dr Worden in his cross-examination, if his approach to interventions by human beings were to be extended to its logical conclusion, the group litigation itself would be categorised as a Horizon countermeasure. In my judgment, that would be the logical consequence of his approach. Plainly this litigation cannot properly be seen as a countermeasure. His approach to countermeasures is not a sensible one; he mixed up both genuine countermeasures such as TIN; design functions such as DEP; and manual corrections as well, and treated them all in the same way. He was wrong to do so.
862. It was also somewhat unhelpful for some of his countermeasures, such as later correction of user errors or UEC, to be so widely defined as to include both manual checks – including by the SPM themselves – and the Post Office and Fujitsu’s both automated and manual processes. Another criticism of Dr Worden’s approach to countermeasures relates to one which Dr Worden described in his paragraph 463 as being “one of the most important countermeasures in Horizon”, and that is manual inspection of data. I do not see how this can be a countermeasure in Horizon as defined at all. Dr Worden’s summary states that “Any large business IT system is used by many people, who view its outputs and check them against each other for consistency, and against their own knowledge of the business. Subpostmasters, watching their branch accounts, were a key component of this.”
(emphasis added)
863. The inherent illogicality in this is, in my judgment, obvious. SPMs, viewing their own branch accounts, which are created or are part of the Horizon system, cannot in my judgment be included as a countermeasure in the Horizon system (even if one ignores entirely the definition of the Horizon system in the Horizon Issues). The fact that a SPM can or may notice that (say) their accounts are (as the SPM sees it) wrong, or that there is discrepancy of some thousands of pounds (or more, or less) in relation to a transaction that they may not recognise, cannot sensibly be described as a countermeasure in Horizon in the way that Dr Worden does. As Mr Coyne explained, this “assumes that the person scrutinising the data is able to identify the correct source of data to be relied upon, in order to rule out what may or may not be erroneous in the first instance. Additionally, its utility as a countermeasure is heavily reliant upon the person scrutinising the data already knowing there is an issue that requires data inspection.” It also obviously relies upon a human element, which necessarily depends upon the skills or knowledge of the person. When this is put with the lack of notification to SPMs generally by Fujitsu and/or the Post Office of existing bugs (and what the effect of those bugs was), which was a feature of the evidence, it can be seen that there is no basis for including this in any list of such measures.
864. The effect of some, but not all, of the bugs might be susceptible to being noticed in this way, for example if there were a “doubled up” transaction in the same financial figure at the same time – the SPM might remember that a customer performed a single transaction in the way that Mr Patny did with MoneyGram as set out at [135] above. But that would depend both upon the type of bug, the type of transaction, the memory of the SPM and also crucially the amount. Another bug, Dalmellington, had some noticeable effects as the evidence in this case shows. Other impacts of other bugs would not be noticed in the same way. This shows the random nature of what Dr Worden describes as a countermeasure. In my judgment, Dr Worden was quite wrong to include this as a countermeasure. I find that something that is a “manual corrective fix” cannot sensibly be described as a Horizon countermeasure.
865. One of Dr Worden’s other countermeasures was “testing good practice”, what he termed TGP. The purpose of testing any system is not to prove that it is correct; rather it is to prove (or attempt to prove) that it is not, a point explained by Dr Worden. Examples of this are regression testing, user testing and testing of edge cases. Dr Worden relied upon what Mr Membury in particular had given evidence about in his statement. I have summarised Mr Membury’s evidence at [500] to [507] above, including his serious omissions regarding E&Y and their concerns. Dr Worden included what was essentially a caveat in his first report, namely at paragraph 315 where he said in his section dealing with development and testing of Horizon, specifically in relation to Mr Membury:
“His witness statement describes several different audit processes, designed to ensure by independent external review that quality is managed effectively in Horizon (Witness Statement of Mr William Membery, 28 September 2018). If these audits have taken place as described in his witness statement, and if the results are broadly as he describes them, then that would increase my confidence that the quality of Horizon has been effectively managed.”
866. He also stated that “a large system such as Horizon can only be robust if sufficient attention has been paid to issues of quality in its development, testing and support.” I have already identified that the experts are agreed that Horizon as it is now is relatively robust. The documents also show that the approach of Fujitsu seems to be an increase in focus upon such issues since about 2016 onwards. Testing and evaluation is in three stages (taking this from another of Dr Worden’s appendices):
1. Unit (or component) testing takes place once the product is developed or procured and ensures that the product conforms to its requirements.
2. Link (or integration) testing verifies that the product interworks with other major components. This level of testing is generally performed on a complete release.
3. Acceptance testing proves to the Post Office's client that the development meets its functional requirements and to the Post Office that it may be brought into use without impacting on the existing solution or other applications.
867. This is then followed by what is called release management, which may be either major or incrementally using interdependent components. This approach to testing is entirely standard in the IT industry. No sensible programmer – or company involved in a system such as Horizon - would introduce changes without testing their impact. However, there is a paucity of documentation that demonstrates the degree to which Fujitsu’s testing regime was successful. Dr Worden, in both his report and appendices, identifies in general terms what was done and why it was important; there is no document available to the experts that demonstrates (for example) Fujitsu’s test-pass percentages, or how specific failures were managed and re-tested.
868. I do not consider testing to be a countermeasure. I consider testing to be an essential part of the foundation of any system. I would go further and say that comprehensive testing is standard in the IT industry.
869. Finally, so far as security or SEC is concerned, Dr Worden explained that “any system that could be easily subverted would not be robust. Horizon is secured mainly through ‘separation of duties’, user authentication, access control and audit.” The evidence of fact made it clear that the degree of access available to the whole of 3rd line support at SSC was far wider than intended, and was subject to criticism by E&Y. Further, the extent of the powerful APPSUP privileges was extraordinarily wide.
J. Conclusion on Expert Evidence
870. I prefer the expert evidence and approach of Mr Coyne to that of Dr Worden. This is for the following reasons.
871. Dr Worden took a partisan view of the evidence of fact, preferring that of the Post Office’s witnesses (by which I mean in this instance predominantly Fujitsu) to that adduced by the claimants. This was most obvious in respect of his treatment of Mr Roll, but also occurred in respect of the claimants’ evidence of individual experiences of events caused by Horizon. No independent expert should do this. One example of this was the important issue of whether Fujitsu could inject transactions under Legacy Horizon without the SPM in question being aware of this. Dr Worden stated that he had “established” that these would be visible to SPMs. Mr Roll’s evidence of fact was directly to the contrary. Dr Worden had dealt with this by stating “So, in my opinion, Mr Roll could not have made these changes to branch accounts “without the Subpostmaster knowing”. In other words, in this expert’s opinion, the evidence of a witness of fact (and a very important one at that) was simply not correct in fact.
872. However, in this case, even the Fujitsu witnesses eventually accepted (but prior to the trial, in their supplementary witness statements) that this had been possible, and that Mr Roll had been correct. Dr Worden’ supplementary report dealt with this by stating that there was a lack of understanding or clarity on this point. He stated that “it seems to me that I require further factual information before I can comment on this evidence” and also that “Mr Rolls new evidence does not cause me to alter the opinion expressed at paragraph 1119 of my first report, when commenting on Mr Roll’s first witness statement, that he could not alter branch accounts without the Subpostmaster knowing”. His first opinion in paragraph 1119 was wrong, and in my judgment obviously wrong, and this should have been unequivocally corrected both in his 2nd report and also in his cross-examination. The extract of this in [843] above makes it clear that he did not accept that.
873. The sequence is therefore as follows. Mr Roll gave written evidence he had done something, and Fujitsu gave written evidence this was not possible. Dr Worden “established” that Mr Roll could not have done what he said he had done, and plainly preferred the Fujitsu evidence. He therefore adopted a position of which party’s evidence of fact was to be preferred. More factual evidence was then served and Fujitsu witnesses accepted this could be done, in contradiction to their earlier evidence. Dr Worden declined to “comment on this evidence” and remained of the view that Mr Roll could not have done that which he said he had done – in other words, his factual evidence was incorrect.
874. No independent expert giving evidence to the court should do this. It is an obvious preferring of the evidence of fact of the party instructing him, added, in this case, to a refusal or failure to accept further evidence of fact to the contrary which subsequently emerged. This is even though the issue of remote access was accepted by the Fujitsu witnesses in supplementary statements.
875. Mr Coyne did not do this. Indeed, he was fair and balanced in identifying where there were such differences, and he was also clear that he did not prefer one side’s evidence of fact to the other side. This is the correct approach for an expert to adopt.
876. I find Dr Worden’s approach to the factual evidence I have identified concerning Mr Roll a particularly egregious failure to maintain the necessary standard of impartiality. Injection of transactions remotely into Legacy Horizon was an important factual point in this litigation, and indeed in the whole history of the ongoing lengthy dispute between the Post Office and the claimant SPMs. Further, in his expert declaration, Dr Worden went wider than the standard wording and specifically referred to an earlier decision in another case. He stated at paragraph 1194 the following:
“I understand that my duty in providing this report and giving evidence is to help the Court on matters within my expertise, and this duty overrides any obligation to the party by whom I am engaged or the person who has paid or is liable to pay me. I confirm that I have complied and will continue to comply with this duty. I have not assumed that any particular version of events is true and I have had regard to the case of Imperial Chemicals Ltd v Merit Merrell Technology Ltd [2017] EWHC 1763 (TCC) in producing my report”.
877. That particular case (of which there are a number of judgments, on liability and quantum) concerned an expert and his preference for the evidence of fact of the party who had instructed him. I had stated in that judgment at [74] that “An expert's role is not to decide issues of fact themselves, and choose what facts to believe and what not to believe”. Specific reference was made by Dr Worden to this judgment in his expert declaration. It might therefore be thought that the requirement upon him as an independent expert not to take a view on which factual evidence he preferred was, or should have been, at the forefront of his consideration. In any event, such an approach by the court to how experts should approach evidence of fact is not particularly ground-breaking. Independence requires an objective consideration by an expert of both sides of the factual evidence. It is the court’s role, not that of an expert, to decide which evidence of fact is accepted and which is not. Notwithstanding this, Dr Worden plainly accepted the initial factual evidence of Fujitsu and rejected the factual evidence of Mr Roll. Then when Mr Parker accepted Mr Roll’s account, Dr Worden still would not do so. He was quite wrong to do this on both occasions.
878. Further guidance on the approach of an expert, if guidance be needed, is also provided in [237] of a later judgment in the same ICI v Merit series, namely [2018] EWHC 1577 (TCC) in which I stated:
“1. Experts of like discipline should have access to the same material. No party should provide its own independent expert with material which is not made available to his or her opposite number.
2. Where there is an issue, or are issues, of fact which are relevant to the opinion of an independent expert on any particular matter upon which they will be giving their opinion, it is not the place of an independent expert to identify which version of the facts they prefer.”
879. Dr Worden went so far as to state, in respect of his consideration of the variations in robustness of Horizon over time, that “in my opinion, as expressed in section 8, the Claimants’ shortfalls are not caused by bugs in Horizon, or any lack of robustness in Horizon.” The question of whether the claimants’ shortfalls was caused by bugs was not one of the Horizon Issues as I have explained above, but in any event, Dr Worden’s conclusions that the specific losses were not caused by bugs/lack of robustness was only reached by means of his statistical approach in section 8 of his first report. This was so heavily based on assumptions that were either not valid, or conceptually flawed, that his conclusions were not valid either. I would go further – the whole process he adopted so far as his statistical exercise was concerned was both invalid and of no evidential value.
880. He also relied – in my judgment heavily - upon information from Mr Jenkins. Mr Jenkins was not even identified as one of his sources of information in section 1.3 of his first report, headed “Sources of information”. This means that Dr Worden was given access to information that was not made available to his opposite number, contrary to the point at (1) above in [878] above. Although there were some references throughout the text of the report to Mr Jenkins, Dr Worden did not routinely identify where he had relied upon Mr Jenkins. He also provided a great deal more information about this contact with Mr Jenkins in his oral evidence than he did in his written reports. In his cross-examination, he identified, when asked about a passage, that he had obtained that information from Mr Jenkins, which plainly took place before the first report was served as he accepted on his first day in the witness box it was “a year ago”. This was not clear in the report itself. One example of this was paragraph 654.2, where the report was dealing with the effect of the Receipts/Payments Mismatch bug, and Dr Worden stated “Because the operation involved was apparently not a double-entry operation on the BRDB, the countermeasure of checking the double-entry constraint DEA did not catch it”. This information came from Mr Jenkins, but until Dr Worden was asked this in cross-examination, no reader would be able to tell this. The involvement of Mr Jenkins in this explanation in his report was simply hidden. Nowhere was there a note or summary of all the information that had been given to Dr Worden by Mr Jenkins. In this litigation in particular, and given the involvement of and information provided by Mr Jenkins, who knew so much about the Horizon system, such a note or summary was, in my judgment, essential. This was particularly important given there was no witness statement from Mr Jenkins. Dr Worden had been provided with, and had used, information from Mr Jenkins in addition to the witness statements served by the parties.
881. Dr Worden also introduced the concept of “lasting” and “transient” effect of bugs or defects on branch accounts. There was no such distinction in the Horizon Issues themselves. A further difficulty with this approach is that Dr Worden confirmed that even an impact on branch accounts that was corrected several months later by a TC would, so far as he was concerned, be of “transient” impact. A different way of stating the same concept is that, so far as he was concerned, “transient” could be of quite long effect. Given TCs are corrections generated outwith the Horizon system and are effectively corrective actions issued by the Post Office itself, this moved his consideration away from the generic Horizon Issues into the realm of the efficacy of the Post Office business systems as a whole, not only including Horizon but their human processes too. That is not what the Horizon Issues were concerned with, and he frankly admitted in his cross-examination that had he not done this, his opinion evidence would have been rather different. Mr Coyne did not use lasting and transient effect, and correctly addressed the actual Horizon Issues themselves, as drafted and agreed by the parties and as approved by the court.
882. Dr Worden also adopted this different approach to impact upon branch accounts. Dr Worden only considered an effect or impact on branch accounts where a discrepancy loss (or gain) was not rectified by a correction such as a Transaction Correction, as set out in the entry in the 2nd Joint Statement about the experts’ different approaches. This ignored the word “potential” in the Horizon Issues themselves. In my judgment, that was a wholly unhelpful approach, and also had the effect of minimising or reducing the number of bugs that have an impact upon branch accounts. This can be demonstrated by considering the following hypothetical example.
883. Consider a situation where there is a software bug, which in this instance I shall call H. The effects of H are such that they cause the branch accounts of a particular branch Post Office to suffer an unexplained loss of £999.99. This occurs, and the SPM of that branch contacts the helpline and goes through to SSC, who create a PEAK and identify this and after a few weeks of investigation, work out that this is an unexplained software bug which they have not come across before. A KEL is therefore also created identifying H. Future incidents of H occurring would (or should) lead to new PEAKs which will also (or should) link to the same KEL, and to the same bug. Fujitsu report to the Post Office about the existence of H, which they have now discovered, and understand is a software bug. The Post Office accept that the sum of £999.99 has arisen in the branch accounts as a result of H and issue a TC to the branch in that sum, to correct the unexplained loss in that sum in the branch accounts. If one were to use Dr Worden’s interpretation of impact upon branch accounts, by reason of the issue of the TC, H would not be a bug, error or defect with an impact upon branch accounts. Such an approach is, in my judgment, wholly artificial and incorrect. It also defies the proper meaning of the words.
884. Mr Coyne also considered substantially more PEAKs and KELs than Dr Worden, by a factor of almost ten. Mr Coyne stated that he and his team had considered a broad total of about 7,500. Dr Worden had only considered a far smaller sample. Mr Coyne’s far more vigorous and thorough exercise relied far less on the process of extrapolation inherent in Dr Worden’s approach, who was inevitably basing his opinion on a far smaller sample than Mr Coyne. Given the breadth of the resources being deployed by the Post Office in conducting this case, I find it difficult to understand why Dr Worden restricted his consideration of PEAKs and KELs in this way. However, I consider that the conclusions of Mr Coyne, based on the far more substantial consideration of these important documents, are more reliable than those of Dr Worden.
885. To be fair to Dr Worden, he entirely accepted that he and Mr Coyne had approached their evidence differently. He stated the following when explaining why he had not supported Mr Coyne’s information requests.
“A. Well, basically I think it became evident, and it is evident now, that two experts took very different approaches, and my approach was top down, understand the architecture and work down through things like KEL. So as far as I was concerned, I had plenty of information to go on. And Mr Coyne's information requests didn't strike me as things that, yes, I have got to really see that because I was -- had a different priority. I was trying to do a top down understanding of the architecture and top down look at robustness, and so on and so forth. And so I had plenty of documents to look at, basically. So I think it is the different approaches taken by the two experts that led to lack of overlap.”
886. This continued:
“Q. Sorry, not having supported the very many requests he had to make?
A. Yes, I felt his interests were different from mine and I really had plenty to do.
Q. And you initially only looked at KELs, didn't you, you didn't actually look at PEAKs?
A. No, I looked at KELs and I looked at PEAKs where they were relevant. I felt that KELs were a more distilled form of information. I felt they were sufficient in many ways, especially when you go and look at the PEAKs.
Q. You looked at one or two PEAKs referred to in the KELs you looked at?
A. Yes. Look at some of them where you feel the KEL doesn't tell you enough.
Q. And Mr Coyne was looking at PEAKs and KELs?
A. I think in the first report he was looking mainly at KELs, like me, and in his second report he turned on PEAKs.”
887. The reason that PEAKs are of great assistance is that there is a great amount of further information contained in them than in the KELs, and they also give a far more complete picture of the number of different occurrences of a bug. Both types of documents are however useful. I do not know why Dr Worden should have seen Mr Coyne as having “different interests” to him.
888. Dr Worden also entirely seemed either to misunderstand (or restrict his consideration of) Horizon Issue 1, which concerned bugs, errors and defects having the potential to cause apparent/alleged discrepancies or shortfalls in branch accounts or transactions, or undermine the reliability of Horizon accurately to process and to record transactions. When challenged about his approach to this, including the absence of the word “cause” and the use of the word “potential” in the issue itself, he stated the following:
“A. What's the distinction between having the potential and actually doing? I mean, if it doesn't actually do, then in some sense it didn't have the potential.”
889. This is not the correct, or conventional, use of the word “potential”. It also looks at the matter in reverse. In the context of the Horizon system, whether bugs, errors or defects in that system have, or had, the potential to impact branch accounts or transactions, should focus one’s attention upon the possible effect of those bugs, errors or defects. If, of course, a particular bug actually has had the effect of in fact impacting branch accounts, then by definition it must also have had the potential to do so. Actual effect is a sub-set of potential effect. It means impact upon branch accounts must have actually happened. However, potential effect does not require that impact to have actually occurred. The two terms are not synonymous. Dr Worden therefore moved away from the actual issues before the court, and approached a great deal of his evidence from the point of rather different issues, which he seemed to have narrowed down himself. I consider such narrowing of the issues to be unwarranted.
890. In their opening submissions, the claimants summarised this different approach as follows:
“We respectfully will invite the court to adopt an approach more consistent with Mr Coyne's approach which is to look at what actually happened with particular examples and trace them through to a reasoned and careful conclusion and then from the ground up, as it were, draw inferences upwards rather than from an overarching hypothesis downwards.”
891. The “overarching hypothesis” was that of Dr Worden. Because he had taken such an overall view, and an overall view on the most recent performance and functionality of the Horizon System (which is now HNG-A as of February 2017, and is no longer HNG-X), the degree of assistance his evidence provided to the Horizon Issues across the whole life of the system, both Legacy Horizon and Horizon Online, was more limited than that of Mr Coyne.
892. In this group litigation, when the first individual claims are being tried out, further analysis will be required of the actual branch accounts of individual claimants. That stage of the litigation will deal with actual branch losses, if any, that are said by the claimants to be due to the actual impact of the bugs, errors and defects which exist. It is at that stage that findings will be made about actual impact upon the branch accounts of the different claimants. Some of the bugs and defects are agreed by Dr Worden and Mr Coyne, and those that are not agreed are resolved by me in the Technical Appendix. It is against the background of those findings that individual claims will be resolved. The purpose of the Horizon Issues trial was to resolve the generic issues, as the parties were poles apart about even the existence of such bugs, errors and defects, and whether they had the potential to act upon branch accounts and transactions in the way alleged by the claimants. Dr Worden moved away from consideration of those specifically drafted and agreed Horizon Issues and answered rather different ones.
893. I also consider that Dr Worden, in his approach to countermeasures, insufficiently differentiated between measures which were part of the Horizon system and those that were not. He also included items in his countermeasures that plainly ought not to have been included as such, for example those the result of human intervention or involvement, and manual inspection of data by SPMs.
894. In my judgment, Mr Coyne emerged more or less unscathed from his four days of cross-examination by leading counsel for the Post Office. That is not to criticise the cross-examination itself, which was very thorough and careful. It is just that what the Post Office set out to demonstrate, including lack of impartiality on Mr Coyne’s part (if not positive bias), lack of reliability in his conclusions, and the other detailed criticisms of him, simply was not achieved. He was described as “evasive” in some of the Post Office’s closing submissions, a term which I find is somewhat wide of the mark in describing how he gave evidence in the case before me, and how he answered questions. He was also criticised in the same submissions as adopting “an approach [that] was simply to look for bugs without giving a proper sense of scale, context or even balance”.
895. In the Post Office’s closing submissions, there is a section headed “Judicial criticisms of Mr Coyne’s approach” where reliance is placed on a judgment in Scotland, called CGI IT UK Ltd v Agilisys [2018] CSOH 112 by Lord Bannatyne. In that case, Mr Coyne had appeared as an expert and he was criticised by the judge for an unbalanced view, amongst other things. The judgment was handed down on 4 December 2018. The fact that he had been judicially criticised in that case was not specifically put to Mr Coyne. However, the substantive criticisms of him, that he gave unbalanced and selective evidence, and the other criticisms, were put to him, and he did not accept them. The point that he had been previously criticised in a judgment ought specifically to have been put by reference to that case, so that he could answer this point. However, in any event I have read (as invited in the Post Office’s submissions) the entire Scottish judgment, in which the evidence of Dr Hunt, Mr Coyne’s opposite number, was preferred to his evidence. I have done this with particular care, and I have also considered all the criticisms made of him in that judgment, as well as the ones put to him in his cross-examination before me. The Scottish judgment provides a full explanation of the shortcomings of his approach in that case. I have also taken this prior judicial criticism into account in weighing up his expert evidence in this trial.
896. The Post Office, amongst other things stated that “it is submitted that the examples set out above [ie in the Lord Bannatyne judgment] show the same tendency to reverse and switch in response to difficult points.” In other words, it is submitted that Mr Coyne adopted the same approach in the Horizon Issues trial before me, as he had in the earlier case in 2018. I have considered the criticisms of Mr Coyne with particular care in the light of this previous judicial criticism, even though I do not have the benefit of Mr Coyne’s explanation to the specific points arising out of that case. I do not find his evidence in the Horizon Issues trial suffers from the same deficiencies as Lord Bannatyne found in that earlier case. In particular, one of those criticisms in CGI, that he had failed to consider material documents and matters, simply does not remotely apply here. He and his team had done an extraordinarily thorough job of examining many thousands of PEAKs and KELs, and many more than Dr Worden did. I do not find that Mr Coyne reversed his evidence, or switched, or that any of the other criticisms of him are made out. It may be that he has changed his approach as a result of his experience in that case or judgment, or there may be other reasons for it. However, I do not find the criticisms made of him to be made out in this case.
897. In my judgment, given the Horizon Issues, which expressly include “bugs, errors or defects”, the approach of an IT expert to start by looking in detail at the large volume of the Fujitsu PEAKs and KELs for bugs is not only entirely sensible, it is verging on essential. I reject the criticism that Mr Coyne’s evidence was given without having any sense of scale, context or balance. I conclude that Mr Coyne was impartial in his evidence in the Horizon Issues trial, and was giving the court evidence that was entirely uninfluenced by the exigencies of litigation or the interests of the party who called him. I find that he was independent, and that his conclusions in this case on the Horizon Issues are reliable.
898. There were some subjects upon which he became a little flustered in his cross-examination, for example on the final afternoon when he was taken to task for the entry in the 2nd Joint Experts’ Statement on an agreed passage where he and Dr Worden had agreed that “the number of distinct bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in branch accounts, is between 12 and 29”. That latter number had come from Mr Coyne and he wished to reduce the number, initially to 13 but then upon reflection to 21. Mr de Garr Robinson made a great deal of this, as one would expect of a skilled cross-examiner, but in reality it is a simple point, as I have explained above. It did show a change of position because the number of bugs changed as I have explained. However, it is easy to overestimate the degree to which making an error on figures in such circumstances does, or does not, represent a lack of reliability on the part of an expert who has performed a substantial body of work. It must be remembered that this was an agreed entry in an agreed Joint Statement that was 44 pages in length (the experts agreed four separate statements, running to 100 pages in total including appendix). I deal with the detailed analysis of each alleged bug in more detail in the Technical Appendix, but the fact that the number did not remain an immutable “29” in all circumstances is not, in my judgment, a point against Mr Coyne. His attempts to ensure the final number was correct was not as smooth as he might have liked, and it gave Mr de Garr Robinson an opportunity to deploy his forensic skill, but in reality it did not impact upon Mr Coyne’s reliability.
899. Mr Coyne was widely attacked for a lack of partiality, and indeed, for positive bias in favour of the claimants. I do not find those complaints or criticisms to be well founded. They are not based on any real substantive instances that can be shown in any specific respect to represent Mr Coyne demonstrating any lack of impartiality. They are not made out. The points made are, in my judgment, insufficient grounds to dent his credibility as an independent expert whose evidence can safely be relied upon in resolving the Horizon Issues. He had not been previously instructed to undertake work for either party, and he had no previous involvement in the history of the wider dispute between the Post Office and SPMs concerning the Horizon system and its reliability. I found his evidence before me to be neutral and independent, and not influenced by the fact that he was instructed for the claimants.
900. Dr Worden’s decision, very late in the day, to prepare his 3rd Report, appears – by reason of the dates - to have coincided with the dismissal of the recusal application. This matter was not explored at all in cross-examination and so it is not necessary to consider the timing of Dr Worden’s realisation that he could perform a new exercise. Given that his oral evidence to me was that he was advised to serve that 3rd Report directly upon the court, and effectively was left to draft the covering email to the court himself, I am reluctant to criticise him for this unusual approach unduly. This is therefore not something that I weigh in the scales to any appreciable degree in deciding whether his opinion evidence and conclusions are more, or less, reliable than those of Mr Coyne.
901. I do not wish it to be considered that I am being universally critical of Dr Worden in all respects. He had done a substantial amount of detailed analysis in his two reports, and the overall knowledge that the court had, at the conclusion of the cross-examination of both experts, was very much greater than it had been prior to the commencement of the Horizon Issues trial. Dr Worden discovered some bugs himself, bugs that neither the Post Office and/or Fujitsu either knew about, or had admitted the existence of, prior to Dr Worden’s involvement in the litigation. He also took a sensible and considered view of some elements of the documentation, the importance or relevance of which had not been admitted by the Post Office earlier in the litigation, such as the KELs. Some of the matters that were agreed by Dr Worden with Mr Coyne in the 1st to 4th Experts’ Joint Statements saved a considerable amount of court time. His entries in particular in respect of the Bug Table, where he considered the detailed contents of a number of PEAKs and KELs, were useful as well.
902. However, Dr Worden had become closely involved with the Post Office’s case in the litigation. In my judgment he took a partisan view of the evidence of fact of Mr Roll, which he should not have done, and he also seemed to be acutely aware of the Fujitsu position. For example, if there was a problem in Horizon that originated with Riposte, rather than part of the system written by Fujitsu, he was always enthusiastic to make clear this differentiation in Fujitsu’s favour. It was not relevant to the Horizon Issues whether bugs, errors and defects in Horizon were caused by Fujitsu or Riposte (provided by Escher, a separate company), or any other part of the system. These are all points that lead to the same conclusion, namely that in my judgment his expert evidence was not entirely independent of the Post Office’s and/or Fujitsu’s interests, and his conclusions were substantially less reliable than those of Mr Coyne. I do not know whether he had found himself subconsciously influenced by his discussion with Mr Jenkins or not, but regardless of the reason, his conclusions were not reliable and I reject them in favour of those of Mr Coyne.
903. In all the major respects therefore, and taking account of all the points made both in cross-examination and submissions by both parties concerning their own, and the opposing side’s, IT expert, I prefer the evidence and conclusions of Mr Coyne to those of Dr Worden.
The number of established bugs, errors and defects in Horizon
904. These are included in summary form in Appendix 2, together with whether they were present in Legacy Horizon or Horizon Online (or both) and their approximate years of operation. For a detailed analysis, the Technical Appendix to this judgment (which is Appendix 1) should be consulted.
K Audit Data
905. The experts are agreed, in item 4 of the points of agreement referred to above in respect of Horizon Issue 3 as part of the 3rd Joint Statement, that the Post Office does not consult the audit data. The actual text of that entry is the “Post Office does not consult the full audit data (unfiltered ARQ Data) before deciding how to handle discrepancies and issuing Transaction Corrections.” Regardless of this agreement, it is obvious on the factual evidence that the Post Office does not do so. Indeed, throughout both the Common Issues trial and the Horizon Issues trial, it has been increasingly obvious that the Post Office uses sources other than the audit data when it is challenging SPMs about what they have actually done in their branch Post Offices. Various terms are used for the audit data. The 3rd Joint Statement uses “full audit data” and “unfiltered ARQ Data”. In places the term “core audit data” and “core audit store” was used, which refers to the same data and store. I shall refer to it as audit data.
906. The issue of audit data arises in the following way. Audit data is a complete and accurate record of everything that has occurred, which in the context of Horizon means including a full record of keystrokes used by a SPM (or assistant) in the branch. This accurate record is kept in what is called the audit store. This is a secure place for the keeping of such data. It is vital to the proper operation of a system such as Horizon that such accurate audit data is kept.
907. In his evidence Mr Coyne referred to the concept of “Write Once Read Many”, which has its own acronym, WORM. Dr Worden accepted that this was “pretty common” in the industry. It means, effectively, that audit data is only written once – indeed, it must be only written once. That is the whole point of audit data. However, it is written (or in other words, recorded) so that it can subsequently be read, and the second part of this concept, “Read Many”, means that it can be read many times. The audit data is there, in my judgment, so that it can be consulted when required. The experts’ agreement makes it clear that the Post Office does not consult it before deciding how to handle discrepancies and issue TCs.
“Q. But did you see any documents of any sort indicating or referring to the stream of data flowing on a continual basis out of the audit store into Post Office's management systems?
A. No, but that's not how things would work. If Post Office wanted to get access to the data in the audit store they would go to a screen or go to an application on their computer and they would run the request for that data.
Q. Mr Coyne, I would like to suggest to you that it is completely unrealistic to think that a separate sealed core audit store of the sort we're talking about should be cracked open hundreds of times a day in preference to using management information systems which are designed for that precise purpose?
A. I think the word "sealed" is misleading and the concept of cracking something open to get access to it I think is misleading as well.
Things in an audit store are only -- can be written to and only written to once, and the term that's often used is write once read many, WORM. So the process is written to once, but people can read from that store on many occasions.
Q. But just to be absolutely clear, you had not and indeed you have not seen any documents suggesting that Post Office had the ability to gain access to the audit store on its own systems, had you? There was no design facility, there was no -- there were no lines of communication between the audit store and Post Office in any document you had ever seen, correct?
A. No, it looks as if the majority of the references to audit database access was from Fujitsu personnel.
Q. And one final thing. Would I be right in thinking that now that you understand how the audit store actually works and the costs and delays associated with extracting data on a large basis from the audit store, would you accept that it would be disproportionate to be using the audit store as a basis for making decisions on transaction corrections in every single case?
A. Yes, it would seem that it would be very expensive and very slow to access the audit store, and effectively for the number of transaction corrections you couldn't do that, and therefore you accept that you make decisions on the management information systems rather than the audit store.”
(emphasis added)
909. The Post Office therefore relied upon the fact that the audit data was effectively kept by Fujitsu; that accessing it was akin to “cracking open” something that was a “separate sealed” store; and that it would be disproportionate to use it to make decisions on transaction corrections in every case.
910. The first and third of those points are accepted; the second one Mr Coyne described as misleading. I agree with that response to all three of those points. I also find that the concept of “cracking open” the “sealed store” is rather off the point of the very purpose of audit data, and the uses to which it both can and should be put.
911. The following points are, in my judgment, important on this issue.
1. Audit data is necessary so that there is an accurate record of what has occurred on the system, including what a SPM has done in branch, in terms of the keystrokes actually used. The use of the word “audit” makes this clear, and also this is not a controversial point.
2. The need to consult such an accurate record does not arise in every case where a transaction correction is considered or issued. The management data is sufficient for a large number of transaction corrections.
3. However, where there is a dispute between the Post Office and a SPM about branch accounts, with the arguments on both sides about what or who is to blame, the audit data should be consulted. That is one of the purposes of having it in the first place. I can think of no sensible reason not to consult the audit data in such a scenario.
4. Doing so is not equivalent to “cracking open” a “sealed store”. The audit store is sealed in the sense that data, once in the store, should not be capable of being changed, or written to again. However, reading that data is not “cracking open” the store. It is consulting or reading the audit data, one of the main (if not the only) purposes of having such audit data in the first place.
5. The evidence in both this trial, and the Common Issues trial, where the Post Office cross-examined a number of SPMs on events in their branch accounts by using a variety of management information, other than the audit data, makes clear to me just how important it is to use the audit data, rather than other sources including management information. The management information is confusing, contradictory, has been shown to be wrong and requires numerous assumptions or a “take it from me” type of approach on the part of a questioner. It is rarely agreed what that management data shows. The audit data, by its very nature, will be far superior and the best evidence available of what has occurred on Horizon. It should be consulted in circumstances where there is a dispute between the Post Office and a SPM.
6. Finally, even the Post Office’s own witnesses were not agreed on what the management data would, or would not, show. At one point Mrs Van Den Bogerd said that Credence (which is management data) would record keystrokes in branch. This was then contradicted by another Post Office witness who said Credence did not do this. Whether the lack of understanding on the part of Mrs Van Den Bogerd was widespread or not within the Post Office is unclear to me. It is however clear to me that Credence data does not in all circumstances record all keystrokes performed in branch by a SPM. The audit data does.
“It is a central principle of Horizon that the Core Audit Database acts as a secure 'gold standard' for branch accounts (countermeasure SEK) and that the audit record can only contain events which originated at the counter - either in customer transactions or monthly balancing.”
913. That is an accurate description in my judgment. I can see no sensible or justifiable reason for the Post Office’s reluctance to consult the audit data in cases of serious dispute with SPMs, in particular the types of dispute which form the subject of this case, and without doubt, any dispute that involves criminal proceedings against SPMs. I also consider the same point applies in relation to internal Post Office proceedings that lead to the suspension or termination of SPMs. Acceptance of that point does not mean that that the audit data has to be consulted for every transaction correction issued by the Post Office. As can be seen from the passage of cross-examination at [908] above, the Post Office effectively jumped to the issue of whether audit data should be consulted before issuing every single TC. Mr Coyne accepted it was not necessary to do that. That is not the same as accepting it should not be consulted in circumstances where there was a dispute between the Post Office and SPMs.
914. Mr Coyne had identified issues with using Credence data. There was a one-hour difference in the time stamps used between Fujitsu and Credence, which can hardly have helped sensible investigations when SPMs raised queries, but there is more to this than that. The E&Y review in March 2011 identified various issues with Credence, including weak change controls within the back-end of the systems which allowed Logica developers (the third-party provider) to move their own uncontrolled changes into the production environment, which included both Credence software code and the data within Credence used for what was called “audit evidence” but which has to be differentiated from what I am referring to audit records in the audit store. There was a lack of further documentation to approve fixes and patches applied to Credence outside of the release process, which meant that linking changes to issue tickets to record the original request for the bug fix was not possible.
915. A concern was also noted by E&Y in respect of front end change process weakness. The passage states:
“During our walkthrough of user administration of the front end of Credence we noted several users with administrator rights, including some generic users (this is noted below as a separate point). These users have the access rights to create and amend reports, including those which may be relied upon for audit evidence. These users can change report design, and processing without documented request, test or approval. When users have the rights to change reports that are used by the business for reconciliation, exception reporting or other processing, there is the risk that the reports are manipulated either intentionally or accidentally.”
916. There is also at least one specific occasion, considered in the evidence during the trial, which shows that the Credence data does not show the correct position. This was put to Ms Van Den Bogerd. This was an occurrence at Lepton in October 2012. It led to something that was referred to throughout the trial as the Helen Rose Report or Rose Report, as the author of the report into it was called Helen Rose. I have dealt with it to a degree at [227] above. That report records that:
“A transaction took place at Lepton SPSO 191320 on the 04/10/2012 at 10:42 for a British Telecom bill payment for £76.09; this was paid for by a Lloyds TSB cash withdrawal for £80.00 and change give for £3.91. At 10:37 on the same day the British Telecom bill payment was reversed out to cash settlement.
The branch was issued with a Transaction Correction for £76.09, which they duly settled; however the postmaster denied reversing this transaction and involved a Forensic Accountant as he believed his reputation was in doubt.”
(“duly settled” means the SPM paid the Post Office that sum).
The Credence data showed that the SPM had reversed the transaction. By consulting the audit data, Mr Jenkins discovered that he had not. This was expressly confirmed, both in the Rose Report and also by Ms Van Den Bogerd in her cross-examination.
917. This shows that the management data is not entirely reliable. Had the SPM at Lepton simply paid the approximately £80 and not engaged the forensic accountant – in other words, had he not been prepared to pursue this matter – the Post Office would simply have received that sum from the SPM to which it was not entitled, and the Rose Report would not have emerged.
918. The issue of remote access also further complicates (or rather, weakens) the reliability of management data. All of these concerns reinforce the view that I have in any event, that audit data (as in, that stored in the audit store) is what should be used in the circumstances that I have identified.
919. One point that occupied the parties a great deal in the Horizon Issues trial, upon which there was some evidence, and upon which Mr Coyne formed a view based on his analysis of some contemporaneous documents, is that Fujitsu raised a charge to the Post Office for audit data requests (also called “ARQ requests”) above a certain number per year. The numbers that were discussed were in the amount of hundreds of pounds per request. The parties could not agree on what these charges were, although it was accepted that there were a number each year that were either free, or the costs of which were included in the contract sum (depending upon which way one looks at this matter). The charging structure will, in terms of the amount per request above a particular number, have changed over the years in any event.
920. That charges are raised by Fujitsu to the Post Office is not an adequate answer, in my judgment, to the Post Office’s failure to consult or provide the audit data in cases such as those in [912] above. There are some contemporaneous references within Post Office documents suggesting this may have been a disincentive in some cases to raising ARQ requests of Fujitsu. Further, there are numerous references within Post Office documents (at a high level) of the very great cost to the Post Office of the Horizon system generally, that cost being paid to Fujitsu. Fujitsu were said in one document to see the contract with the Post Office as a “cash cow”. This entry came in a document (parts of which were redacted, which I record simply for completeness) dated 17 January 2017 which were the minutes of the Post Office Group Executive Meeting of that date. Ms Van Den Bogerd was “in attendance” for two items only, numbered 6 and 7, but not in the list of those “present” by which I assume she was not there for the whole meeting. The Post Office Chief Executive and Group Legal counsel were there, together with a number of others.
921. There is only one agenda item which is not redacted, namely item 5, IT Strategy. Parts of the minutes relevant to the Horizon Issues are as follows:
“RISKS
• Need to be clear for the Board what operational risks we are facing and how we are mitigating those risks, are there any choices we can make to accelerate the solution to get inside the risk appetite.
• Already making these changes to the areas of risk, but will make this clearer. Need a strategic discussion at GE on choices in the IT strategy
• Horizon - cannot accelerate work on the data centre refresh, getting under control. FJ running this well, risk review & declaration every month. RH will kick again but doing as much as we can.
• Credence - Support and maintenance has moved to Accenture but on old hardware. Single platform if goes, goes completely. Driving pressure into Accenture to get more capability. Accenture picked up difficult pass and the additional costs at the moment. Starting to see light at the end of the tunnel. RH having difficult conversations, flag up any issues, assumptions and risks to GE. KPMG engaged to undertake a GAP analysis to produce an IT control framework.
• What are the risks of the 'hop' to thin client not working? RH - need to prove thin client works and the 'hop' is a safe way to test we could revert to HNGA if the 'hop' fails. All nervous as we are dealing with legacy systems. More comfortable that we are not changing Back & Front office at the same time, but still using Horizon. SSKs have proven that thin client can work, it is the journey to implement that is vital.
• Fujitsu - negotiations will be tough as POL do not hold the power. FJ see the contract as a cash cow, so need to persuade them that working with POL to migrate to cloud technology is to their benefit against a 'too good' contract. They may say make us whole but we cannot give additional work because of the procurement risk. Only choice if they will not work with us is to start building an alternative. Come back to GE with an update on the FJ negotiations plan”
(emphasis added)
922. This document shows that Fujitsu, at least so far as the Post Office are concerned, are seen by the Post Office as believing that the contract with the Post Office is “too good” and a “cash cow”, such that Fujitsu may not even “work with us” [ie the Post Office] in terms of improvement. Part of that very attractive commercial arrangement (attractive for Fujitsu) may be charges that Fujitsu is entitled contractually to raise for ARQ requests. Part of an attempt to reduce the expense of the contract by the Post Office may be a reluctance to raise such requests. Dr Worden described this document as “chiming” and said that “I have always had the impression from the governance structure and the documentation and so on that Fujitsu were not short of budget, really”. By that he meant Fujitsu were well-remunerated, and this remuneration was paid by the Post Office.
924. Finally on this point, I am surprised that the desirability - if not the actual and basic need - to consult the audit data is a controversial point. In my judgment it is not only good practice to consult the audit data, given the very purpose of audit data within a complex IT system such as this one, but it is also obvious common sense. There is little point in having audit data if it is not consulted in the circumstances that I have identified above. I appreciate that the audit data itself will not be immediately comprehensible to some SPMs. There may also be charges raised by Fujitsu in respect of such requests. However, neither of these are reasons for the failure to consult it. I also make no findings on whether Fujitsu are, or are not, entitled to raise charges and if so how much. Those latter two points were not fully argued before me, and Fujitsu were not represented.
L. Overall Conclusions
926. Those claimants who gave examples of specific incidents specified them as well as they could, and in my judgment truthfully and accurately. It was usually Ms Van Den Bogerd who, for the Pot Office, attempted to demonstrate that this was not what had occurred, often by means of assertions that if the SPMs had followed procedures properly, what they said happened, simply could not have happened. The Fujitsu witnesses gave some limited evidence of fact in relation to specific instances and some specific PEAKs and KELs. The Post Office’s counsel did their best to challenge the factual accounts that were given by the claimants. I have provided the relevant findings in respect of those matters above. I indicated at the Case Management Conference on 22 February 2018 that the Horizon Issues should be capable of resolution with limited, if any, evidence of fact. Both parties chose to submit factual witness statements and called factual evidence. The factual evidence submitted by both the parties was, in my judgment, limited as expected and required. With the exception of the matter to which I have already referred concerning the evidence of Mr McLachlan and Mr Henderson at the PTR, the extent of each party’s evidence of fact was not subject to challenge prior to commencement of the Horizon Issues trial.
927. The factual evidence of specific instances was of assistance in coming to conclusions on the Horizon Issues. Indeed, I found some of the factual evidence to be of great assistance. That of Mr Roll and Mr Godeseth was extremely useful. The latter, one of the Post Office’s main witnesses and the Chief Architect of Horizon, was sufficiently damaging to the Post Office’s case on the Horizon Issues that they were, essentially, forced almost to disavow him, and the Post Office’s closing submissions were highly critical of the accuracy of his evidence. As a further example only, Mr Latif gave evidence that he had performed certain basic routine steps to transfer the sum of £2,000 between terminals, and (in accounting terms) that sum had – to use his word – “disappeared” from the Horizon system. The Post Office evidence challenging this was from Mrs Van Den Bogerd, and essentially amounted to an assertion that if certain steps had been followed by him, this should not, or could not, have happened. That would be all well and good in an ideal theoretical world that did not pay any attention to reality, but it entirely ignores Mr Latif’s primary evidence of fact. I accept that primary evidence from the claimants’ witnesses in this trial as I have explained in Part D. Mr Latif had been a trainer trusted by the Post Office to train other SPMs, as explained at [98] above. Other references in the documents show both Romec engineers, and even Post Office auditors, being recorded in PEAKs as witnessing events supporting the type of occurrences that underpin the claimants’ case. These were ignored by those at Fujitsu tasked with completing these PEAKs, and conclusions are shown in those PEAKs being drawn by Fujitsu that flew in the face of what had occurred. Mr Latif’s experience is extraordinarily similar to the Dalmellington bug, which is numbered 4 in the Bug Table. Mr Latif did not operate core and outreach branches, therefore his experience is slightly different to what happened at Dalmellington. However, if one is to take an objective and sensible view of the evidence of a previously trusted SPM, whom the Post Office itself has used to train other SPMs for some years, who stated that he did something correctly and X occurred, then meeting that with a bare assertion that X simply cannot have happened is not particularly sensible, nor is it persuasive.
928. The approach by the Post Office to the evidence of someone such as Mr Latif demonstrates a simple institutional obstinacy or refusal to consider any possible alternatives to their view of Horizon, which was maintained regardless of the weight of factual evidence to the contrary. That approach by the Post Office was continued, even though now there is also considerable expert evidence to the contrary as well (and much of it agreed expert evidence on the existence of numerous bugs).
929. This approach by the Post Office has amounted, in reality, to bare assertions and denials that ignore what has actually occurred, at least so far as the witnesses called before me in the Horizon Issues trial are concerned. It amounts to the 21st century equivalent of maintaining that the earth is flat.
930. When real world examples such as Mr Latif’s are put together with the expert evidence that I have accepted – or even with Dr Worden’s lower figure for accepted bugs of 11 different ones – it can be seen that this institutional obstinacy by the Post Office amounts to little more than repeated assertions that the Horizon system (both Legacy and Online) cannot be to blame for the claimants’ experiences, coupled with (for some) challenges to the claimants’ witnesses because the Post Office simply cannot accept their factual accounts. The findings that I have made, on the evidence in the Horizon Issues trial, show that the reality is rather different, and the existence of the bugs, errors and defects that I have found to exist do have the effect explained by Mr Coyne.
931. The approach that I have adopted to the evidence can be considered as follows. Firstly, to consider the factual evidence, the relevant contemporaneous documents, and applying the burden and standard of proof, reach evidential findings. I have considered the challenges to that evidence, and if the specific instances of factual witnesses explaining what they experienced could be explained (as the Post Office sought to do with some of the SPMs) by them simply not being reliable, remembering incorrectly, having been careless or even (for some) being criminal. I have also considered all of the Post Office’s factual evidence in this trial, the explanations of the Post Office’s witnesses, and reached my factual conclusions. I have considered the expert evidence on its merits, and then tested my conclusions against the evidence of fact that was adduced by both parties, and considered whether my judgment on the evidence of fact matched and supported, or contradicted, or in any way affected or influenced my conclusions on the expert evidence. In a sense this was to consider the factual evidence, and the expert evidence that I accepted and preferred, separately, and see if they both reached the same conclusion. Secondly, the evidence can also be approached in a different sequence. This is by firstly considering the specific factual evidence, and then considering the expert evidence, and only in the light of the conclusions on the factual evidence. Alternatively, one could simply decide the Horizon Issues on the expert evidence alone, although that would be far less preferable and would ignore entirely the important factual evidence of the Fujitsu witnesses, in particular Mr Godeseth. However, even if that third way were to be adopted, the same answers to the Horizon Issues would, in my judgment, be reached.
932. Whichever way is adopted, it must be remembered that any claimant can only ever have personal experience of one small element of the Horizon system on a number of occasions. The overview and historical performance of Horizon, both Legacy Horizon and Horizon Online, has to come from Fujitsu witnesses and the two experts. Dr Worden and Mr Coyne had far more of an overview of the life of the system, as did Mr Godeseth and Mr Parker. Mr Roll had experience of it in its early years, but his evidence was what led to the correct picture on remote access finally emerging.
933. I have done all of these exercises. Regardless of whichever approach is adopted, the same conclusions are reached. In this respect, all roads lead to Rome. In my judgment, the evidence of the factual witnesses accepted by me entirely supports and corroborates the conclusions reached by Mr Coyne. Indeed, I would go further than this, and state that Mr Godeseth’s evidence alone is enough to support and corroborate Mr Coyne’s conclusions. When that is put together with the evidence of Mr Roll, and the concessions that were obtained from the Fujitsu witnesses (in the circumstances of their performance as witnesses, to which I have already referred) it is clear to me that the correct conclusions to be drawn on the Horizon Issues are those drawn by Mr Coyne, save and to the extent that I have modified them in any specific respect.
934. I consider that the evidence of the Fujitsu witnesses in particular, both former and current, has been of considerable assistance to me in resolving the Horizon Issues. Mr Roll, Mr Godeseth, and even in his own way Mr Parker (though not in the way that Mr Parker himself would have intended) have all provided clear evidence of the problems with the Horizon system, the bugs, errors and defects within both Legacy Horizon and Horizon Online in its HNG-X form, the way that these problems were (or were not) dealt with, and the way that Fujitsu had powers which, until shortly before the trial started, Fujitsu sought to keep from the court, and may not even have fully disclosed to the Post Office. Because the extent of these powers was kept secret in this way, the Post Office finds itself now having made misleading public statements previously. If one looks back to an earlier case management hearing and reconsiders how Fujitsu, through the Post Office, sought to portray the contents and lack of importance and relevance of PEAKs and KELs, then it can be seen that there has been a pattern of considerable defensiveness over the Horizon System. There has certainly been a lack of transparency, and a lack of accuracy in description.
935. I have explained above at [172] how the Post Office sought to draw a particularly glowing reference in Fujitsu’s favour from the evidence of Mr Roll in their closing submissions. The Post Office stated that from his evidence “….a clear picture emerged of Fujitsu as an organisation which was thorough, professional and conscientious and which took considerable care to ensure that matters were properly investigated and dealt with.” I have already stated that this is not a conclusion that can be drawn from the evidence of Mr Roll. Nor is it a conclusion that, in my judgment, can sensibly be drawn from the totality of the evidence taken together, both factual and expert, in the Horizon Issues trial. As an example, on too many occasions are there PEAKs that conclude at one point that user error cannot be to blame, only for a misleading closure code to be used for such a PEAK. Fujitsu personnel routinely refer in such documents to the known existence of bugs, without this (so far as the documents deployed in the trial are concerned) being communicated to the SPM in question in these terms. In places there is even debate at Fujitsu shown in the documents about whether the Post Office and/or SPMs should be told. I do not see how a thorough, professional and conscientious organisation can have produced for disclosure in this litigation so many thousands of KELs during 2019 itself, both during and even after the trial. I reject that description; it is an inaccurate description of Fujitsu and/or its investigative motivation.
936. I consider, as explained in the Technical Appendix, that Legacy Horizon was not robust, and that although Horizon Online in its HNG-X form was better than Legacy Horizon (not least, I consider, because Riposte was no longer part of Horizon) its robustness was questionable, and did not justify the confidence placed in it by the Post Office in terms of its accuracy. HNG-A is a different matter, and the experts are agreed that it is far more robust than Horizon in earlier times. On the face of the relevant KEL, it is not possible to say whether the Drop and Go bug, number 28 in the Bug Table, which occurred in June 2017, was HNG-X or HNG-A. However, if the latter, it is one of only two such items in the Bug Table that relate to the existing version of the Horizon system.
937. I turn therefore to the points identified by Males LJ in Simetra Global Assets Ltd and another v Ikon Finance Ltd and another [2019] EWCA Civ 1413 to which I have referred at [
74] above. I have already dealt with the first point, succinctness (obviously lacking in a judgment of this length). The second point is a requirement to consider all the evidence. I have done this, both for and against each of the cases advanced by both the claimants and the Post Office. I have not recited every single disputed fact as to do so would lead to an even longer judgment, and substantially delay production of this judgment well into 2020, and which would be contrary to the overriding objective. The third point is to identify the issues, consider the evidence in respect of the issues, and give reasons. For this trial, the Horizon Issues were agreed by the parties and approved by the court; the review of the evidence has been longer than is ideal, but I have considered it all, whether recited in this judgment, or the Technical Appendix, or not. The fourth point is somewhat salutary, namely that fairness requires that a judge should deal with apparently compelling evidence, where it exists, which is contrary to the conclusion which he proposes to reach.
938. In my judgment, there is no such evidence advanced by the Post Office. It is pointless to speculate on how such compelling evidence might have been presented, if there had been any. The evidence that has been advanced for the individual occurrences, for example by Mrs Van Den Bogerd, can be seen once tested in cross-examination to amount to unfounded assertion and, sometimes, an expressed desire, even in 2019, to go away and do more research. This is not apparently compelling evidence to the contrary to my conclusions. Additionally, the time for further research by the Post Office is long past. The Post Office’s approach to evidence, even despite their considerable resources which are being liberally deployed at considerable cost, amounts to attack and disparagement of the claimants individually and collectively, together with the wholly unsatisfactory evidence of Fujitsu personnel such as Mr Parker. The Post Office evidence also includes a very high level overview of Horizon by its expert which amounts to little more than a claim that it has worked quite or very well, most of the time. This is effectively what Dr Worden’s section 8 exercise consists of. Counsel for the Post Office at one point put to Mr Coyne that there was more likelihood of an SPM being hit by lightning than of having a bug, error or defect in Horizon cause a branch account loss in their branch. Mr Coyne did not agree with that, and neither do I. Indeed, such a point wholly ignores the enormous number of PEAKs that show that branch accounts were potentially impacted on a great many occasions.
939. Also relevant are the other points which Males LJ went on to consider, namely the importance of contemporaneous documents. He stated the following on this subject at [48] and [49] of Simetra:
(emphasis added)
941. They also contain some very telling expressions, not in the context of a Fujitsu employee giving evidence as a Post Office witness in this trial, but at the time, with such unguarded comments as Anne Chambers in February 2006 stating that “this problem has been around for years and affects a number of sites most weeks” and “this appears to be a genuine loss” on another occasion (as examples only). These entries are at odds to the publicly stated position by Fujitsu both then and indeed later. In my judgment, the PEAKs and KELs deployed in the Horizon Issues trial, and there were very many, are consistent with my conclusions on the evidence and the answers that I have arrived at to the Horizon Issues. Obviously no account was taken in any of the evidence, and no account is taken by me in this judgment, of the 5,000 KELs that Fujitsu only provided and disclosed in September 2019, months after the trial had ended.
942. The Post Office’s internal Horizon Online Induction Training slides and accompanying notes dated 7 December 2009 form a useful snapshot of how Legacy Horizon was viewed. The purpose of HNG-X was “to deliver a significant reduction in the total annual cost of ownership of Horizon, whilst ensuring the system remains fit for purpose in the 21st Century.” These materials make it clear that this programme had been identified in 2005 and was "to deliver a significant reduction in the total annual cost of ownership of Horizon". Legacy Horizon was seen within the Post Office as being very expensive, although its overall cost is not relevant to the Horizon Issues. The notes to these materials state:
“However, we need to continue to manage firmly any over-expectations of the frontline that Horizon Online will deliver improved functionality - they may see this as a missed opportunity so will not cure all the issues and problems that users have with Horizon although where practical, and at no extra cost, we have smoothed away a number of "rough edges".
(emphasis added)
There was no significant upgrade or improvement in functionality intended as part of HNG-X.
943. Another type of document that is illuminating in terms of the Horizon Issues are internal Fujitsu briefing documents, and to a lesser extent also some internal Post Office emails, that show the same approach to, or view of, Horizon within those organisations as are drawn by my conclusions. One example relates to the so-called Ping fix. After this was brought in, the number and value of TCs fell dramatically so far as Camelot/the lottery was concerned. This appears at [427] to [428] of the judgment on the Common Issues, Judgment (No.3). In 2007 there had been the following value of TCs in respect of the lottery: approximately (to 1 decimal point, which represents £100 thousand) £22.8 million; in 2008 there were £12.5 million; in 2009 there were £12.0 million; in 2010 this became £11.3 million; and in 2011 £4.5 million. In 2012, the year the Ping fix was introduced, these fell below £1 million for that year.
944. The Introduction to a Post Office document of 2011 entitled “Transaction Acknowledgements – End to End Incident Management and Data Flow” explains the following purpose to the Ping fix:
“Introduction
The PING Project was part of the Back Office Efficiency Programme within Post Office Ltd.
The objective of this project was to automate a process that converts Client transactional data files (e.g. Paystation), into a data feed to the branch, i.e. a Transaction Acknowledgement file.
The branch will then process the data electronically on Horizon Online and thus avoid non compliance issues and the resulting cost to the business to process queries as a result.
Please note - The term PING is an internal Post Office Ltd term and branches and helplines will know this new functionality as 'Transaction Acknowledgements'.”
(emphasis added)
945. The compliance issues were caused by the fact that the lottery terminals were on one computer system, namely that operated by Camelot, whereas Horizon was the system operated by the branch that was selling the Camelot products, and there was no electronic transfer of data in to Horizon in respect of Camelot sales. The “objective” in the Post Office document makes it clear that the Ping fix is a required improvement to the functionality, to overcome issues caused by having two different systems which meant that there was (prior to the Ping fix) no automated process to convert the client transactional data files into the data feed necessary for Horizon for the branch accounts. This has nothing to do with correcting excessive carelessness or fault on the part of SPMs. It is, in my judgment, about remedying a deficiency in the functionality of Horizon. That document also made it clear, because there are express entries to this effect, that outages might mean that the system would not deal with the matters sufficiently or accurately. A template message for SPMs is even included in the document for when this occurs, although that template message would not inform SPMs that outages would lead to inaccuracies. It is plain that outages would indeed lead to this, as this is mentioned in many places in the document, and acknowledged by its authors. This information was not however, based on the templates, to be given to SPMs.
946. A theme contained within some of the internal documents is an extreme sensitivity (seeming to verge, on occasion, to institutional paranoia) concerning any information that may throw doubt on the reputation of Horizon, or expose it to further scrutiny. One entry in a document that makes it clear that the Post Office itself had already recognised this is contained in a document authored by Mrs Van Den Bogerd, entitled “Extracts from Lessons Learned Log” and dated 11 November 2015. One entry under “issues identified” was as follows in respect of the Post Office’s behaviour up to that date:
"Failure to be open and honest when issues arise eg roll out of Horizon, HNGx migration issues/issues affecting few branches not seemingly publicised."
(emphasis added)
947. The Post Office’s guard may have been down when that document was written; it predates issue of the first claim form by about 6 months. I consider that it is the type of document that Males LJ had in mind when he spoke of a witness’ guard being down, when their true thoughts are plain to see. It plainly records Mrs Van Den Bogerd’s view of the Post Office’s approach to Horizon when it was written in November 2015.
948. In July 2016, following the disclosure of the Dalmellington bug, the following emails went to and from very senior people in the Post Office. On 1 July 2016 the Chief Executive, Paula Vennells stated to Alisdair Cameron and Rob Houghton, in the context of an article that had appeared on the internet about problems with Horizono:
“Subject: The Dalmellington Error in Horizon / problemswithpol
Dear both,
This needs looking into please.
https://problemswithpol.wordpress.com/2015/11/10/the-error-in- horizon/?iframe=true&theme_preview=true
[This is a reference to an internet post about a conviction of an SPM]
Can I have a report that takes the points in order and explains them.
Tim McCormack is campaigning against PO and Horizon. I had another note from him this am which Tom will forward, so you are both in the loop.
We must take him seriously and professionally.
This particular blog is independent of Sparrow but clearly related in that it appears to present similar challenges that were raised in the course of the scheme.
I'm most concerned that we/our suppliers appear to be very lax at handling £24k. And want to know we've rectified all the issues raised, if they happened as Tim explains.
Thanks.
Paula”
949. Mr Houghton of the Post Office sent it on to someone else on the same day, copied to Mrs Van Den Bogerd. This stated:
“I need an urgent review and mini <taskforce> on this one. It probably needs to link up heavily with Angela’s work as FSC are mentioned extensively - Angela cfi. I don't know how we respond to this but can we section a few inside people to get all over it and give me/Al/ Paula evidence and understanding.”
950. However, the same recipients also received the following message from Mr Houghton later on the very same day:
“Can you stand down on this please? [A redacted section then follows]
Any specific actions and I will revert.
My apologies.”
951. The email chain is heavily redacted and therefore the reasoning behind this volte face within the Post Office is not shown, nor was it explained by any Post Office witness.
952. Thus the entirely understandable initial reaction of the Chief Executive of the Post Office, that a point by point investigation was required in respect of specific criticisms of Horizon in respect of one particular SPM who blamed Horizon for shortfalls and discrepancies, appears to have been very swiftly shut down and not pursued due to a decision at what must have been very high levels within the Post Office. I can think of no justifiable reason why the Post Office, institutionally, would not want to address the Chief Executive’s points and investigate as she initially intended, and find out for itself the true situation of what had occurred.
953. Both the recipients of the Chief Executive’s email of 1 July 2016 above are specifically named in the Technology Strategy Update Decision Paper of 30 January 2017 in which the following is stated specifically. Mr Houghton is the Author of the paper, and Mr Al Cameron is “the Sponsor”. Part of that document records:
“This document forms an update to the IT Strategy approved in July 2016 by the PO Board. In July we outlined that IT was not fit for purpose, expensive and difficult to change.”
The same document also states that "We need to quickly rationalise and resolve misaligned contracts enacted to support legacy IT, obsolescence, a lack of PO technical competence, particular focus on Fujitsu and Accenture” .
(emphasis added)
954. This therefore means that the decision taken not to investigate further in July 2016 and to “stand down”, in itself a surprising decision for a reputable institution to take given all the circumstances, was taken at broadly the same time as a conclusion was reached that the IT “was not fit for purpose”. The reference to the “lack of PO technical competence” is a concern, given [75] of the Technical Appendix which makes it clear that currently the Post Office can introduce new clients into Horizon without requiring Fujitsu input. The level of competence may of course have changed since 2016. The IT Strategy, which is referred to in this Update Decision Paper, and outlined that “the IT was not fit for purpose” and which was approved in July 2016 by the Post Office Board is, in my judgment, entirely consistent with my conclusions on the evidence. It is not, however, consistent with the Post Office’s response to the claimants’ pre-action letter, or the Defence in the group litigation, or the evidence adduced by the Post Office in the Horizon Issues trial. The Post Office’s solicitors’ response to the pre-action letter is dated 28 July 2016, very lengthy, and states that “the investigations to date have consistently pointed towards human error or dishonest conduct in branches as the most likely cause of shortfalls.”
955. Nor is it consistent with the way the Post Office explained, in its oral opening submissions, that the Post Office had been investigating incidents earlier in terms of bugs. So far as Dalmellington is concerned, the oral submissions were as follows:
“….even the documents that your Lordship saw this morning with the Dalmellington bug, your Lordship will see the rigour that's applied. There's concern that postmasters aren't given advice that might be incorrect. The rigour associated with that process and the determination of Fujitsu and the other people involved, the other stakeholders, to get to the bottom of what happened is quite striking in my submission.”
956. Dalmellington was investigated in 2015, and the outcome of that investigation is available to this court. The outcome of that investigation is far from favourable to the Post Office’s position on bugs, errors and defects, and the robustness of Horizon. The oral submission that the Dalmellington bug was investigated rigorously (which it was) does not easily sit with the documents showing that one year later the Post Office specifically decided not to investigate other incidents at all, and to “stand down” those tasked with investigating it. This is notwithstanding that the Chief Executive’s initial (and in my judgment entirely proper, if not the only proper) reaction was that those allegations needed to be looked at “seriously and professionally”, that an investigation was needed, and that an urgent review and mini task force was required. Nor does it sit easily with the way that other bugs that were present in both Legacy Horizon and Horizon Online were investigated by Fujitsu.
957. Further, a conclusion in terms that the IT is “not fit for purpose” is not something that would have been reached lightly in an IT strategy document, or approved lightly by the Post Office Board. The Post Office Board is a serious level within the organisation. The Board are not likely to be involved in, nor to have brought to their attention, matters that are anything other than serious, considered and fully researched. This conclusion in this internal contemporaneous document has not driven my conclusions on the evidence in the slightest. Rather, it is the other way around. I have reached my conclusions on the evidence independent of contemporaneous references such as this one, but then tested my evidential conclusions against these to see if there is a contradiction. However, there is none, and they are entirely consistent.
958. There are only very isolated Post Office documents that are inconsistent with my conclusions. I shall deal with only one. On 17 August 2015 the BBC Panorama programme broadcast a programme which contained allegations about Horizon. The contemporaneous response to that by the Post Office in its statement to SPMs started with the following passages, which I shall reproduce. The bold emphasis was present in the original:
“The Post Office wholly rejects extremely serious allegations repeated in BBC’s Panorama programme of 17 August 2015. The allegations are based on partial, selective and misleading information.
· The Post Office does not prosecute people for making innocent mistakes and never has
· There is no evidence that faults with the computer system caused money to go missing at these Post Office branches
· There is evidence that user actions, including dishonest conduct, were responsible for missing money”
959. The first of those bullets point is for another day, and is not for this court in any event, for the reasons explained at [60] to [66] above. This judgment deals with the detailed background and technical evidence behind the second bullet point. The confidence expressed by the Post Office in 2015, in that second bullet point, in the computer system and its accuracy is not consistent with the evidence I have accepted, the expert evidence on the number of bugs in the Horizon system (both Legacy Horizon and HNG-X) and my findings. This Post Office document was however drafted for public consumption, and this must explain the inconsistency. To the extent that the second bullet point conflicts with my findings in this judgment, that public statement in 2015 by the Post Office is factually incorrect. This is not to make any findings on any individual claim, or on the cause of money said to be missing in the accounts of any particular branch Post Office or individual claimant SPM.
960. In my judgment, a number of both the Post Office’s own internal documents, and Fujitsu ones too, namely those that were not drafted for public consumption, plainly support my conclusions on the evidence. Further, certain matters that have emerged in the Horizon Issues trial – such as discussions within Fujitsu itself as to whether the Post Office should be told certain detrimental information about the Horizon system, and the Post Office’s own decision at the highest level not to investigate certain matters as recently as 2016 – are of great concern. The Post Office has gone to great lengths over the years, and spent a great deal of time and a huge amount of public money, in defending the performance of Horizon. It is also the case that the Post Office must have been reliant on Fujitsu to a certain degree in terms of being provided with accurate information of a technical nature. That is not only obvious from the evidence, but has also been agreed by the experts in the 3rd Joint Statement. That accuracy from Fujitsu has not always been available, as demonstrated by this judgment.
961. However, regardless of that – and this judgment does not deal with who, if anyone, at the Post Office knew precisely what about Horizon, and when, as these are not part of the Horizon Issues – in my judgment a number of the specific internal contemporaneous documents, rather than being contrary to my conclusions on the evidence, are entirely consistent with those conclusions. I do not know if the Post Office Board, who were told in July 2016 that its own IT was not fit for purpose, will be particularly surprised at the findings on the expert evidence in the Bug Table, and the number of admitted bugs. It does not much matter, for the purposes of resolving the Horizon Issues, whether they are surprised or not.
962. The experts’ agreements in particular have been of great assistance, but everything has been considered. This judgment, together with the Technical Appendix, is of substantial length, and to recite everything that has been deployed by both sides, in terms of evidence, submission and reference to all contemporaneous documents, is neither necessary nor desirable.
963. It must also be remembered that Horizon as it is today, or at least in the last couple of years since it became HNG-A, is a very different system to earlier times. Legacy Horizon ceased to be used in 2010. Modern Horizon Online as it is today is not the same as the online system that was introduced in 2010; it is different and more robust, on the agreed expert evidence, to the system as it was even when the litigation started with the issue of the first claim form on 11 April 2016. It became HNG-A in 2017, rather than HNG-X, and runs on a completely different Windows platform. The dates of the bugs in the Bug Table are (between them collectively) from 2000 to 2018, although there are far fewer in recent years. Having said that, some – such as Bug 8, Recovery Issues, and Bug 28, Drop and Go, have been experienced up to 2018 and 2017 respectively. The latter is a reference data bug and hence easily fixed. The Technical Appendix at [421] provides a summary of the different bugs from Legacy Horizon and Horizon Online. It can be seen that there were more in Legacy Horizon than HNG-X in any event, and far fewer in HNG-A. In any case, the cut-off date for claims to be issued in the group litigation is 24 November 2017, as extended by paragraph 15 of the 1st Case Management Order sealed on 27 October 2017.
964. The Post Office has been very concerned as to the outcome of this litigation on its wider business. It described the group litigation as long ago as its Opening Submissions in November 2018 for the Common Issues trial as representing “an existential threat” to that business. I do not consider that these answers to the Horizon Issues represent a significant threat to the Post Office’s entire business. Findings in this judgment as to the performance and robustness of Legacy Horizon from 2000 to 2010, and then of Horizon Online (in both its forms, HNG-X and HNG-A) from 2010 to 2018 are not findings on the Horizon system as it exists at the date of this judgment. These findings cannot be routinely applied to the way that HNG-A operates as at December 2019. It is agreed by the experts that the Horizon System in its HNG-A form is now relatively robust. This judgment is a historical analysis of the Horizon System as it relates to the period in question in the group litigation, not a judgment upon Horizon HNG-A as it is today.
M. Answers to the Horizon Issues
965. This section is intended as a very brief summary of what has gone before, both in this judgment and also the Technical Appendix. It does not replace the previous 964 paragraphs or that appendix. The detailed passages themselves should be consulted for a more detailed consideration of the issues.
966. I therefore turn to the answers to the Horizon Issues themselves. For convenience, I shall reproduce the specific issue above each answer, in order to avoid the reader having constantly to turn back some hundreds of pages to remind themselves. I shall also retain the headings and sub-headings. The wording of the Horizon Issues was agreed by the parties and approved by the court.
BUGS, ERRORS AND DEFECTS IN HORIZON
Accuracy and integrity of data
967. Issue (1): To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and 24 of the GPOC and referred to in §§49 to 56 of the Generic Defence to have the potential (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?
968. Answer: It was possible for bugs, errors or defects of the nature alleged by the claimants to have the potential both (a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, and also (b) to undermine the reliability of Horizon accurately to process and to record transactions as alleged by the claimants.
969. Further, all the evidence in the Horizon Issues trial shows not only was there the potential for this to occur, but it actually has happened, and on numerous occasions. This applies both to Legacy Horizon and also Horizon Online. It has happened under both the HNG-X and HNG-A iterations of the Online system, but far less frequently under the latter than the former. Indeed, there are only isolated instances of it happening in respect of HNG-A, which the experts agree is a better system than either of the other two iterations of Horizon.
970. I accept the claimants’ submissions that, in terms of likelihood, there was a significant and material risk on occasion of branch accounts being affected in the way alleged by the claimants by bugs, errors and defects. This is amply demonstrated by Dr Worden’s evidence. He accepted that there was strong evidence of at least 12 bugs causing a lasting discrepancy in branch accounts. This conclusion was reached even within the context of his unjustified creation of both “transient” and lasting impact, which limited his consideration of this, and is an approach which I have rejected. This number of bugs was very much smaller than the number contended for by Mr Coyne, and the number which I have found in Part E of the Technical Appendix, but even on that smaller number, was itself significant. Dr Worden’s reliance upon TCs as a countermeasure led to a failure by him to follow the agreed wording of the Horizon Issues, as TCs are not part of the Horizon system. Given TCs are part of the way that the Post Office corrects the branch accounts (and branch accounting statements) produced by Horizon, the only way that a fully precise answer could be given for the number of times such impacts have occurred would be if the figures were available for the years of operation of Horizon, demonstrating how many TCs each year had to be issued in respect of the effects on branch accounts of such discrepancies. Figures would also be required of the number of TCs that were upheld when disputed by SPMs.
971. However, those figures are simply not available for the majority of the life of both Legacy Horizon and Horizon Online. Nor is there any data centrally recorded by Fujitsu of the number of bugs, errors and defects which were discovered by dissatisfied SPMs reporting incidents to Fujitsu, with percentage outcomes of what the investigations by the SSC led to, in terms of result. Such data as there is, namely Mr Parker’s summation of closure codes of PEAKs, is wholly unreliable and I have rejected it. Nor are the closure codes that were used by Fujitsu accurate in any event. The evidence of fact was that full and accurate records of this nature are simply not kept by Fujitsu. Had they been, both experts could (and probably would) have used such data as a starting point for their investigations. The sheer scale of the number of TCs issued by the Post Office each year – which is over one hundred thousand for many of the years that are the subject of the group litigation – supports my conclusion that there was a significant and material risk of inaccuracy in branch accounts as a result of bugs, errors and defects in the Horizon System (both Legacy Horizon and HNG-X).
972. Issue (2): Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as described in (1) above and if so how?
973. Answer: Although the experts were agreed that the extent to which any IT system can automatically alert its users to bugs within the system itself is necessarily limited, and although Horizon has automated checks which would detect certain bugs, they were also agreed that there are types of bugs which would not be detected by such checks. Indeed, the evidence showed that some bugs lay undiscovered in the Horizon system for years. This issue is very easy, therefore, to answer. The correct answer is very short. The answer to Issue 12 is “No, the Horizon system did not alert SPMs”. The second part of the issue does not therefore arise.
974. Issue (3): To what extent and in what respects is the Horizon System “robust” and extremely unlikely to be the cause of shortfalls in branches?
975. Answer: This issue or question is in two parts. I shall deal with them sequentially. The first is the extent and in what respects the Horizon system is “robust”. The experts are agreed that HNG-A is relatively robust, and I have found that the system as it is in 2019 is far more robust than it was prior to 2017. The Technical Appendix in general, and [444] of that in particular, should be consulted for a more complete answer on robustness. However, the Horizon Issues considered the whole life of the system from 2000 to date. In summary terms only, Legacy Horizon was not remotely robust. The number, extent and type of impact of the numerous bugs, errors and defects that I have found in Legacy Horizon makes this clear. Further, Dr Worden considered that TCs were an essential countermeasure. If TCs are excluded from consideration of the Horizon system, which they must be, then the answer becomes even more stark, even on the evidence of the Post Office’s own expert.
976. HNG-X, the first iteration of Horizon Online, was slightly more robust than Legacy Horizon, but still had a significant number of bugs, errors and defects. This was particularly so in the years 2010 to 2015, but also after that. This is clearly demonstrated from the findings in the Technical Appendix. The functionality of Legacy Horizon was not therefore materially improved when the system moved to the online version, although one of the major differences between Legacy Horizon and HNG-X was the former’s use of Riposte (which was not used at all in HNG-X). I find that the robustness of HNG-X was questionable and did not justify the confidence routinely stated by the Post Office (prior to February 2017) in its accuracy. HNG-A is far more robust than either of the previous two iterations of the system. I have taken into account, in reaching these conclusions, not only the evidence of bugs, errors and defects, but also the lack of records or logs at Fujitsu over time over the use and controls of the powerful access roles permitted to Fujitsu personnel. This deficiency in reporting and controls seems to have been corrected in about 2017. This adds to my conclusion (and that of the experts) that HNG-A is more robust than HNG-X. As the experts agreed, this increase in robustness has, in part, developed from Post Office discovering bugs, errors and defects in live use and then applying fixes and improving monitoring. This is important, as it means that the robustness today of HNG-A cannot be relied upon by the Post Office in justifying, or being equated to, robustness in HNG-X. The same point applies to HNG-X and Legacy Horizon.
977. I accept Mr Coyne’s evidence that there are far more TCs than expected even now, compared with comparable systems in the banking and finance sectors. However, it may be that this is the only way in which it can now sensibly be operated. TCs are outside the scope of the definition of the Horizon system in any event. The experts are agreed that HNG-A is robust, and far better than it was in either its Legacy Horizon or HNG-X form.
978. Turning to the second part of the issue, namely to what extent and in what respects is the Horizon system extremely unlikely to be the cause of shortfalls in branches. I accept that this presupposes a shortfall in a branch, and then requires consideration of whether Horizon is extremely unlikely to be cause of that shortfall. By shortfall, this issue means an unexplained discrepancy of the type explained by the claimants’ witnesses, and also recorded in numerous PEAKs. In my judgment, there is a material risk that such a shortfall in a branch’s accounts was caused by the Horizon system during the years when both Legacy Horizon and HNG-X were in use, which is 2000 to 2010 and 2010 to 2017 respectively. For each of those iterations of the system, the Horizon system was therefore not “extremely unlikely” to be the cause of shortfalls in branches, as contended for by the Post Office. I realise that is a double negative, but I consider wording the answers as closely as possible to the agreed issues is likely to be of greater assistance to the parties.
Controls and measures for preventing / fixing bugs and developing the system
979. Issue (4): To what extent has there been potential for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?
980. Answer: The experts were agreed that there are a number of actual reported errors in data recorded within Horizon arising from each and all of (a) data entry, (b) transfer or (c) processing of data. Therefore, not only does the potential for these errors exist, but this has in fact happened, and this has led to financial discrepancies in branch accounts. The experts were also agreed that bugs, errors and defects identified in relation to Horizon Issue 1 were often relevant to Horizon Issue 4 in that they are ultimately errors arising from the processing of data in Horizon.
981. The “extent” to which this has occurred has not been fully assessed by the experts. However, Dr Worden considered this issue to be a subset of Horizon Issue 3, in respect of which I have effectively or substantially accepted the claimants’ case, at least so far as Legacy Horizon and HNG-X are concerned.
982. The experts do not agree as to its ‘extent’ in terms of actual effect, although that is not what the issue asks. It was agreed by the experts that there was evidence within the PEAKs and KELs of bugs/errors/defects within Horizon arising from parts (a), (b) and (c) of this issue that occurred that caused financial discrepancies, as well as some that occurred without causing financial discrepancies. It was also agreed that reference data is critical to the operation of Horizon. I have explained in the Technical Appendix what reference data is, and the important part it plays both in the software and operation of the system. It is agreed by the experts that errors in reference data have led to discrepancies in branch accounts. It has also been agreed by the experts that errors in third-party data have led to discrepancies in branch accounts. The Post Office does not have control over errors in third-party data, but given these have an impact upon branch accounts, unless they are corrected, then those errors will also have an adverse effect upon a SPM’s branch accounts.
983. I do not consider that the extent of the actual effect can be fully answered in precise mathematical terms on the evidence as it is. This is not least because in order to do so, the experts would have to consider the full data set of KELs, and several thousand KELs were produced by Fujitsu during, and well after, the trial. However, a precise mathematical answer is not required. The answer to the issue as posed is that there is a material risk for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data in Horizon in both the Legacy Horizon and HNG-X forms.
984. Issue (5): How, if at all, does the Horizon system itself compare transaction data recorded by Horizon against transaction data from sources outside of Horizon?
985. Answer: This is largely automated. It has not however been automated throughout the whole life of Horizon from 2000, as shown by the Ping fix in respect of the Lottery/Camelot. The experts’ agreement on reconciliation states inter alia that “Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's clients is largely automated. Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/Error Notices to the” SPMs. What this means is that if the data in Horizon was wrong (due to bugs, errors or defects within Horizon) and did not reconcile with third-party data, then “manual corrective fixes” and the issuing of TCs would be undertaken, which would not be automatic.
986. Some comparison of transaction data from some clients of the Post Office is done within the Horizon system and other client data is compared outside the Horizon system. Again, the evidence did not deal in a comprehensive way with each such method for every one of the Post Office’s many hundreds of clients. Nor do I consider that such evidence would be required properly to answer this issue.
987. Issue (6): To what extent did measures and/or controls that existed in Horizon prevent, detect, identify, report or reduce to an extremely low level of risk the following:
a. data entry errors;
b. data packet or system level errors (including data processing, effecting, and recording the same);
c. a failure to detect, correct and remedy software coding errors or bugs;
d. errors in the transmission, replication and storage of transaction record data; and
e. the data stored in the central data centre not being an accurate record of transactions entered on branch terminals?
988. Answer: The experts were agreed that there were many measures and controls within Horizon that existed to prevent, detect, identify report or reduce the risk of varying errors. However, the experts were also agreed that whilst Horizon contains measures and controls for detecting system integrity concerns, these automatic mechanisms have been shown to have failed in the past. The experts did not agree as to the ‘extent’ of prevention, detection, identification, reporting or risk reduction that the automatic and manual control measures delivered. There were numerous specific matters explored in some detail in the trial that clearly demonstrated such failures, such as the Callendar Square bug, the Receipts and Payments Mismatch bug, gaps and duplicates in entries in the audit store, keying and mis-keying errors (which were referred to in different PEAKs) and fixes which introduced other issues. The doubling of the foreign currency discrepancy that was put to Mr Godeseth was a specifically stark example, even though Fujitsu – inexplicably in my judgment – chose to attribute this to user error.
989. I do not consider that the measures and/or controls that existed in Legacy Horizon, or HNG-X prevented, detected, identified, reported or reduced to an extremely low level of risk any of the sub-items at (a) to (e). Indeed, I reject the Post Office’s case that there was such an extremely low level of risk. That is, in my judgment, a wholly complacent and unjustified position that has existed for many years, based on my findings in the Technical Appendix.
OPERATION OF HORIZON
Remote Access
990. Issue (7): Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon remotely (i.e. not from within a branch)?
991. Answer: Yes. This answer is substantially agreed. Both the Post Office and Fujitsu could read data remotely. Fujitsu could access all transaction data recorded by Horizon. The experts agreed that Fujitsu needed remote access for support purposes. Fujistu has direct access to the branch database. In Legacy Horizon, this was the messagestores held on the correspondence servers; in Horizon Online, it is the BRDB. In Horizon Online, Fujitsu can also download logs stored on the counters. The Post Office has access to transaction data which is copied to its Management Information System or MIS. In Horizon Online, the Post Office has read-only access to branch data on the BRDB which includes the levels of cash at a particular branch.
Availability of Information and Report Writing
992. Issue (8): What transaction data and reporting functions were available through Horizon to Post Office for identifying the occurrence of alleged shortfalls and the causes of alleged shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the Horizon system?
993. Answer: This issue is substantially agreed. The Post Office had access to data and systems that were not available to SPMs. The role of the Transaction Processing System (or TPS) is to gather the transactions taking place in the branches, and to pass them on both to Horizon’s back-end systems such as APS and DRS. These would also be passed on to other IT systems in the Post Office, such as Credence and a succession of systems based on SAP, which have finally culminated in what is called POLSAP. The Post Office therefore has a range of Horizon reports and its own management information systems (MIS). Dr Worden stated that he expected the MIS to include analytical facilities which were not required by, and so were not available to, Subpostmasters. None of this was controversial by the end of the trial, or in issue between the experts.
994. The Post Office would also be able to use standard database reporting tools to retrieve and analyse information about branch transactions. It could also obtain audit data as explained in Part K of this judgment. A request had to be made to Fujitsu for this audit data, however. The Post Office could not simply access such audit data itself without recourse to Fujitsu, who had a charging structure in place in respect of such requests.
995. Importantly in my judgment, the only source of actual key stroke information (in the sense of what buttons had been pressed in the branch) was to be found within the audit data. Credence does not provide this. I also consider that although the Post Office did have access to “the causes of alleged shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the Horizon system”, this had to be obtained through Fujitsu. This should be recorded for completeness. The evidence does not show the Post Office IT department either being capable of investigating directly itself, or if it was capable of doing so, actually undertaking this. All investigations recorded in PEAKs and KELs show that Fujitsu did this. The terms of the experts’ agreement in this respect was that the Post Office was “reliant upon Fujitsu for diagnosis of whether the occurrence of shortfalls was caused by bugs/errors or defects within the Horizon system.” I find that the Post Office plainly was reliant upon Fujitsu in this way.
996. Issue (9): At all material times, what transaction data and reporting functions (if any) were available through Horizon to Subpostmasters for:
a. identifying apparent or alleged discrepancies and shortfalls and/or the causes of the same; and
b. accessing and identifying transactions recorded on Horizon?
997. Answer: The answer to this Issue is also substantially agreed. The experts agreed that the causes of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or transaction data available to SPMs. Other causes of apparent or alleged discrepancies and shortfalls may be more difficult, or impossible, to identify from reports or transaction data available to the SPMs, because of their limited knowledge of the complex back-end systems. I would add that the subject matter of every PEAK referred to in the Bug Table, and the sort of problems that initiated the creation of each of those PEAKs, plainly could not be investigated by the SPM or SPMs in question in each case on the material that Horizon made available to those SPMs.
998. In Judgment No.3 I had identified certain features or characteristics of both the SPMC and the NTC, the contracts under which SPMs were engaged (dependent upon the dates of their engagement). The experts were not aware of these contractual findings when they reached any of their expert agreements, as that judgment was not provided even in draft until just before the Horizon Issues trial started. It was handed down formally at the end of the first week of factual evidence. It is of interest, I consider, that the two experts considered at least the essence of one of the implied terms (although not in contractual terms) in their agreement that “identification of the causes of apparent or alleged discrepancies required the co-operation of PO staff and SPMs”. Co-operation is an important part of any investigation, as it is an important part of the contractual relationship between the Post Office and SPMs.
999. SPMs did have access to many reports relating to the different aspects of running a branch (such as currency exchange, stock and so on). They also had access to transaction logs that would show individual transactions, and event logs, but these were only available for a few weeks (initially this was for 6 weeks or 42 days, then this changed to 60 days). It was possible for a SPM to print these out on paper till rolls. There was evidence in the Common Issues trial from both claimant SPMs and Post Office witnesses about how unwieldy and non-user friendly this was. Indeed, in that earlier trial one of the Post Office witnesses used the size of the Super Court in the Rolls Building (where both the Common Issues trial and the Horizon Issues trial took place) graphically to demonstrate how long such a till roll would be even just for a day in a busy branch.
1000. Because the reports and data available to SPMs were so limited, their ability to investigate was itself similarly limited. The expert agreement to which I refer at [998] above makes it clear in IT terms (based on the transaction data and reporting functions available to SPMs) that SPMs simply could not identify apparent or alleged discrepancies and shortfalls, their causes, nor access or properly identify transactions recorded on Horizon, themselves. They required the co-operation of the Post Office.
Access to and/or Editing of Transactions and Branch Accounts
1001. Issue (10): Whether the Defendant and/or Fujitsu have had the ability/facility to (i) insert, inject, edit or delete transaction data or data in branch accounts; (ii) implement fixes in Horizon that had the potential to affect transaction data or data in branch accounts; or (iii) rebuild branch transaction data:
a. at all;
b. without the knowledge of the Subpostmaster in question; and
c. without the consent of the Subpostmaster in question.
1002. Answer: I shall answer this by reference firstly to Fujitsu, and then to the Post Office.
1004. The injecting of transactions into branch accounts by Fujitsu, and the visibility of these (as in whether the SPM could tell these had been done in this way, due to the use of a counter position greater than 32) was very controversial prior to the trial. By the time the Fujitsu witnesses were called, they had accepted that what Mr Roll had said was correct, and Mr Godeseth confirmed this would look as though the SPM had done it.
1005. The Post Office: this could only be done by the Post Office utilising its Global User role. This permitted the Post Office to inject transactions without the knowledge or consent of the SPM. However, it could only be done by someone who was physically present in the branch, as that is the only way Global User could be used by the Post Office. The answer therefore for the Post Office is “yes” only in respect of that part (relating to injecting transactions) of (i)(a) (b) and (c). With that exception, the answer for the Post Office is “no” to the other eight permutations. The qualifying statement has to be made that whoever at the Post Office was exercising Global User rights would have to be physically present in the branch. This would therefore be likely to mean that a SPM or their assistant would at least know someone was in their branch doing something in terms of access to their branch accounts. A description of “Global User” is provided at [674] above. The Post Office’s contended for answer to this Issue was “No” to all nine permutations so far as the Post Office (rather than Fujitsu) is concerned. This is wrong, and ignores the existence of the Global User role.
1006. The Balancing Transaction Tool, and how many times it was used, occupied a certain amount of time at the trial, as its existence had arisen when the experts were preparing their reports and had led to inter-solicitor correspondence. The majority of the factual evidence on this is dealt with in the section of this judgment dealing with Mr Godeseth’s evidence. This tool enabled Fujitsu specifically to insert a transaction into branch accounts. The one documented occasion of its use was done with the consent of the SPM in question. However, in my judgment, this issue was a sideshow. This is because the design intent in the Low Level Design document in relation to this tool had plainly been overtaken, over time, as the numerous other times it was used showed, based on the messages in fact inserted. Secondly, and more importantly, the very powerful roles identified at [1003] above enable those with such roles to do far more than someone simply using the Tool.
1007. Issue (11): If they did, did the Horizon system have any permission controls upon the use of the above facility, and did the system maintain a log of such actions and such permission controls?
1008. Answer: Given the answers to Issue 10 above, this issue does arise and requires answering. There were permission controls, but the roles were very wide and in my judgment they were not controlled. This included but is not limited to the lack of any proper logs. Although at an earlier stage in the trial, the Balancing Transaction tool was focused upon, it emerged that there were other roles and tools available to Fujitsu in addition to the Balancing Transaction tool which enabled Fujitsu to achieve the same result, namely injection and alteration of transaction data. Indeed, not only were these roles (such as APPSUP and the other roles identified in the answer to Issue 10 above) more powerful, but they were widely available and in my judgment effectively unaudited. The whole of SSC had the APPSUP role for many years, and internal Fujitsu documents recorded that they were not supposed to have that role. There were a large number of personnel within SSC. Whilst some Privileged User Logs do exist, these logs did not exist prior to 2009 at all. Prior to July 2015, they only recorded when a user logged on and logged off, with no recording of what they were doing. This is a plainly inadequate record. Even after July 2015, the experts agree that such logs are not a useful source of evidence about remote access, due to their content (more accurately, this should be their lack of content). These deficiencies in the control environment were identified as long ago as before the audit report by E&Y in 2011, such that the deficiencies were recorded as being in existence in 2011. Even so, this was not promptly corrected.
1009. OCPs do evidence documentation of remote alteration of transactional and other branch data. However, on the evidence available at the trial, these OCPs constitute what could be described as a piecemeal and ad hoc process, and a not insubstantial percentage were issued retrospectively. Further, in many cases there is evidence that the person raising the OCP was the person who was to monitor it. In my judgment, and given the powerful nature of the roles, this method of control is inadequate.
1010. The view of Dr Worden, the Post Office’s own IT expert, who as I have identified is a man of considerable experience in the IT world stretching over decades, is that the number of Privileged Users and SSC users who can create a Balancing Transaction seems high. The lack of control of these roles, the recording of access, the deficiencies in the logs, and the concerns expressed by E&Y, demonstrate in my judgment an inadequacy of controls. Roles were not properly controlled even after concerns were raised by E&Y in the 2011 Audit, particularly in respect of the APPSUP role.
1011. The answer therefore is that there were permission controls; they were however inadequate. Logs were maintained but only after 2009. From 2009 to July 2015 they did not record actions, only whether a user had logged on and logged off. Even after July 2015, the experts agree that such logs are not a useful source of evidence about remote access, due to their content (more accurately, this should be their lack of content). In my judgment, this amounts to a deficiency in controls.
1012. Issue (12): If the Defendant and/or Fujitsu did have such ability, how often was that used, if at all?
1013. Answer: Due to the nature of Fujitsu’s record keeping, the experts could not provide any confident evidence on this subject of frequency. Mr Coyne established that the APPSUP role was used 2,175 times between 2009 and 2019. Dr Worden confirmed that this was consistent with his review of the Privileged User logs and he equated this to about one person using this role each working day during that period. The Fujitsu witnesses stated this would be in a “support role” but very little detail is available about what this was, or why such regular access of someone with such a powerful role was required. In any event, the Fujitsu witnesses of fact had not reviewed the Privilege User Logs in any detail.
1014. Indeed, the lack of evidence provided in respect of this issue supports and corroborates my conclusions on Horizon Issue 11 which precedes it. The inability of the Post Office, and/or the experts for both parties, to be able to provide a clear and precise answer to what is, in essence, a very simple question (even though the actual answer might be a large number) directly arises from Fujitsu’s plainly inadequate records. On the evidence from the two experts, the answer should be in approximate terms only and would be about once a day over the whole life of the system, to the extent that any records are available.
1015. Issue (13): To what extent did use of any such facility have the potential to affect the reliability of the Branches’ accounting positions?
1016. Answer: The design abilities of the roles to which I have referred was very wide – as a single example only, the experts agreed that anyone with the APPSUP role could pretty much do whatever they wanted. The answer to this issue is therefore that these facilities had the potential to affect the reliability of a SPM’s branch accounts to a material extent. Further, the evidence shows clearly that there were instances when this in fact occurred, which goes wider than the issue posed (which asks about potential) but which I find is also relevant to the Horizon Issues as a whole.
Branch trading statements, making good and disputing shortfalls
1017. Before I turn to Horizon Issues 14 and 15, there is a point that is relevant to the group litigation as a whole that can conveniently be made at this stage. Firstly, the answers to both of the remaining two Horizon Issues are largely agreed by the experts. This means, almost by definition, that the subject matter of those issues ought not to be controversial.
1018. However, the fact that a substantial part of Issue 14, in particular, was so controversial for so long is, in my judgment, notable. The Post Office ought to have known how its own system works. It is agreed by the experts that an SPM cannot record a dispute on Horizon at all. Indeed, this point was so very clearly conceded by Mr Godeseth when he was asked about it in cross examination that it was remarkable. He explained that there was no “dispute” button on Horizon, and that this was something expressly and deliberately done by design. Yet there were two substantial points in issue in the Common Issues trial on this very same subject. One was the ability of a SPM to dispute the figures shown on Horizon. The second was whether, in law, a branch trading statement had the effect of what is called a settled account between agent and principal (which would mean that certain legal consequences would flow if it were). Judgment No.3 found in favour of the claimants, and against the Post Office, on both those points. It is therefore more than a considerable surprise to see that, as a matter of evidence in the later Horizon Issues trial, those points should never have been put in issue by the Post Office at all.
1019. In complex litigation, it is important that points that are not, in reality, contentious, are not put in issue if there is no proper factual basis for doing so. If, as a matter of reality, any particular point is not controversial, putting it in issue will simply increase time and costs. A SPM cannot be criticised for “accepting” an entry in their accounts if they have no choice but to accept that entry. Nor can a branch trading statement be given the status of a freely agreed (and therefore settled) account if it is no such thing. I wish to remind the parties to this litigation that points that should not be in issue ought to be conceded at a sufficiently early stage, such that time and legal costs are not spent in argument on an erroneous factual basis.
1020. Issue (14): How (if at all) does the Horizon system and its functionality:
a. enable Subpostmasters to compare the stock and cash in branch against the stock and cash indicated on Horizon?
b. enable or require Subpostmasters to decide how to deal with, dispute, accept or make good alleged discrepancy by (i) providing his or her own personal funds or (ii) settling centrally?
c. record and reflect the consequence of raising a dispute on an alleged discrepancy, on Horizon Branch account data and, in particular:
(i) does raising a dispute with the helpline cause a block to be placed on the value of an alleged shortfall; and
(ii) is that recorded on the Horizon system as a debt due to Post Office?
d. enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch Trading Statement after 2005?
e. enable or require Subpostmasters to continue to trade if they did not complete a Branch Trading Statement; and if so, on what basis and with what consequences on the Horizon system?
1021. Answer: This is substantially agreed by the experts. The Horizon system can compare cash and stock figures, but it has no facility to record a dispute. The comparison is done by the system comparing its electronic records of what cash and stock is held in a branch, with the corresponding figures inputted by the SPM at the end of a trading period.
1022. The functionality of the system is as follows. The SPM inputs, at the end of each trading period, the relevant figures for the cash and stock that are physically present. The Horizon system identifies any discrepancy and the system invites the SPM to transfer the amount of that discrepancy into the local suspense account and continue “to roll over”, or to discontinue this operation. Rolling over is required before the next trading period can commence. For discrepancies of less than £150, the SPM has to “make good” that amount, by adding money to the till (if it is negative) or removing it (if positive). For negative sums in excess of £150, the SPM has the choice of making good (by adding the money) or “settling centrally” which basically just means seeking time from the Post Office to pay. This is extensively explained in the Common Issues judgment, Judgment No.3. This is not done as part of the Horizon system. A slightly different process was available prior to 2005 which was called cash accounting, but the process was broadly the same, although “settling centrally” was not available under cash accounting. Making good would cause the actual cash position to increase by the amount put into the till from the SPM’s own resources. If, say, there was a negative discrepancy of £100 between the actual cash position (the cash physically present in the till), and the derived cash position (shown on Horizon) then making good would lead to the figures being the same. The meaning of “settling centrally” simply means paying the amount to the Post Office over time (or having it deducted by the Post Office from the remuneration paid to the SPM).
1023. A SPM had to do this in order to “roll over” to the next accounting period. This means that a branch trading statement had to be completed if the SPM wished to open the branch for the next trading period. The branch trading statement had to be done on the figures for cash and stock shown on the Horizon system. There was extensive evidence of fact in both the Common Issues trial and the Horizon Issues trial that this had to be done. Unless it was, the branch could not trade in the next trading period.
1024. So far as disputed amounts are concerned, an SPM cannot dispute a discrepancy or any figure on Horizon, or record on Horizon that they have raised a dispute. Horizon records figures even when they are not correct, and there is evidence of figures on Horizon being known to be incorrect by the SPM and the Post Office, or both. A dispute is normally raised by a SPM through contacting the helpline, which is a telephone service which was widely referred to in the Horizon Issues trial. There is a finding in Judgment No.3 that by contacting the helpline an SPM was registering a dispute with the Post Office. The helpline and its function are however outside the scope of the Horizon Issues, and the helpline does not form part of the Horizon system.
Transaction Corrections
1025. Issue (15): How did Horizon process and/or record Transaction Corrections?
1026. Answer: This too is substantially agreed in terms of technical content, and I have explained in the body of the accompanying judgment, that TCs are outside the definition of the Horizon system.
1027. The creating of TCs is a manual process. Dr Worden relied upon TCs as underpinning his conclusion on robustness of the Horizon system. The creation of TCs is no part of the Horizon system. Without TCs, which are very numerous, the Post Office system of accounting with its SPMs would be even more problematic that it has been. There have been over 100,000 TCs issued each year since 2006. This is more than 2,000 per week.
1028. The Post Office has a central accounting function that comes to a decision that an adjustment has to be made to a branch’s accounts, as the branch transaction data (which comes from Horizon) is not consistent with the data the Post Office has received from its clients or suppliers. This adjustment is the result of a human process and is made by way of the issuing of TCs to a particular branch. This is done without the involvement of SPMs and the decision to do so is taken outside the Horizon system. In the early days of Horizon they were called Error Notices.
1029. A TC is then sent by the Post Office to the SPM and the SPM processes such TCs by accepting them. By that acceptance, the correction that is the subject of the TC will enter the branch accounts on Horizon. Disputes to TCs are done by the SPM contacting the helpline and again this is outside the Horizon system, which does not record that dispute. If the dispute is upheld (as in, the original TC is accepted by the Post Office as having been wrongly issued) then another TC will be issued to correct it. That subsequent TC will correct the effect of the first TC in the branch accounts when it is accepted by the SPM. The issuing of the subsequent TC is also done outside the Horizon system.
1030. The Post Office does not have comprehensive records of how many TCs that have been challenged by SPMs have been upheld. I have provided at [859] and [860] above an explanation of part of the 3rd experts’ joint statement in respect of TCs, and the percentage of TCs relating to Santander that the Post Office said were upheld. 77% of these were upheld; which plainly means that 23%, or 1 in 4, of those disputed were wrongly issued. I will also record here for completeness (although they are not TCs) that Transaction Acknowledgments or TAs are also used for making corrections to branch accounts. To use the Lottery and Camelot as an example, the issuing of TAs is now automated following the so-called Ping fix. However, a SPM has no option whether to accept TAs or not; they have to be accepted in branch. This was agreed by a number of the Post Office’s factual witnesses.
Appendices:2. Summary of bugs, errors and defects 3. Glossary |