England and Wales High Court (Technology and Construction Court) Decisions

IN THE HIGH COURT OF JUSTICE
BUSINESS AND PROPERTY COURT OF ENGLAND AND WALES
TECHNOLOGY AND CONSTRUCTION COURT (QB)

		Royal Courts of Justice 7 Rolls Building, Fetter Lane, London, EC4A 1NL
		17th January 2022

COVID-19: This judgment was handed down remotely by circulation to the parties' representatives by email. It will also be released for publication on BAILII and other websites. The date and time of hand-down was 10.00am on 17th January 2022.

Mr. Justice Eyre:

1. These proceedings arise out of the Defendants' procurement of a new supercomputer for use by the Second Defendant. The Claimant was an unsuccessful tenderer in that process and alleges that there were breaches of the Defendants' obligations under the Public Contract Regulations 2015.
2. By its current application the Claimant seeks permission for expert evidence in the field of high performance computing and identifies seventeen issues (set out in the Annex to this judgment) which it says should be the subject of such evidence. The Defendants accept that some evidence in that field is appropriate but disagree with the Claimant as to the extent to which such evidence is necessary and the issues in respect of which it should be allowed.
3. The project to provide a new supercomputer for the Second Defendant is rightly characterised by the Claimant as being exceptional in both its value and scope and the following is inevitably a somewhat condensed description of the process and of the issues between the parties.
4. The new supercomputer is expected to be the most advanced supercomputer in the world which is dedicated to weather prediction and climate change. The estimated contract value was £854m and the project as a whole is supported by £1.2billion of government funding.
5. The submission of initial proposals had been followed by two competitive dialogue stages and then by the submission of the final tenders. Package B of the Invitation to Participate in the Phase Two Dialogue Stage ("the ITP") had set out the Defendants' requirements for the supercomputer. At B.2 the Defendants had said that the two test supercomputer systems and the development supercomputer system to be supplied were to be "architecturally equivalent" to the main supercomputer system. There is a dispute as to the interpretation of section B2.5. In very short terms the issue is whether the detailed requirements set out in the third and fourth sentences of that section were additional to the requirement of architectural equivalence stated in the second sentence or were a definition of that requirement.
6. The Claimant was one of only two tenderers who submitted a final tender. However, the Defendants found the Claimant's tender to be non-compliant with the stated requirements. This resulted from the Claimant's tender being scored at 0/5 in each of three categories on the basis that the proposed development supercomputer system was not architecturally equivalent to the main supercomputer system.
7. The Claimant challenges that aspect of the procurement. It says that there were manifest errors in the evaluation of its tender and in the finding that the proposed development supercomputer was not architecturally equivalent to the main supercomputer. Alternatively it is said that the Defendants' interpretation of the requirement of architectural equivalence was contrary to the obligation under the Regulations for the procurement to be conducted with transparency and consistency. The Defendants, it is alleged, either made the decision on the basis of undisclosed requirements or interpreted the requirement of architectural equivalence in a way which would not be transparent to the reasonably well-informed and normally diligent tenderer. It is also said that it was disproportionate for the Defendants to determine that the Claimant's tender was non-compliant rather than to seek further clarification.
8. The Defendants deny liability saying that the requirement of architectural equivalence was interpreted correctly; that the Claimant's tender was scored correctly because the proposed development supercomputer did lack the required architectural equivalence to the main supercomputer; and that the conclusion that the tender was non-compliant did not involve either error or any other breach of the Regulations.
9. A nine-day trial limited to the issues of liability, causation, and sufficiently serious breach is listed for 9^th May 2022. At that trial the court will have to determine the correct interpretation of the concept of architectural equivalence in the context of: the ITP, the tender process as a whole, and the understanding of the RWIND tenderer. It will then have to determine whether, in the light of that interpretation, there was manifest error in the finding that the Claimant's tender was non-compliant by reason of a lack of architectural equivalence between the development supercomputer and the main supercomputer.
10. It is common ground that high performance computing is a recognised field of expertise. It is also common ground that the court will need some assistance by way of expert evidence. This is clearly right and I note that in her judgment following the case management conference O'Farrell J said "there may be some limited expert evidence; I do not underestimate the technical issues involved in the claim, given its subject matter." Similarly, at the cost-budgeting hearing Fraser J accepted that this is "a highly technical field". There is no dispute that expert evidence will be needed to enable the court to understand the technical background. The Claimant characterises this as glossary evidence and the dispute before me was essentially as to the extent to which expert evidence going beyond such explanation of technical terms and the like was appropriate.

The Principles to be Applied.

11. The parties are agreed that the starting point is Coulson J's summary of the applicable principles as set out in BY Development & others v Covent Garden Market Authority [2012] EWHC 2546 (TCC).
12. The claimants in that case challenged the procurement process undertaken by the defendant in relation to the redevelopment of the New Covent Garden Market. The claimants had been unsuccessful tenderers and asserted that the defendant's assessment of their bid had contained a number of manifest errors, alternatively that the decision to award the contract to a rival bidder was unfair or the result of unequal treatment of their bid. Coulson J was determining the claimants' application for permission to adduce expert evidence in two fields. First, expert planning evidence as to the degree of planning risk involved in the claimant's planning strategy and as to the compliance of the claimant's tender with published planning policies and related planning matters. Second, expert financial evidence as to the financial risks in the competing bids.
13. Coulson J emphasised the limited role of the court when determining a procurement challenge made on the basis of manifest error or unfairness. He explained, at [8], that:

"[the court] is reviewing the decision solely to see whether or not there was a manifest error and/or whether the process was in some way unfair. The court is not undertaking a comprehensive review of the tender evaluation process; neither is it substituting its own view as to the merits or otherwise of the rival bids for that already reached by the public body."

14. The judge then noted, at [11] - [14], the approach taken in judicial review cases where it is "very rare" for expert evidence to be admissible in such cases. Coulson J took account of the distinction which Collins J had drawn in R (on the application of Lynch) v General Dental Council [2003] EWHC 2987 (Admin) between expert evidence which sought "to explain what is involved in a particular process and how complicated that process is" and expert evidence which sought to "opine that it was irrational for the public body to reach the conclusion it did". The former might be admitted in very rare cases "to explain the technical terms and concepts" but the latter "would involve an illegitimate usurpation of the court's function" and was inadmissible.
15. Coulson J noted that expert evidence had been admitted in the case of Henry Bros (Magherafelt) Ltd v Dept for Education for Northern Ireland [2011] NICA 59 but explained that the evidence in question there "went solely to the applicability of one of the relevant criteria, against which the bids were considered, rather than any wider issues concerning the tender process as a whole."
16. The principles to be derived from the authorities were summarised thus at [20] and [21]:

"20 In summary, I consider that the authorities demonstrate that, where the issues are concerned with manifest error or unfairness, expert evidence will not generally be admissible or relevant in judicial review or procurement cases. That is in part because the court is carrying out a limited review of the decision reached by the relevant public body and is not substituting its own view for that previously reached; in part because the public body is likely either to be made up of experts or will have taken expert advice itself in reaching the decision; and in part because such evidence may usurp the court's function.

21. All of that said, however, I believe that it goes too far to say that expert evidence can never be admissible in public procurement cases concerned with manifest error. In some cases, it may be required by way of technical explanatory evidence (Lynch). In addition, there may be other cases where, unusually, such evidence is both relevant and necessary to allow the court to reach a conclusion on manifest error. That may be particularly so where the particular issue is specific and discrete, such as a debate about one of the criteria used in the evaluation (Henry Bros) or complex issues of causation (Harmon). .."

17. In the light of those principles Coulson J, at [22], formulated the question to be addressed when considering the admission of expert evidence as:

"Is this a claim where the technical background is so complex that explanatory expert evidence is required, and/or is this an unusual case where expert evidence on some or all aspects of the tender evaluation process is required in order to allow the court to reach a proper view on the issues of manifest error or unfairness?"

18. In answering that question Coulson J found that the proposed expert evidence was not admissible. He characterised the questions which had been postulated for the experts as relating to "elements of the evaluation itself" rather than addressing "background or technical issues" (at [24]). That characterisation did not mean that the expert evidence was necessarily inadmissible but did mean, at [27], that the question became one of:

"whether, despite the general rule outlined above, this is an unusual case where in all the circumstances, the court should admit opinion evidence as to the way the tender process unfolded and the merits or otherwise of the defendant's evaluation of the respective bids."

19. Coulson J concluded that the case did not fall into the unusual category where such evidence would be admissible. This was because the questions would permit a re-opening of the evaluation process, see [31], and would involve the experts giving their opinions on whether the defendant's rejection of the claimant's bid was manifestly wrong. Such evidence was inadmissible because it "would usurp the function of the court", see [34].
20. It follows that where a procurement exercise is being challenged on the grounds of manifest error or unfairness expert evidence will not normally be admissible. It may exceptionally be admitted where it is required to explain technical concepts or where it is necessary for the court to reach a proper conclusion on the question of manifest error.
21. It is common ground that some expert evidence is appropriate in the circumstances of this case by way of explanation of the technical background so as to enable the court to understand the context of this procurement exercise. I agree with that analysis. The Claimant accepts that some of the questions which it proposes should be addressed by the experts go beyond such technical explanation or glossary evidence. The crucial question here is the extent to which evidence going beyond such explanation is admissible as being in the second of Coulson J's potentially admissible categories, namely as evidence which it is necessary for the court to have in order to reach a proper decision on the questions in issue.
22. In my judgment the authorities show that the following considerations are relevant when considering the ambit of that second category.
23. The first and pervading consideration is that the case must be truly exceptional before such evidence will be admissible. As Coulson J explained this is a consequence both of the limited nature of the exercise being undertaken by the court in determining a challenge based on alleged manifest error and of the inadmissibility of expert evidence which would have the effect of usurping the court's function of determining whether the relevant decision was the consequence of manifest error. It is only where "there are particular reasons why, on the facts of the case in question" that such evidence is necessary that its admission will be justified (see in BY Development at [29]).
24. It follows that the court must truly need such evidence in order to reach a proper conclusion before the evidence can be admitted. It is not sufficient that the evidence is potentially relevant, helpful, or desirable. The circumstances must be such that in the absence of the proposed expert evidence the court will be unable to reach a reach a proper conclusion on the challenge being made to the procurement exercise in question. In that regard the court must also remember that its task in considering such challenges is a particular and limited one. The consequence of that is that the question has to be whether the court can safely perform that particular and limited role without the evidence in question. However, if expert evidence is necessary for the court to reach a proper conclusion then it must be admitted. In their written submissions Ms. Sloane QC and Mr. Suterwalla supported that proposition by reference to CPR Pt 35.4 and to Roth J's approval in Phones 4U Ltd v EE Ltd & others [2021] EWHC 2879 (Ch) of the approach taken by Warren J in British Airways PLC v Spencer [2015] EWHC 2477 (Ch). However, the proposition derives really from the point of principle that if there is potentially admissible evidence which is truly necessary in order for the court to reach a proper conclusion on the issues before it then that evidence is to be admitted absent special circumstances.
25. A related point is that although the admission of such evidence is an exceptional course it is a permissible one. There will be some cases where expert evidence in Coulson J's second category will properly be admissible and the court must be alert to the possibility that the case being considered is one of those exceptional cases.
26. It is also to be remembered that expert evidence is necessarily opinion evidence. Moreover, the question of the admission of such evidence will normally only be a matter of debate where there is likely to be a difference of expert opinion. In cases where the meaning of particular technical concepts or the proper approach to a particular exercise is uncontentious then the parties are likely to be able to agree such matters and to provide the court with an agreed position (whether in the form of an agreed expert opinion or a formal glossary or the like) setting out the background against which the decision is to be made. Thus the admission of expert evidence is not precluded by the fact that it is opinion evidence on a matter where the opinions of experts differ. That is really a statement of basic principle which only needs to be enunciated here because at points in the correspondence the Defendants' position appeared to be that the proposed expert evidence was not admissible because it involved the expression of opinion. The true question is whether the relevant issue is one on which the court needs the particular opinion evidence.
27. The exercise of deciding whether the expert evidence is necessary for the court to reach a proper conclusion will inevitably be highly case and issue specific. The court is required to consider with care the nature of the procurement exercise in question; the particular terms of the challenge to that exercise; and the particular topics on which it is sought to adduce expert evidence.
28. Expert opinion evidence is not admissible if it is an expression of opinion on the very issue which the court has to determine. In this case that is the question of whether there was manifest error in the Defendants' conclusion that the Claimant's tender lacked the required architectural equivalence between the two supercomputer systems. That limitation on the admissibility of expert evidence can be seen in Coulson J's approval in BY Development at [13] of Collins J's explanation in Lynch of the illegitimacy of opinion evidence on the irrationality of a decision which is subject to a judicial review challenge. Moreover, at [34], Coulson J regarded the fact that the expert evidence he was considering would involve the experts expressing their opinion as to whether the defendant's decision was manifestly wrong as a separate reason for its inadmissibility. In Circle Nottingham Group v NHS Rushcliffe Clinical Commissioning Group [2019] EWHC 3635 (TCC) HH Judge Stephen Davies characterised the inadmissible evidence as that which "would have the effect of the experts descending into the arena and giving opinions on the matters in issue between the parties."
29. Ms. Sloane contended that notwithstanding those passages there could be cases where exceptionally evidence as to the merits of the evaluation decision could be admissible as falling within Coulson J's second category. In support of that argument Ms. Sloane pointed to Coulson J's formulation of the issue before him at [27] of BY Development (which I have set out at [18] above). She said that this demonstrated an acceptance that in an appropriate case there could be expert evidence as to the merits of the evaluation of a tender.
30. In addition Miss. Sloane relied on the judgment of O'Farrell J in Bombardier Transportation UK Ltd & others v London Underground Ltd [2018] EWHC 2926 (TCC) where at [83] the judge said:

"… Given the technical nature of the procurement under challenge, it is likely that expert evidence will be required. I recognise that it is rare for expert evidence to be permitted in procurement cases. In procurement challenges the Court is concerned with the lawfulness of the process rather than reassessing the decision. It is not required to make its own determination as to the winning tenderer and therefore in many cases, expert evidence is not appropriate. However, in this case, the Court will be asked to determine whether LUL's evaluation of at least two of the technical proposals was within the range of reasonable conclusions open to it. If, as is likely based on the pleadings, the parties disagree on what would be a reasonable range of conclusions, it is difficult to see how that could be determined in the absence of expert evidence."

31. Miss. Sloane said that in those words O'Farrell J contemplated as possible, indeed as likely, expert evidence on the question of whether the evaluation of the tender in question had been within the range of reasonable conclusions.
32. Neither of those passages bears the weight which Miss. Sloane sought to place on them and neither of them cause me to doubt the accuracy of the proposition I have set out at [28].
33. Coulson J's words at [27] of BY Development must be read in the context of his judgment as a whole. When read in that context those words were not a departure from the proposition expressly enunciated that the court would not admit expert evidence usurping the court's function and directly addressing the question whether the decision which had been reached was manifestly wrong. The dividing line between such evidence and evidence which relates to the merits of the evaluation may be a narrow one but it is a real division and while expert evidence as to the latter may exceptionally be admitted that in the former category may not be.
34. It is right that at first sight O'Farrell J's words suggest that expert evidence might be admissible on the question of whether a particular decision was within the range of reasonable conclusions and so on the very issue of the existence or otherwise of manifest error. It is, nonetheless, important to see the context in which those words appeared. O'Farrell J was addressing two applications. One, from the defendant, for the lifting of the automatic suspension on entry into a contract with the successful tenderer and one, from the claimant, for an expedited trial. In deciding upon those applications O'Farrell J considered the time which it would take properly to prepare the case for trial and the time which the trial would take. It was in that context that the judge's reference to expert evidence was made.
35. Although O'Farrell J clearly had the general principles to which I have referred well in mind she was not in fact making a decision as to whether expert evidence should or should not be admitted. The issue of the extent to which expert evidence would be appropriate does not appear to have been argued before her and she was not referred to the authorities to which I have been referred. I do not understand O'Farrell J to have been purporting to lay down any general proposition of principle and the brief almost parenthetic passage to which I have been referred in the Bombardier judgment does not alter my assessment of the applicable principles.
36. In a related argument Miss. Sloane made reference to the nature of a manifest error. She said that expert evidence can be adduced to establish that there was a manifest error and that such evidence can involve the expression of an opinion as to the existence or otherwise of a manifest error. The former proposition is correct and I agree that in determining whether to admit expert evidence the court must have regard to what is meant by a manifest error and what is needed for the court to be able properly to determine that question. However, the latter proposition is not correct. It is contrary to the authorities I have already cited and as I will now explain it does not follow from the cases on which Miss. Sloane relied as to the approach to a manifest error.
37. The characterisation of an error as a manifest error is a reference to its materiality to the decision under challenge and to the incontrovertibility of the assertion of the error once matters are properly understood: per Coulson J in Woods Building Services v Milton Keynes Council [2015] EWHC 2011 (TCC) at [17]. There Coulson J summarised the effect of the approach enunciated by Morgan J in Lion Apparel Systems v Firebuy Ltd [2007] EWHC 2179 (Ch) and by Green J in Gibraltar Betting & Gaming Association Ltd v SSCMS & others [2014] EWHC 3236 (Admin). The requirement of centrality or materiality is met if the error "goes to the heart of the impugned measure [or decision] and would make a real difference to the outcome" (per Green J in Gibraltar Betting & Gaming Association at [100]). The test for obviousness or incontrovertibility is not that the error is immediately apparent to a person lacking knowledge of the technicalities of the particular field (including the judge) but whether when the facts, including any technical questions, are properly understood there is no scope for the view that the decision in question was properly made and where the error falls outside any appropriate "margin of appreciation". Expert evidence might be needed, as HH Judge Stephen Davies said in Circle Nottingham at [17] summarising the effect of Law Society v The Lord Chancellor [2018] EWHC 2094 (Admin), to "make obvious to the unskilled person, including for such purposes the judge, that the reasoning which led to the decision was vitiated by a technical error which, once explained, could be shown to be incontrovertible."
38. The question of whether a particular decision was vitiated by a manifest error remains one for the judge and one as to which expert evidence is not admissible. Expert evidence might be needed and hence admissible to show the materiality or centrality of the error. Thus it might be that expert evidence will be needed to show the way in which the asserted error affected the chain of reasoning leading to a particular decision and/or its materiality in that chain of reasoning. It might also be needed as a step in enabling the judge to reach a conclusion on the incontrovertibility of the error. Thus such evidence might be necessary to enable the judge to understand why it is being said that the decision was erroneous and to assess the competing contentions in that regard. Expert opinion evidence will not, however, be admissible on the question of the incontrovertibility of the error which is a question for the court. As already noted the distinction might be a fine one but it is a real one.
39. The practical operation of the correct approach is illustrated by Fraser J's decision in Bop-Me Ltd v SSHSC [2021] EWHC 1817 (TCC). The challenge there was to the defendant's decision in a procurement exercise in relation to face-masks. The claimant sought permission for expert evidence on four issues. Fraser J declined permission on three of those issues but allowed it on the fourth. The judge applied the approach set out in BY Development and concluded that the technical background was not so complex that expert evidence was needed to explain it. It followed that the case did not fall into the first of Coulson J's exceptional categories. Fraser J then considered the second category and the question of whether expert evidence was needed to enable the court to reach a proper conclusion on the issue of manifest error. He took the view that three of the four topics on which the Claimant sought to adduce expert evidence were "matters of interpretation and construction" with the consequence that expert evidence was not needed. However, the fourth was the question of what had to be included for a testing report to comply with the relevant British Standard (which did not itself identify the contents of such a report). Fraser J noted that "the contents of an industry compliant testing report, is not a matter upon which the court will have any knowledge of its own." In those circumstances he concluded that "expert evidence on the industry standard and expected contents of testing reports under the British Standard… will assist the court in determining whether the Secretary of State was entitled to conclude that the claimant's offer was defective in this regard". In those circumstances expert evidence on that issue was admitted. It will be seen that Fraser J distinguished between expert evidence addressing what was standard and expected in the relevant industry and the court's separate determination on the correctness or otherwise of the defendant's decision in the light of that expectation.
40. Although Miss. Sloane made some reference to authorities and statements of principle in respect of the admission of expert evidence in other areas those did not really advance matters. To be admissible in a procurement case expert evidence must satisfy the requirements for the admission of such evidence in civil litigation generally. The critical question is whether the evidence in question satisfies the additional requirements arising from the nature of procurement cases and set out in BY Development.
41. Similarly the argument that expert evidence is needed to ensure the parties are on an equal footing can have only very limited weight in procurement cases. Ms. Sloane said that if the Claimant was not permitted to adduce expert evidence it would be disadvantaged as against the Defendants who would be able to call evidence from its evaluators one, at least, of whom would be put forward as an expert. That apparent imbalance is a consequence of the exercise in which the court is engaged namely that of deciding whether the decision was based on a manifest error rather than whether the decision was objectively right or wrong. It follows that an inequality of arms is not of itself a factor operating in favour of the admission of expert evidence. See in this regard per Coulson J in BY Development at [28] – [29] and per HH Judge Stephen Davies in Circle Nottingham at [18].
42. The preceding authorities and considerations relate to cases where procurement decisions are challenged on the ground of manifest error. A separate potential ground of challenge is that of the failure of a procurement exercise to satisfy the requirements of equal treatment, transparency, and consistency. Such a challenge can be closely connected with one alleging manifest error (as will be seen that is the position here). In determining such challenges the court has to consider whether the requirements imposed on tenderers would be comprehensible to a reasonably well-informed and normally diligent tenderer and how the requirements would have been understood by such a RWIND tenderer. As was explained by the Supreme Court in Healthcare at Home Ltd v The Common Services Agency (Scotland) [2014] UKSC 49 (see [1] – [11], [14], and [26] – [27]) the RWIND tenderer is a hypothetical construct akin to the man or woman on the Clapham omnibus and as such is a means of describing a standard applied by the court. As a consequence the court will not admit expert evidence to establish how a RWIND tenderer would have understood the criteria of a particular procurement exercise. Expert evidence may be admissible if it is necessary to explain technical terms; the context of the procurement exercise; or the circumstances of the industry in question. That is in order to enable the court to put itself in the position of the RWIND tenderer but expert evidence going beyond that and setting out an opinion as to the understanding of a RWIND tenderer is not permissible.

The Application of those Principles to the proposed Expert Evidence.

43. I have set out in the Annex to this judgment the seventeen questions in answer to which the Claimant seeks to adduce expert evidence. In the time since the application was made and to some extent in the course of the hearing before me there has been some modification and refinement of the proposed questions and the Annex is based on the Claimant's consolidation of the questions it originally proposed and the questions proposed by the Defendants. There were modest differences between the Claimant and the Defendants on the precise wording even of some of the questions which they agreed were admissible. Ms. Sloane and Ms. Hannaford QC sensibly focused on the points of principle and not on the precise wording of the questions. Both counsel were agreed that once the court had determined the topics on which expert evidence was admissible the parties should be able to resolve their limited differences as to the precise wording to be used. That is a sensible and proportionate approach. In setting out my decision on the individual questions I will address the admissibility of expert evidence on the subject matter of the question. In those instances where the difference as to wording between the Claimant and the Defendants involves a question of principle I will determine that dispute. In those instances the form of the question appearing in the Annex will be the wording I have approved. Where the difference is in reality as to questions of style or form I will normally leave that for resolution between the parties and where I conclude that the question is inadmissible I will not normally address academic differences of wording.
44. I will set out my decisions on the particular questions in turn but the following general conclusions are the basis of the approach I have taken to those questions.
45. In light of the generally applicable principles which I have summarised above it is necessary to have regard to the nature of the procurement exercise here; to the nature of the challenge being made by the Claimant; and to the issues which will have to be determined at the liability trial.
46. I have already noted that the background to this procurement is that of high performance computing. That means that the court will have to make its decision in the context of technical matters of computing, of computer structures, and of the performance and capacities of different computer systems. In the light of that I agree with the parties that expert evidence will be needed to explain the relevant technical terms; to explain the tasks to be performed by the proposed supercomputer; and to enable the court to understand the capacities and structures of the relevant computers.
47. The Claimant's challenge is to the Defendants' decision that the development computer proposed in the Claimant's tender was not architecturally equivalent to the main supercomputer system. The Claimant says that there was a manifest error in the Defendants' conclusion that there was not architectural equivalence and in scoring the Claimant's tender as 0/5 in the relevant parts of the evaluation. The assessment of whether there was a manifest error by the Defendants will depend in large part on the court's conclusions as to the meaning of architectural equivalence (or rather the court's conclusions as to how that term would have been understood by the RWIND tenderer) and the application of that meaning to the relationship of the proposed development computer to the main supercomputer system.
48. It is common ground that architectural equivalence is not a term of art in the sense of being a technical term or a term used generally in the field of high performance computing with a particular meaning. Accordingly, there will be no scope for expert glossary evidence setting out the generally accepted meaning of that term. Rather the court will have to come to a view as to how the RWIND tenderer would have understood that term when it appeared in the IPT. That exercise will involve the court in reading the IPT in its technical context and doing so through the eyes of the notional RWIND tenderer. As in three of the proposed issues in the Bop-Me case that exercise will be one of construction and interpretation. Expert evidence will be necessary to enable the court to be in the position of the RWIND tenderer. That will be expert evidence explaining the context of the procurement; explaining the meaning of relevant technical terms; and potentially explaining the capacities of different computer systems but it will not be expert evidence giving the expert's opinion of the RWIND tenderer's understanding of the term architectural equivalence.
49. Having determined how the RWIND tenderer would have understood architectural equivalence the court will then need to consider whether there was a manifest error in the Defendants' assessment that the Claimant's proposed development computer was not architecturally equivalent to the main supercomputer and/or that the Claimant's tender was properly scored at 0/5 in the relevant questions. That exercise will overlap with the assessment of the Claimant's contention that the Defendants' decision was based on undisclosed criteria. The argument will be that if, as the Claimant contends, the finding of non-compliance on the ground of the absence of architectural equivalence was not justified then that decision must have been the result of manifest error or the undisclosed application of other criteria. In order properly to determine that question the court will need expert evidence as to the structure and capacities of the development computer proposed by the Claimant and of the main supercomputer (to the extent that such evidence has not already been utilised in providing the context for the determination of the meaning of architectural equivalence). The court will not, however, need expert evidence as to whether the systems were architecturally equivalent nor as to whether the Defendants were in error in finding that there was not architectural equivalence. Those are the central matters which the court will need to determine and in relation to which expert evidence would be an illegitimate usurpation of the court's role. The court is capable of making that determination properly and without expert evidence as to the correctness or otherwise of the evaluation. That is because armed with expert evidence as to the meaning of the technical terms and as to the capacities and structures of the different systems and applying its conclusion as to the RWIND tenderer's understanding of architectural equivalence the court will itself be able to assess whether the finding of non-compliance did or did not involve a manifest error. In short equipped with that material the court will itself be able to decide whether the similarities and differences between the proposed development computer and the main supercomputer meant that they were or were not architecturally equivalent. In addition it is to be noted that expert evidence as to the correctness or otherwise of the Defendants' finding that there was not architectural equivalence will almost inevitably involve the expert in expressing an opinion as to the meaning of architectural equivalence. As already explained that is a matter for the court and one on which expert evidence is not admissible. Thus expert evidence as to the correctness of the evaluation would be an impermissible intrusion into the court's role in two respects.
50. It follows that expert evidence will be admissible as to the explanation of technical terms; the context of the procurement; and the capacities and structures of the relevant computer systems but not as to the equivalence or otherwise between them.
51. Question 1. The parties are agreed that expert evidence on this topic is necessary. I also agree. This is expert evidence providing an explanation of the technical background and such an explanation is necessary in the context of this complex and highly technical field. Permission is given for expert evidence in this regard.
52. Question 2. This also is explanatory evidence and is necessary to enable the court to understand the background to the dispute. In the course of their discussions there was an issue between the parties as to the inclusion of the words "in the context of the Procurement" in the preamble to the question. I am satisfied that the inclusion of those words is appropriate to ensure that the expert evidence is focussed on the background relevant to the issues the court will be considering. Permission is given for expert evidence in this regard.
53. Question 3. Again this is necessary explanatory evidence. The wording appearing in the Annex involves a little revision to the language of the parties' drafts. The terms to be explained were referred to in questions U6.2 and U6.3 of the IPT and the Defendants are right in their submission that the experts should be referred to those questions. Permission is given for expert evidence in this regard.
54. Question 4. This is also necessary explanatory evidence. As in the preceding question the Defendants are right to say that there should be reference to particular questions in the IPT. Again permission is given for expert evidence in this regard.
55. Question 5. This is further explanatory evidence where again the Defendants are right to say that the reference to further parts of the IPT is to be included. The Defendants took issue with the inclusion of the terms "sufficiently large", "sufficient licences", "initial porting work", and "ongoing development work". The Defendants said that these are not technical terms and that the issue for the court in respect of these terms is how they would have been understood by the RWIND tenderer. That is a matter on which expert evidence is not admissible. I disagree with that characterisation of these terms. The terms are all used in question B2.5 or in the notes thereto. I am satisfied that when seen in context the first three are technical concepts in respect of which expert explanation is required if the court is to understand what is meant by the requirement in question. Thus the requirements are that the development supercomputer system be "sufficiently large to run a single copy of the um-global-nwp benchmark" that it "include sufficient licences for 5 concurrent Compiler instances"; and that the system is to be provided to "allow initial porting work to the new Supercomputer System". The court cannot form a proper view on how the RWIND tenderer would understand those terms without expert explanation as to what is involved in running the benchmark, in the compiler instances, and in initial porting work. The position is less clear-cut in relation to "ongoing development work" but the reference is to such work "that is not appropriate for an operational system". I am satisfied that the extent of the need for development and the form it would take and in particular which such work would be and which would not be appropriate for an operational system are technical matters outside the court's expertise where expert evidence is necessary and permissible. It follows that permission is given for expert evidence in this regard.
56. Question 6. Permission is refused for expert evidence in respect of this question. In asking the expert to opine on the relevance of the various concepts to "architectural equivalence" the question is necessarily calling upon the expert to express an opinion on the meaning of "architectural equivalence". The indication that the relevance of the concepts is to be explained "from a technical perspective" does not avoid this difficulty. It is for the court to determine how the RWIND tenderer would have understood "architectural equivalence" and determining the relevance of particular concepts to that term is a matter for the court. Expert evidence as to the alleged relevance of the concepts as opposed to their meaning will not assist the court in that exercise and trespasses on a central issue for the court to determine.
57. Question 7. This question suffers from the same failings as question 6 and permission is refused for the same reasons.
58. Question 8. The parties are agreed that this is necessary explanatory evidence. I also agree and permission is given for expert evidence in this regard.
59. Question 9. Questions 9 and 11 cover similar ground and invite the experts to provide technical explanatory evidence. I am satisfied that such evidence is necessary. There was a degree of debate as to whether both questions were needed. I am satisfied that one question covering this aspect of the case will suffice and in the Annex at question 11 I have set out the text of an expanded question 11 making particular reference to the matters mentioned in question 9. In those circumstances permission is given for expert evidence addressing question 11 in the form set out in the Annex and question 9 is no longer required.
60. Question 10. The Claimant characterises this question as eliciting technical evidence which will assist the court in determining the relevance of the listed characteristics to the concept of "architectural equivalence" and also in determining whether the Claimant's tender should have been scored at 0/5 in the relevant parts of the evaluation. In my judgement the evidence which the Claimant seeks to elicit goes beyond mere explanation. Instead answering the question would inevitably involve the experts in stating an opinion on the meaning of "architectural equivalence" and on whether the Defendants were in error in concluding that the difference in the listed characteristics meant that there was no "architectural equivalence". Such evidence trespasses on the very point the court is to decide. It is moreover not needed to enable the court to reach a proper decision on the matters in issue. Having received expert evidence about the meaning of the various technical terms and the context of the exercise the court will be able to reach a proper conclusion as to the RWIND tenderer's understanding of "architectural equivalence" without evidence of the expert's opinion of the relevance of particular characteristics to that concept. Having done that and being informed by the expert evidence as to the particular characteristics and the similarities and differences between the proposed systems the court will be able to reach a proper conclusion as to manifest error. It follows that permission is refused for expert evidence in respect of this question
61. Question 11. For the reasons already given when addressing question 9 permission is given for expert evidence in relation to this question in the revised form of question set out in the Annex.
62. Question 12. This question was proposed by the Defendants. The Claimant agrees that it should be asked but points out that an answer would involve the experts in giving their opinions on matters similar to those in the questions posited by the Claimant. This shows, the Claimant says, the appropriateness of the questions formulated by the Claimant. In her oral submissions Ms. Hannaford accepted that the question had been proposed by the Defendants but said that, on reflection, it could be seen to have been infelicitously worded and she said that it should not be permitted to the extent that an answer would require the expert to enter the arena and usurp the court's function.
63. Permission is refused for expert evidence in relation to this question. To the extent that an answer would involve the experts in giving simply a technical explanation it would be duplicative of the answers to questions 8 and/or 11. That is because a technical explanation of the listed characteristics and of their relevance to the performance of a supercomputer system and/or a technical explanation of the similarities and differences between the systems would be expected to include an explanation of the technical matters relevant to porting and to development work. To the extent that the technical explanation given in answer to the earlier questions does not cover that aspect then question 12 would have the practical effect of requiring the expert to opine on whether the Defendant was right to score the Claimant's tender at 0/5 in the relevant part of the evaluation on the basis of a lack of architectural equivalence or a failure to meet the stated requirements. That would involve the expert opining on the meaning of architectural equivalence and on whether there was a manifest error in the Defendants' evaluation. Thus the question is either duplicative of the other questions and so unnecessary or to the extent that it goes beyond questions 8 and 11 it is an illegitimate usurpation of the court's function and gives evidence which is not required for the court to reach a proper conclusion for the reasons already given.
64. Question 13. This mirrors question 6 and is impermissible for the same reason namely that it is inviting the expert to opine on the meaning of "architectural equivalence". The expert could not answer the question without stating his or her opinion as to the meaning of that term. Moreover, in context the question amounts to inviting the expert to opine on whether there was error in the evaluation. As already explained that would be a usurpation of the function of the court and is not required for the court properly to determine the case. Accordingly, permission is refused for expert evidence in relation to this question.
65. Question 14. This question repeats, in relation to particular aspects of the systems, the approach adopted in questions 6, 7, and 13. It is impermissible for the same reasons and permission is refused for this question as well.
66. Question 15. An explanation of the meaning of "porting" and of "initial porting work" will have been given in answer to question 5. A further explanation of those terms is not needed. To the extent that the expert's opinion on the meaning of "fully porting" adds anything to that earlier explanation it will in reality amount to the expression of an opinion on the correctness or otherwise of the evaluation made by the Defendants and as such would amount to the expert opining on the question for the court. Accordingly, permission is refused for this question.
67. Question 16. At first sight this question is expressed in terms of a technical explanation. However, on proper analysis it calls for the expert to express an opinion on the merits of the evaluation process. Sub-paragraph (a) inevitably involves the expert in addressing the merits of the evaluation by commenting on whether the Defendants had the information needed to make a calculation as to the time for the relevant processes. Answering the question in sub-paragraph (b) would also inevitably require the expert to express an opinion on the merits of the evaluation. The expert evidence which would come in answer to this question as a whole would not be identifying a particular matter which would then be an element in the contention that there was manifest error. Rather it would be the expression of an opinion as to the result of the evaluation. The issue then becomes one of whether the court can safely and properly perform its limited role without that evidence. Is the evidence truly necessary for the court to perform its task?
68. I am satisfied that the court can properly perform its task without expert evidence in answer to this question. The court will have expert evidence as to the capacities and performance of the different systems and as to the tasks they were to perform. I have already explained how that evidence will enable the court to determine the meaning of architectural equivalence and whether there was manifest error in the conclusion that there was not architectural equivalence. The question is directed to the issue of whether the use of different systems would result in an increase in the time taken for the identified processes. The Defendants' conclusion in that regard was a significant element in the evaluation. I am satisfied that the court will be able to decide the issue of whether or not there was manifest error in that regard without this expert evidence. It will be able to do so by reference to the expert evidence already being provided as to the capacities and performance of the different systems. It will not require this expert evidence which would involve the expert in expressing an opinion on the correctness of the evaluation and on whether it was based on sufficient information. On that basis permission is refused for this question.
69. Question 17. This is also a question answering which would involve the expert in opining on the outcome of the evaluation and expressing a view as to whether or not the conclusion reached was erroneous. It trespasses on the very question the court will have to resolve and is, moreover, not necessary for the court to be able properly to perform its role. As with the preceding question I am satisfied that the expert evidence provided in answer to the permitted questions will enable the court to reach a proper conclusion on the question of manifest error in this respect and that expert evidence as to whether the requirement could or could not be met in particular circumstances is not needed. Permission is refused for this question also.
70. It follows that permission is given for expert evidence answering questions 1, 2, 3, 4, 5, 8, and 11 in the form those questions appear in the Annex but refused for the other proposed questions.

The Further Conduct of the Case.

71. There is to be a further Case Management Conference on 28^th January 2022. The Claimant has proposed a timetable for expert evidence which would have the consequence of all such evidence being in its final state by 29^th April 2022. I do not understand that timetable to be controversial between the parties but it would mean that the expert evidence would be finalised only shortly before the trial listed for 9^th May 2022. In my judgement that timetable creates a risk that matters will not be properly prepared in time for the trial and gives no leeway for any unavoidable slippage of the dates. I invite the parties to submit a revised and shortened timetable for that expert evidence which I have approved. That timetable is to be submitted for approval either at the hearing to deal with matters consequential upon this judgment or if earlier, as it is likely to be, at the forthcoming Case Management Conference (and subject to the views of the judge conducting that hearing).
ANNEX

Question 1.

Please provide a technical explanation of the following supercomputer systems in the context of the Procurement:

a) The Main Supercomputer Systems, namely the Generation 1a, Generation 1b and Generation 2 Supercomputer Systems

b) The Porting System

c) The Development Supercomputer System

d) The Test Supercomputer Systems

Question 2.

Please provide a technical explanation of each of the following components of a supercomputer system, in the context of the Procurement:

(a) Processors, including:

(i) Qualification processors;

(ii) Engineering sample processors;

(iii) Pre-production processors

(b) Memory

(c) Interconnect (including PCI Express)

(d) Storage

(e) Cores

(f) Nodes

(g) Blades

(h) Motherboards

(i) Platforms

Question 3:

Please explain each of the following concepts and technical terms in the context of the Procurement and the stated requirements of Questions B2.5 , U6.2 and U6.3 specifically and High Performance Computing:

(a) operational suite

(b) parallel suite

(c) a programme's runtime and scale.

Question 4:

With regard to a code or set of codes, please explain each of the following processes/ technical terms below in the context of High Performance Computing and the Procurement and Questions B2.5, U6.2 and U6.3 specifically:

a) compilation / compiling

b) running

c) assessing

d) configuration and "final configuration";

e) tuning

f) optimisation / optimising

g) validation; and

h) verification

Question 5

Please explain each of the following concepts and technical terms below in the context of the Procurement and the stated requirements of Questions B2.5 , U6.2 and U6.3 specifically and High Performance Computing:

(a) code

(b) architecture

(c) binary compatibility;

(d) software compatibility;

(e) a benchmark, such as the um-global-nwp benchmark

(f) sufficiently large to run a single copy of the um-global-nwp benchmark at the committed performance;

(g) sufficient licences for 5 concurrent Compiler instances;

(h) debugging and profiling capabilities;

(i) high performance storage;

(j) porting;

(k) initial porting work;

(l) ongoing development work.

Question 6:

Insofar as not already addressed by Q5 above, please explain, from a technical perspective, the relevance of each of the concepts in Q5 above to "architectural equivalence" between computer systems in the context of B2.5

Question 7:

Please give your opinion on whether anything else is relevant, from a technical perspective, to the concept of "architectural equivalence" between systems in the context of B2.5 and High Performance Computing.

Question 8:

Please explain each of the following characteristics of CPUs and the relevance of the same to the performance of a supercomputer system:

(i) Processor (CPU) computational core count;

(ii) Processor (CPU) per-core performance;

(iii) memory;

(iv) memory speed;

(v) memory generation: DDR4 and DDR5;

(vi) memory bandwidth;

(vii) memory channels;

(viii) Interconnect (including PCI Express) generation

Question 9:

Please provide a comparison of the listed characteristics in Q8 above in (a) the Ice Lake CPUs and (b) the Sapphire Rapids CPUs in the context of the Claimant's tender response.

Question 10:

In your opinion, are those listed characteristics (individually or in combination within a computer system) capable of being relevant to the requirement of "architectural equivalence" between computer systems in the context of B2.5 and HPC and, if so, how?

Question 11:

Please explain the similarities and differences in the characteristics referred to in Q8 above, making particular reference to the similarities and differences of the CPUs, between:

(a) the Development Supercomputer System proposed by the Claimant at Final Tender (based on Intel's IceLake processors); and

(b) the Generation 1a Supercomputer System proposed by the Claimant at Final Tender (based on Intel's Sapphire Rapids Processors).

Question 12:

Please explain in the context of the Procurement and B2.5, U6.2 and U6.3 specifically, the implications of the similarities and differences explained in Q11 above for:

(a) porting onto the Generation 1a Supercomputer System; and

(b) development work on the Development Supercomputer System.

Question 13:

Please give your opinion on the relationship, from a technical perspective, between the similarities and differences explained in Q11 above and the concept of "architectural equivalence" as set out in B2.5.

Question 14:

In your opinion, is (a) a motherboard and (b) a platform capable of being relevant to the requirement of "architectural equivalence" between computer systems in the context of B2.5 and HPC and, if so, how?

Question 15:

Please explain the terms "initial porting", "porting" and "fully porting" in the context of the stated requirements of U6.2 and HPC.

Question 16:

In relation to the time necessary to complete: (a) "initial porting", "porting" and "fully porting"; and (b) the processes set out in Question 4 above, in the context of the stated requirements of U6.2 and High Performance Computing, please give your opinion on:

(a) What information would be necessary in order to be able to calculate that time.

(b) Whether using the Ice Lake CPUs in the Development System and the Sapphire Rapids CPUs in the Generation 1a Supercomputer System would affect that time and if so, why and to what extent.

Question 17:

In relation to a requirement that "processors along with other aspects of the Development System needed to be such that codes configured to run on the Development System and then moved to the Main System would use the same quantity of resource and have the same runtime when run with that same configuration" please give your opinion on:

(a) Whether it is possible for that requirement to be met where the Development System and Generation 1a Supercomputer System are not identical:

(i) If this is possible, please explain the necessary conditions or requirements to permit this to occur; and

(ii) If this is not possible, please explain why.

(b) Whether it is possible for that requirement to be met where the processors used in the Development System and the Generation 1a Supercomputer System are not identical:

(i) If this is possible, please explain the necessary conditions or requirements to permit this to occur; and

(ii) If this is not possible, please explain why.