England and Wales High Court (Patents Court) Decisions

IN THE HIGH COURT OF JUSTICE
BUSINESS AND PROPERTY COURTS OF ENGLAND AND WALES
INTELLECTUAL PROPERTY LIST (ChD)
PATENTS COURT

		Royal Courts of Justice The Rolls Building 7 Rolls Buildings Fetter Lane London EC4A 1NL
		29/11/2019

 


Mr Justice Birss :

1. This is a patent action about United Kingdom patent GB 2 365 712 entitled "Computing device with improved user interface for applications". The patent was granted on 30^th July 2003 based on an application made on 27^th July 2001 claiming priority from a British filing GB 0019459 on 28^th July 2000. The original proprietor was Symbian Ltd. Symbian was a joint venture between various telecommunications companies including Ericsson and Nokia. The patent was acquired by the claimant, Conversant, as part of a portfolio of patents.
2. Conversant brings this patent infringement claim against the defendants (Apple). The allegation of infringement concerns aspects of the user interface in various models of Apple's iPhone, running various versions of its iOS operating system. The trial focussed on the iPhone X product running iOS 11.4 and iOS 12, and on two aspects of the user interface, called Widgets and Home Screen Quick Action Windows (HSQAWs). There is no need to consider any other products, iOS versions or functions (or third party apps) at this stage. That can be done on an enquiry if necessary.
3. The defendants are respectively the UK company operating the UK retail arm for the group, an Irish company which operates the Apple online store in the EU and the US parent company for the group as a whole. There is not now any need to distinguish between these companies.
4. Apple denies infringement of the patent and contends that the patent is invalid on various grounds. By trial the major grounds were allegations of obviousness based on two items of prior art (an extract from the book "Windows 98 for Dummies" about Microsoft's Outlook Express product and a cellular phone device called SIMON produced by IBM in 1994); an AgrEvo-obviousness argument; and points on added matter. There was a pleaded point on insufficiency but it was not pressed. Apple did not raise any objection relating to the exclusions from patentability, sometimes called patent eligibility, such as those relating to programs for computers or presentations of information (Art 52 EPC).
5. Conversant applied unconditionally to amend the claims. The amendments are opposed on added matter grounds (Art 123(2) EPC/ s76(2) of the 1977 Act). Conversant invited me formally to make the amendments in any event and then if, contrary to Conversant's case, I found the amended claims invalid on any ground, to revoke the amended claims. In other words Conversant invited me not to consider the opposition to the amendments distinctly. Apple did not object to this course. I will do it this way, although in most cases it is not wise to roll things up like this. One reason it works here is that there are no objections to the amendments based on clarity or support (Art 84 EPC/s14(5)(b), (c)) nor on extension of scope of protection (Art 123(3) EPC/ s76(3)(b)). Notably and despite some of what was advanced in argument by Apple, there is no objection to the introduction of the term "smart phone" on clarity grounds.
6. The only claims which need to be considered in this judgment are claims 1 and 2 as proposed to be amended. Shown as a mark up from the claims as granted, they are in this form:

1. A computing device smart phone comprising a display screen, the computing device smart phone being able configured to display on the screen a main hierarchical menu system,

wherein the device smart phone is also able configured to display, in addition to the main hierarchical menu system, application summary windows for each of several different applications,

in which each summary window serves as a summary of a particular application by virtue of displaying a limited list of (i) several commonly used functions offered within that particular application and/or (ii) stored data commonly accessed by that particular application,

wherein the smart phone is configured to, for a given application:

display, in a summary window for the given application, both a limited list of several commonly used functions offered within the given application and stored data commonly accessed by the given application;

display the said commonly used functions offered within the given application without opening the given application;

display the said stored data commonly accessed by the given application without opening the given application; and

open the given application when an item of the said displayed commonly used functions or the said displayed stored data is selected.

2. The computing device smart phone of Claim 1 in which selecting a commonly used function listed in the summary window causes the related given application to open and that selected commonly used function to be activated.

7. The only relevance of claim 2 is that if one of the added matter objections to claim 1 succeeds, an amendment to merge claim 1 into claim 2 cures it.
8. Conversant called Mr Andrew McGrath as an expert witness. He is a user interface/user experience (UI/UX) designer with extensive experience of designing the interfaces for mobile phones. His experience was sufficiently close to the priority date so that any difference in timing does not matter on the facts of this case. He was then working on mobile telephone user interfaces for a network operator Hutchison 3G UK Ltd (now known as Three or 3 or 3 Mobile) who were looking to identify mobile phones to work on their network. The fact he was not working directly for a phone maker does not matter.
9. Mr McGrath studied Product Design at The Glasgow School of Art, graduating in 1989. He obtained an MA in Industrial Design from Manchester Metropolitan University in 1991 and then joined British Telecom's R&D department. By 2000 Mr McGrath was a Technical Design Manager leading a team of user experience designers. In that year he joined the mobile telecoms business of the Hutchison group. By the time Mr McGrath left in 2004 he was Head of User Interface. His later roles have also been focused on mobile device user experience.
10. Apple called Professor Russell Beale as an expert witness. He is Professor of Human Computer Interaction at the School of Computer Science, University of Birmingham. He has extensive experience in the field of human computer interaction. Prof Beale graduated from York University in 1986 with a degree in Physics and obtained a D Phil in Computer Science at York in 1991. He joined Birmingham University in 1991 as a lecturer and has been associated with the university ever since. From 1996 to 2000 Prof Beale worked partly at Birmingham and also did some travelling, worked at the University of Waikato in New Zealand and then was Chief Executive of aQtive Ltd, a company focussed on agent-based internet searching. From 2000-2002 Prof Beale had a leave of absence from Birmingham and worked in a series of IT based start up businesses. He returned to full time academic work at Birmingham in 2002. Prof Beale's primary work is using artificial intelligence to assist people in their interaction with computers in general. A number of his projects have involved mobile devices.
11. One of Apple's submissions was that it is not accurate to say Mr McGrath was working in user interface design for mobile phones at the priority date. That is wrong for the reason I have explained. Apple contended that Mr McGrath was very well prepared. If this was a criticism, I reject it. Apple also contended that Mr McGrath found it difficult to answer questions put on assumptions. So he did but I did not detect there any reason to reduce the value of his evidence. Much of the cross-examination involved multi-layered assumptions, not always explicit. Apple submitted that the effect of Mr McGrath's approach in this respect was that some of his evidence was not helpful or constructive. I reject that. He was doing his best to answer the questions put to him.
12. Apple also submitted Mr McGrath had an "exceptionally narrow view" of the relevant field and "seemed to be heavily influenced by his particular personal experience working for a mobile phone operator". That is also not a good point. Unlike Prof Beale, at least Mr McGrath was actually working on the user interface of a product within the claims at the relevant time. The issue of the width of the field will be dealt with below.
13. By contrast Apple submitted Prof Beale was a model witness. In my judgment he was no more or less a model witness than Mr McGrath was.
14. Conversant submitted Prof Beale fell far short of the standard the court expects of an expert witness, adopted the role of an advocate for Apple's case, maintained technically indefensible positions (such as a point on applications within the Outlook Express prior art) and was prepared to cling to an obviousness analysis when its defects were pointed out to him. Conversant suggested that some of the cause of this may have been the way he was instructed rather than his own fault. I reject these criticisms of Prof Beale's evidence too. Prof Beale was an enthusiastic witness who wanted to ensure the court understood his views. He was not an advocate for Apple. He did not accept Conversant's case on applications in Outlook Express but gave coherent reasons for his view. He maintained his opinion about obviousness because he did not accept the points put to him and gave coherent reasons why not. None of this means that I will (or will not) accept every reason given by Prof Beale, or for that matter Mr McGrath. But Prof Beale's approach to his oral evidence did not undermine the weight which I might attach to his opinions.
15. In my judgment, contrary to submissions of each party, Mr McGrath and Prof Beale were good witnesses. They are experts in their chosen fields and they were seeking to help the court in their answers to counsel's questions. Although ultimately they are experts working in the same field, which is concerned with the interaction between people and computer equipment, their experience and focus was very different and so they came at this case in very different ways. Mr McGrath's experience was that of a hands on team of product designers, focussed in particular on traditional mobile telephones. Prof Beale's experience was much wider in that it was concerned with the human computer interaction field as a whole. As an academic Prof Beale taught those who would go on to undertake design work on user interfaces such as those of mobile devices, and he also carried out his own research in the field, some of which was concerned specifically with mobile devices.
16. I am grateful to both Prof Beale and Mr McGrath for their evidence in this case.

The patent – an introduction

17. There was a stark difference between the parties' cases as to the correct identity of the person skilled in the art. However the issue is also tangled up with the claim amendments and claim construction. Accordingly I will commit the heresy of starting by taking a look at the patent as granted without first defining the skilled reader of the document. The debate is really about whether the invention is about mobile phones (whatever that means) or computers in general or somewhere in between. The proposed amendments to claim 1 limit the claim from a computing device in general to a smart phone.
18. It is also necessary to state now what meaning I give to various terms. The reasoning will be dealt with below. At the priority date:

i) A "mobile telephone" meant a phone which used the then current cellular telecommunications systems (GSM/2G). The art knew that 3G/UMTS technology was coming. A representative example of a mobile telephone of the time is the Nokia 3310. It was actually launched a month or so after the priority date but that makes no difference on the facts of this case:



ii) PDA stands for Personal Digital Assistant. These were handheld mobile computer devices of the time. Well known examples were the Psion Organiser and the Palm Pilot. They were not telephones in any sense and the term mobile telephone did not include them. An example of the screen of a PDA at the time (the HP Jornada) is:



iii) There were devices which were a combination of PDAs and mobile telephones. One example was the Nokia Communicator range. These Nokia Communicators had two screens. When the device was shut it looked like a large mobile phone, with a screen and keyboard similar to the Nokia 3310 above. The device could be opened and turned to "landscape". Then it was in effect a kind of PDA. The PDA interface had a small typewriter like keyboard and a second display screen roughly the same size as the device. The phone could be used via the PDA interface too.

iv) The term smart phone did not have a precise meaning. It was understood to convey the idea of a phone with extra features. They could be: larger and colour screens, games, email and the ability to know the phone's location. Another related term at the time was "feature phone". A device called the Ericsson R380 was launched in 1999. It was another combination of PDA and mobile phone but with a single screen. There was a flip top keypad. When closed the flip top covered most but not all of the screen, so what was left visible was a conventional sized mobile phone screen. When the flip top keypad was open it exposed a larger area of the screen and a PDA style interface. I have found this was a smart phone.

19. The title of the patent, which has been set out already, is wide. It refers to a "computing device" without qualification. The background section of the patent describes the "Field of the Invention" as follows:

"This invention relates to a computing device with an improved user interface for applications. The term 'computing device' refers to any kind of device which can process and display information. The invention has specific application to mobile telephones. The term 'mobile telephone' refers to any kind of mobile device with communications capabilities and includes radio (mobile) telephones, smart phones, communicators, PDAs and wireless information devices. It includes devices able to communicate using not only mobile radio such as GSM or UMTS, but also any other kind of wireless communications system, such as Bluetooth."

[p1 ln9-16] (the page numbers refer to the published page numbers)

20. This is entirely general. Although mobile telephones are discussed as being something to which the invention has specific application, the term "computing device" is as wide as one could imagine. Furthermore the term mobile telephone itself is also defined very widely and includes PDAs. Thus the definition in the patent is materially wider than the meaning the skilled person would give the term if they had not read the patent. The definition in the patent also refers to "wireless information devices", which would encompass WiFi devices which again would not necessarily be thought of as telephones in any traditional sense. The term smart phone is not defined any further in the patent. The skilled reader would give it the meaning I have mentioned above.
21. The next section of the patent is a section called Description of the Prior Art. This is expressed to be directed to problems facing designers of computer devices with small screens. For present purposes the point to note is the reference to the wide term "computing devices". I will come back to the detail below but also notable in this section is that the problems which are actually discussed there are described in the context of the tiny screen of the conventional mobile telephones of the time (see image above). I use the term "tiny" as someone writing today, 19 years later.
22. Next is a summary of the invention section, which includes a consistory clause corresponding to claim 1. Claim 1 as granted can be seen from the marked up version above. The claim is wide. It is to any computer device. It is not limited to mobile telephones (by whatever definition), nor even to devices with small screens.
23. There is then a detailed description section which includes figures. Although the detailed section is drafted in a wide way, it is tolerably clear that what is being focussed on is a problem which arises with the then conventional mobile telephones with what I have called tiny screens. For example Figures 1 and 3 of the patent are:

24. The application summary window referred to in the claim is shown in Fig 3. For the priority date (July 2000), these screens are a bit larger than the conventional tiny screen of the mobile telephones on the market. They were the sort of screen which those in the art knew was coming in the not too distant future and was being considered by designers at the time. That applies to either party's formulation of the skilled person.
25. The final paragraph of the detailed description summarises the position quite clearly:

"As explained above, the present invention is particularly useful for mobile telephones because of their user interface constraints. However, it may also be used in environments outside of mobile telephony. For example, desktop and laptop PCs may also benefit from the present invention. The present invention may therefore be used in any computing environment, including both keyboard and keyboard-less devices."

[p8 ln21-26]

26. One can now see what has probably happened. The problem, to which the actual invention the inventor really came up with was a solution as far as they were concerned, was a problem on mobile telephones, because of their small screens. However having made their "subjective" invention, the inventor (and no doubt the patent attorneys working for the inventor's employer) recognised that it might have a much wider potential application, to computing devices in general. So the patent is written in the way it is. However today it does not suit the patentee to maintain such a wide claim. Hence the amendment to bring the whole thing back to where it started – ideally to mobile telephones. However even then there is a problem because in a fit of enthusiasm, the term mobile telephone itself was given a very wide definition in the specification. So the patentee has used the rather vague term "smart phone", aiming as it does to catch the well known Apple smart phone product, the iPhone.

The person skilled in the art

27. The question at this stage is how to define the person skilled in the art. Conversant contends that the person is a user interface designer for, and specifically for, mobile phones, by which they mean "true" mobile phones rather than things within the scope of the definition in the patent. Apple contends that the skilled person is a user interface designer with knowledge and experience of computing devices more generally, including other mobile devices. Note the term "mobile devices" in Apple's formulation is a wide one. It includes things which are not phones such as PDAs with no phone function.
28. Apple says, not unreasonably, look at the patent as a whole – it is not limited to mobile telephones. The field of the invention is broad. Conversant says, not unreasonably, although the patent refers to a broad range of computing devices, look at the claim as proposed to be amended, it is limited to smart phones.
29. In terms of evidence, after cross-examination the position is tolerably clear. There were real teams, working at the priority date, whose job was the design of the user interface / user experience of true mobile telephones. These teams did not design the user interface of a desk top computer nor did they design the user interface of what were then PDAs. There was evidence that for the Nokia Communicator which had two screens, one team would work on the phone interface while a distinct team at the same company worked on the PDA interface.
30. There were clearly real groups working on smart phones, as is demonstrated not least because Ericsson's R380 smart phone was launched before the priority date. The evidence did not get into this in any detail, I think because Apple's case was much wider and Conversant's narrower. I infer from the evidence as a whole that a group working on a smart phone like the Ericsson R380 may well have included two sub-groups, one focussed on the phone interface and the other on the PDA interface but even if that was so, they will have worked very closely together.
31. There were also real people working in the computing field in general with wider human computer interaction (HCI) expertise. Prof Beale was at the time an example of that. He also taught such people. Furthermore, people with the wider HCI kind of experience will have joined mobile phone UI/UX teams and brought their wider experience to bear. That was Prof Beale's evidence which I accept. Nevertheless, other members of the team will not have had the wider HCI perspective of Prof Beale's kind of person. They will have had a narrower focus of the kind proposed by Mr McGrath.
32. The person skilled in the art is a legal construct used to provide an objective legal standard by which various legal questions can be answered. Nevertheless the court will always have regard to the reality of the position at the time (Schlumberger v EMGS [2010] EWCA Civ 819). Apple's case seems to involve a point of principle that the way to identify the skilled person as a matter of law is to look at the field the patent itself locates the invention in and posit a person in that field as the relevant person. The problem with that approach is that one could end up in this case with a person working in the field of (say) PDAs, even though they are no longer within the claims. The point is wrong because a patent is taken to be directed to those with a practical interest in its subject matter (Catnic v Hill & Smith [1982] RPC 183). Its subject matter is the invention, and the invention is what is defined in the claims (s125 of the Act). It follows that while it will be unusual, there is nothing wrong in principle for the effect of a claim amendment to mean that the notional person skilled in the art relevant to an amended claim may be different from the one applicable to the unamended claim.
33. Therefore, applying these principles to the facts of this case, for the purpose of assessing the claims as proposed to be amended, the skilled person is someone with a practical interest in smart phones.
34. Conversant's formulation, while closer to the mark, is also wrong. The team is one working on smart phones not merely on mobile phones.
35. A different point could be that as a matter of fact real teams at the priority date who worked on other things such as desktop computers, were also interested in designing the user interface of smart phones. However the balance of the evidence was to the contrary. At the priority date there were real teams focussing on smart phones.
36. Another different point relates to the prior experience of members of the notional skilled team. Even if I am wrong and the team is to be regarded as one working only on a mobile phone and not the wider smart phone idea (Conversant's case), at the risk of repetition I find that such a team would include both members with the kind of narrow experience supported by Mr McGrath and also members with the wider HCI experience supported by Prof Beale. The UI/UX field relating to mobile telephones was a new and specialist field at the time as Mr McGrath said, but the work was not only done by Mr McGrath's type of person but also by people with the wider HCI experience Prof Beale described, possibly having studied the subject at university in the kind of course run by the Professor. Their experience will include concepts taken from desktop computers, PDAs and other fields. Whether it is a team concerned with smart phones or only true mobile phones, the notional skilled team will include members with both kinds of experience.

Common general knowledge

37. Prof Beale referred to a wide range of books, journals and conferences which he said formed sources of common general knowledge. A striking example was something called the MITRE guidelines. These were a vast collection of 994 guidelines published by the US Airforce 15 years before the priority date. Mr McGrath's evidence was that the skilled person by his definition, would never have heard of them. I find they were not common general knowledge of the relevant skilled team. The farthest I am prepared to go is that the member of the skilled team with Prof Beale's wider HCI background might dimly remember the existence of this tome but that is all. They would not regard it as a source relevant to their work in the skilled team. Apple submitted that the fact the MITRE guidelines were referred to in a (single) ETSI document from 1992 (which was not itself a standard) was relevant evidence. It was not.
38. Prof Beale also referred to the Nielsen heuristics. I am not satisfied that this body of work formed part of the common general knowledge of the skilled team either. The member of the skilled team with Prof Beale's wider HCI background would know they existed but I was not convinced the skilled team would employ them as a tool or an input into the design process. At best from reading and listening to Prof Beale's evidence, although the individual heuristics are written as rules, I rather think those in the HCI field used them as a means to classify a pre-existing interface or as a teaching aid, but neither is the relevant issue.
39. There were books and journals about HCI and the member of the skilled team with Prof Beale's wider HCI background would be aware of them in general terms, and no doubt have their own favourites. Even the member of the skilled team with Mr McGrath's narrower focus would be aware that works of that kind existed. As I understand the way this case is put, it is not necessary to go through the individual books or journals identified and make findings of fact about them.
40. There were also some conferences or workshops but designing the user interface for mobile phones was a new field at the time and there were very few of them which were directly relevant.
41. I do not believe anything turns on a particular statement in any of the various books, journals or the proceedings of any conferences or workshops.
42. Another possible source of common general knowledge was the experience members of the skilled team gained from their own use of desktop computers, PDAs and phones. This was a source of common general knowledge. The general characteristics of the user interfaces in desktop computers, PDAs and mobile phones formed part of the skilled team's common general knowledge for that reason alone.
43. I have already referred to the different kinds of device – mobile phones, PDAs and desktop computers. The term "form factor" is a description of the relevant properties of these devices. The form factor of a mobile phone like the Nokia 3110 was that it was a small, portable device designed to be used with one hand. It had a tiny screen. Core functions were the phone itself and the ability to do SMS messaging. There were a few other applications on the device. Nevertheless part of the common general knowledge was that more were coming. Devices with this form factor had a specific set of design constraints: small screen; limited input means; a need to be easy to use "on the go" and in particular with one hand; and a manufacturer's user interface paradigm.
44. The point relating to the manufacturer's user interface paradigm bears some explanation. The idea was that all phones from a single source (for the sake of argument Nokia) were designed with a user interface which worked in more or less the same way as the other devices from the same source. Thus a user would find it easy to use another Nokia phone even though it was a different model from the one they were used to. Apple suggested this latter point was a purely commercial matter and therefore irrelevant. I disagree. This was a real aspect of user interface design. No doubt it existed largely for commercial reasons but it was a tangible, practical characteristic of the devices.
45. The form factor of a PDA was a bit different. These were handheld portable devices with larger screens than the tiny screens of true mobile telephone. They were not designed for one handed use. They often had a touch screen which could be activated with a stylus. A finger could be used too but for various reasons, including the then quality of touch screen technology, a stylus was better. They could have a small keyboard. They could have soft keys (as could true mobile telephones). A soft key is a key beside the screen whose function can be programmed in software so that in one context it does one thing while in other circumstances it does something else. PDAs were like portable computers and had office functions.
46. The form factor of desktop computers (for this purpose including laptop/notebook computers) was much larger. They could be carried but were not held in the hand. There was a full keyboard and a mouse. Mr McGrath said that a mobile phone UI/UX designer had to "unlearn" concepts applied from this very different paradigm. I accept that but not entirely. It goes too far. The truth is a matter of degree. The idea applies most to the difference between the user interface on a desktop computer and the one on a true mobile telephone. The handheld use of the mobile phone "on the go" and its tiny screen made it a very different interface from a desktop. The point was much less relevant to the difference between desktops and PDAs. It is true that a PDA is not designed for use "on the go" but that is only one factor.
47. Part of the common general knowledge included devices which combined the form factors of true mobile phones and PDAs. The well known examples at the priority date were the communicators – which had two interfaces.
48. The skilled team understood there were differences between the user interface of a PDA and one for a true mobile phone. However a team working on a smart phone at the priority date would be well aware of both and would not think that ideas relevant to one interface were necessarily of no relevance to the other. It would depend on the specifics.
49. Turning to specific features of user interfaces, the WIMP paradigm (described below) was part of the common general knowledge of the skilled team. They were aware of its use on desktop computers, and probably PDAs.
50. Using menus was part of the common general knowledge, relevant to all form factors.
51. Another aspect of the common general knowledge was the concept of using notifications. This was part of the common general knowledge applicable to all form factors. Some notifications were driven by the computer and indicated that an event had occurred, others could be summoned by the user, such as a pop up window from the status indicator in the task bar of a desktop computer.
52. An example from the Nokia Communicator, is as follows:



53. This shows a notification window on the right hand side open on top of the contacts directory. The rectangle of the notification cuts across the word "directory" which is written underneath. The notification is displayed when the user first opens the cover of the device. It tells the user about stored data (unread faxes etc.). It also happens to illustrate the use of soft keys. These are the grey ovals on the right. They are not on the screen, they are part of the device itself. The user can tap the oval key beside "View" to view the highlighted data (faxes) or could tap the oval key beside "Cancel" to close the notification.
54. I find that this image embodies a number of concepts which were part of the common general knowledge of the skilled team although the particular window itself was not. The common general knowledge concepts it illustrates are: the idea of a notification which could pop up in defined circumstances as an alternative to what is on the main screen. The circumstances could be when the user does something or when the user indicates they want it to appear e.g. by pressing a button. The notification includes stored data. From the user's point of view, it is presented without opening the application to which the stored data relates (e.g. the fax app or the mail app). By presenting the information without opening the application, the user is given information which relates to that application without having to "cognitively commit" to the application. The image also illustrates the use of soft keys. All of this, individually and in combination in a window, was common general knowledge.
55. Apple submitted that this shows stored data and commonly used functions being presented together. One needs to take care with that. Of course the notification window does present functions to the user but they are really choices related to the notification itself – like viewing the data referred to or cancelling the notification – rather than functions offered because they are commonly used inside a particular application (like "create new mail" would be). Of course, one can also "View" mail in a mail application.
56. Another example of the same idea given by Prof Beale was a Calendar Reminder in Windows CE (an operating system for PDAs). This was:

57. The Calendar Reminder is a notification which pops up in appropriate circumstances. It displays data. The available functions relate to the notification itself.
58. In this context Apple also referred to an image from the Simon prior art but it was not common general knowledge at all. Apple also referred to the "Today" screen from the HP Jornada PDA. The device was common general knowledge but I am not convinced the particular lay out of its Today screen was.
59. Apple submitted that displaying data and functions was common general knowledge. That is too broad. The common general knowledge included the idea of displaying data in a pop up type notification such as a window. Summoning or being presented with the notification did not involve opening the application to which the data related from the user's point of view. With the data, there would be functions presented associated with the notification itself. I do not accept that the common general knowledge included the idea of displaying a window for an application which contained both a commonly used function (or functions) offered within that application, and also stored data related to that application.
60. The focus of notifications is to present data to the user. Prof Beale also described a feature called contextual menus. In fact he meant a particular kind of thing (see below) but it was given this wide generic name. That was an unhelpful name because in a sense all menus are contextual. It was also unhelpful because the way the arguments developed involved taking advantage of the wide name.
61. The thing Prof Beale was talking about was a feature some user interfaces included which allowed a user to summon a short cut to commonly used functions for a particular object on the screen that the user was interested in. On a desktop system a common approach was to provide a mouse with two buttons. The left button is the normal one. When a user puts the cursor on an icon and clicks the left button, the application opens. However when the user puts the cursor on an icon and clicks the right button, a small pop up menu of common functions appears. That is what Prof Beale referred to as a contextual menu. There were numerous variations embodying the same concept. But to call it simply a contextual menu leads to confusion. I will call this the idea of a user summoned shortcut menu. This idea was part of the common general knowledge of the skilled team. With an eye on the issues in this case, it is notable that this user summoned shortcut menu idea did not present data, only functions. On the other hand, like the notifications, it did allow a user to see the options without opening the application. It was an alternative to opening the application by whatever normal means the system presented.
62. Mr McGrath's evidence was that while notifications in a general sense were part of the common general knowledge of his skilled person, the kinds of examples I have given above did not reflect that knowledge because they are not taken from the mobile phone form factor. Similarly Mr McGrath's evidence was that user summoned shortcut menus (he did not use the term) were a feature of the desktop interface only. I do not accept that this evidence reflects the common general knowledge of the skilled team for three different reasons. First the team is concerned with smart phones and not just mobile phones and so will have had knowledge of and interest in the interface in the PDA form factor. Second the team will include members with wider HCI experience, which included all the matters described above, even if they were only found on desktops. Third the team had its own experience of the user interface of desktop computers (and probably PDAs) as users of those systems. A user interface designer necessarily pays attention to the user interface on any device in a manner an ordinary person would not.
63. Conversant pointed out that Apple had not presented any examples of pop ups on a tiny mobile phone screen at or before the priority date. There was an effort by counsel for Apple to cross-examine on an abandoned prior art reference called Frederiksen but that was plainly not common general knowledge. I am sure the skilled team would know that pop up menus had not been implemented on true mobile phones so far. That was because of the tiny screen size. However the skilled team would be well aware of the concept and would know it was something available to employ as screens were becoming larger. The idea of a pop up either covering the whole screen or leaving part of the previous context visible were common general knowledge.
64. Another concept described by Prof Beale was called "Overview Detail". I am not satisfied that a concept of that generality was common general knowledge at the priority date. Prof Beale did use that expression as part of his later work but I am not satisfied it was used in 2000. A concept which had been described (by Tidwell) was "Overview Beside Detail" in which overview information is presented with detailed information beside it, but that necessarily assumes that the display is wide enough to achieve this. Later (2011) the wider expression "Overview Plus Detail" was used. Whether the concept of "Overview Beside Detail" was common general knowledge at the prior date probably does not matter. I hold that it was part of the common general knowledge as something in the desktop form factor.
65. A concept in mobile telephone user interfaces was the idea of giving the user a short cut through the phone's menu system by allowing them to learn a series of keys (such as a sequence of numbers) to reach a given item in the menu system. This was common general knowledge. It was also well known that as the number of applications grew, the complexity of such menu systems would increase.
66. In terms of the trends in user interface design at the priority date, Mr McGrath referred to a "road map" of devices which he regarded as the kind of things the skilled team would see as the future of mobile phones at the priority date. These included what Mr McGrath said were smart phones and feature phones. There were the consumer oriented devices. They would have somewhat larger screens than what were at that time on true mobile phones but were still to be used "on the go". They would have more applications such as a calendar and email. Apple did not agree the actual planned devices were common general knowledge because the companies' plans were not publicly known. I agree that specific details of devices were not public at the relevant time but the broad concepts set out above were well known and public. While accepting that "consumer stuff" would be looked at, Prof Beale emphasised email as something which was important at the time. I agree. All of this was common general knowledge.
67. A team interested in putting an email application on a smart phone at the priority date would be interested in how that had been done on other small devices. That would include mobile telephones. As an illustration of the general point, Mr McGrath flew to Japan to learn about a contemporary device called the iMode from the major company DoCoMo, which was new and successful at the time. However the skilled team would not only be interested in the user interface in true mobile telephones. Their interest would be much more general than that.
68. Another relevant aspect of common general knowledge was that while 2G/GSM was the prevailing cellular technology, nevertheless 3G/UMTS had been standardised and was coming. It would allow higher data rates and internet connectivity. That is one reason email was important.
69. In summary, the sort of device the skilled team at the priority date would be working on, based on their common general knowledge, would have increased functionality as compared to a true mobile phone. The functions would include a calendar, contacts and email. It would have a larger screen, albeit the screen would still be fairly small and with limited resolution. The input capability would be limited – so there could be a keypad but in addition soft keys or a stylus were part of the common general knowledge. Use "on the go" is one aspect but would not be regarded as the only way the device might be used. The device would have a main hierarchical menu system of some kind for allowing the user to have access to its functions.

The patent

70. I refer back to the brief introduction to the patent above. The "Description of the Prior Art" section summarises the problems addressed by the invention as follows:

"One of the problems facing the designers of computing devices with small screens is how to allow the user to navigate quickly and efficiently to access data and activate a desired function. Computing devices with small screens tend to need data and functionality divided into many layers or views: for example, the small display size of mobile telephones has conventionally meant that several hierarchies of functions have to be offered to a user. "

[p1 ln19-24]

71. The patent describes the idea that the interface can be thought of as having many layers and that, when a mobile telephone includes several different applications the user interface can be complex and hard to learn, especially for new users. An example is given for a "contacts" application (see fig 1 above), as follows:

"the user normally has to first of all locate, then start/open the required application and then may need to navigate to the required function (e.g. create a new contact entry) or cause the required stored data (e.g. display names beginning with the letter `A') to be displayed. This process can seem slow, complex and difficult to learn, particularly to novice users"

[p1 ln 29 – p2 ln2]

72. Therefore, the patent describes, whereas with conventional user interfaces a user may need to scroll around and switch views many times to find the right data or function, an effective user interface would enable users to "readily and rapidly" access the right data or function.
73. There is a brief discussion about the organisation of hierarchical menus in some mobile phones and then a reference to short cuts offered on some mobile phones which involve memorising keyboard input sequences. However these have the drawback that they appeal only to expert users.
74. The invention claimed in the claims is presented as the solution to these problems. Claim construction will be addressed below. The invention involves using an "application summary window" displayed in addition to the main hierarchical menu system. The application summary window provides a limited list of several commonly used functions within that application and/or stored data commonly accessed for it. An example of the application summary window can be seen comparing figures 1 and 3 above. The messages application is one of the applications in the main hierarchical menu system (see fig 1). The application summary window for that application is shown in fig 3. It displays data (3 unread emails etc.) and commonly used functions (e.g. create message). The patent describes this as a 'snap-shot' view of an application which brings together a limited list of functions and can also display commonly accessed stored data. The patent explains that the user is then able to directly select that data or function and that this will cause the application to open so that the user is then presented with a screen in which the data or function of interest is prominent. Accordingly:

"This saves the user from navigating to the required application, opening it up, and then navigating within that application to enable the data of interest to be seen or a function of interest to be activated."

75. The detailed description part of the patent describes all this in more detail. Of some significance is the first paragraph in that section which refers to the advantage of "ease and speed" of navigation particularly on small screen devices and draws a comparison between a user being able to get to the summary window in just two steps, which is "far faster and easier" than conventional navigation approaches and which means that only three steps may be needed from start up to reaching the required data/function.
76. At page 5 various ways of causing the application summary window to appear are discussed including allowing the highlight on the main hierarchical menu to rest for a certain amount of time on the name of an application (e.g. 1.2 seconds), or by voice activation, a soft key, a "right scroll" or "right cursor" key or a press and hold function. At page 6 ways of dismissing the application summary window are mentioned including a time out or using a scroll command in the opposite sense to the one which called up the window in the first place.
77. The idea of preserving context for the user is discussed at the bottom of page 6. At page 7 further ideas are described such as the mobile telephone learning what functions or data are most likely to be of interest to a given user, allowing the user to define them, using the mobile phone's knowledge of the environment in which it is situated and varying the content of the summary depending on what the user has been doing before they open the application summary window.
78. The detailed section ends with the paragraph also quoted in the introductory section above and is followed by the claims.

Claim construction

79. Claim 1 as proposed to be amended is set out above. There are a few points of construction but before getting to those the general outline of the claim can be understood.
80. The claim is to a device (a smart phone) with a screen. It is a product claim not a method claim. The device has to be configured to do various things in certain circumstances. However it will still be a device within the claim even when the device is switched off.
81. It is configured to do three things. First, it is configured to display a main hierarchical menu system. Second, it is also configured to display things called application summary windows in addition to the main hierarchical menu system. The "application summary windows" are plural because there must be at least two available to be displayed, one each for different applications. They do not have to be displayed together but the device has to be configured so that each could be displayed in appropriate circumstances. The third thing the device is configured to do is perform four defined tasks for a given application. An "application" is a coherent set of functions and data from the user's point of view. The user's point of view is an important aspect. Identifying something as an application is not done by delving into the specifics of how software is coded or how processes run in an operating system or stay resident in computer memory.
82. The application summary window is defined as serving as a summary of a particular application. It must display a limited list of either several (more than one) commonly used functions within that application or stored data commonly accessed for that function or both.
83. Although one aspect of the meaning of "display, in addition to" is in dispute, what I do not believe is disputed is that part of what the inventor was getting at in using these words was to require that the application summary windows are in addition to the main hierarchical menu in the sense that they exist as an alternative to the "normal" way of accessing the application through the main hierarchical menu.
84. The four tasks the device must be configured to perform are defined for a given application. The first task is to display both some functions and stored data for that application. Thus read as a whole the claim does not cover a case in which the phone has available numerous application summary windows for different applications if each of them only ever displays either data alone or functions alone. On the other hand, if at least one of the application summary windows does display both, the device will be within the claim even if all the other application summary windows do not do this. The plurality of application summary windows demanded by the claim does not apply to the requirement to display both data and function.
85. The second and third tasks correspond to one another and require that the data and function be displayed without opening the application. Like "application" itself, the concept of opening an application is understood from a user's perspective and does not refer to the details of how the computer works.
86. The fourth task is to open the given application when the user selects one of the displayed functions or data.
87. Notable by its absence is any reference in or after the fourth task to activating the function if what was selected was a function. That is in claim 2. Apple contends its absence from claim 1 is added matter. Conversant does not agree but as a fall back Conversant submits if claim 1 is to be revoked for this added matter then the invalidity is cured by joining claim 2 into claim 1. I will address the added matter point below but I can say at this stage that Conversant is right that the amendment to claim 2 would cure it in any event.
88. The outstanding issues of construction relevant to what I have to decide are:

i) smart phone

ii) main hierarchical menu system

iii) … display, in addition to ...

Smart phone

89. Conversant, supported by Mr McGrath's evidence, submitted that while the term smart phone did not have a commonly accepted meaning at the priority date, the term did have currency for those working in the mobile telephone field as referring to a more advanced form of the "Roadmap" model. It had a mobile phone form factor and more advanced characteristics that were anticipated at the priority date including: 3G, GPS for location tracking, relatively bigger screens with better resolution, colour, cameras and, going forward, cameras with better resolution, Bluetooth, improved internet (browsing), improved games, MP3 players, improved access to email, video and video messaging. Conversant's case is that this is what the skilled reader would understand the patentee to have used the words in the patent to mean. It did not include devices with touchscreens because the ones which were available (on PDAs) were poor and unreliable.
90. Apple, supported by Prof Beale, agreed that the term did not have a precise meaning but submitted that at the priority date, a smart phone was understood by the skilled person to be a device capable of good connectivity for both voice and data, and of having computer like features but with the form factor of a mobile phone, PDA or Pocket PC. It did not include the ability to customise the phone with additional applications like the modern-day smart phone. To that extent the definition has changed between then and now.
91. Prof Beale put forward three examples of devices called smart phones contemporaneous with the priority date. One was the Ericsson R380 (discussed above). The others were a Qualcomm pdQ which apparently combined a Palm PDA and Qualcomm digital phone and a Samsung SCS which also operated as both a cell phone and a handheld computer based on Windows CE. There was not much evidence about the latter two.
92. Mr McGrath did not accept that any of these would be regarded as a smart phone at the priority date. Based on his definition there were no extant smart phones at all at the priority date, although they were coming.
93. In terms of how the skilled team would understand the term, one way of characterising the difference between Mr McGrath and Prof Beale is that Mr McGrath's smart phone is a consumer oriented device which is not yet on the market but is being actively designed now whereas Prof Beale's definition includes a more office oriented device combining a mobile phone and PDA, some of which have already been launched.
94. I find that, leaving aside the patent specification at this stage, the skilled team at the priority date presented with the Ericsson R380, which describes itself as a smart phone, would have no difficulty with that description. They would not think it was wrong or inappropriate. It is a smart phone because it is more than a normal phone. It has a number of extra features, including a calendar, contacts and email. The fact it can offer a PDA-like interface does not mean it is not a smart phone. Nor is the fact that it could be seen as rather office oriented, as opposed to oriented for the mass consumer market. To the skilled person at the priority date (by any definition of that person) the term "smart phone" did include the future feature phones/smart phones referred to by Mr McGrath which the skilled person would be aware of in general terms but it was not limited to that definition. It included devices like the Ericsson R380.
95. I do not believe it is significant either way but if necessary to do so I would hold that the Qualcomm pdQ and Samsung SCS were also smart phones.
96. Another term in use at the time was communicator. Nokia used that term as the name of their product which had two screens. No doubt some skilled people would have included the Ericsson R380 within the idea of communicators as well but not in a way which meant it was not a smart phone. The usage overlapped.
97. Turning to the specification, the term smart phone only appears in the list on page 1 which is the extended definition of mobile telephone. Conversant contended that the items in this list would be understood as distinct and therefore mutually exclusive so that, for example, a device could not be both a communicator and a smart phone. I do not agree. The skilled reader would not think the patentee was seeking to use language in this way. The members of the list are part of a deliberately expansive definition of mobile telephone. Until Conversant made the claim amendment the individual items had no other significance. A smart phone is self evidently also a "radio (mobile) telephone". The skilled team whose reaction to the Ericsson R380 was as I have described it above would not have a different reaction to the term smart phone in the patent.
98. Therefore I reject Conversant's narrow case on the meaning of smart phone. It is not limited to devices which only had even the slightly larger but still small screens of Mr McGrath. By the priority date it referred to a hand held device and so had the form factor of a true mobile or PDA. It had to be able to be used as a phone with mobile (i.e. cellular) capability. The screen could be any size, such as at least the size of the flip open screen in the Ericsson R380. I do not accept Mr McGrath's evidence about the size of smart phone screens. Mr McGrath also used the expression "phone first". I do not accept that concept as a limitation to smart phone. As long as it has a phone function, the device could properly be called a smart phone.

Main hierarchical menu system

99. Apple, supported by Prof Beale, contended that a menu was a list of textual items. So a menu of available applications would be a text list of those applications – messages, mail, contacts etc. The user indicates which item they want by selecting it from the list such as by highlighting the item in some way and pressing a key, tapping on the screen or moving a mouse, to indicate the selection. Apple and the Professor accepted that a main hierarchical menu system by the priority date could include pictograms giving a visual indication of the meaning of the item in the menu but they did not accept that a scheme like the iPhone, in which the available applications are presented to the user by a group of icons on a screen, was a menu.
100. The difference between what Apple accepts as a main hierarchical menu and what it does not can be seen from the two images below. Each shows the display at the point at which the user is presented with a choice of applications to select and open. On the left is a mobile phone from at or close to the priority date and on the right is the home screen of the Apple iPhone:

          

Philips Azalis 238          Apple iPhone

101. In both cases the user is presented with pictograms which visually represent the individual application. In the Philips Azalis there is text above a soft key in the middle ("Wap") which is the name of the application pointed to by the arrow. In the iPhone there is text under the pictogram. Apple accepts that the Philips has a main hierarchical menu but argues that the iPhone does not. The reason is because of the nature of the icons in the Apple screen.
102. The argument is that in interface design in general there is something called the WIMP paradigm (Windows, Icons, Menus, Pointers). Microsoft's Windows product is an example. Some years ago the WIMP paradigm replaced the text command based way of interacting with computers. To be a true icon in the WIMP paradigm the icon is an object with which the user can interact separately from whether they interact with the application it represents. So an icon can be moved, deleted, selected with other icons and perhaps renamed. These properties or most of them are true for the icons in the iPhone but are not true for the mere pictograms in the Philips phone. On the other hand, in the WIMP paradigm a menu is a text based list on which the individual items can be selected. There is a spectrum between menus and icons but they do not overlap. This is Prof Beale's evidence.
103. Mr McGrath accepted that within the WIMP paradigm, which was well known, icons and menus were different, but he did not agree that on a mobile phone one was in the WIMP paradigm and he maintained that a menu did not have to be textual. Nor did he agree than an icon had to be an "object".
104. The skilled reader would know and understand the WIMP paradigm but would not think it was relevant to the patent. The reader would not think the patentee used the term "menu" to invoke the WIMP paradigm at all. The reader would understand that the main hierarchical menu system was simply the means by which the user is to be presented with the relevant options and would be expected to navigate around it by selecting items from the menu. They would not think it had to be limited to text. It could include pictograms with no text at all and it would still be a main hierarchical menus system even if the pictograms were "icons" or "objects" as strictly so called within the WIMP paradigm.

… display, in addition to ...

105. The debate here is whether the phrase in the claim requires the main hierarchical menu system still to be visible in some way when the application summary window appears. The language does not involve any terms of art. The patent depicts the application summary window in figures 2 and 3. Figure 3 is above. Figure 2 is:

106. The specification at p6 ln25 explains that having the application summary window open with one item of the main hierarchical menu above and below it (in this case Launcher and Phone) preserves context for the user.
107. This is relevant to a point on added matter. The words "display, in addition to" are not in the application as filed. Apple contends that their inclusion adds matter if the claim is not limited to the case in which the main hierarchical menu and application summary window are both visible in some way when the application summary window opens. I do not agree. I find that the claim language, as Conversant contends, is not limited to a case in which the main hierarchical menu and application summary window are both visible. It would cover a case in which the application summary window appeared without the main hierarchical menu being visible. However I do not agree there is any added matter. True it is that the application only discloses examples in which the two appear together. It certainly never positively teaches that the application summary window could appear without the main hierarchical menu, but neither does the patent. The claim covers such an eventuality but does not disclose it (AP Racing v Alcon [2014] EWCA Civ 40, referring to AC Edwards v Acme [1992] RPC 131). The fact that in the circumstances the user's context was not preserved does not matter. The claim does not require the user's context to be preserved in this way. Nor can there be any intermediate generalisation since no feature has been taken out of the context in which it was disclosed at all. Moreover the only matters of context relied on by Apple were: (a) "mobile telephone" – but given the wide definition that does not make a difference; and (b) "drop down" menus – but the application as filed as a whole is clearly more general than that and is not suggesting menus have to be drop down menus.

Added matter

108. The two remaining added matter points are (i) a point called "for a given application" and (ii) the point about opening and not activating.

for a given application

109. Apple submits as follows:

"250. There is also no teaching in the Application as Filed that, for a given application, the device must display in the summary window both a limited list of commonly used functions and stored data. However, this is what the proposed amended claim 1 now requires. The Application as Filed made clear that an application summary window could display common functions within an application and/or data stored in that application. In other words, there was a choice: a summary window could display functions or data or both. The proposed amended claim maintains that choice in defining summary windows generally, but adds a further requirement that, the device (the smart phone) must be configured to display at least one summary window with both functions and data. There is no teaching in the Application as Filed that the device should be configured in this way. That is new technical information about the invention that the skilled reader learns for the first time from the proposed amendments. This is a further reason why the Patent as proposed to be amended is bad for added matter."

110. I do not accept this submission. The application as filed disclosed the idea that an application summary window should display functions and/or data. In other words functions alone, data alone or both functions and data. As a matter of disclosure all three options are taught. Claim 1 as filed expressly referred to the application summary window as comprising (i) functions and/or (ii) data. No new information is disclosed in new amended claim 1. The three options are the same but the claim has now been limited to the case in which at least one of the application summary windows is "and". That is all. (The application never said they all had to be the same.)

opening and not activating

111. The four tasks at the end of claim 1 as proposed to be amended end with opening the application. This could be when a user has selected a function. The claim does not say that the function itself has to be activated. That comes in claim 2. This is not a covers as opposed to discloses case. As written the amended claims clearly do disclose the idea of opening the application when a function is selected without necessarily activating the function. If it is not disclosed in the application, as Apple contend it is not, then it would be added matter.
112. Conversant refer to two passages in the application. The two passages are as follows:

"Preferably, where the summary window for a given application shows data or a function of interest, the user can directly select that data or function; this causes the application to open and the user to be presented with a screen in which the data or function of interest is prominent. This saves the user from navigating to the required application, opening it up, and then navigating within that application to enable the data of interest to be seen or a function of interest to be activated."

[p3 ln15-21] my emphasis

"… When an item in the App Snapshot is selected (e.g. by being highlighted and then selected using a conventional selection technique such as pressing a right cursor), the device displays the relevant data in the application details view, or displays the relevant screen offering the relevant functionality. The required application may be automatically opened when the item in the App Snapshot is selected. The App Snapshot can therefore display data from an application and functions of that application without actually opening the application up: only once a user has selected an item in the App Snapshot associated with a given application does that application have to be opened. For example, when `Create Messages' in an App Snapshot is selected, then the messaging application is opened up; that application does not however have to be opened up prior to that stage."

[p5 ln19] my emphasis

113. I agree with Conversant. These passages disclose the idea of opening the application when a function is selected without necessarily activating the function itself. There is no added matter.

Infringement

114. Somewhile ago in the proceedings Apple admitted that the iPhone is a smart phone within claim 1. They were right to do so.
115. There are two relevant kinds of function windows on the iPhone which need to be addressed at this stage. The iPhone devices are configured to display them in appropriate circumstances. The two functions are called Widgets and Home Screen Quick Action Windows (HSQAWs). No issue of infringement by equivalents has been raised. If the relevant iPhone is within the claims then Apple infringes under s60(1) of the 1977 Act.
116. Examples of Widgets are these:

Screen A	Screen B	Screen C

(This diagram and the following one are better seen in colour)
117. Screen A either shows two Widgets or one Widget and one HSQAW. Whatever they are, behind them is the home screen blurred out to indicate context. The two Widgets have appeared together. They relate to an application called Notes. Normally to make an application open the user taps lightly on the relevant icon appearing on the home screen. In this case the Widgets have appeared because instead of tapping lightly on the Notes icon, the user pressed their finger onto the Notes icon gently but a bit more firmly and for a longer period of time. This calls up the Widgets. In this example functions are shown in the top Widget and data in the bottom one. The firmer finger press is called a "3D Touch" in Apple parlance.
118. Screen B shows one Widget, again for the Notes application. This has arisen because the user went from the home screen to a Search screen. The Search screen resides notionally left of the home screen and so a user reaches it by swiping appropriately with their finger on the home screen to notionally push the home screen away to the side. When that happens Widgets can appear along with the Search function. The user can tell the phone which Widgets they wish to see here by adjusting the settings.
119. Screen C shows a single Widget on the screen of a locked device. The screen behind the Widget is what the phone looks like when it has been woken up (my language) so that the screen is no longer blank, but the user has not entered their password or PIN to unlock it. In this mode the iPhone displays Widgets if the user has set them. In that respect it is like B.
120. The other function said to infringe is the Home Screen Quick Action Window. Examples are:

Screen D	Screen E	Screen F

121. Some of these windows look similar to Widgets but it is convenient to classify them as HSQAWs. In each of these three screens the HSQAW appears when a user performs a 3D Touch on the relevant icon on the Home screen. The HSQAW appears in front of a blurred image of the home screen which was there before the HSQAW was summoned, in order to maintain user context. In Screen D the icon pressed was the Messages application (shown not blurred to the bottom right of the window). In Screen E the application was Notes and two windows have appeared. It may be the right way to characterise this screen is as showing a widget which appears with an HSQAW but it does not matter. In Screen F the home screen was a small group of commonly used applications suggested to the user by the phone using a function called "Siri Suggestions". The Messages icon was pressed.
122. The only issues are (i) main hierarchical menu system and (ii) display in addition to. On the constructions of these terms I have arrived at above, the device configured this way satisfies claim 1. The main hierarchical menu system is the set of icons on the home screen from which the user can select which application they wish to open. The application summary windows are the windows described above. The fact that the main hierarchical menu system (home screen) is not always visible when these windows appear does not take Apple out of the claim.

Obviousness

123. The right approach (Pozzoli) is to identify the inventive concept, identify the person skilled in the art and the common general knowledge, identify the differences between the claim and the cited item of prior art, and then ask whether the claimed invention is obvious in that light.
124. Claim 1 has been construed above. In terms of an inventive concept, the heart of it is the idea of making available application summary windows in a smart phone. The application summary windows have the various characteristics defined above. They present, as Mr McGrath put it, the "best of the app". One point of some importance is that the claim does require at least one application summary window to display both function and data.
125. The person (team) skilled in the art and the common general knowledge have been identified above.
126. I will identify the differences and consider obviousness in relation to each citation separately but at this stage it is useful to dwell on what the context is in which the skilled team is operating. The problem they are seeking to solve is how to accommodate the likely increase in the number and sophistication of applications which will have to be made available on smart phones as compared to traditional mobile telephones. This was a current problem at the time for real skilled teams and no hindsight is involved in posing it.
127. A point was raised by Conversant about what the skilled person is doing when they read the cited prior art. The law is clear that the document is read with interest (e.g. Dr Reddy's v Eli Lilly [2008] EWCA 2345 (Pat) Floyd LJ at paragraph 70). Having read it with interest the invention may be obvious and if so the patent is invalid. Conversant's point was that they detected in Apple's cross-examination that a point was put that the skilled person is actually told that they must read the document with interest, but that would be wrong in law. Whether Apple did put it that way does not matter. I agree with Conversant that that would not be the right approach in law. The skilled person when they read the cited prior art is not being told or being given a hint that a solution to a problem might be found in that document. Otherwise there is a real risk of injecting hindsight into the analysis.
128. The evidence from each expert was based on their own definition of the skilled person. Some of the cross-examination of each did involve questions addressed to the other defined skilled person. I have concluded that neither sides' definition is the right one. That does not mean the evidence is all irrelevant, since the detailed reasons each expert gave for various points can still be examined, and may or may not be accepted or applied to the skilled person as defined in this judgment.

Windows 98 for Dummies – Outlook Express

129. The text relied on was the second edition of the book published in 1999. The book as a whole is about 300 pages long. The relevant pages are in Chapter 13 about using Windows 98 applications at p274-284, i.e. a passage 6 pages long. It is legitimate to rely on an extract from a book, rather than imagining the skilled person reading the whole book, as long as the extract is coherent. The coherent section here starts at p275 under the heading "Managing E-mail with Outlook Express".
130. From the point of view of the skilled person reading these pages with interest, they would understand that they were reading a description of how to manage email on a desktop computer. Despite the very significant differences between the user interface of desktop computers as opposed to smart phones, the skilled person would not dismiss it without further ado just because they were focussed on smart phones. They would be aware that Microsoft's Windows 98 was a well known operating system for desktop computers. They might well know Outlook Express itself, as an email application Microsoft made available bundled with Windows.
131. The book contains images of the screen and explanatory text. It is convenient to show two screens from the book, figs 13-5 and 13-7. The first is Fig 13-5 as follows:



132. Here one can see an image of the whole desktop of the computer. The task bar runs along the bottom with the start button on the bottom left and blocks indicating open applications in the middle. One block is marked Paint Shop Pro. The block marked Outlook Express is shown as if it is a button which has been pushed inwards because that is the window which is presented. The rest of the screen is taken up by the Outlook Express window screen. This is the screen a user gets when they open the application (unless they have previously ticked the box near the bottom to tell Outlook Express to start by going straight to the email inbox). It has a title at the top (Outlook Express), then a menu running horizontally just underneath (File, Edit etc.) and then underneath that there is a group of four buttons to click on, with text and a visual indication of the functions they perform. Down the left is a graphic representing various folders and under that a block for contacts. Most of the screen is taken up with a page entitled Outlook Express with the user's name and then three sections are below – Email, Newsgroups and Contacts. On the right is a box for "tip of the day".
133. The same start screen is also shown in Fig 13-7, which is the screen focussed on by Apple. The figure in the book just shows the Outlook Express window and not the task bar but it is clear that that is just for convenience in the book. It is:



134. The significance of this representation (and in fact fig 13-5 too) is that here a combination of current data and commonly used functions are presented together to the user in a kind of application summary. So under the heading "E-mail" there is data: "1 unread Mail message", and there is a commonly used function: "Create a new Mail message". The user can go straight to the unread mail or to create a new mail message by clicking on the underlined text. This is "in addition to" using the main menu system.
135. Apple submitted, supported by Prof Beale, that in terms of claim 1 what is presented here is three applications – E-mail, Newsgroups and Contacts – as well as application summary information for each application. It is true that these three functions are themselves coherent subsets of functions and data within that application but the crucial thing is the user's point of view. Focussing on fig 13-7 alone potentially leads to trouble because that presents as if the only thing the user will see is a window entitled Outlook Express. However in fact, even though it may well dominate, for the user that window always exists on a desktop arrangement in which it is clear from the user's point of view that the application is Outlook Express. Prof Beale accepted that this desktop arrangement was well-known to the skilled person and accepted that the blocks on the bottom also indicate that Outlook Express is an application. The professor effectively described the court's task as an exercise about drawing a box around certain things to decide if they amount to an application. I accept that but I do not accept Prof Beale's opinion about where to draw these notional boxes in this case. From the user's perspective the application (in terms of the patent claim) is Outlook Express.
136. The differences between what is disclosed and what is claimed are the following. The device this system runs on is a desktop computer. It is not a smart phone. There is a single application summary window which displays both stored data and several commonly used functions. Outlook Express has its own main hierarchical menu system, for which this is an alternative, although that main hierarchical menu is not really the main hierarchical menu of the device. When this application summary window is displayed, the application (Outlook Express) is already open, unlike claim 1.
137. In my judgment it is a clear case. The skilled team is considering the problem of multiple applications to be added to the smart phone. One of the applications they are likely to want to consider putting on a smart phone is email. Nevertheless, having read these pages with interest the skilled team would be left thinking they had seen an interface developed for a completely different world from the one they are considering. Outlook Express is for the user interface on a desktop computer. It is as far as one could be from the user interface of a smart phone. They would know that email had been implemented on mobile devices including PDAs. Even though those are not phones, they are at least mobile devices and have a form factor relevant to the smart phone project. They would be regarded as of much more relevance to the problems the team had to solve. PDAs had email but they did not have an interface which looked like this part of Outlook Express. The skilled team would also know of the failure of the Windows CE, which was an attempt to squeeze the Windows desktop user interface onto a PDA.
138. In describing how his skilled person would react to seeing these Outlook Express screens, Mr McGrath said they would think it was a mess. I agree. It is a complicated and busy screen which contains so much detail one could not possibly compress all that down into a smart phone.
139. Part of Prof Beale's approach here depended on his definition of the skilled person as having a wider remit than the one I have decided upon but even on that wider basis I was not convinced by the professor's reasons. In my judgment his approach here involved an element which looked backwards at these screens, to see that they exhibit features which one can identify as relevant to the invention. It is a worthy approach to the study of user interface design but I do not believe it is the way a skilled team in this art would operate.
140. I reject this obviousness case.

The Simon cellular phone device

141. The cited prior art is a manual for the Simon cellular phone device. It was published in 1994. The device was produced by the computer company IBM and the phone company Bell South. To someone at the priority date the device was a rather brick-like unit. It was built like a telephone hand set and had a stylus based touch screen. There was no separate physical keypad. The unit itself can be understood from the following picture taken from the manual:

142. To a skilled team in 2000 this would obviously be an old product. In fact it was not a commercial success. Strictly speaking the team would not have known that but they would know that they had never seen it before. The Simon device was not part of the common general knowledge of a team in 2000, even as a well known failure.
143. What the skilled team would immediately see, having read the manual with interest, is that this was an early attempt at what they would regard as a smart phone, but using old technology. The technology was almost ancient even by 2000 standards. Nevertheless in my judgment the skilled team would be interested. Notably, given the problem the skilled team faced of dealing with multiple applications on a smart phone, they would see that here was an effort to do exactly that. The manual describes Simon in the following way:

"Your Simon is a cellular phone with personal organization and communication capabilities."

144. In 2000 that would be apt to describe aspects of the smart phone that the skilled team were working on.
145. The Simon device had a single display screen but presented two main interfaces to the user, a Phone screen and a Mobile Office screen. They are:

146. The skilled team would not only be interested in the Phone screen. They would also be interested in the Mobile Office screen not least because that is where multiple applications are presented to the user.
147. One of Simon's applications is Mail. The skilled team would look at them all but it is not hindsight to focus on Mail since that is one of the functions the team would be interested in. From the main menu screen, selecting the Mail application leads to the following screens. First presented is the one on the left. The user can navigate accordingly. The screens are:

148. Notable for the purposes of this case is that the left hand screen presents a window relating to an application with both commonly used functions and stored data together. The functions include "Send and Receive" and "Create Mail" and the stored data includes "3 Received Messages" etc.
149. In terms of email technology, at this time email generally operated as "pull" email in which the email client software on the local computer (desktop or mobile device) had to actively contact the central mail server to find out if any mail was available to be received and then fetch it. That is one reason why Simon had a function called "Receive", as the skilled team would understand. By 2000 a different form of email system was becoming available (and was common general knowledge) called "push" email in which the central mail server "pushes" new email notifications to the local device without the local device having to ask for it.
150. Comparing the prior art and the claim, although it does not matter whether Simon is strictly a smart phone within the claim or not, I would hold that it was. The first mail screen in Simon is a summary window which presents both stored data and commonly used functions at the same time. However it is not an alternative way of accessing the functions of the mail application, it is the way to access the mail application. Simon does not have any other windows which could be said to be an application summary window and so the plurality aspect of claim 1 is not satisfied.

Is the invention obvious?

151. What struck me when each expert was cross-examined about the first Mail screen in Simon was that they said the same thing. They (and their defined skilled person) would be interested in the way that screen presents functions and data together. (It was not just either witness parroting the legal test.) I believe that represents the reaction of the skilled team, operating without hindsight. To a UI/UX person in 2000 it is a new and interesting idea. It conveys the concept that in the confined space of a handheld device there is utility in presenting to the user a few commonly used functions and some stored data together at the same time at the start of their interaction with an application. Conversant made the point that the functions shown were apposite to what was shown at that level. So they were, but that does not alter the analysis.
152. The reason Apple sought to rely on Outlook Express above was because that too presented data and functions together but it did so in a very different context. With Simon however, here is the very context the skilled team is interested in, and the function/data combination would be understood by a UI/UX person as being presented in order to address the limitations of the interface. The fact that the underlying technology of Simon was old does not matter because what is interesting is the user interface. Identifying it here is not a backwards looking exercise.
153. Now as a matter of common general knowledge, by 2000, the skilled person is well aware of the idea of notifications (about stored data), pop ups, short cuts and/or user summoned shortcut menus (which are of functions). These are all ways in which the user can be given information or possible functions without opening the relevant application.
154. It would be obvious for the skilled team, in the light of their common general knowledge, to realise that they could use the mode of presentation shown here in Simon in a kind of window selected by the user in some way without opening the application. The skilled team's device would retain its ordinary main hierarchical menu system in order to access the application in the conventional manner. The size of screen they were considering on a smart phone could be comparable to the one in Simon although it would be more likely to be quite a bit smaller, but even in that case the idea would still be obvious. Also, an interest in wanting to be able to use the device "on the go" does not make it any less obvious.
155. Mr McGrath did not accept the invention was obvious to his skilled person but an important aspect of his reasons derived from his view of the experience and knowledge of that person and I have not accepted that. The skilled team I have identified had a much wider knowledge and experience than Mr McGrath's person. I have not accepted Prof Beale's definition either, but as I have explained already, none of this means the evidence as a whole does not allow the court to come to a conclusion.
156. Conversant referred to what it called a long pause Prof Beale gave when he was asked why the skilled person would "suddenly start thinking about shortcuts". I recall that passage. In my judgment it was not an indication that Prof Beale was stumped. It was a careful witness who appreciated the question was an important one and was giving his considered opinion. Conversant also submitted his answer did not make sense. I disagree. The professor gave his answer in his own terms but, put in terms of patent law, his answer was that things like short cuts were part of the common general knowledge. I accept that. Conversant also took a point on paragraph 169 of the Professor's first report but I do not accept that the distinction drawn by Conversant undermines the force of the evidence as a whole.
157. The idea would not just apply to the Mail application but to other applications too. The fact it was only disclosed for the Mail application in Simon does not make it inventive to apply the idea to multiple other applications too. I reject Conversant's case that this involves any sleight of hand or hindsight. The key is the genuine interest the skilled person would have in the first Mail screen.
158. The result is the invention claimed in claim 1 (and claim 2 for that matter). There is no other independently valid claim and the patent therefore is invalid.
159. In reaching this conclusion I have not placed weight on Prof Beale's evidence about the principle that designers aimed to "design the right thing" and "design the thing right". I do not doubt that they did but I do not accept the approach he described plays any part in the analysis.
160. Conversant took a point about the fact that Prof Beale did not seem to have visualised the end point of what he thought was obvious until asked about it in cross-examination. That was a weakness of his approach to Outlook Express but I do not agree the same point applies to Simon because of the similarity in form factor. At most Simon has a PDA form factor, but that was within the ambit of the skilled team's thinking.
161. Conversant also asked the rhetorical question – if it was obvious why was it not done before? If this invention was a ground breaking idea which revolutionised the industry and solved a long standing problem then there might be something in the rhetoric, but it is not. It is a worthwhile idea, but it is one which was obvious over an item of prior art which was not itself common general knowledge. The rhetorical question does not help.
162. Conversant accepted, rightly, that if claim 1 was obvious, no other claim was independently valid.

Agr Evo-obviousness

163. I will not spend a lot of time on the Agr Evo-obviousness attack because it is not well founded. A convenient way of characterising the technical contribution made by the claim is that it provides an improved method of navigation to what had been called "the best of the app", i.e. the stored data and commonly used functions. It is improved because it is a better, quicker and/or easier method. The user does not have to use the feature, but the device is still a better device because it makes this function available so that some users can use it. I am satisfied the patent discloses this contribution, that it is a contribution made by the claimed invention and that it supports the claim as it stands.
164. The fact the patent refers to the number of steps in a hierarchical menu is not a way of working out whether there is a technical contribution. No doubt it is possible to infringe badly and set things up so that one can reach the data or function in fewer clicks using the main hierarchical menu. However the way the main hierarchical menu is set up is up to the designer. There is no need for the claim to limit the number of steps the user must take to reach the data or functions concerned.
165. Apple emphasised that Mr McGrath accepted a point made by Prof Beale that it was possible to put the invention into practice in a way in which it will not always reduce the number of steps the user has to take to arrive at the data or function, as compared to the number they had to take in the prior art. That is also not relevant. It does not alter the fact that the invention presents an improved user interface because it provides a way of setting up an alternative route for the user to use, as well as the main hierarchical menu, which is capable of being a better, quicker and/or easier method. That is an advantage.
166. Apple submitted that claim 1 was not limited to a situation where, once the application was opened, the user is presented with a screen where the data of interest is prominent or the function of interest has been activated. Therefore the user will be required to take further steps in that case. I suppose that is so but again it is of no relevance. The claim does have to include this limitation to be commensurate with the technical contribution. This is not a claim consisting of a chemical formula covering millions of discrete chemical compounds, which may or may not have the biological function on which the invention rests. This claim describes the essential features of the invention in a way the skilled person can understand as fairly supported by the disclosure. It would be possible for the skilled person to put it into practice in a way which does not work very well, but that is not the test.
167. Finally, I do not accept the suggestion from Apple that if it makes a contribution at all, then it is only in the context of Mr McGrath's "true" mobile phones. It is correct that Mr McGrath said that the problem identified in the patent is only interesting on a small screen with limited input means. However Prof Beale's evidence on obviousness over the prior art was that the step he regarded as obvious was worth taking irrespective of screen size. I accept that aspect of Prof Beale's evidence (which is not the same thing as saying it was obvious).

Conclusion

168. Apple's iPhone product would infringe the patent if the patent was valid, but it is not. The patent is invalid because the claims are obvious over the Simon prior art. All the other invalidity arguments advanced by Apple do not succeed.

Postscript

169. On receiving the draft judgment Conversant submitted that:

"with our English v Emery Reimbold & Strick [2002] EWCA Civ 605 obligation in mind, we wish to draw to the Court's attention that when dealing with the case of obviousness over the SIMON manual, the Court has not dealt with our arguments that (a) without hindsight, the skilled person had no reason or motivation to take that screen out of the carefully designed SIMON menu system in which it sits; (b) taking the 'mode of presentation' in the top menu of the SIMON Mail app outside the app (i.e. without opening the app) would be counterintuitive; (c) the whole analysis was driven by hindsight; and in particular (d) the final step of applying this 'mode of presentation' to each app is pure hindsight (notwithstanding §157), a point supported by what actually happened in PDA user interfaces: the multiple notification box (as illustrated in §52 of the draft Judgment)."

170. I believe the judgment deals with all these points. Points (a) and (b) do not face up to genuine interest shown by both experts in the first Mail screen which I have found reflects the thinking of the skilled person. I do not accept point (c) and have explained why in my judgment this analysis is not infected by hindsight. Point (d) is not an English v Emery point.