AI developments and usability Linus Wretblad (IPscreener / Uppdragshuset, SE)

1. New practice of AI tools

2. IPscreener explores AI to boost innovation
Prior art search
Classification
Assisted Reading
Decision support

3. Review of Innovation workflow vs performance
$12,000
Avg. Cost of Patent
from Filing to Grant
+ = $616,000
Est. total Cost /
Patent application
$604,000
Avg. R&D Spend Per
Patent Application Filed

4. For evaluation measurementis the key

5. A Baseline measurement model setup
SCORE CALCULATION
How many of the
patent documents
cited by the examiner
are found by the tool
within a specificed
number of hits
shown?
Position Hit List
1 US2015053224 A1
2 WO2011075652 A1
3 US2017151364 A1
4 US2019224085 A1
5 US8622252 B2
6 EP1583232 A2
7 US6953484 B2
8 US2017151363 A1
9 US2003126692 A1
10 US9375739 B2
11 US2004045098 A1
12 US9938072 B2
13 US2019082808 A1
14 GB2011548 A
15 US2010270399 A1
16 US6502697 B1
17 US2006198692 A1
18 EP1239606 A1
19 US6634037 B2
20 WO2016092108 A2
Simple
Example
R10=50%
R25=100%

6. What parameters for measurements?
X/Y citations
only vs
including A
citations?
Spread among
classes/years/
assignees?
Number of
topics per
area or set?
Effect of
machine
translations vs
language?

7. Baseline metrics; recallperformance per class
0,0
10,0
20,0
30,0
40,0
50,0
60,0
70,0
80,0
A01B
A01N
A23G
A41B
A43D
A47C
A61D
A62B
B01D
B04C
B21B
B22F
B24B
B26B
B27M
B30B
B41K
B43M
B60H
B60T
B61L
B63G
B65D
B67D
C01G
C06B
C07K
C09C
C10K
C12M
C21B
C25C
D02G
D06B
D21B
E01F
E04C
E06B
F01L
F02M
F04D
F16H
F21L
F23G
F24C
F27B
F41F
G01F
G01R
G03D
G05F
G06Q
G09B
G10L
H01B
H01Q
H02N
H03L
H04R
Recall ratio of the number of citations cited in the patent office examination report that were
also found by IPscreener within top 100 hits. The scores are based on 175 000 EP & WO
applications, where title + abstract of the document were used as query text.

8. Grouped metrics; recall perdomain
Recall ratio of the number of citations cited in the patent office examination report that were
also found by IPscreener within top 100 hits. The scores are based on 5000 EP & WO
applications per WIPO domain, where title + abstract of the document were used as query text.
0,0%
5,0%
10,0%
15,0%
20,0%
25,0%
30,0%
35,0%
40,0%
45,0%
50,0%
R10 R25 R50 R100

9. Customer metrics;recall on Portfolio
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
A61K A61P C07D C07C C07K
Recall 100 long Recall 100 short
Recall ratio of number X/Y citations from patent office report found by IPscreener within top 100
hits. The scores are based on 3907 patent applications from latest 10 years, title/abstract vs full
text input.
Input format Recall 10 Recall 25 Recall 50 Recall 100 Avg length
Long 31,29% 39,39% 44,05% 48,88% 120293
Short 23,99% 30,59% 34,86% 39,72% 2007

10. 1) What to enter; Performancevs text length
Performance on Recall 100 for different inputs lengths (characters) of queries (patent
applications) sampled across random technical domains.

11. 1) User analysis; Performancevs text length

12. 2) Effect of user influencing the AI search
0,00%
10,00%
20,00%
30,00%
40,00%
50,00%
60,00%
70,00%
Recall 100 Mechanical Recall 100 Biotechnology
Short Full
Short input was in both cases abstracts with an average 1k characters. Long input was the full
text, where Mechanical had an average of 34k characters and Biotechnology 89k characters.
Green is a manually optimized input text and potential ranking/parameters added..

13. 3) What effectof MT on AI performance?
Analysis based on 1966 families where input performance from native or manual translation of
full text was compared to machine translated family members from Japanese and Mandarin.
Average text input text length 38326 characters.
0,00%
5,00%
10,00%
15,00%
20,00%
25,00%
30,00%
35,00%
40,00%
Recall 10 Recall 25 Recall 50 Recall 100
Native english MT of Mandarin MT of Japanese

14. Challenge: Patents most boring document ever?

15. Support elementsof searching & reading
Right Document(s) Right Passage(s) Right Interpretation(s)
QUERY: A disposable mug for
serving hot liquids such e.g.
coffee where the mug is
provided with an isolative
shell. By this protective
arrangement, a user may hold
the cup with the bare hands
even though the content is
too hot
ANSWER: …an insulated
container jacket, the
arrangement adapted for a
one-time usage for heated
fluids by comprising an outer
frusto conical shaped sleeve
having disposed separated
channels…

16. AI Interpretation; segments & relevancy
Abstract
Description
Claims
Green (Background)
Techniques for making
disposable mugs
Red (Similar)
Disposable coffee mug
with thermal isolation
Yellow (Relevant)
Disposable mugs having
several spaced layers

17. AI on Interpretation; segments & relevancy
Nex generation AI tools categorize, selects & group
relevant paragraphs rather than documents

18. AI on Interaction; The interfacing of data

19. Case review– A complex text input

20. Case review- Hit list & innovation landscape
Cited X document
in patent office
report is the hit
number 11

21. Case review- AI assist reading in fulltext
The AI zooms
automatically in on
the most relevant
text passage (right)
and claim (left)

22. Case review- The examiner citation vs AI selection
The AI is highlighting
the most relevant text
part, within relevant
examiner citation

23. What to expect ahead? More AI on usability!

24. Final aim – boost and qualify innovation

25. Thanks For Listening!
Questions?
linus@ipscreener.com
www.ipscreener.com