Evidence of insufficient quality of reporting in patent landscapes in the life sciences

Despite the importance of patent landscape analyses in the commercialization process for life science and healthcare technologies, the quality of reporting for patent landscapes published in academic journals is inadequate.

Patents in the life sciences are a critical metric of innovation and a cornerstone for the commercialization of new life-science- and healthcare-related technologies. Patent landscaping has emerged as a methodology for analyzing multiple patent documents to uncover technological trends[1], geographic distributions of patents[2], patenting trends and scope[3], highly cited patents[4] and a number of other uses[5]. Many such analyses are published in high-impact journals[6,7,8], potentially allowing them to gain high visibility among academic, industry and government stakeholders. Such analyses may be used to inform decision-making processes, such as prioritization of funding areas, identification of commercial competition (and therefore strategy development), or implementation of policy to encourage innovation or to ensure responsible licensing of technologies. Patent landscaping may also provide a means for answering fundamental questions regarding the benefits and drawbacks of patenting in the life sciences, a subject on which there remains considerable debate[9,10,11] but limited empirical evidence.

The aim of the patent-landscaping process is to capture a set of pre-defined patent documents and to analyze them in some manner (Box 1). In general, the process comprises three main stages: (i) designing and conducting searches, (ii) data cleaning and curation, and (iii) data analysis and interpretation[5]. This process is analogous to the process for conducting systematic reviews of academic literature; however, in patent landscapes, the patent documents replace academic articles. Similar to systematic reviews, patent landscapes are important in allowing higher-level insights to be drawn that could not be achieved by analysis of the documents in isolation. Unlike systematic reviews, however, patent landscapes are often published under different guises than those for original research articles, possibly leading to a lack of emphasis on the disclosure of the key pieces of information that underpin the analyses and conclusions.

Box 1: Definition of a patent landscape
A landscape is an analysis of the relationships between multiple sets of indicators or of those indicators measured against temporal, technical or spatial dimensions. In the case of a patent landscape, at least one of the indicators is patent publications or some aspect thereof. A landscape seeks to encompass an entire population of relevant data rather than a random sample drawn from that population (adapted from ref. 5).

As in any type of research, to allow for reproducibility and effective evaluation of accuracy and quality, it is essential that studies are reported adequately. The specific methodologies used, and the results generated, must be reported; clear justification and the aims of the investigations need to be provided to assess the validity of any conclusions, and any funding or conflicts of interest should be disclosed to assess potential biases. The importance of adequate reporting has been heavily reinforced and policed in healthcare-related research[12,13,14], including in systematic reviews[15]; however, outside of healthcare-related research, discussion has been more limited. Despite the important potential implications of patent landscapes from a scientific, commercial and political perspective, criticism has emerged over a lack of standardization, transparency, clear justification of the methodology, and reporting of the ad hoc investigations being conducted[16,17].

Here, we provide the first systematic assessment of the quality of reporting in patent landscapes in the life sciences that have been published in academic journals. Given the heterogeneous nature of the methodologies used for, and the questions addressed by, patent landscaping[16], consistent methodological practice may not be practical or useful, and this could indeed stifle innovative methodologies from emerging. Clarity of reporting, however, is essential regardless of the approach taken and the hypotheses tested, and assessment of reporting clarity should represent a significant component of the peer review process and should be expected in scientific articles. Patent landscapes are published in a variety of journal types of different impact factors, and as various different article types (e.g., review articles and original research articles). The relationship between these indicators and reporting quality is also explored.

Results

Search results, study inclusion and study characteristics. Searches and screening resulted in a total of 81 full-text studies for inclusion in this systematic review (Fig. 1; Supplementary Methods; Supplementary Tables 1 and 2). After the removal of duplicate records, we screened the abstracts of 3,348 articles for relevance and found 577 records that could not be excluded based purely on reviewing the abstracts and titles and that were assessed as full texts, where available. We excluded 380 articles because, after deeper examination, they were not patent landscapes or reviews (n = 201), there was no mention of a search for patents (n = 78), they were not life science focused (n = 100), or they were not journal articles (n = 1). An additional potentially relevant 116 articles were identified, which we were unable to exclude based purely on abstract or title, but for which the full text was not available.

Quality of reporting

In 81 articles assessed for compliance with respect to 20 items that were considered together to represent an adequately reported study, none of the articles reported all of the items included in our checklist that were relevant to them. Mean compliance across all articles was 64% (s.d. ± 15%). Table 2 lists the number and percentage of articles reporting each item.

Table 2: Number and percentage of articles that have reported items in the reporting-quality checklist (n = 81; n = 70 for items with asterisks (*))

The percentage compliance was not normally distributed (Shapiro–Wilk test of normality, W = 0.93392, P ≤ 0.001), and normality could not be achieved through transformation of the data. Therefore, Kruskal–Wallis tests were used for comparisons of compliance against to article and journal type, and a Spearman’s rank correlation coefficient was calculated for compliance with respect to the SCImago journal rank (SJR; http://www.scimagojr.com/journalrank.php). No correlation between SJR and compliance was seen (Fig. 3a). Mean compliance in research, review and ‘other’ articles were 70% (s.d. ± 8%), 55% (s.d. ± 16%) and 60% (s.d. ± 17%), respectively (Fig. 3b). A significant effect of article type on compliance was observed. Mean compliance in scientific and ‘other’ journals was 64% (s.d. ± 16%) or 63% (s.d. ± 11%), respectively, which did not differ significantly (Fig. 3c).

Discussion

The final step in conducting academic research is not the analysis of data but the communication and dissemination of the research in a manner that allows the approach taken to be determined and reproduced, and the validity of the findings to be assessed[18]. The primary mechanism of this dissemination and communication within academia is via journal publications, which represent the means by which the quality of a study is assessed. Our analysis provides evidence that the quality of reporting in patent landscapes in the life sciences in such publications is generally inadequate.

Reporting of methods and results. Many articles do not report critical methodological items. Without full reporting of the following methodological items—eligibility criteria of patents to be included; search strategy; databases, dates, patent offices and components of patents searched; software used; patent selection process; and details of analysis conducted (Table 2, items 9 to 17)—reproducing and validating a patent landscape methodology is, in our opinion, not possible. Eight articles (9.9%) reported all of the methodological items that were applicable to them. Just one article (1.4%) reported that patent selection was blindly reproduced, and 24% of articles listed the patents included in the study, i.e., the data set on which the conclusions and analyses were based.

The lack of reporting of key methodological items and results challenges the interpretation of the conclusions and renders the ability to reproduce studies (e.g., to update them) impossible in many cases. A patent landscape represents a considerable amount of effort, and without clarity over their methods and results, the impact of this effort can be substantially reduced; greater clarity would allow additional analysis to be conducted by researchers in the future, maximizing the potential benefit of the research[19]. Improvements in the reporting of the methods and results could allow for meta-analyses of patent-landscape-related papers and for independent researchers to use the results in their own investigations for other purposes.

Potential for conflicts of interest. Conflicts of interest and funding sources are also relatively poorly reported items (45%). Patents are inherently linked to commercial interests; therefore, the declaration of conflicts of interest and any funding sources is important in the evaluation of potential biases. In clinical trials, lower quality of reporting has been associated with increased effect sizes[20]. Although there is no effect size, as such, to evaluate in patent landscaping articles, incentives for the portrayal of inaccurate information by authors or funders could easily be conceived, and the promotion of transparency for conflicts of interest and funding should therefore be encouraged.

Explaining reporting quality. Our findings are not limited to lower-quality journals. It is generally assumed that the higher the quality of, and therefore the quality of reporting in, an article, the higher the likelihood that it will published in a high-impact journal[18,21]. However, no statistically significant correlation was observed between the SJR and the quality of reporting in patent landscaping articles (Fig. 3a), suggesting that reporting quality is not currently an important determinant in the publication of patent landscapes in high-quality journals. This is unlike other areas, in which a number of studies conducting similar analyses in different fields have found marked relationships between impact factor and reporting quality[22,23], and these may be due to the fact that there are a limited number of well-reported patent landscapes on which to model reporting.

Small but significant differences in reporting quality are associated with different article types. Perhaps expectedly, reviews are less well reported than research articles (Fig. 3), which indicates that structured paper formats might be more conducive to complete reporting. Given that patent landscapes may just be one component of an otherwise narrative review, structured reporting, as in many research articles, is unlikely to be practical for all papers. However, emphasizing that the patent-landscaping component of any paper represents research that others may wish to rely on for further analysis could help to improve reporting, and methodologies could be reported in supplementary materials.

Analysis types. Patent landscaping has previously been described, for the most part, as relatively simplistic[5], and much of the analysis recorded and observed in our study supports this. A large proportion of papers presents numbers of patents over time, patents per geographical region, patents per assignee or inventor, or other count data (Fig. 2). There is no inherent problem in the use of such analyses; however, patent numbers and other such measures cannot alone portray all of the information that may be useful or required by the reader. For example, it is possible that within one research area, there are a large number of patents that focus on a very narrow set of inventions, whereas in another research area there may be very few patents with broad scope. Without some consideration of the information within the patent documents, discerning these differences would be challenging. With an appropriate discussion of limitations, these issues could be allayed; however, limitations were discussed in less than half of the included papers (46%).

In some cases, more advanced analytical approaches have been used to interrogate patent documents in more detail, although these still raise some concerns. Software was used to thematically cluster patent documents and present the outcome visually in ‘cluster maps’ in some papers (28%). From such data, areas of high patent activity are often identified[1,3,24] and may be used to identify ‘gaps’ in the technology or research landscape[25]. Other papers have mentioned the use of patents to identify gaps, although not through an explicit methodology[26,27]. Proprietary software is often used to generate cluster maps, but the algorithms underpinning such software are rarely discussed or detailed in the papers using them, and in some cases the algorithms underpinning the software do not appear to be publicly available at all. Without detailed analysis of patent claims by an expert, the identification of gaps in research or technologies is difficult to ascertain with existing methods; however, very few papers conducted any form of claims analysis (2%). As mentioned above, an appropriate discussion of the limitations would be useful in addressing these concerns.

Outlook. The findings of this systematic review are congruent with similar investigations that have been conducted in other fields reporting omissions in methods[28,29] (including statistical methods[30]), incomplete presentation of data thereby preventing analysis in the future[31] and inadequate statements for conflicts of interest[32]. The findings also provide empirical evidence for statements previously made in relation to the quality of patent landscapes[5]. To address reporting issues in other fields, a great number of reporting guidelines have been developed, including checklists of items that should generally be reported, and which together represent an adequately reported study. Introduction[33] and endorsement[34] of guidelines have been associated with improved quality in the reporting of clinical trials. Our study provides the empirical justification for the development of a guideline to improve the quality of reporting in patent landscaping articles, which is registered as being under development on the Equator Network website (http://www.equator-network.org). The purpose of the guideline is to improve transparency and standardization of reporting to allow reproducibility, comparability and accurate evaluation of patent landscapes.

In the meantime, researchers involved in patent landscaping should aim to improve reporting quality, motivated by the fact that it will ultimately increase the impact of the research and provide a greater contribution to the scientific community. Authors have a responsibility to ensure that information is clearly presented to readers in all forms of research, and patent landscapes should not be considered an exception. Journals and reviewers, too, are responsible for ensuring that reporting is adequate and must act as the gatekeepers by working with authors to ensure that transparency is achieved. We are not suggesting that overwhelming the reader with methodological details and extensive results in every article is required; supplementary materials can and should be used to ensure comprehensive reporting. With improved transparency and comprehensive reporting, patent landscaping will continue to provide useful insights, critically, in a manner that allows them to be reproduced and fairly evaluated.

Limitations. This systematic review focused solely on the reporting quality of patent landscapes in academic journals. It should be noted that patent landscapes are commonly reported outside of academic journals, such as those conducted by government bodies or industry. The quality of reporting in certain available publications35 in general appears to be quite good, perhaps due to the lack of limitation with regards to document length in comparison to academic papers. However, even seemingly detailed landscapes lack full disclosure of search terms[36,37] and contain only superficial explanations of the algorithms employed[38]. The findings of this study may still, therefore, be useful in the context of reporting studies outside of academia.

Additionally, we were unable to access the full texts of 116 potentially relevant articles that could not be excluded on the basis of the abstract alone. There is no reason to suspect that these articles would have provided evidence contrary to that presented.

Conclusions

Patent documents are an exceptionally rich source of information that can and should be mined and analyzed for a number of purposes. The breadth of possibilities for analysis of patent documents may preclude the development of standardized methodologies, and as such this may not be possible. However, without adequate reporting, the full value of such analyses will not be realized, and even the most rigorous and elegant investigations may be limited in reach because they simply cannot be reproduced and critically evaluated. With transparent reporting and consideration to the other observations made in our paper, the value of patent landscape studies within and beyond the academic community could increase considerably.

Author contributions

J.A.S. and Z.A. contributed equally to this work. A.J.C. and D.A.B. contributed equally to this work.

References

1. Roberts, M. et al. Nat. Biotechnol. 32, 742–748 (2014).
2. Clark, K. et al. Vaccine 29, 4086–4093 (2011).
3. Johnson, T.S. et al. Recent Pat. DNA Gene Seq. 7, 2–12 (2013).
4. Panja, S. et al. J. Intellect. Prop. Rights 20, 39–50 (2015).
5. Bubela, T. et al. Nat. Biotechnol. 31, 202–206 (2013).
6. Fiala, J.L.A. & Lowery, D. Nat. Rev. Drug Discov. 15, 8–9 (2016).
7. Egelie, K.J., Graff, G.D., Strand, S.P. & Johansen, B. Nat. Biotechnol. 34, 1025–1031 (2016).
8. Paradise, J., Andrews, L. & Holbrook, T. Science 307, 1566–1567 (2005).
9. Jaenichen, H.-R. & Pitz, J. Cold Spring Harb. Perspect. Med. 5, a020941 (2014).
10. Rai, A.K. & Sherkow, J.S. Nat. Biotechnol. 34, 292–294 (2016).
11. Edwards, A. Nature 533, S70 (2016).
12. Schulz, K.F., Altman, D.G. & Moher, D. Br. Med. J. 340, c332 (2010).
13. Altman, D.G. & Moher, D. in Guidelines for Reporting Health Research: A User’s Manual (eds. Moher, D., Altman, D.G., Schulz, K.F., Simera, I. & Wager, E.) 1–13 (John Wiley and Sons, Ltd., 2014).
14. von Elm, E. et al. Lancet 370, 1453–1457 (2007).
15. Liberati, A. et al. Br. Med. J. 339, b2700 (2009).
16. Grant, E., Van den Hof, M. & Gold, E.R. World Pat. Inf. 39, 3–10 (2014).
17. Gold, E.R. & Baker, A.M.J. Law, Info. Sci. (20) 1–22 (2012).
18. Jordan, K.P. & Lewis, M. Musculoskelet. Care 7, 137–142 (2009).
19. Chalmers, I. & Glasziou, P. Lancet 374, 86–89 (2009).
20. Schulz, K.F., Chalmers, I., Hayes, R.J. & Altman, D.G. J. Am. Med. Assoc. 273, 408–412 (1995).
21. Garfield, E. Can. Med. Assoc. J. 161, 979–980 (1999).
22. Devereaux, P.J., Manns, B.J., Ghali, W.A., Quan, H. & Guyatt, G.H. Control. Clin. Trials 23, 380–388 (2002).
23. Montané, E., Vallano, A., Vidal, X., Aguilera, C. & Laporte, J.R. BMC Clin. Pharmacol. 10, 2 (2010).
24. Arshad, Z. et al. Expert Opin. Drug Discov. 11, 321–332 (2016).
25. Sastry, K.R., Rashmi, H.B. & Badri, J. J. Intellect. Prop. Rights 16, 139–153 (2011).
26. Cucoranu, I.C., Parwani, A.V., Vepa, S., Weinstein, R.S. & Pantanowitz, L. J. Pathol. Inform. 5, 16 (2014).
27. Swamy, H.M. et al. Recent Pat. DNA Gene Seq. 6, 64–71 (2012).
28. Glasziou, P., Meats, E., Heneghan, C. & Shepperd, S. Br. Med. J. 336, 1472–1474 (2008).
29. Reveiz, L. et al. PLoS One 5, e12484 (2010).
30. Fleming, P.S., Koletsi, D., Polychronopoulou, A., Eliades, T. & Pandis, N. J. Dent. 41, 265–270 (2013).
31. Chan, A.-W., Hróbjartsson, A., Haahr, M.T., Gøtzsche, P.C. & Altman, D.G. J. Am. Med. Assoc. 291, 2457–2465 (2004).
32. Kesselheim, A.S., Wang, B., Studdert, D.M. & Avorn, J. PLoS Med. 9, e1001280 (2012).
33. Ziogas, D.C. & Zintzaras, E. Ann. Epidemiol. 19, 494–500 (2009).
34. Turner, L. et al. Cochrane Database Syst. Rev. 11, MR000030 (2012).
35. UK Intellectual Property Office Informatics Team. Eight great technologies: a summary of the series of patent landscape reports. (Intellectual Property Office, Newport, UK, 2014). https://www.gov.uk/government/publications/eight-great-technologies-the-patent-landscapes
36. Chilton, V., Mantrand, N. & Morel, B. Patent landscape report: microalgae-related technologies. (World Intellectual Property Organization, Geneva, 2016). http://www.wipo.int/publications/en/details.jsp?id=4042&plang=EN
37. UK Intellectual Property Office Informatics Team. Graphene: the worldwide patent landscape in 2015. (Intellectual Property Office, Newport, UK, 2014). https://www.gov.uk/government/publications/graphene-the-worldwide-patent-landscape-in-2015
38. UK Intellectual Property Office Informatics Team. Eight great technologies: regenerative medicine. (Intellectual Property Office, Newport, UK, 2014). https://www.gov.uk/government/publications/eight-great-technologies-regenerative-medicine
39. Moher, D., Liberati, A., Tetzlaff, J. & Altman, D.G. Ann. Intern. Med. 151, 264–269, W64 (2009).

Acknowledgements

This study is funded primarily by the UK Medical Research Council (J.A.S.). The authors wish to thank the following organizations for their contributions, as funding and events partners, to the CASMI Translational Stem Cell Consortium (CTSCC), and without whom the consortium and the benefits it will bring to stem cell translation would be constrained: the SENS Research Foundation, GE Healthcare, the Center for Commercialization of Regenerative Medicine (CCRM), Sartorius Stedim Biotech (formerly TAP Biosystems), Lonza, the California Institute for Regenerative Medicine (CIRM), UK Cell Therapy Catapult, the US National Institutes of Health Center for Regenerative Medicine, the New York Stem Cell Foundation (NYSCF), ThermoFisher Scientific, Eisai, Medipost (US), Medipost (Korea), Celgene, Roche and Oxford Biomedica. This study is funded primarily by the UK Medical Research Council (J.A.S.). D.A.B. gratefully acknowledges personal funding from the Oxford Musculoskeletal National Institute for Health Research Biomedical Research Unit (NIHR BRU), the Saïd Foundation and the SENS Research Foundation. Z.A. gratefully acknowledges support from CTSCC. The authors would also like to thank the EQUATOR Network for their generous support.

Author information

Affiliations

James A. Smith and Andrew J. Carr are in the Nuffield Department of Orthopedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK

James A. Smith, Zeeshaan Arshad and David A. Brindley are at The Oxford–UCL Centre for the Advancement of Sustainable Medical Innovation, University of Oxford, Oxford, UK

Zeeshaan Arshad and Hannah Thomas are in the School of Medicine, University of St. Andrews, St. Andrews, UK

David A. Brindley is at the Department of Paediatrics, University of Oxford, Oxford, UK.

Competing interests

D.A.B. is a stockholder in Translation Ventures Ltd. (Charlbury, UK) and IP Asset Ventures Ltd. and Biolacuna Ltd. (Oxford, UK) and Biolacuna Ltd. (Oxford, UK), companies that among other services provide biomanufacturing, regulatory, and financial advice to pharmaceutical clients. J.A.S. has consulted with IP Asset Ventures Ltd. and Biolacuna Ltd. D.A.B. also is subject to the CFA Institute’s codes, standards, and guidelines, so he must stress that this piece is provided for academic interest only and must not be construed in any way as an investment recommendation. Additionally, at time of publication, D.A.B. and the organizations with which he is affiliated may or may not have agreed to and/or pending funding commitments from the organizations named herein.