Measuring the effectiveness and impact of an open innovation platform

Today, most pharmaceutical companies complement their traditional R&D models with some variation on the Open Innovation (OI) approach in an effort to better access global scientific talent, ideas and hypotheses. Traditional performance indicators that measure economic returns from R&D through commercialization are often not applicable to the practical assessment of these OI approaches, particularly within the context of early drug discovery. This leaves OI programs focused on early R&D without a standard assessment framework from which to evaluate overall performance. This paper proposes a practical dashboard for such assessment, encompassing quantitative and qualitative elements, to enable decision-making and improvement of future performance. The use of this dashboard is illustrated using real-time data from the Lilly Open Innovation Drug Discovery (OIDD) program.

Introduction

The concept of Open Innovation (OI) was first coined by Henry Chesbrough [1] to describe the paradigm by which enterprises allow free flow of ideas, products and services from the outside to the inside and vice versa in order to remain competitive, particularly in rapidly evolving fields where there is abundant, relevant knowledge outside the traditional walls of the enterprise. Initially, this idea was applied to the emerging realms of computer hardware and software, but since then it has spread to a number of areas in industry, academia and government. According to Chesbrough, there is a continuum from fully open to fully closed enterprises, with the degree of openness depending upon a number of internal and external factors affecting that particular enterprise, including intellectual property (IP) and level of regulation [1]. Today, not only has research and publication into the topic of OI grown exponentially, but typical OI methodologies – including idea competitions, customer immersion and crowdsourcing – are routinely understood and leveraged to seek, refine and spur innovation in a variety of industries [2]. However, given its complexity, highly regulated environment, and reliance on IP concepts [3–7], the Pharmaceutical industry has been relatively slow to adopt OI approaches, particularly in the R&D arena.

Notwithstanding the undeniable barriers to implementation of OI methodologies in pharmaceutical R&D, the literature seems to indicate a shift from the debate around the concept of OI to focus instead in the discussion of actual examples of OI approaches. These approaches were developed by different research institutions worldwide with an emphasis on the description of collaborative models that have shown a certain level of success, and the analysis of the best practices underlying such emerging models [8–12]. At its core, this shift accentuates the significant pull for change driven by the well-known shortcomings of the traditional, closed model for pharmaceutical R&D, paired with the actual usefulness of OI models when correctly deployed and implemented. It could be argued that the recent proliferation of OI approaches in pharmaceutical R&D is actually a manifestation of the inherent openness of the scientific discovery process, and that the implementation of OI methods in pharmaceutical R&D is no longer a philosophical problem of ‘whether to do it’ but instead becomes a logistical problem of ‘how to do it’[11,12].

One way to understand the apparent paradox between the difference and complementarity of traditional closed versus OI models in pharmaceutical R&D is to consider that the long-term objectives of both approaches are actually the same – that is, the successful production of new medicines. The main contrast between the two approaches can then be described in terms of different short-term goals and tactics to achieve this end. Thus, the traditional closed innovation model generates, refines and develops all inputs, ideas and hypotheses inside the organizational walls. In contrast, the OI models seek to enrich the diversity of available options by incorporating external inputs, ideas and hypotheses and by possibly refining and developing them internally and externally. Secondly, organizations large and small may not have all the expertise they need within their walls to resolve a particular challenge. In this manner, OI programs attempt to leverage external parties to develop a large, broad pool of opportunities and talent complementary to those available internally. Likewise, while traditional R&D concentrates on the systematic prioritization of internal opportunities to focus on those options that survive the internal decision-making funnel, open innovation casts a wider net to capture the largest possible library of options before the utilization of prioritization and selection mechanisms that help to focus on the most interesting or promising opportunities. Finally, traditional innovation flow tends to be depicted as a linear process, whereby promising opportunities move forward and others get put back on a shelf; open innovation requires an iterative process in which inputs may be continually screened and recycled in future searches [13].

Moreover, the efficiency of the traditional R&D paradigm tends to be measured using long term economic indicators (such as number of IND/launches, or cost per IND/launched medicine) that rely on retrospective historical analyses that incorporate the entire R&D process from idea inception to commercialization [14–16]. In contrast, the evaluation of OI models used in early drug discovery struggles to benefit from such historical analyses given (a) the relative novelty of the models and (b) the difficulty of segregating the contributions of internal versus external inputs with respect to the outcomes of the enterprise at large. More importantly, we argue that macroscopic, retrospective and economic leading indicators traditionally used to measure R&D productivity provide an inadequate way of measuring the efficiency of the early phases of drug discovery in general and of OI approaches in particular.

The pharmaceutical R&D process

The pharmaceutical R&D process is generally depicted as a series of sequential stages progressing in linear fashion from hypothesis generation and validation through identification of a clinical candidate, followed by clinical development and finally commercial distribution [15]. While the overall process can be understood as linear, there is a clear inflexion point at Candidate Identification (CI) with a marked differentiation between phases before and after CI, with respect to the types of activities implemented and the manner in which progress is achieved [Chart 1].

The R&D process after CI is focused on collecting, understanding and leveraging vast amounts of information about a single molecular entity that is considered competent to test the clinical hypothesis in humans, followed by the careful design and implementation of the clinical experiments required to demonstrate its safety and efficacy in the clinical setting. Thus, development activities after CI are indeed linear and sequential, defined by pre-established protocols and subject to high levels of external regulation and therefore data collection. Performance metrics usually chosen to evaluate the pharmaceutical R&D process in its entirety often emphasize the post-CI phase and are focused on clinical success/attrition, overall published costs and dates, and other public domain or retrospectively collected information which more or less aims to describe the progress of a single Candidate Molecule from its identification as such to its approval for distribution as a medicine [14–16].

In contrast, the R&D process before CI takes place iteratively through the repetition of a series of ‘design-test-analyze’ learning cycles fueled by the constant evaluation and prioritization of thousands of distinct molecular entities undergoing biological tests which are carefully designed and selected to provide some sort of indirect indication of future clinical relevance. At this stage, there are massive amounts of information collected on many new molecular entities, but relatively little is known about each one – and rarely does one single chemotype follow a linear path of progression to the end game [17,18]. The process through which diverse chemotypes are evaluated, prioritized and refined – while at the same time probing the biological system of interest – is labyrinthine, fraught with detours and dead ends. The overall goal becomes the generation and incorporation of all information into progressively better biological understanding and better structural designs that are capable of regulating the system in a manner that is most consistent with the underlying clinical hypothesis. Thus, the exercise of calculating leading indicators of success or process performance at this time counts with the unavoidable difficulties that (a) the process has an inherently cyclical nature, and (b) the actual chemical entity that may eventually reach clinical development is not yet known, nor are its properties. Meaningful metrics must then be capable of estimating the efficient transmission of information to enable option prioritization with minimal possible investment, regardless of the final outcome.

How should OI models be measured?

Open innovation approaches in pharmaceutical R&D are quite varied, based on the details of execution and/or implementation chosen by each company. For example, some of the factors contributing to this variety may include the nature of the arrangements between partners, the actual exchange taking place within the partnership, the type of incentives used and the short-term goals the partners intend to achieve [4,7,9,11,12]. On one hand, this variety underscores the vibrancy of an emerging field and contributes to the concept of a balanced ‘ecosystem’ with plenty of offerings for all interested participants. On the other, it also increases the complexity of isolating common measurable elements within such a heterogeneous mixture.

Approaches to measure the degree of ‘openness’ in the biopharmaceutical industry that are based on a retrospective analysis of publicly available information have been published [19]. Likewise, a general framework for the assessment of OI approaches incorporating IPOO metrics (inputs, process, output, outcomes) but also considering instrumental, conceptual and symbolic metrics has been described [20]. We have previously proposed an OI framework for biopharmaceutical companies composed of a set of qualitative factors, which incorporate aspects of operational complexity and degree of ownership and control [13]. Within the context of the OI models specifically applied to early drug discovery, there remains a need for the description and publication of generally applicable, easily implementable metrics to facilitate the determination of whether a novel OI business approach is adding value to the ‘traditional’ innovation process within a given organization. This understanding is of paramount importance to the survival of any OI approach, not only for the enablement of decision-making within the program per se, but also for the justification of its very existence within the organization.

With an emphasis on the avoidance of high-level metrics focused on the overall R&D process, which use retrospective analyses of economic returns as described elsewhere [14–16], we now propose a framework incorporating four broad categories that can be applied to the assessment of any OI program focused on the early stages of drug discovery: Investment, Pipeline Health, Return, and Culture & Capabilities [Chart 2]. Although there are a variety of different OI strategies currently being used in the market, this framework can be adapted to assess the key performance indicators of specific programs of interest.

Investment

The first category for assessing OI programs in pharmaceutical R&D is Investment, or resources that are being used to support the program. All OI programs require some sort of investment, whether it is being set up as a unique organizational unit or leveraging the resources of other units within the organization. Within the early drug discovery space, broad groupings that may be helpful to consider include compound inventorying costs, data collection and lab costs, dedicated human resources, collaboration incentives, and supporting technology (platform, software, website, etc.) costs. In addition, startup costs may be considered (though these will vary significantly between different programs), and other costs will be shared with other areas within the wider organization. (One example of shared resources might be using empty instrumental capacity for an OI-driven experiment in a traditional R&D unit.) Thus, the specifics of how shared investments may be allocated, whether or not to include startup costs, and other decisions will depend on the particular program being assessed. In the broadest sense, the Investment category will provide insight into the overall resources being used for the purpose of advancing OI-based ideas. This assessment category may also guide decisions by Leadership on whether to scale back or increase investments based on budgetary or operational needs in synergy with internal-based experimentation.

Pipeline health

The second category to consider in evaluating an OI program in pharmaceutical R&D is the Strength of its Opportunity Pipeline. One of the primary goals of any OI program is to source a wide range of ideas and opportunities from a diverse audience, in order to expand the accessible research substrate beyond the internal teams within the organization. Broad groupings to consider include the number and/or variety of proposals received (e.g. new targets, new chemical structures, and new enabling technologies), collected data, as well as the ability for data to be shared across the organization, changes in data generation over time, and published scientific results. The opportunity pipeline category will shed light on the diverse nature of opportunities being received and how these opportunities are being moved forward through the program alongside already available internal opportunities. These features allow Leadership to understand how OI programs are increasing possibilities beyond the work being done through more traditional R&D means and approaches.

Returns

Generally, the returns from early drug discovery are not realized in dollars or clinical validation since clinical trials, let alone product commercialization, are yet to occur. As the ultimate proof of medical and commercial success is poorly linked to its leading indicators at this stage of the process, Returns from OI programs largely take the form of hypotheses tested (proven or rejected) and new research outcomes available to the organization (conclusions or future directions). Potential groupings of measurable outcomes to consider within this space could be the type and variety of therapeutic areas benefitting from the external inputs, the percent of opportunities being moved forward towards advanced evaluation, and the number and type of follow-up research collaborations that result. These returns demonstrate the added value being produced through the OI program vis-à-vis the traditional approaches, and support continued investment in the program. Undoubtedly, a strong program will produce valuable returns both to the pharmaceutical company as well as the partnering institutions in order to remain sustainable. Follow-up collaborations generated from OI programs deliver value to both partners and combine internal and external resources to help advance promising opportunities along the arduous R&D discovery process while also informing other research throughout both organizations.

Culture & capabilities

Our final proposed assessment category is focused on the formation of the external network of participants, their adherence to the OI program, and the type and variety of internal capabilities being offered to fuel the pipeline of opportunities. All OI models thrive when they are able to tap into diverse networks which provide access to ‘non-liquid’ innovation, that is, innovation not readily accessible by more traditional means. In its broadest sense, the Culture of the OI program is reflected in its ability to form and sustain a network, and the Capabilities offered by both the organization and the participants become the reciprocal incentives that make the network possible. In the case of OI programs focused on early drug discovery, a diverse and healthy network is particularly relevant as it produces wider divergence of possible opportunities for consideration at the outset where the investment is minimal; in turn, the capabilities of the organization are leveraged to help differentiate and advance the most promising opportunities produced by the capabilities of the participants. Specific examples of cultural metrics would be the geographic diversity and types of participants and program satisfaction scores. In terms of capabilities, the number and diversity of offerings included in the program are important factors to consider. Overall, these metrics show how successful OI practices result in the creation of a satisfied and engaged network, and to which extent are the internal capabilities successfully leveraged to incentivize the network.

Case study: the Lilly Open Innovation Drug Discovery program

In order to test and substantiate the metrics framework described above, we obtained detailed data from the Lilly Open Innovation Drug Discovery (OIDD) [21]. Lilly is regarded by many to be a pioneer in the implementation of open innovation models in the pharmaceutical industry, with a long, documented history of collaborative approaches along the entire R&D value chain [22–24]. The OIDD program, first launched in 2009 [25], is a web-based platform that allows external investigators access to Lilly’s early drug discovery scientific capabilities to increase the pool of potential opportunities for further development available to Lilly. The program targeted participation by academic laboratories and smaller, independent companies wishing to partner with Lilly on early drug discovery projects. In exchange for enabling the external investigators’ research, Lilly obtains first right of negotiation to access promising opportunities. Within the OIDD program, the fundamental concerns that are central to the invention process are addressed directly at the outset, by explicitly establishing within the legal framework of the program that all IP rights remain with the submitter at all times [26]. To date, several hundred scientists at over 400 affiliated institutions in more than 30 countries worldwide have joined the program and are actively engaged in program participation, and details regarding first-generation collaborative outcomes based on the screening offering have been recently described in the literature [27]. Given the high level of external acceptance of the program and the internal impact already realized, the company recently expanded the available offerings by introducing additional engagement models based on capabilities other than compound screening. Thus, participants are able to access computational tools for designing molecules with drug-like properties, submit libraries to be synthesized remotely using Lilly’s automated equipment and request a library of proprietary compounds to test emerging biological assays [28].

For the purposes of the scope of this publication, we focused our analyses on the longest established offering, OIDD Screening [29], which enables participating scientists to submit chemical compounds for experimental evaluation in Lilly’s proprietary biological assays. Submitters electronically enter chemical structures, also known as virtual compounds, into the OIDD user interface, applying an algorithm that allows an automatic selection process to be run in structure-blinded fashion in order to protect the IP rights of the submitters [30]. Once a compound is determined to be eligible for screening, the submitter ships a physical sample to Lilly’s laboratories for testing across a variety of projects or modules associated with specific therapeutic areas of interest to Lilly. Each module incorporates a number of tiered in vitro biological assays (primary, profiling and confirmatory), each of which generates scientific results or datapoints that are then electronically shared with the submitter in real time. As compounds make progress down each module’s testing scheme, associated results are constantly evaluated and used in selecting the compound subset with the desired biological profile: a final set of promising compounds called confirmed actives is then identified [31]. The full package of accumulated datapoints for a given confirmed active compound forms the basis for the establishment of a follow-up collaborative agreement, or a possible publication by the submitter. At this stage, follow up collaborations take many forms depending upon the specific module and data package for the active, as well as the interests and capabilities of the submitter and/or institution. Broad categories of collaborations are those focused on biology (chemical scaffold evaluation or material scale-up) or on chemical diversity (sample acquisition and compound harvesting).

As the OIDD Screening offering has been continuously running since the program inception in 2009, there is a robust set of continuous data available for evaluation and analysis. Within this context, we have applied the concepts previously illustrated on Chart 2. For the remainder of this paper, the term OIDD program will be synonymous with the Screening offering as described above.

Investment

Note: In our analysis of the OIDD program, we have excluded one-time startup costs such as website and platform software development, initial marketing, publications and infrastructure and have instead focused on understanding and optimizing run rates.

The primary investment categories for the OIDD program include (1) the cost of shipping compound samples from affiliated organizations, (2) generating datapoints, (3) collaboration incentives, and (4) dedicated full-time employees, or FTEs. These categories, detailed below, provide an understanding of the recurrent investment needed to operate OIDD since its inception, and allow effective business planning and identification of opportunities for increased efficiency.

First, shipping costs include sending empty vials from Lilly to the submitter, and returning filled vials to the Lilly site for evaluation. Second, the program utilizes datapoint generation as the actual program currency used to attract participation from submitters. Regarding OIDD datapoint generation, we must note an important nuance that differentiates this investment from billable expenses such as shipping costs: the total amount invested in datapoint generation is not an additional operational expense required to support OIDD activities – rather, this amount results from the estimation of available capacity captured by OIDD research activities, fitting within the natural ebb and flow of ongoing internal research activity, already paid for by normal business operations – that is, sunk costs for instrumentation, plus reagents/materials and FTEs.

Third, we calculated full spend on collaboration incentives, derived from all follow-up agreements regardless of type. As noted in Fig. 1, about half of the follow-up collaborations signed have relied exclusively on some sort of in-kind contribution, requiring no further financial support from Lilly. These collaborations leveraged existing Lilly internal infrastructure and/or scientific capabilities, applied to the completion of experimental work relevant to the collaboration. The remaining balance, or incentive collaborations, comprises those cases where the relevant experimental work is instead completed by the partner, after the transfer of financial support from Lilly.

The above categories taken altogether, the OIDD program has invested approximately $7.1M to date since its inception ($150 per compound), as a mix of operational and capabilities investment.

The fourth and final investment to consider is FTE count, which serves as an indicator of program growth and evolution over time. As shown in Fig. 2, the OIDD program infrastructure was originally established with IT and scientific business functions alone – before scientists were directly involved. As the program grew beyond the original proof of concept and the number and type of scientific modules included increased over time, the composition of the dedicated team reached its current distribution – an almost even split between employees from IT, Business, and Science functions. Once a critical mass is reached, the number of dedicated FTEs may be scaled up or down based on budgetary and other internal considerations. The OIDD program investment thus described should be viewed in the context of the other evaluation categories, discussed below.

Pipeline health

The second dashboard category is Pipeline Health, which focuses on the progression of opportunities from inception to conclusion. Within the context of the OIDD program, this pertains to the compound structures moving through submission, filtering and acceptance; followed by the associated compound samples progressing through registration, testing, and selection of confirmed actives; and culminating in the negotiation and execution of follow-up collaborations for promising opportunities. Important factors to consider in this category include the pass/fail rate at each phase of the process; the types of therapeutic areas benefitting from external opportunities identified by the program; and the number and types of follow-up collaborations being established.

Over the life of the program, OIDD program participants have uploaded more than 400,000 chemical structures to the platform and submitted almost 300,000 for consideration by Lilly, of which about 140,000 structures were accepted by the filtering algorithm. These virtual compounds resulted in the registration of almost 50,000 physical compound samples received at Lilly labs for biological testing. Considering all sources, including external vendors or internal synthesis, this accounts for approximately 10% of the total set of registered chemical entities. Statistical analysis of OIDD inputs over the life of the program demonstrates that on average 500 new compounds are registered each month, resulting in a steady increase in the OIDD compound collection over time [Fig. 3]. These figures help Leadership understand the relative value proposition added by OIDD to internal teams, and also demonstrate the efficiency of the platform at narrowing down a large number of varied opportunities to focus on those with the most potential interest to internal teams. A detailed measure of the level of increased chemical diversity accessed by the OIDD program has been discussed elsewhere [29].

The scientific results generated through OIDD have a double value proposition in that they can be used both by Lilly and by program participants – that is, individual investigators and affiliated institutions who want to publish their research. Each datapoint represents ‘knowledge gained’ or ‘hypothesis tested’ as discussed above, which can be leveraged by both partners at the same time for different purposes (decision-making vs publication). The submitter obtains all datapoints in real time through the OIDD platform, while Lilly scientists are able to identify opportunities to advance internal projects. Additionally, as each compound sample can be tested multiple times in many different modules while inventory remains available, Lilly has the possibility of generating repeat opportunities from each compound accepted and tested with the ultimate goal of identifying previously unknown compounds with a promising biological profile warranting advanced in vivo testing. To date, the OIDD program has generated more than 1.8 M datapoints as a result of the testing of more than 45,000 crowdsourced compounds, with clear predomination of Oncology and Endocrine/Cardiovascular therapeutic areas as a reflection of the number and types of Lilly projects launching active screening campaigns over the life of the program.

The accumulated datapoints can also be viewed in terms of the phase during which they were produced, with nearly 69% of all data generated during the primary screening phase and 26% as secondary screening. As the complexity and cost of assays increases in that order, this is also a demonstration of applied efficiency of the OIDD program to operationalize a ‘phased investment’ approach. As fewer, most promising opportunities are advanced to later phases, the data generated in the confirmatory phase drops to 5% of total datapoints.

Returns

The almost 50,000 registered compounds are tested in a variety of project modules spanning therapeutic areas of internal interest. For each module, a subset of confirmed actives identified by the testing scheme as the most promising opportunities with the desired biological profile is chosen for advanced evaluation [31]. To date, approximately 1800 confirmed actives have been moved forward, with approximately two thirds of those associated with Oncology projects [Fig. 4]. As indicated above, this simply reflects the preponderance of Oncology projects in the Lilly active project pipeline over the life of the program but it does speak to the alignment of OIDD with internal priorities. Given the specific design features of the OIDD platform, about 90% of confirmed actives are classified as small molecules, with the remainder falling into Natural Product, Peptide, and Chemical Probe categories. As an important reference point, this total number of confirmed actives yields an approximate 4% confirmed hit rate over all projects, which provides useful perspective with respect to typical success indicators for early drug discovery.

As some of the confirmed actives identified by OIDD projects move forward to the collaboration negotiation phase, selected submitters may be invited to have these actives added to the Lilly collection for further screening, synthesis and other evaluation. In these cases, larger samples are purchased by Lilly and become available to a wider audience for future internal research uses, therefore providing additional opportunities for collaboration. An additional 1.2% of registered compounds have been added to the Lilly collection via this route, increasing its chemical diversity and providing a further endpoint for the crowdsourcing model [32].

Culture & capabilities

Creating a sustainable network of potential partners and a culture of participation is of paramount importance for an OI approach regardless of the enterprise context. Within Pharma, a vibrant OI community is a leading indicator of the potential richness of inputs – a diverse cohort of external partners, ideas and hypotheses. Assessing the number of affiliated institutions and submitters is perhaps the best indication of the size of the OI network and a measure of the diversity of potential opportunities. As of the end of 2016, OIDD had 576 registered users, with 388 of those considered active participants. As seen in Fig. 5, there was 38% affiliation attrition from 2014–2015 following the revision of the Program Agreement in March 2015, which expanded the scope of the agreement between Lilly and OIDD affiliated institutions [26]. One trend that can be seen in the data is the increased number of active submitters overall, which also shows expansion in the use of OIDD offerings from within affiliated organizations. The average annual affiliation growth rate is 28.4%, and the average annual submitter growth rate is 57.6%, demonstrating strong user intake in the years since program inception.

Other factors that might be of interest to further understand the network composition include types of participant organizations and their geographic distribution. Currently, half of the participating institutions in OIDD are based in the U.S. and about a third is in Europe, with the remaining 20% distributed across all continents. This distribution is also reflected within the context of follow-up collaborations, underscoring the unbiased nature of the crowdsourcing approach and the value of continued investment in leveraging platform technology to enable access to traditionally untapped geographies.

Another way to look at the OIDD culture is to analyze participant survey results. OIDD institutions were administered a questionnaire that contained a variety of questions on topics such as their experiences with the OIDD program and the technology platform, and their overall satisfaction. The results indicate that the vast majority (86%) of survey respondents felt highly satisfied with the program, as measured by their likelihood to recommend OIDD participation to others.

In addition, by segmenting out the survey results by type of responder (e.g. Academic vs Small Company) or by type of respondent role (Administrator vs Investigator), some interesting differences become apparent. For example, a larger percentage of Small Company participants rated the program highly when compared to Academic participants. The Small Company participants also felt more familiar with, and had more frequent use of, the program offerings and platform. Contributing to this difference is a lesser extent of separation of roles (Administrator vs Investigator) in the industrial versus the academic setting, pointing to interesting cultural differences between customer segments. These results have been helpful to program Leadership as they develop strategies for expanding and managing the growth of OIDD offerings going forward, and may influence approaches used in future collaborative negotiations with potential partners.

Finally, consideration of capabilities within the OIDD context is understood in terms of offerings and modules, as described above [33]. As program capabilities are the central component of participation incentive, there is clear understanding by Lilly Leadership of the need to maintain a sustainable flow of scientifically relevant modules and assays. We looked at the evolution of both program modules and assays over time and observed a decline in the number of assays but only a slight fluctuation in the number of modules (projects). This reflects team-level decisions on which individual screens are going to be included in the primary, secondary and confirmatory stages: thus, we inferred that the decline reflected a continuous push for increased program efficiency and a more deliberate use of datapoints produced for relevant decision-making. On the other hand, the relatively steady number of modules over time reflects on Lilly Leadership’s decisions to expand the scope of the program well beyond the original small selection of projects which were included at the outset [25], to better leverage the value of the crowdsourcing approach across a greater variety of drug discovery areas.

Conclusions

The pharmaceutical R&D environment – particularly in its earliest stages – has been traditionally known as closed and secretive, specifically regarding its protection of intellectual property. At the same time, the need for efficiency and a streamlined R&D process has driven the industry towards change and increased openness. OI approaches in pharmaceutical R&D have the potential to increase the pool of potential opportunities by incorporating starting points beyond the bias of internal knowledge, while also helping introduce efficiencies. Furthermore, OI programs inherently support an iterative process of innovation, allowing for greater number of exchanges associated with potential opportunities, which is consistent with the iterative nature of the early drug discovery process.

However, it is challenging to measure the effectiveness of an OI approach in early drug discovery due to the long cycle times, high attrition rates and complexity of the overall process. The proposed dashboard described in this article offers an approach that accommodates the need to quantify the value and impact of early OI programs by focusing on leading indicators that can be measured and controlled: the resource dedication to the program, the robustness and advancement of identified opportunities, and the interaction/sentiment between the organization and its external partners.

As an example of this proposal, the dashboard was applied to data facilitated by the Lilly Open Innovation Drug Discovery program (OIDD). The OIDD program includes confidentiality and intellectual property provisions for all participants, which addresses a fundamental challenge in OI approaches. The total number of OIDD affiliated institutions and submitters is a good indication of network size and measures the diversity of potential opportunities that the organization can secure – a diverse cohort of partners generates more inputs, ideas and hypotheses. To date, the OIDD program has generated more than 1.8 M datapoints as a result of the testing of almost 50,000 crowdsourced compounds, yielding approximately 4% confirmed actives across all projects. This value is comparable to typical early drug discovery success rates, higher for some projects.

In the case of Lilly, our application of this strategy has also generated insights around the importance of the technological infrastructure that is needed to support OIDD offerings and the ability of the program to: (1) facilitate the exchange of information with external participants while ensuring the protection of their IP, (2) have mechanisms in place to track generated opportunities, (3) integrate with internal systems in order to understand its complementarity with traditional R&D efforts, and (4) have flexibility to quickly incorporate new modules as the program offerings develop over time. Looking forward, the pharmaceutical industry should continue to test, leverage, and benefit from models that improve drug discovery and R&D processes, while effectively quantifying metrics to pragmatically and continuously evaluate these programs – things we believe the proposed dashboard offers.

Conflict of interest

All authors have no conflicts of interest to declare.

Acknowledgements

The Lilly authors wish to thank Alan Palkowitz, PhD for his vision, leadership and sponsorship; Timothy Grese, PhD and Jim Myers, JD, PhD for their thorough technical review and editorial comments; Andy Dahlem, PhD for his support of this article; and the entire OIDD team for their hard work, commitment and ingenuity.

The Deloitte authors wish to thank Pierce Goldsmith, Rachel Skelly, and Yao Zhang PhD for their contributions to the data analysis and research found in this article.

References

1. H.W. Chesbrough Open Innovation: The New Imperative for Creating and Profiting from Technology Harvard Business School Press, Boston (2003)
2. H.W. Chesbrough, W. Vanhaverbeke, J. West New Frontiers in Open Innovation Oxford University Press (2014)
3. J. Hunter, S. Stephens Is open innovation the way forward for big pharma? Nat. Rev. Drug Discov., 9 (2010), pp. 87-88
4. M. Bianchi, et al. Organisational modes for Open Innovation in the bio-pharmaceutical industry: an exploratory analysis Technovation, 31 (2011), pp. 22-33
5. Y.L. Bennani Drug discovery in the next decade: innovation needed ASAP Drug Discov. Today, 16 (2011), pp. 779-792
6. I. Khanna Drug discovery in pharmaceutical industry: productivity challenges and trends Drug Discov. Today, 17 (2012), pp. 1088-1102
7. A. Schuhmacher, et al. Models for open innovation in the pharmaceutical industry Drug Discov. Today, 18 (2013), pp. 1133-1137
8. J. Hunter Collaboration for innovation is the new mantra for the pharmaceutical industry. Spring 14 Drug Discov. World (2014) (http://www.ddw-online.com)
9. D. Ehrismann, D.D. Patel University-industry collaborations: models, drivers and cultures Swiss Med. Wkly. (2015), 10.4414/smw.2015.14086 (http://www.smw.ch/content/smw-2015-14086/)
10. J. Bentzien, et al. Crowdsourcing in pharma: a strategic framework Drug Discov. Today (2015), 10.1016/j.drudis.2015.01.011
11. M. Reichman, P.B. Simpson Open innovation in early drug discovery: roadmaps and roadblocks Drug Discov. Today (2015), 10.1016/j.drudis.2015.12.008
12. D. Holmes A new chapter in innovation Nature, 533 (2016), pp. S54-S55
13. Executing an Open Innovation Model: Cooperation is Key to Competition for Biopharmaceutical Companies; Deloitte website, Accessed 6 September 2016, http://www2.deloitte.com/content/dam/Deloitte/us/Documents/life-sciences-health-care/us-lshc-open-innovation.pdf.
14. I. Kola, J. Landis Can the pharmaceutical industry reduce attrition rates? Nat. Rev. Drug Discov., 3 (2004), pp. 711-716
15. J.W. Scannell, et al. Diagnosing the decline in pharmaceutical R&D efficiency Nat. Rev. Drug Discov., 11 (2012), pp. 191-200
16. J.A. DiMasi, et al. Innovation in the pharmaceutical industry: new estimates of R&D costs J. Health Economics, 47 (2016), pp. 20-33
17. J.P. Hughes, et al. Principles of early drug discovery Br. J. Pharmacol. (2010), 10.1111/j.1476-5381.2010.01127.x
18. Research & Development in the Pharmaceutical Industry, OIDD website, Accessed September, 2016. https://openinnovation.lilly.com/dd/includes/pdf/R_and_D_White_Paper_Final.pdf.
19. F. Michelino, et al. Measuring open innovation in the bio-pharmaceutical industry Creativity Innovation Manage. (2015), 10.1111/caim.12072
20. M. Erkens, et al. Measuring open innovation: a toolkit for successful innovation teams Performance, 6 (2014), pp. 12-23
21. OIDD website, What is OIDD? https://openinnovation.lilly.com/dd/what-is-oidd/ Accessed September, 2016.
22. The story of Eli Lilly’s open innovation journey: how one company developed a mature model PDMA Visions, 34 (2010), pp. 19-22
23. M. Lyles, Y. Lai, C. Dhanaraj Innovation Without Walls: Alliance Management at Eli Lilly and Company Harvard Business Review (2007)
24. P.K. Owens, et al. A decade of innovation in pharmaceutical R&D: the chorus model Nat. Rev. Drug Discov., 14 (2015), pp. 17-28
25. J.A. Lee, et al. Open innovation for phenotypic drug discovery: the PD2 assay panel J. Biomol. Screen., 16 (6) (2011), pp. 588-602
26. OIDD Website, Program Agreements, https://openinnovation.lilly.com/dd/getting-started/program-agreements.html.
27. A. McGilvray Compound screening: fresh hunting ground Nature, 533 (2016), pp. S65-S67
28. OIDD website, What We Offer, https://openinnovation.lilly.com/dd/what-we-offer/ Accessed September 2016.
29. M. Alvim-Gaston, et al. Open innovation drug discovery (OIDD): a potential path to novel therapeutic chemical space Curr. Topics Med. Chem., 14 (2014), pp. 294-303
30. OIDD Website, Protecting Your Intellectual Property, https://openinnovation.lilly.com/dd/what-is-oidd/protecting-your-intellectual-property.html.
31. Within Lilly, the term “confirmed active” excludes so-called PAIN compounds that interfere with assays or have promiscuous activity, thus over-inflating success rate. For an excellent description of PAINs, see Baell J., Walters M.A. (2014) Chemistry: Chemical Con Artists Foil Drug Discovery. Nature 513(7519):481–3. DOI: 10.1038/513481a.
32. OIDD Website, Compound Acquisition, https://openinnovation.lilly.com/dd/what-we-offer/compound-acquisition.html.
33. OIDD Website, OIDD Screening, https://openinnovation.lilly.com/dd/what-we-offer/screening.html.