Cloud Hosting

1. Executive summary

The Cyber Security Academic Start-up Accelerator (CyberASAP) programme is funded by the Department for Digital, Culture, Media and Sport (DCMS) and delivered by the Knowledge Transfer Network (KTN) [footnote 1]. It supports the commercialisation of UK cyber security research and helps academic researchers to turn ideas into fully rolled-out commercial projects by developing the academics entrepreneurial skills. It has two phases as set out in Figure 1:

Figure 1: CyberASAP overview

This evaluation report relates to the CyberASAP programme which was initially piloted in 2017/18 and has been delivered over five years to 2021/22, with one cohort of researchers each year. [footnote 2]

Conclusions and recommendations against each of the core evaluation questions are outlined below.

Was the programme delivered as intended?

The programme has evolved each year and delivery has taken place as intended in the Memorandum of Understanding (MoU) between DCMS and Innovate UK.

What worked well, or less well, for whom and why?

There are several examples of elements of CyberASAP that worked well, including:

There were also several examples of areas for development. This included:

Recommendation 1: we recommend that decisions to fund CyberASAP are communicated at least 9 months in advance (rather than the current timeframe of c. 4 months) to allow sufficient time for programme promotion with potential entrepreneurs from underrepresented groups.

Recommendation 2: we recommend that CyberASAP engages with university Technology Transfer Offices (TTOs) to identify potential candidates for the programme.

What can be learned from the delivery methods used? / Were there any unexpected or unintended issues in the delivery of the intervention? / How did external factors influence the delivery and functioning of the programme?

The majority of survey respondents (98%, n=54) were satisfied or very satisfied with the programme structure.

However, COVID-19 limited face-to-face interaction and qualitative feedback suggested this was detrimental to relationship building between participants and with investors. Other factors that impacted delivery across all years of the programme included that:

How can the existing programme be improved to become more effective?

The existing programme could be improved by:

Recommendation 3: we recommend that SMART outcomes are further developed for CyberASAP and there is detailed reporting on these. KTN to develop based on the CyberASAP ToC metrics and agree with DCMS. Evidence should be collected by KTN for progress against each using both published data and primary research with participants. For example, those noted in the DCMS MoU with KTN for Year 6 and including:

The programme has delivered business impacts. To date 108 projects have been supported by CyberASAP [footnote 6] and of these 12 were categorised as spinouts; 2 were acquired by other firms and 5 developed a patent.

The DCMS funding has leveraged 12,382,895 in additional investment (for example corporate / private investment, acquisition, VC investment, angel investment, and seed funding / equity fundraising).

Programme participants were able to meet with organisations, however it will take time to understand the impacts from these meetings and any relationships.

It has also been successful in developing entrepreneurial skills and confidence, including for participants who did not progress to the proof-of-concept stage however completed part (a) or all of phase 1.

To what extent has the programme been successful in commercialising academic research / accelerated the process to commercialisation?

Overall, 26% (n=12 of 47 tracked participants) have spun out companies successfully [footnote 7], some of whom provided their project outputs as an open-source product and through published articles. In addition, survey respondents reported an increased capability to commercialise their research following participation in the programme as:

Nevertheless, participants still found it difficult to commercialise their research after the programme had finished and felt that further support is required, either from their university and / or by having further support CyberASAP. Specific examples included ensuring that universities and TTO staff maintain commitment to commercialisation throughout and are as committed to the commercialisation of the Intellectual Property (IP) as the investors. Based on the contribution analysis conducted CyberASAP has had:

While 77% (n=41) of the 53 respondents who answered the question had not yet received investment following the programme, some of those will only be at the stage of seeking investment. Therefore, the above contribution analysis is only part of the answer. It will be important to continue tracking progress of the projects after the programme ends to measure longer term impacts.

Recommendation 4: we recommend that the outcomes and impacts from the programme continue to be tracked. It should be a requirement within the delivery partner contract to provide evidence of the capability and business outcomes being achieved for participants at one, two and three years after completing the programme to provide evidence of the longer-term benefits.

We recommend DCMS set up a monitoring template that covers all the outcome and impact measures expected from the programme. This should be completed by the delivery partner for participants when they finish the programme and then at 6, 12 and 24 months after completion.

To what extent does the evidence suggest future funding would be more effective if targeted differently (at less well funded universities, for example)?

The premise of the current CyberASAP model is to attract and identify the most promising commercial opportunities from different parts of the UK academic research base and to make funding opportunities accessible to universities from all regions as well as those outside the Academic Centres of Excellence in Cyber Security Research (ACE-CSR) and Russell Group.

The project has been successful in achieving this as 82 of the 108 (76%) participating universities from non-Russell Group institutions and 84 (77%) outside of London.

Due to the programme both reaching several non-Russell Group universities and there being a lack of notable difference in outcomes between Russell and non-Russell Group universities (as discussed below), a focus on the best commercial opportunities should remain.

To what extent does the success of the programme differ between different cohorts, types of firm/idea, university?

There is no notable difference in outcomes between cohorts, projects or between universities, including Russell Group and non-Russell Group institutions (with one exception). The only notable variations were:

In addition, there is a similar proportion of the 108 Russell Group and non-Russell Group university projects that are in development, licensed and acquired. [footnote 9] However, a higher difference in the amount of spin-outs achieved was identified between the two groups. KTN feedback suggests this is likely to be the result of Russell Group universities typically being better funded and possessing mature and well-established commercialisation capacities. [footnote 10] This is shown by the proportion of Russell and non-Russell group universities that have:

Do participants join other cyber growth programmes after completing CyberASAP?

Evidence obtained from stakeholder interviews indicated that most of the CyberASAP participants who fully completed all phases of the programme did not go on to other cyber growth programmes. Those who did not progress through all stages of CyberASAP tended to either (a) re-apply for CyberASAP; (b) progress to the London Office for Cybersecurity Advancement (LORCA); (c) progress to the Cyber Runway programmes, or (d) access other sources of funding / development programmes. This was in part due to CyberASAP helping them to better understand the schemes available, suggesting the programme plays an important role in the wider cyber growth and innovation ecosystem.

What are the additional or unintended benefits of the programme?

Delivery partner feedback suggests the programme has helped to change the way universities approach commercialisation as some are now more willing to be pragmatic about how much equity they will receive. For example, Royal Holloway University applied a policy which assigned IP to the start-up company in exchange for an under 10% share to make the deal more attractive to investors. [footnote 11] KTN have also suggested the programme is starting to see repeat participation from universities who are beginning to change their approach in this area.

RSM Consulting LLP were commissioned by the Department for Digital, Culture, Media and Sport (DCMS) [footnote 12] to undertake independent evaluations of the CyberASAP, Cyber Runway and UKC3 programmes. The evaluations will help DCMS to understand the impact of these programmes and the findings will be used to inform the development of future interventions.

The CyberASAP programme supports the commercialisation of UK research into cyber security and helps academic researchers to turn ideas into fully rolled-out commercial projects by developing the academics entrepreneurial skills. In doing this, it recognises the barriers that academics face when commercialising research, including the lack of dedicated time available to research the market and to validate potential products.

This evaluation report relates to the CyberASAP programme which was initially piloted in 2017/18 and has been delivered over five years to 2021/22, with one cohort of researchers each year. [footnote 13]

The evaluation incorporates delivery and performance across all years to date.

The evaluation methodology was agreed with DCMS and includes the following stages:

Scoping phase

(1) Project initiation meeting: the project commenced with a project initiation meeting involving the evaluation team and DCMS to: (1) review and agree the evaluation methodology and timetable; (2) discuss access to relevant information and (3) finalise arrangements for project management and progress updates.

(2) Desk research and analysis: a review of the strategic and delivery context for the programme and mapping was conducted to identify other sources of funding available to support the commercialisation of cyber security academic research.

(3) Review of programme documentation setting out rationale for funding measures: review of the programme business case; MoUs between DCMS and Innovate UK; Key Performance Indicators (KPIs) agreed between DCMS, Innovate UK and KTN; and previous research / theories of change relating to the programme to identify the rationale for the intervention and the outputs and impacts expected from it.

(4) Development of ToC: an online workshop was facilitated with DCMS staff involved in the business case for funding and the design and management of the programme to test and refine the draft ToC and associated metrics. The final ToC (see Appendix B Theory of Change) was used to inform the research tools that were developed, specifically the participant survey and guides for the participant, delivery partner and case study interviews. (5) Evaluation plans for each programme: an evaluation plan was developed detailing the design and approach being taken to address the evaluation questions. This was informed by the ToC and outlined how each of the ToC metrics would be measured.

Data collection

(1) Analysis of programme monitoring information / impact information and published data: to inform the assessment of programme performance against its core KPIs (as per the ITT) and those in the agreed ToC.

(2) Surveys and consultations: this involved:

Survey methodology: the survey was designed by RSM UK Consulting in collaboration with DCMS to collect evidence against the key evaluation questions and ToC metrics. As it was not possible for participant details to be shared with RSM UK Consulting without consent, an online survey link was distributed via KTN, with subsequent reminders by email and targeted telephone follow-up to ensure a representative sample across regions and cohorts.

Note: where n= is used during survey analysis, it is referring to the number of respondents responding in a certain way / to a specific answer choice, rather than the entire respondent base. Where applicable, a base number has been provided in figure titles or in text to provide more general information on total number of respondents. This base number will occasionally vary from the overall survey participant number of 55 depending on relevance of the question and if respondents choose not to answer.

Participants survey profile:

Table 1: CyberASAP participants respondents by region (base number = 55)

Table 2: CyberASAP participants respondents by cohort (base number = 55)

(3) Counterfactual: nine interviews were completed with those participants successful in applying to phase 1 of CyberASAP but did not proceed to phase 1(b) or phase 2.

(4) Case studies: four in-depth case studies were developed to provide qualitative insight into the benefits of participating in the programme. These were selected to provide a representative sample across regions, cohorts, and stage of idea development and are shown in Table 3.

Table 3: Case Studies

Analysis and reporting

(1) based on the ToC, 6 contribution statements were developed describing the outcomes CyberASAP intends to achieve and how

(2) based on the data collected in the previous stages the strength of evidence was assessed against each contribution statement, as well as evidence of any other factors that have contributed

The strength of evidence was determined by reviewing:

High strength of evidence includes:

(3) The contribution of CyberASAP to the expected results as described by each contribution statement was assessed as strong, some, or negligible, with:

Reporting: included a progress presentation, interim and final reports, a final presentation, and a closing workshop with DCMS which will act as a learning event.

Limitations

Counterfactual: it was not feasible or appropriate to contact those who were unsuccessful in their application to the CyberASAP programme to form a counterfactual group as their characteristics were too dissimilar to complete a robust regression discontinuity analysis - based on information provided by KTN due to:

The absence of a robust counterfactual means that caution should be applied to over interpreting the results of the impact evaluation, in that it is not possible to rule out the possibility that some of these impacts will have occurred under the counterfactual.

Therefore, it was agreed with DCMS that a qualitative counterfactual approach would be applied by completing interviews with participants who did not complete all phases of the programme in order to identify what they did instead and if / how the programme impacted on this.

This section details the strategy and delivery context, and the rationale for the CyberASAP programme, as well as providing an overview of other programmes in this space.

The CyberASAP programme was expected to contribute, or has the potential to contribute, to several key national strategies, as set out below.

Table 4: Strategic context

National Cyber Strategy 2022 - focuses on strengthening the UK Cyber Ecosystem

While the CyberASAP programme was designed before the National Cyber Strategy 2022 was developed, it contributes to the objective of fostering and sustaining sovereign and allied advantage in the security of cyberspace-critical technologies through ensuring the UK becomes more successful at translating research into innovation and new companies in the areas of technology most vital to our cyber power.

The UK cyber security sector is growing rapidly as outlined in the UK Cyber Security Sectoral Analysis published in 2022. This is shown by:

The UK has a reputation as a global leader in cyber security research, with 19 ACE-CSR, four Engineering and Physical Sciences Research Council National Cyber Security Centre (EPSRC-NCSC) Research Institutes, four Centres for Doctoral Training, the Centre for Security Information Technologies (CSIT) and the PETRAS National Centre of Excellence in Cyber Security of Internet of Things (IoT). he 2022 UK Cyber Security Sectoral Analysis also notes that investment in cyber security firms has increased, with over 1.4 billion being raised in 2021 across 108 deals. In addition, the sector is playing a critical role in responding to emerging cyber threats and challenges, and the rapid proliferation of connectable products.

However, research in 2020 found that long-term investments in other nations, especially the USA, France and Germany, are leading to the development of large clusters of research excellence. This can pose a threat to maintaining the UKs position as a leading nation for research and innovation in cyber security, given a potential brain drain from the UK. It suggests a need for the UK to further invest in cyber security research in various forms, including clusters of research excellence in cyber security; doctoral research funding to train future research and development (R&D) leaders in cyber security; and national research facilities.

The UK Innovation Strategy highlights that UK universities have become more effective at attracting investment and bringing ideas to market in recent years and of the top ten universities ranked by levels of funding raised by spinouts, the UK has five. However, this trend needs to be expanded beyond a small group of research-intensive UK universities, ensuring that technology transfer skills and expertise of a broader range of universities are enhanced to make the sector more accessible for investors.

The challenges within the sector and the innovation landscape include:

CyberASAP is part of a wider ecosystem of cyber security growth and innovation programmes across different stages of the innovation pathway.

Figure 2: Cyber security growth and innovation programmes

CyberASAP is the first stage in the innovation pathway focused on pre-seed and proof of concept ideas. It supports the commercialisation of UK cyber security research into fully rolled-out commercial projects. It is complemented and followed by programmes that support companies at different stages of the business lifecycle to:

These interventions are also complemented by other government and private sector initiatives with a cyber security element, illustrated in the following table.

Table 5: Mapping of other programmes

There are several programmes available to support innovation within the UKs wider cyber security ecosystem. However, there are no other initiatives focused primarily on the pre-seed, concept stage as, while Cyber Runway Launch aims to support the establishment of new companies in the sector, CyberASAPs main focus is on addressing the challenges faced by academics in the commercialisation of research.

The CyberASAP programme was initially set up as a pilot in 2017 based on research by KTN into barriers for the commercialisation of research in cyber security. The programmes development is outlined in the following table.

Table 6: CyberASAP development summary

Source: Information provided by KTN to RSM UK Consulting (February 2022)

The evaluation incorporates delivery and performance across all years to date.

The programme has consisted of two phases based on a similar structure used in the ICURe programme, the proof-of-concept phase was added after the pilot year as it was felt this was missing in the original ICURe model. This included the:

A two-stage selection process is used to both (1) ensure there is a wide selection of ideas and universities involved and not only those most likely to succeed and (2) for industry experts to filter out those that do not have a robust idea / concept that is viable to take forward to phase 2.

Figure 3: CyberASAP overview

The initial ICURe pilot programme had a simple set of KPIs. This included:

From Year 2 these were developed further as KTN introduced a logic model to measure the social and economic impacts of the programme based on Year 2 activities and programme design. Key programme objectives included (based on the 2021/22 delivery year):

Figure 4: CyberASAP Year 5 (2021/22) objectives

This section details the CyberASAP programme governance structure; key stakeholders; the application process; how the programme is delivered; and reporting requirements. It focuses on assessing whether the programme was delivered as intended and what could be improved in delivery.

In Year 1 (2017) the delivery partners were:

The governance structure for the CyberASAP programme in Years 2 5 is outlined below:

Go here to read the rest:
Evaluation of the Cyber Security Academic Startup Accelerator - GOV.UK