Innovation in data science is often lost between industry and academia, a new report, Data Science: From Academia to Industry, offers crucial suggestions for improving this process.

The “Valley of Death” represents the gulf between industry and academia, this is the point at which learnings from cutting-edge research are not carried over to the operational, day-to-day activities of industry. Great ideas and lessons do not make it across this “valley”.

The challenges and opportunities of the “valley” were explored by the team at Energy Systems Catapult and Arenko Ltd. (an operator of flexible assets in the UK energy market that focuses on algorithms and automation) in a joint report Data Science: From Academia to Industry.

Understanding the landscape

To understand the landscape of academic impact in the energy industry, we conducted desktop research and interviewed over 30 professionals from both academia and industry. These professionals provided insight into their difficulties in collaborating and transferring knowledge. They also shared insights about what worked well, or poorly, in their interactions across the valley of death.

We identified several themes during this stage including:

• disappointment with collaborations between academia and industry;
• when developing projects together, academia and industry often don’t fully understand the differences in work culture and end up choosing suboptimal ways of working together;
• objectives and outputs deviating from the original proposal.

We supplemented the interviews with a literature review which revealed a disconnect between industry needs and academic practices.

The methodologies outlined in most papers are often not clear enough to enable reproduction, and without open data and/or code, it is often virtually impossible to verify the results and methods. For industry practitioners with limited time and resources, these instant roadblocks make it far less likely that a company will apply academic research.

A gap in the education market

There is an important training gap in coding, a skill set that is desperately needed given the rapid digitalisation of the energy sector.

Although data science courses develop rigorous skills in modelling and advanced algorithms there is very little content on applying the latest programming approaches, e.g., code review, structure, or version control. Of ten data science-focused master’s programs from major universities in the UK, we found only two that had dedicated introductory courses to programming and none of them had any dedicated intermediate or advanced courses.

In other words, unless self-taught, such graduates entering the workforce will be lacking in the skills to develop operational code in industry teams. Discussions with those in the energy sector have highlighted this as a major difficulty when recruiting new data science talent.

Improve collaboration

The research found several areas that could prove fruitful for improving collaboration, including:

• creating a better understanding of each other’s cultures, businesses, and objectives;
• moving away from short-term projects and developing longer-term strategic relationships between business leaders and academics;
• focus on making academic research open and reproducible;
• better utilise and listen to industrial experience and knowledge in shaping courses.

Bridging the gap

Through a combination of dedicated taskforces, webinars, and publications, Energy Systems Catapult is actively working to bridge the gap between academia and the energy industry. For example, through the Energy Data Taskforce (EDTF), a taskforce commissioned by the UK Government, Ofgem and Innovate UK, we worked to develop an integrated data and digital strategy that helps unlock the opportunities of a modern, decarbonised and decentralised Energy System for the benefit of consumers.

While our work with the Energy Digitalisation Taskforce (EDT) – the EDTF’s successor – is helping to modernise the energy system to unlock flexibility and drive clean growth towards Net Zero carbon emissions by 2050. The EDT considers the market design, digital architecture and governance of a modern digitalised energy system.

We’re also proactively encouraging sectoral collaboration. Our Value in Energy Data: Webinar Series for instance aims to help bridge the gap across academia, industry and policy within the energy sector, a key objective of Energy Systems Catapult. We are very proud and privileged that the best and brightest from the world of data and digital agree to share their insights with us. The series has covered everything from improving power grid operations, how open source tools help the utilisation of data, the importance of weather data to energy system modelling and many others.

To further encourage open and transparent ways of working with data, we recently partnered with National Grid Electricity Distribution (NGED) (formerly Western Power Distribution) on a series of data science challenges to engage the energy community, demonstrate the value of open data, encourage open science, and solve some of the key problems facing distribution network operators.

Energy Systems Catapult worked with NGED to host a range of Presumed Open Data challenges to demonstrate the value their data can bring to innovators and the wider community. These challenges:

• Highlighted many data driven problems that are facing and challenging network operators.
• Released unique and valuable datasets to the research community.
• Demonstrated state-of-the-art methods who shared their approaches and techniques via our wrap-up webinars.
• Created new outputs including peer-reviewed academic publications and open access code for advanced algorithms.
• Produced a growing list of energy data science enthusiasts and a network of skilled individuals who can help drive the digitalisation of the energy system.

Learn more

The learnings and challenges outlined above and more can be found in the series of reports we developed. The main report considers the core aspects and challenges of bringing innovative ideas from academia to industry in data science.

*A version of this blog first appeared on Climate Change AI.