4 important success factors for data science projects in your company

Data science projects can be very challenging. The basic challenge often lies not in the fact that they are particularly complex, time-consuming or lengthy, as with most software projects, but in the minds of the managers.

As a service provider whose mission is to bring data science and artificial intelligence to medium-sized companies and to use the untapped potential, we at Ailio are often faced with a very specific challenge:

“We are developing something with technologies that are abstract for you and we cannot promise you the exact result or whether the challenge can be solved at all.”

At first this doesn't sound like a particularly management-friendly pitch and we'll of course do our best to package it in a sexier and more comprehensible way.

In essence, however, the statement is true - data science projects always have a research and development (R&D for short) character. Every company is different - even if they come from the same industry and have a similar data and IT strategy. The quality, number and form of data is always different and has a massive impact on the approach, effort and success.

In short: you usually just don't know beforehand and you just have to do it. As a rule, good results can be achieved with a manageable amount of effort; the amount of data available to an SME is sufficient for most use cases and, with a good evaluation, the use case usually does not fundamentally fail.

But you just don't know beforehand... and false promises have a taste like they would say in southern Germany.

So can we ensure that the projects are a success and make the investment decision as attractive as possible?

1.Multi-stage project plan – From deep dive to PoC to MVP and release

Basically, it is advisable to think in small, risk-minimizing steps and not to try to plan the big picture straight away. Do you have an exciting use case and data for it? First, a deep dive into the data to really understand it, get questions answered, see what is still missing and needs to be expanded... this usually takes a maximum of a week and has many concrete advantages. Starting with the fact that the company understands its own data.

We then recommend a short proof of concept in which a minimal version is implemented that proves that the basic assumption is correct and feasible. If the deep dive turns out that the data structure and infrastructure initially has to-dos to enable the company to carry out DS projects... this is still necessary before the PoC as required.

A proof of concept should ideally take between 5 and 20 days, depending on the use case, thereby keeping the investment required for it slim.

At the end of these phases, you have invested around 15-30 project days, which is a manageable investment for most companies, with the result of having a clear picture of your own data, having it transformed and optimized and knowing exactly what can be implemented with it and to what extent.

Based on the PoC findings, a concrete project plan for the MvP can usually be drawn up and more concrete promises can be made... so we are slowly becoming suitable for management. From here we essentially have a normal software project.

2. Get stakeholders and users on board in advance

Many data science projects are actually doomed to death before they are even started. Especially in the corporate environment, you wouldn't believe how much we have already developed, which, even though it was successful and would be very useful to use, never went beyond the PoC phase.

Why is that? In the end, data science projects change the everyday work of people who are often never asked about them or involved in the development. This means that the initially good idea was never promoted and accepted internally. This is where the nasty surprise comes at the end when the person responsible for the department that is supposed to help with the project isn't really behind it and the employees don't understand what it's all about or are even afraid of it.

3.Ensure access and get technical partners on board

Same game as point 2, only on a technical level... a company is able to massively increase the costs for a data science project for itself and at the same time minimize the probability of success because the data science team does not receive the access and support it would need due to departmental and responsibility thinking as well as a lack of willingness to cooperate on the part of the company's IT. A DS project rarely works on its own. Often the results have to be integrated into the software, APIs have to be addressed, data has to be made available and explained or perhaps even recorded at all.

4.Faith

That's one thing about believing in data science... you can laugh at it or take it seriously. Ultimately, the goal of data science projects is to make data-driven and fact-based decisions. You develop software that sees all the data, recognizes it and can create connections that the human brain can never overview or understand. In the end, the decision as to how the results of the software are handled is often still in human hands. In the end, data science has little to do with belief as it is the path away from manual work and gut feeling decisions to automation and fact-based approaches. There needs to be a rethinking at the organizational level to understand that even if the results are not calculable and predictable, they will certainly be more concrete and reliable than pure gut feeling. An understandable idea, but one that still gives many inexperienced decision-makers in the data science environment a stomach ache when making an investment decision.