We combine smart algorithms and sharp experts to deliver cutting edge solutions.

Jan Bosch article about Machine & Deep Learning: Non-Critical Deployment

2019-03-19

Starting with non-critical deployment of ML/DL components is critical to learn about the challenges the company might experience as it moves to the use of ML/DL components in more critical deployments where potentially exceptional, unpredictable and erratic behavior severely undermines customer value derived from products, solutions and services
In our research (see reference below), we have identified that companies that have reached this step experience a number of main challenges. These challenges are associated with the four stages of working with ML/DL components, i.e. assemble dataset, create model, train & evaluate and deploy. Below we discuss the key challenge in each stage.
During the first stage where the company needs to assemble the data sets, organizations experience three main challenges. The first is that data that would be needed for training and validation is spread out across the company and needs to be collected from a variety of different data silos. The associated challenge is of course that the data from different silos may easily use different semantics and schemes for similar or related data items. Second, in most cases, ML/DL models required labelled models for supervised learning. Although the company may have the data available, it often is far from obvious to deduce what labels need to be associated with each data item. Finally, available data sets typically are assembled for specific purposes, causing the data to not be representative of reality, but rather contain a significant over-representation of specific cases or data.
During the second stage the engineers are concerned with creating a model that is aligned with the problem at hand and that generates the desired output, such as a classification or prediction. Whereas during experimentation and prototyping, any model that achieves a some level of accuracy is acceptable, in this step the model will be exposed to customers. This requires the quality of the model to be higher, but most companies lack the skills and competencies to improve on a basic model. Doing so requires the ability to analyze which elements, algorithms or layers in the model cause the lack of accuracy as well as the ability to take corrective action to address the problem.
The training and evaluation stage is concerned with training and evaluating the model defined in the previous stage. The key challenge here often is the availability of data for training and evaluation. Although approaches such as k-fold cross-validation exist and more experienced data scientists will know how to use these, in practice the company is in the early stages of adopting AI/ML/DL solutions and the amount of available talent in the company tends to be limited.
The deployment stage is on the receiving end of the challenges experienced in the previous stages and this frequently results in a significant training-serving skew. This means that the model performs significantly worse in deployment than in training. This is typically caused by a difference between the data used during training and the data served during operations.
Concluding, companies evolve through a number of steps when adopting AI/ML/DL models. In this article we discussed the challenges that companies experience in the second step where the company deploys the first ML/DL models in non-critical parts of products, solutions and services. The main challenges are concerned with assembling labelled data sets of sufficient quality and quantity as well as the skills of engineers to improve under-performing models. These challenges may cause a significant training-serving skew when models get deployed.
The purpose of this article was to outline the challenges in order to help companies adopt ML/DL solutions while avoiding the traps that we outlined. Machine and deep learning offer fabulous technology that can provide incredible results and benefits. However, it comes with significant engineering challenges some of which we have outlined in the above. Good luck!
Reference: Lucy Ellen Lwakatare, Aiswarya Raj, Jan Bosch, Helena Holmström Olsson and Ivica Crnkovic, A taxonomy of software engineering challenges for machine learning systems: An empirical investigation, XP 2019 (forthcoming), 2019.

Written by Jan Bosch Jan Bosch

Jan Bosch article about Machine & Deep Learning: Experimentation Stage

This week I got the opportunity to speak at the initiative seminar organized by the Chalmers AI Research center (CHAIR). The ...

Read more
Synteda today announce that it is acquiring Auqtus

The acquisition will significantly strengthen Synteda’s position in all IT sectors through Visual GUI Testing and Augmented ...

Read more
Article by Stefan Byttner about Intelligent machines and systems

Stefan Byttner, CTO of Synteda, talks about machines and humans in a article for Halmstad newspaper . https://www.halmstad7d ...

Read more

Synteda is a business partner in helping your company embrace the power of artificial intelligence and machine learning.

Our mission is to assist companies in leveraging the era of artificial intelligence effectively and achieve new advantages by delivering innovative solutions, training and advisory services.

We combine your data, business objectives and domain expertise with our cutting edge machine learning techniques and specialist competence to optimize your business.

Computer Vision

We transform computers into expert observers by enabling them to identify objects, analyze scenes and activities in digital or physical visual environments using proprietary code.

Machine Learning

Our analysts and engineers combine your data, objectives and domain expertise with our cutting edge and evolving algorithms to optimize your business using custom tailored solutions.

Biometrics

By utilizing innovative algorithms our team builds unique biometric solutions for your business. Our algorithms are based on several years research using robust multimodality biometric systems for recognition in composition to provide a new level of security and accuracy.

Data Science

Understanding the complex relationships between algorithms, business and statistics enables us to offer a specialst consultancy service that help you get the most from your data.

Careers

Right now we are looking for 1 talent. We often use continuous selection, which means means that interesting candidates can be contacted for an interview the same day we publish the position. Don't hesitate, send us your application right now!

Card image cap
Developer with computer vision skills
Published 2019-03-13

We are looking for a talented Computer Vision/Machine Learning developer for development of innovative object detection algorithms.

Learn more and Apply

Didn't find anything that seemed like a good fit? No worries! We are always open to talking to interesting candidates with the right profile that want to work with us. Create a quick unsolicited application by leaving your name and contact info below and we will be in touch!

What can machine learning do for your business?

You need more than just common wisdom and intuition to make the most out of your data and business moving forward.

We believe our expertise in machine learning and artifical intelligence will help your company identify opportunities, create value and optimize processes.

Leave your contact information below to let us know you want to know more and we will be in touch.

Partners