News

Catch up on our latest news, developments and insights

Jan Bosch article about Machine & Deep Learning: Non-Critical Deployment

Starting with non-critical deployment of ML/DL components is critical to learn about the challenges the company might experience as it moves to the use of ML/DL components in more critical deployments where potentially exceptional, unpredictable and erratic behavior severely undermines customer value derived from products, solutions and services In our research (see reference below), we have identified that companies that have reached this step experience a number of main challenges. These challenges are associated with the four stages of working with ML/DL components, i.e. assemble dataset, create model, train & evaluate and deploy. Below we discuss the key challenge in each stage. During the first stage where the company needs to assemble the data sets, organizations experience three main challenges. The first is that data that would be needed for training and validation is spread out across the company and needs to be collected from a variety of different data silos. The associated challenge is of course that the data from different silos may easily use different semantics and schemes for similar or related data items. Second, in most cases, ML/DL models required labelled models for supervised learning. Although the company may have the data available, it often is far from obvious to deduce what labels need to be associated with each data item. Finally, available data sets typically are assembled for specific purposes, causing the data to not be representative of reality, but rather contain a significant over-representation of specific cases or data. During the second stage the engineers are concerned with creating a model that is aligned with the problem at hand and that generates the desired output, such as a classification or prediction. Whereas during experimentation and prototyping, any model that achieves a some level of accuracy is acceptable, in this step the model will be exposed to customers. This requires the quality of the model to be higher, but most companies lack the skills and competencies to improve on a basic model. Doing so requires the ability to analyze which elements, algorithms or layers in the model cause the lack of accuracy as well as the ability to take corrective action to address the problem. The training and evaluation stage is concerned with training and evaluating the model defined in the previous stage. The key challenge here often is the availability of data for training and evaluation. Although approaches such as k-fold cross-validation exist and more experienced data scientists will know how to use these, in practice the company is in the early stages of adopting AI/ML/DL solutions and the amount of available talent in the company tends to be limited. The deployment stage is on the receiving end of the challenges experienced in the previous stages and this frequently results in a significant training-serving skew. This means that the model performs significantly worse in deployment than in training. This is typically caused by a difference between the data used during training and the data served during operations. Concluding, companies evolve through a number of steps when adopting AI/ML/DL models. In this article we discussed the challenges that companies experience in the second step where the company deploys the first ML/DL models in non-critical parts of products, solutions and services. The main challenges are concerned with assembling labelled data sets of sufficient quality and quantity as well as the skills of engineers to improve under-performing models. These challenges may cause a significant training-serving skew when models get deployed. The purpose of this article was to outline the challenges in order to help companies adopt ML/DL solutions while avoiding the traps that we outlined. Machine and deep learning offer fabulous technology that can provide incredible results and benefits. However, it comes with significant engineering challenges some of which we have outlined in the above. Good luck! Reference: Lucy Ellen Lwakatare, Aiswarya Raj, Jan Bosch, Helena Holmström Olsson and Ivica Crnkovic, A taxonomy of software engineering challenges for machine learning systems: An empirical investigation, XP 2019 (forthcoming), 2019.

Written by Jan Bosch, 2019-03-19 Read more

Jan Bosch article about Machine & Deep Learning: Experimentation Stage

This week I got the opportunity to speak at the initiative seminar organized by the Chalmers AI Research center (CHAIR). The key message in my presentation was that working with artificial intelligence (AI) and specifically machine & deep learning (ML/DL) constitutes a major software engineering challenge that is severely underestimated by companies that start to experiment with machine and deep learning. Although I have discussed some of the challenges in an earlier blog post, we have continued to conduct research in this area and we have collected additional data concerning the specific challenges. In addition, we have developed a model that captures how companies typically evolve in their adoption of AI/ML/DL. In the figure below, we show the steps that companies typically evolve through. In this and the upcoming posts, I intend to discuss the challenges associated specifically with each step. This is based on an article that recently was accepted for publication in the proceedings of the XP 2019 conference. Figure 1: How use of AI/ML/DL evolves in industry As the figure illustrates, the first step that most companies engage in is experimentation and prototyping. In this case, the work on machine & deep learning models is conducted purely in-house and without any connection to the products and services that the company offers to its customers. The work with basically any ML/DL approaches follows the process shown in the figure below. In the simplest view, here are four stages, i.e. assemble data-sets (or data pipes), create models, train & evaluate and, finally, deployment. There are two iterative processes. The inner loop is concerned with the typical activity of creating a model, training it, evaluating it and then tweaking the model with the intent of increasing accuracy and reducing error rates. The outer loop illustrates the periodic or continuous retraining of models based on the most recent data and the subsequent continuous deployment of models into operation. Figure 2: The basic ML/DL development process Our research shows that companies experience various challenges in each of the steps of the process and that these challenges depend on the evolution stage where the company finds itself. For companies in the experimentation and prototyping stage, I’ll describe the key challenge in each process step. For the “assemble data-sets” step (which in later stages becomes the data pipelines step), the very activity of assembling the right data-sets for training and validation purposes often proves to be a significant challenge. Although all companies tend to drown in data, this data often has unclear semantics and the way it has been collected is often unclear, resulting in data-sets that are not necessarily representative of the operational data that would be used during operations. As a data point: a company that I visited recently claimed that more than 90% of all effort in the data analytics team went to assembling data-sets and setting up reliable data pipelines. Although easily underestimated, this is a major challenge. The “create models” step is concerned with creating ML/DL models that perform well for the data that the problem domain is characterized by. As a well performing model is highly dependent on the characteristics of the input data, any issues during the previous step automatically affect the quality of the model. In addition, especially in this early stage, often companies experience a lack of talent with experience, exacerbating the situation. The “train & evaluate” step typically struggles with the fact that establishing the problem specification and desired outcome as well as having data-sets that capture a solid ground truth that can be used as a reference for training and evaluating models. As a consequence, it can prove to be difficult to determine which model is superior as well as whether any of the models is of sufficient accuracy. Due to the nature of this stage, that is no deployment mechanism yet. The challenges with setting up a deployment mechanism are discussed in future articles discussing the higher stages in the evolution model. Concluding, the first stage in adoption ML/DL in your products, systems and solutions is concerned with experimentation and prototyping. During this stage, the predominant challenge is the establishment of data-sets of sufficient quality as a basis for model creation, training and evaluation. These data-sets need to be representative of the data that will, during operations, come through the data pipelines. Our research shows that companies struggle with data quality in this stage and the subsequent steps in the development process are negatively affected. So, get going with ML/DL yesterday, but focus your energy where it counts: high-quality data sets. Reference: Lucy Ellen Lwakatare, Aiswarya Raj, Jan Bosch, Helena Holmström Olsson and Ivica Crnkovic, A taxonomy of software engineering challenges for machine learning systems: An empirical investigation, XP 2019 (forthcoming), 2019.

Written by Jan Bosch, 2019-03-19 Read more

Synteda today announce that it is acquiring Auqtus

The acquisition will significantly strengthen Synteda’s position in all IT sectors through Visual GUI Testing and Augmented Testing technology that will enable accelerated growth and expansion of Synteda’s offering to its customers. Maycel Isaac, CEO and Founder of Synteda: “We are delighted to welcome Auqtus to Synteda, the complementary talents and propositions will help accelerate Synteda’s leadership in the area of Visual GUI testing (VGT) and Augmented Testing (AT). What Auqtus have achieved to date is impressive but together we can invest in realising the full potential of the business. By combining capabilities and leveraging Auqtus knowledge, with academic pedigree behind it, we are confident that we can offer even more innovative and flexible solutions to our customers.” Founded in 2015, Auqtus has established itself as a highly regarded provider of VGT tools based on smart algorithms and AI. VGT, as shown through academic research, is both applicable and feasible in industrial practice and provides positive return on investment compared to equivalent manual testing. However, VGT scripts, like all automated test scripts, are still associated with development, maintenance and execution costs. Auqtus advancements in the area of AT aim to mitigate these challenges even further and foster a new era of testing. An era where humans no longer only instruct tools what to do but rather where humans work along side AI-driven tools to improve their joint testing capabilities in Human-Machine symbiosis. Emil Alégroth, Managing Director Auqtus: “We are thrilled to join an industry leader like Synteda and believe this is a great strategic fit rooted in a shared mission and culture. We at Auqtus have developed and implemented some of the most advanced API libraries for VGT and AT in existence and are extremely proud of what we have accomplished. Together with Synteda, we have the exciting opportunity to further develop our capabilities and deliver even more services and value to customers worldwide to help them solve their everyday test- and quality-related challenges. We are now focussing on a smooth integration.” Synteda managing team: “We knew from the beginning when we met Emil, and rest of Auqtus founders, that this agile, passionate and innovative group with their challenger attitude we love, would be a great fit at Synteda. Together, we will have a rich product offering that allow us to serve even more customers and in better ways than ever before. We are excited about what we can achieve together.” *************************** About Synteda Synteda (www.synteda.com) is one of Sweden’s leading artificial Intelligence and cutting edge technology provider, which wants to revolutionise the human experience for all kind of users. Founded in Gothenburg, Sweden. Synteda gives its consumers the option to enhance their knowledge in machine vision by offers such as Education, advanced Assessments, teams, and product development. For further information, please contact: Maycel Isaac, CEO info@synteda.se

Written by Editor, 2019-03-14 Read more

Article by Stefan Byttner about Intelligent machines and systems

Stefan Byttner, CTO of Synteda, talks about machines and humans in a article for Halmstad newspaper . https://www.halmstad7dagar.se/

Written by Editor, 2018-11-22 Read more