Detect defects in automotive parts with Amazon Lookout for Vision and Amazon SageMaker

According to a recent study, defective products cost industries over $2 billion from 2012–2017. Defect detection within manufacturing is an important business use case, especially in high-value product industries like the automotive industry. This allows for early diagnosis of anomalies to improve production line efficacy and product quality, and saves capital costs. Although advanced anomaly detection systems employ sensors as well as Internet of Things (IoT) devices to collect multimodal data to improve performance, computer vision continues to be a common approach. Detecting anomalies in automotive parts and components using computer vision can be done using normal images, and even X-Ray based images for structural damages. Recent advances in deep learning and computer vision have allowed scientists and manufacturers to develop enhanced anomaly detection systems, including surface defect detection on automotive body panels and dent detection in vehicles.
Anomaly detection typically consists of two tasks: detecting whether a given product is defective or not, and localization of the defect—the process of identifying the exact location of a defect inside of an image. In this post, we show how we designed and implemented an end-to-end pipeline for the following:
Defect detection using Amazon Lookout for Vision, a machine learning (ML) service for identifying anomalies in images using computer vision
Defect localization through supervised and unsupervised approaches using Amazon SageMaker
The supervised approach based on Single Shot MultiBox Detector (SSD) is relevant in use cases where we have sufficient labeled data. However, creating annotations for images can often be expensive and time-consuming. Additionally, anomalous images aren’t usually available at a high scale by definition. For these use cases, we present an unsupervised defect localization method based on a variational autoencoder (VAE).
Data overview
In the automotive space, the most common types of defects shown in the following table can be categorized into the following:
Dimensional – Such as orientation, alignment, and roundness
Structural – Such as holes, screws, and clamps
Surface – Such as scratches, cracks, wear, and roughness
Operational defects – Incompatibility to standards and specifications
In this post, we worked with automotive parts such as wheels and knuckles.
Defects
Types
Details
Dimensional
Dimensions, Shape, Positioning, Orientation, Alignment, Roundness, Corners
Structural
Assembly, Holes, Slots, Rivets, Screws, Clamps
Foreign Objects, Dust, Bur, Swarm
Surface
Pits, Scratches, Cracks, Wear, Finish, Roughness, Texture, Seams-Folds-Laps, Continuity
Operational
Incompatibility of Operation to Standards and Specifications
We utilized the GRIMA X-Ray database GDXray dataset, which includes five groups of images: casting, welding, baggage, nature, and settings (see the following table).
Group
Classes
Images
Casting
67
2,727
Welds
3
88
Baggage
77
8,150
Nature
13
8,290
Settings
7
152
We considered the castings data for this project, which included 2,727 X-Ray images. We excluded four images due to them not resembling wheels or knuckles, leaving us with 984 defective and 1739 non-defective. The following images show some examples from the dataset used in this study.
We used different sample sizes for training and testing due to different requirements for the underlying modeling approaches. The following table displays the number of samples used in each of our models for training and testing.
Model
Training
Testing
Lookout for Vision (classification)
2179
544
Custom Variational Autoencoder (unsupervised)
1739
984
Custom Single Shot Multibox Detector (supervised)
544
1036
Defect detection using Lookout for Vision
Lookout for Vision is an ML service that spots defects and anomalies in visual representations using computer vision. With Lookout for Vision, manufacturing companies can increase quality and reduce operational costs by quickly identifying differences in images of objects at scale. For example, you can use Lookout for Vision to identify missing components in products, damage to vehicles or structures, irregularities in production lines, miniscule defects in silicon wafers, and other similar problems.
Lookout for Vision uses ML to see and understand images from any camera as a person would, but with an even higher degree of accuracy and at a much larger scale. Lookout for Vision allows you to eliminate the need for costly and inconsistent manual inspection while improving quality control, defect and damage assessment, and compliance. In minutes, you can begin using Lookout for Vision to automate inspection of images and objects with no ML expertise required. Using Lookout for Vision to classify images as anomalous (defective) or non-defective is a crucial step in our pipeline for identifying which images need further analysis.
Lookout for Vision process and results
We completed the process of model development using Lookout for Vision in three steps:
We uploaded our sample data to Amazon Simple Storage Service (Amazon S3) into training and testing folders, and we linked them to our Lookout for Vision project.
We trained our classification model through the Lookout for Vision user interface using the uploaded dataset.
When the training was complete, we analyzed the results for the testing set, illustrated in the following figure.
Lookout for Vision enabled us to detect defective images in our dataset and prepare our samples for localizing defective regions in the shortlisted dataset. Lookout for Vision helped identify 183 anomalous images correctly, out of which 123 samples were used for training of the defect localization model. The remaining 60 samples were used for testing. The localization pipeline and results from our approach are described in the following section. Using a system such as Lookout for Vision, which facilitates the identification of defective parts, followed by our defect localization pipeline described in the next section, facilitates an end-to-end defect detection pipeline.
Defect localization pipeline
Defect localization is the process of identifying the exact location of a defect inside of an image. We developed a defect localization pipeline with two options. The first option allows you to provide your data to train ML models. We describe two approaches for training, one using an unsupervised VAE, and another using a supervised SSD method. The second option allows you to use only pretrained or custom models for prediction purposes without training. The following figure illustrates the details of the pipeline developed in this work.
In the first option, we upload the data into an S3 bucket separated into defective and non-defective folders, which are fed into the processing pipeline. The following is a snippet of the VAE training process: