This project contains the source code and scientific work for the master project "Explainable Machine Learning (xAI-Proj-M)" at the University of Bamberg.
The goal of the project is to develop a machine learning application considering the three steps data engineering, model engineering and model evaluation, which are part of the deep learning life cycle. A separate research question is processed for each step but our overall goal is to detect the hand signs paper, scissors, and rock based on images.
Clone the repository on your local device.
- You have Python installed on your machine
- You have Anaconda or at least Miniconda installed on your machine
Open a PowerShell/Terminal window, navigate to the git project folder and setup your environment via
conda env create -f environment.yml
In addition you need PyTorch installed in the environment. This is not included in the yaml-file. To update the existing environment use
conda env update
Enter the following command in your terminal while you are in your project root directory
python src/main.py
The default setting of main.py evaluates our best trainend model on the test set. It displays the confusion matrix and prints out the achievied top-1 accuracy. The model is trained using the EfficientNetV2-S architecture provided by PyTorch which is pretrainend on the ImageNet-1K dataset.
You can change this setting in the main.py file.
These following options are available:
- Train a new transfer learning model:
a) Set the boole variable "train_new_transferlearning_model" = True
b) Set the hyperparameter values you want to use for the grid search hyperparameter tuning in the config file
c) If you want to use also data augmention for training the model set activate_Augmentation=True and the combination you want to use d) Set all other boolean variables to False
e) Run the main.py file again - Train a new baseline model:
a) Set the boole variable "train_new_baseline_model" = True
b) Set the hyperparameter values you want to use for the grid search hyperparameter tuning in the config file
c) If you want to use also data augmention for training the model set activate_Augmentation=True and the combination you want to use d) Set all other boolean variables to False
e) Run the main.py file again - Test an existing model:
a) Change the path of the variable "model_folder" to the model you want to evaluate
b) Change the path of the variable "test_folder" to your image path you want to test on
c) Set the boole variable "test_existing_model" = True and all other boole values = False
d) Run the main.py file again - Display some example images with the prediction of the model:
a) Change the path for the variable "test_folder" to your image folder you want to make predicitons on
b) Set the boole variable "prediction_on_images" = True, all other boole variables = False
c) Run the main.py file again - Display a single image and the prediction of the model:
a) Change the path for the variable "single_image_path" to your image you want to classify
b) Set the boole variable "prediction_on_single_image" = True, all other boole variables = False
c) Run the main.py file again - Display model metrices:
a) Change the path of the variable "model_folder" to the model you want to evaluate
b) Set the boole variable "model_metrices" = True, all other boole variables = False
c) Run the main.py file again - Use LIME to see the areas which the model used to make the prediction:
a) Change the path of the variable "model_folder" to the model you want to evaluate
b) Change the path for the variable "single_image_path" to your image you want to evaluate
c) Set the boole variable "LIME_single_Image" = True, all other boole variables = False - Use saliency to see the areas which the model used to make the prediction:
a) Change the path of the variable "model_folder" to the model you want to evaluate
b) Change the path for the variable "single_image_path" to your image you want to evaluate
c) Set the boole variable "saliency" = True, all other boole variables = False
For our project we combined the following five subdatasets:
-
Roboflow Dataset with 2925 images
-
Kaggle Dataset with 2188 images
-
Kaggle Dataset with 2351 images
-
giusti2018 with 2419 images
-
Own validation set created with all class members with 151 images
-
Own test set created with all class members with 158 images
In total our combined dataset consists of 10192 images. You can find this dataset which splitted into train, test, and validation sets in the folder "combined_dataset". The Datasets 1 to 4 are used for training and validation. We used 80% for training and 20% for validation. In addition, the validation set was extenden with dataset 5. Dataset 6 is used as test set.
The combined dataset is stored in the data_combined folder.The original datasets are stored in the data_orignal folder. The models folder contains the best trained model of the Baseline Model, the Transfer Learning Model, and the Transfer Learning model with Data Augmentation. You can also find there the csv-files with all training information of the models. In the src folder you can find the main.py and all necassary files.
The results of these research questions can be found in the final report.
In this part we want to consider the research question "Which data augmentation methods can increase the model's performance in classifying images of „Rock, Paper, Scissors“ hand signs?"
In this part we want to consider the research question "Can transfer learning bring a benefit on the performance of CNN models for Rock, Paper, Scissors?".
In this part we want to consider the research question "Can interpretable model-agnostic-explanations (LIME) make our CNN models for Rock, Paper, Scissors more explainable?"