Attacking_simulation_CIC-IDS201.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f16f2dfd-c682-4160-a80a-0b6642317929",
   "metadata": {},
   "source": [
    "# Model Extraction Attacks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca85bb04-932c-4477-9a41-63baaa3f8758",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Libs and Confs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80cbd831-6cea-4e68-8ca5-98048ac084e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install tqdm\n",
    "!pip install shap\n",
    "!pip install ipywidgets\n",
    "!pip install -U matplotlib\n",
    "!pip install sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b718af8f-2232-49ad-a883-38e5e2726fc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "from joblib import load, dump   # load models\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import roc_auc_score, precision_score, confusion_matrix\n",
    "import tqdm\n",
    "import time\n",
    "import shap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17114a8d-fbd1-4f47-b83b-7dbd0baec125",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Models\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis\n",
    "from sklearn.mixture import BayesianGaussianMixture\n",
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b96d430-26d5-4e89-8828-fad96e80543f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# conf\n",
    "actual_directory = os.getcwd()\n",
    "dataset_path = os.path.join(actual_directory, \"CIC-IDS-2017\", \"GeneratedLabelledFlows\", \"TrafficLabelling\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "130fedf7-2566-4303-9361-cd7897c1e747",
   "metadata": {},
   "source": [
    "# Data manipulation - CIC-IDS2017"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42f8f74e-8bb3-439f-b4c3-aeaac003730f",
   "metadata": {},
   "source": [
    "Creating samples to be used during the experimentation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bee8d63-ceb8-40af-9edb-ab1e6b1053a5",
   "metadata": {},
   "source": [
    "## CIC-IDS2017"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bf1c2f6-aa56-492b-aad8-992e98135171",
   "metadata": {},
   "source": [
    "https://www.unb.ca/cic/datasets/ids-2017.html\n",
    "\n",
    "Iman Sharafaldin, Arash Habibi Lashkari, and Ali A. Ghorbani, “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization”, 4th International Conference on Information Systems Security and Privacy (ICISSP), Portugal, January 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ceb7ea9-5a94-4b13-8098-1de1fcdb6efe",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Combined Data - CIC-IDS2017"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a7e88a2-cd57-4c91-81cc-652cee95006b",
   "metadata": {},
   "source": [
    "One against others, if we merge all dataset. Every attack was considered 1 and benign flow was considered 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09c73183-e0e5-4048-8af9-49218d798380",
   "metadata": {},
   "outputs": [],
   "source": [
    "''' Function to combine the csv files '''\n",
    "def combine_dataset(path):\n",
    "    filenames = [file for file in os.listdir(path)]\n",
    "    dataframes = []\n",
    "\n",
    "    print('Starting to combine datasets. Wait a moment...')\n",
    "    for file in filenames:\n",
    "        try:\n",
    "            df = pd.read_csv(os.path.join(path, file), encoding='utf-8')\n",
    "        except UnicodeDecodeError:\n",
    "            df = pd.read_csv(os.path.join(path, file), encoding='ISO-8859-1')\n",
    "            \n",
    "        dataframes.append(df)\n",
    "\n",
    "    combined_df = pd.concat(dataframes, ignore_index=True)\n",
    "\n",
    "    combined_df.to_csv(os.path.join(path, 'combined_dataset.csv'), index=False)\n",
    "    print('Datasets combined')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3675658f-8147-4365-914c-1409f3054bc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "''' \n",
    "This block of code receives the merged CIC-IDS-2017 dataset and splits it in groups of data:\n",
    "X_data_model / y_data_model : data used to train the target models\n",
    "X_data_extraction / y_data_extraction: data sample to be used during model extraction attacks\n",
    "\n",
    "So X_data_model / y_data_model was splited in X_train, X_test, y_train, y_test to train and validade target models.\n",
    "'''\n",
    "\n",
    "data = pd.read_csv(os.path.join(dataset_path, \"combined_dataset.csv\"), low_memory=False)\n",
    "# Pre-processing\n",
    "data = data.drop(' Destination IP', axis=1)\n",
    "data = data.drop('Flow ID', axis=1)\n",
    "data = data.drop(' Source IP', axis=1)\n",
    "data = data.drop(' Timestamp', axis=1)\n",
    "data = data.drop(' Source Port', axis=1)\n",
    "data = data.drop(' Destination Port', axis=1)\n",
    "data = data.drop(' Protocol', axis=1)\n",
    "\n",
    "# Replace infinite values to np.nan values\n",
    "data.replace([np.inf, -np.inf], np.nan, inplace=True)\n",
    "\n",
    "# dropping unlabeled rows\n",
    "data = data.dropna(subset=[' Label'])\n",
    "\n",
    "# changing label values. Malicious: 1, Benign: 0\n",
    "data[' Label'] = data[' Label'].apply(lambda x: 0 if x=='BENIGN' else 1)\n",
    "\n",
    "# Missing Values\n",
    "missing_values = (( data.isnull().sum() ))\n",
    "missing_values[missing_values > 0]\n",
    "\n",
    "data['Flow Bytes/s'].fillna(data['Flow Bytes/s'].median(), inplace=True)\n",
    "data[' Flow Packets/s'].fillna(data[' Flow Packets/s'].median(), inplace=True)\n",
    "\n",
    "missing_values = (( data.isnull().sum() ))\n",
    "missing_values[missing_values > 0]\n",
    "# split and normalize data\n",
    "X = data.drop(' Label', axis=1)\n",
    "y = data[' Label']\n",
    "\n",
    "#scaler = MinMaxScaler()\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "X_scaled = pd.DataFrame(scaler.fit_transform(X), columns=X.columns)\n",
    "# split data into data_model, data_extraction. So split data_model into data_model_train and data_model_test\n",
    "X_data_model , X_data_extraction, y_data_model, y_data_extraction = train_test_split(X_scaled, y, test_size=0.1, random_state=42, stratify=y)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_data_model, y_data_model, test_size=0.3, random_state=42, stratify=y_data_model)\n",
    "print(f'Data created\\n X_data_mode shape: {X_data_model.shape}\\n X_data_extraction shape: {X_data_extraction.shape}') \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51e5174c-2bbe-4b06-8a23-97ce59c27444",
   "metadata": {},
   "source": [
    "# Training Target Models - CIC-IDS2017"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4914493-c8c8-42ea-b894-5f4cbdb7e58e",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Target Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f51bdf0e-f4f4-4acf-a209-0a0fce13c872",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Linear Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f26e208b-0668-4bb5-9e2c-d0aec461d35a",
   "metadata": {},
   "source": [
    "#### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be6d9613-a7a6-4ae1-80b3-d13da139735f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_file = os.path.join(os.getcwd(), \"LogisticRegression.joblib\")\n",
    "if os.path.exists(model_file):\n",
    "    print('Model already trained')\n",
    "    lg_model = load(os.path.join(actual_directory, \"LogisticRegression.joblib\"))\n",
    "\n",
    "else:\n",
    "    lg_model = LogisticRegression(solver='newton-cholesky',random_state=42, n_jobs=-1)\n",
    "    try:\n",
    "        lg_model.fit(X_train, y_train)\n",
    "    except Exception as e:\n",
    "        print(f\"We had the {e} error. The model wasn't trained\")\n",
    "    else:\n",
    "        print(f'Model Trained')\n",
    "        # Saving the model\n",
    "        dump(lg_model, os.path.join(actual_directory, \"LogisticRegression.joblib\"))\n",
    "predict = lg_model.predict(X_test)\n",
    "predict_proba = lg_model.predict_proba(X_test)[:,1]\n",
    "\n",
    "lg_cr = (classification_report(y_test, predict, target_names=['Benign flow', 'Malicious flow']),roc_auc_score(y_test, predict_proba), lg_model.score(X_test, y_test)) #saving the classification report to each target model\n",
    "print(lg_cr[0])\n",
    "print()\n",
    "print(f'AUC-ROC: {lg_cr[1]}')\n",
    "print(f'Accuracy: {lg_cr[2]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77688b73-a038-453b-9d73-2bf5f3a7db26",
   "metadata": {},
   "source": [
    "### Non-Linear Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f3f5f1-d458-4068-a2a2-8f0d6bdfa84d",
   "metadata": {},
   "source": [
    "#### Random Forest Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5eb8de1d-3d34-4b04-bc5b-5f2a1d15a7a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_file = os.path.join(os.getcwd(), \"RandomForestClassifier.joblib\")\n",
    "if os.path.exists(model_file):\n",
    "    print('Model already trained')\n",
    "    rfc_model = load(os.path.join(actual_directory, \"RandomForestClassifier.joblib\"))\n",
    "\n",
    "else:\n",
    "    rfc_model = RandomForestClassifier(random_state=42)\n",
    "    try:\n",
    "        rfc_model.fit(X_train, y_train)\n",
    "    except Exception as e:\n",
    "        print(f\"We had the {e} error. The model wasn't trained\")\n",
    "    else:\n",
    "        print(f'Model Trained')\n",
    "        # Saving the model\n",
    "        dump(rfc_model, os.path.join(actual_directory, \"RandomForestClassifier.joblib\"))\n",
    "predict = rfc_model.predict(X_test)\n",
    "predict_proba = rfc_model.predict_proba(X_test)[:,1]\n",
    "\n",
    "rfc_cr = (classification_report(y_test, predict, target_names=['Benign flow', 'Malicious flow']),roc_auc_score(y_test, predict_proba), rfc_model.score(X_test, y_test)) #saving the classification report to each target model\n",
    "print(rfc_cr[0])\n",
    "print()\n",
    "print(f'AUC-ROC: {rfc_cr[1]}')\n",
    "print(f'Accuracy: {rfc_cr[2]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47f53414-2e45-4568-b836-113f9678b74f",
   "metadata": {},
   "source": [
    "#### KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4818d9e-bbe7-41df-ae03-4ca771c8ee0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_file = os.path.join(os.getcwd(), \"KNNClassifier.joblib\")\n",
    "if os.path.exists(model_file):\n",
    "    print('Model already trained')\n",
    "    knn_model = load(os.path.join(actual_directory, \"KNNClassifier.joblib\"))\n",
    "\n",
    "else:\n",
    "    knn_model = KNeighborsClassifier(n_neighbors = 5)\n",
    "    try:\n",
    "        knn_model.fit(X_train, y_train)\n",
    "    except Exception as e:\n",
    "        print(f\"We had the {e} error. The model wasn't trained\")\n",
    "    else:\n",
    "        print(f'Model Trained')\n",
    "        # Saving the model\n",
    "        dump(knn_model, os.path.join(actual_directory, \"KNNClassifier.joblib\"))\n",
    "predict = knn_model.predict(X_test)\n",
    "predict_proba = knn_model.predict_proba(X_test)[:,1]\n",
    "\n",
    "knn_cr = (classification_report(y_test, predict, target_names=['Benign flow', 'Malicious flow']),roc_auc_score(y_test, predict_proba), knn_model.score(X_test, y_test)) #saving the classification report to each target model\n",
    "print(knn_cr[0])\n",
    "print()\n",
    "print(f'AUC-ROC: {knn_cr[1]}')\n",
    "print(f'Accuracy: {knn_cr[2]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "015ff2dc-03d8-4245-9a55-69446151a7ef",
   "metadata": {},
   "source": [
    "#### Quadratic Discriminant Analysis (QDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff834114-4de5-40bb-9d64-55a09123b53e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_file = os.path.join(os.getcwd(), \"QDAClassifier.joblib\")\n",
    "if os.path.exists(model_file):\n",
    "    print('Model already trained')\n",
    "    qda_model = load(os.path.join(actual_directory, \"QDAClassifier.joblib\"))\n",
    "\n",
    "else:\n",
    "    qda_model = QuadraticDiscriminantAnalysis()\n",
    "\n",
    "    try:\n",
    "        qda_model.fit(X_train, y_train)\n",
    "    except Exception as e:\n",
    "        print(f\"We had the {e} error. The model wasn't trained\")\n",
    "    else:\n",
    "        print(f'Model Trained')\n",
    "        # Saving the model\n",
    "        dump(qda_model, os.path.join(actual_directory, \"QDAClassifier.joblib\"))\n",
    "predict = qda_model.predict(X_test)\n",
    "predict_proba = qda_model.predict_proba(X_test)[:,1]\n",
    "\n",
    "qda_cr = (classification_report(y_test, predict, target_names=['Benign flow', 'Malicious flow']),roc_auc_score(y_test, predict_proba), qda_model.score(X_test, y_test)) #saving the classification report to each target model\n",
    "print(qda_cr[0])\n",
    "print()\n",
    "print(f'AUC-ROC: {qda_cr[1]}')\n",
    "print(f'Accuracy: {qda_cr[2]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9405ec99-dd0a-4b89-84d4-0adcb80d1dd3",
   "metadata": {},
   "source": [
    "#### ADABoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad401c27-c405-4a5e-b3f8-3b452ac44eb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_file = os.path.join(os.getcwd(), \"ADABoostClassifier.joblib\")\n",
    "if os.path.exists(model_file):\n",
    "    print('Model already trained')\n",
    "    ada_model = load(os.path.join(actual_directory, \"ADABoostClassifier.joblib\"))\n",
    "\n",
    "else:\n",
    "    ada_model = AdaBoostClassifier()\n",
    "    try:\n",
    "        ada_model.fit(X_train, y_train)\n",
    "    except Exception as e:\n",
    "        print(f\"We had the {e} error. The model wasn't trained\")\n",
    "    else:\n",
    "        print(f'Model Trained')\n",
    "        # Saving the model\n",
    "        dump(ada_model, os.path.join(actual_directory, \"ADABoostClassifier.joblib\"))\n",
    "predict = ada_model.predict(X_test)\n",
    "predict_proba = ada_model.predict_proba(X_test)[:,1]\n",
    "\n",
    "ada_cr = (classification_report(y_test, predict, target_names=['Benign flow', 'Malicious flow']),roc_auc_score(y_test, predict_proba), ada_model.score(X_test, y_test)) #saving the classification report to each target model\n",
    "print(ada_cr[0])\n",
    "print()\n",
    "print(f'AUC-ROC: {ada_cr[1]}')\n",
    "print(f'Accuracy: {ada_cr[2]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "717bf316-3904-4ea9-9b46-55fbbf6f9780",
   "metadata": {},
   "source": [
    "#### MLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb0f32a7-8efe-464c-bf4f-3a190bc47067",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_file = os.path.join(os.getcwd(), \"MLPClassifier.joblib\")\n",
    "if os.path.exists(model_file):\n",
    "    print('Model already trained')\n",
    "    mlp_model = load(os.path.join(actual_directory, \"MLPClassifier.joblib\"))\n",
    "\n",
    "else:\n",
    "    mlp_model = MLPClassifier(hidden_layer_sizes=(100, 50), activation='relu', solver='sgd', learning_rate_init=0.0001, max_iter=10, random_state=42, verbose=True)\n",
    "    \n",
    "    try:\n",
    "        mlp_model.fit(X_train, y_train)\n",
    "    except Exception as e:\n",
    "        print(f\"We had the {e} error. The model wasn't trained\")\n",
    "    else:\n",
    "        print(f'Model Trained')\n",
    "        # Saving the model\n",
    "        dump(mlp_model, os.path.join(actual_directory, \"MLPClassifier.joblib\"))\n",
    "predict = mlp_model.predict(X_test)\n",
    "predict_proba = mlp_model.predict_proba(X_test)[:,1]\n",
    "\n",
    "mlp_cr = (classification_report(y_test, predict, target_names=['Benign flow', 'Malicious flow']),roc_auc_score(y_test, predict_proba), mlp_model.score(X_test, y_test)) #saving the classification report to each target model\n",
    "print(mlp_cr[0])\n",
    "print()\n",
    "print(f'AUC-ROC: {mlp_cr[1]}')\n",
    "print(f'Accuracy: {mlp_cr[2]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26dd8cc5-dab6-48e8-8b61-aa79d170ca1a",
   "metadata": {},
   "source": [
    "#### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29b1d6d0-affc-44e5-8a14-6e74a071a269",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_file = os.path.join(os.getcwd(), \"LogisticRegression_dos.joblib\")\n",
    "if os.path.exists(model_file):\n",
    "    print('Model already trained')\n",
    "    lg_dos_model = load(os.path.join(actual_directory, \"LogisticRegression_dos.joblib\"))\n",
    "\n",
    "else:\n",
    "    lg_dos_model = LogisticRegression(solver='newton-cholesky',random_state=42, n_jobs=-1)\n",
    "    try:\n",
    "        lg_dos_model.fit(X_dos_train, y_dos_train)\n",
    "    except Exception as e:\n",
    "        print(f\"We had the {e} error. The model wasn't trained\")\n",
    "    else:\n",
    "        print(f'Model Trained')\n",
    "        # Saving the model\n",
    "        dump(lg_dos_model, os.path.join(actual_directory, \"LogisticRegression_dos.joblib\"))\n",
    "predict = lg_dos_model.predict(X_dos_test)\n",
    "predict_proba = lg_dos_model.predict_proba(X_dos_test)[:,1]\n",
    "\n",
    "lg_dos_cr = (classification_report(y_dos_test, predict, target_names=['Benign flow', 'Malicious flow']),roc_auc_score(y_dos_test, predict_proba), lg_dos_model.score(X_dos_test, y_dos_test)) #saving the classification report to each target model\n",
    "print(lg_dos_cr[0])\n",
    "print()\n",
    "print(f'AUC-ROC: {lg_dos_cr[1]}')\n",
    "print(f'Accuracy: {lg_dos_cr[2]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db8d0415-0695-4988-a804-426ecc855769",
   "metadata": {},
   "source": [
    "# Complementary Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cc94a41-ddf7-434c-964f-5eac828c499e",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Classification API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06d01785-732d-490c-b49d-16ce6361c257",
   "metadata": {},
   "outputs": [],
   "source": [
    "def features_importance(model_file, n):\n",
    "    '''\n",
    "    Return top n features of a Forest model\n",
    "    :param model_file: name of the saved model\n",
    "    :param n: top n features\n",
    "    '''\n",
    "    \n",
    "    a = load(model_file)\n",
    "    a_importances = a.feature_importances_\n",
    "    a_features_df = pd.DataFrame({\n",
    "        'Feature': (X_test.columns), \n",
    "        'Importance': a_importances\n",
    "    })\n",
    "    a_features_df = a_features_df.sort_values(by='Importance', ascending=False)\n",
    "    \n",
    "    return a_features_df.head(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9933095-87d2-4f63-b94e-21c300ea686e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify_api(target, features, threshold=0.5, type='hard'):\n",
    "    '''\n",
    "    This function opens a model previously saved and uses it to classify some samples.\n",
    "    you can pass the feature line or a matrix of features, each sample a line\n",
    "    \n",
    "    : target: model used to classify feature's samples. This code use scikit way of predict \n",
    "    : features: feature vector to classify in pd Dataset format\n",
    "    : threshold: to use models which use threshold\n",
    "    : type: soft -> output vector with class and other information; hard -> output with only class\n",
    "    '''\n",
    "    actual_directory = os.getcwd()\n",
    "    target_file = os.path.join(actual_directory, target)\n",
    "    if os.path.exists(target_file):\n",
    "        # loading specific model (it needs to get the model file name to be opened)\n",
    "        target_model = load(os.path.join(actual_directory, target))\n",
    "    else:\n",
    "        print('Model file not found!')\n",
    "        return -1\n",
    "    \n",
    "    prediction = target_model.predict(features)\n",
    "    output = []\n",
    "\n",
    "    for result in prediction:\n",
    "\n",
    "        if result <= threshold:\n",
    "            if type == 'soft':\n",
    "                output.append((result, 0)) #benign\n",
    "            elif type == 'hard':\n",
    "                output.append(0)\n",
    "            else:\n",
    "                output.append(-1) #error\n",
    "        else:\n",
    "            if type == 'soft':\n",
    "                output.append((result, 1)) #malicious\n",
    "            elif type == 'hard':\n",
    "                output.append(1)\n",
    "            else:\n",
    "                output.append(-1) #error\n",
    "\n",
    "    if type == 'soft':\n",
    "        return pd.Series(output, dtype=object)\n",
    "    elif type == 'hard':\n",
    "        return pd.Series(output, dtype=np.int64)\n",
    "    else:\n",
    "        return -1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2454442-e119-4fa4-8d2f-980c4d4d0cb3",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db0599f5-7b2e-4c7a-a7e0-4efee1f40e16",
   "metadata": {},
   "source": [
    "## Explainability-Based Feature Agreement (EBFA)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4b8e520-b2f7-4696-a7f0-d7c8369570ed",
   "metadata": {},
   "source": [
    "We are using EBFA metrics to evaluate if target model and surrogate model are deciding their decisions based on same logics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1feb6176-19a2-4e06-bd13-16294a0670af",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Using shap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "422bfc6f-b2c9-4bc3-9516-27166aaa6afe",
   "metadata": {},
   "outputs": [],
   "source": [
    "''' That version is not a black-box version, despite of the version presented in the article '''\n",
    "def ebfa(target, surrogate, X_train, y_train, X_surrogate_train, y_surrogate_train, X_test, y_test, k=10, p=0.0001):\n",
    "    '''\n",
    "    EBFA (Explainability-Based Feature Agreement) Calculation between two models.\n",
    "    This function calculates the top k important features of target model and the surrogate model, and the realation between them.\n",
    "    The result is between 0 and 1. Next to 1 indicates that surrogate model and target model works with the same way.\n",
    "\n",
    "    As SHAP is really expensive, we are going to use part of the train dataset, used to train the model.\n",
    "\n",
    "    :param target: Path to the target model file.\n",
    "    :param surrogate: Path to the surrogate model file.\n",
    "    :param X_train: The dataset for which SHAP values are computed on target model.\n",
    "    :param X_surrogate_train: The dataset for which SHAP values are computed on surrogate model.\n",
    "    :param k: Number of top features to consider for EBFA.\n",
    "    :param p: Size of the subsample of X_train\n",
    "    :return: Average EBFA score across all instances.\n",
    "    '''\n",
    "    \n",
    "    actual_directory = os.getcwd()\n",
    "    target_file = os.path.join(actual_directory, target)\n",
    "    surrogate_file = os.path.join(actual_directory, surrogate)\n",
    "    if os.path.exists(target_file):\n",
    "        # loading specific model (it needs to get the model file name to be opened)\n",
    "        target_model = load(target_file)\n",
    "    else:\n",
    "        print('Target Model not found!')\n",
    "        return -1\n",
    "        \n",
    "    if os.path.exists(surrogate_file):\n",
    "        # loading specific model (it needs to get the model file name to be opened)\n",
    "        surrogate_model = load(surrogate_file)\n",
    "    else:\n",
    "        print('Surrogate Model not found!')\n",
    "        return -1\n",
    "\n",
    "    # subsample used to calculate SHAP of TARGET model\n",
    "    X_train_sample, _, y_train_sample, _ = train_test_split(X_train, y_train, train_size=p, stratify=y_train)\n",
    "    \n",
    "    if len(X_surrogate_train) > 800:\n",
    "        surrogate_proportion = 0.1   \n",
    "    else:\n",
    "        surrogate_proportion = 0.5\n",
    "    X_surrogate_train_sample, _, y_surrogate_train_sample, _ = train_test_split(X_surrogate_train, y_surrogate_train, train_size=surrogate_proportion, stratify=y_surrogate_train)\n",
    "\n",
    "    _, X_test_sample, _, y_test_sample = train_test_split(X_test, y_test, test_size=0.01, stratify=y_test)\n",
    "    \n",
    "    # Create SHAP explainers\n",
    "    #explainer_1 = shap.KernelExplainer(target_model.predict, shap.sample(X_train_sample, 100))\n",
    "    #explainer_2 = shap.KernelExplainer(surrogate_model.predict, shap.sample(X_surrogate_train_sample, 100))\n",
    "\n",
    "    explainer_1 = shap.KernelExplainer(target_model.predict, shap.kmeans(X_train_sample,10))\n",
    "    #explainer_2 = shap.KernelExplainer(surrogate_model.predict, shap.kmeans(X_train_sample,10)) # somente para teste -> deletar\n",
    "\n",
    "    explainer_2 = shap.KernelExplainer(surrogate_model.predict, shap.kmeans(X_surrogate_train_sample, 10))  #_> comentar para teste\n",
    "\n",
    "    # Calculate SHAP values\n",
    "    shap_values_1 = explainer_1.shap_values(X_test_sample)\n",
    "    shap_values_2 = explainer_2.shap_values(X_test_sample)\n",
    "    \n",
    "    # Calculate EBFA for each instance\n",
    "    ebfa_scores = []\n",
    "\n",
    "    importance_1 = ( np.sum(np.abs(shap_values_1), axis=1) / shap_values_1.shape[0] )\n",
    "    importance_2 = ( np.sum(np.abs(shap_values_2), axis=1) / shap_values_2.shape[0] )\n",
    "\n",
    "    # Determine top k features and convert indices to tuples\n",
    "    top_feats_1 = set(tuple(np.argsort(-importance_1))[:k])\n",
    "    top_feats_2 = set(tuple(np.argsort(-importance_2))[:k])\n",
    "\n",
    "    # Calculate intersection and EBFA score\n",
    "    intersection = top_feats_1.intersection(top_feats_2)\n",
    "    ebfa_score = len(intersection) / k\n",
    "    print('Average EBFA:', ebfa_score)\n",
    "    return ebfa_score\n",
    "    # we can return the intersection feature list, return (ebfa_score, intersection). \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f25b0043-afb6-4c81-832f-a82b8309b4d4",
   "metadata": {},
   "source": [
    "# Model Stealing attacks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b81f781c-90b3-42bd-8d0b-1298d3dff2b6",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Attack - Query only - Baseline 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae7b9d48-cf25-4826-9e91-e0ce1a230d1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def substitute_model_a1(target, X, y, X_train, y_train, detection_variation=0.05, threshold=0.5, surrogate_algo='RandomForestClassifier'):\n",
    "    '''\n",
    "    This attacks works only sending some real samples to classifier and retrieving the label. With these pars, we're going to train a substitute model\n",
    "    :Param X: Attack data\n",
    "    :Param y: Attack label data\n",
    "    :Param X_train: The data used to train the target model. This is not a black-box phase\n",
    "    :Param y_train: The labels of data used to train the target model.\n",
    "    '''\n",
    "\n",
    "    # config\n",
    "    max_iter = 600 # max iterations to avoid an eternal loop\n",
    "    num_samples = 10 #stating number of each class sample. eg: 10 benign and 10 malicious samples\n",
    "    surrogate_file_name = 'surrogate_model_a1_'+target \n",
    "    substitute_model_path = os.path.join(os.getcwd(), surrogate_file_name)\n",
    "    \n",
    "    # reset unpaired index and drop unlabeled rows\n",
    "    X = X.reset_index(drop=True)\n",
    "    y = y.reset_index(drop=True)\n",
    "    \n",
    "    label_index = (y != -1)\n",
    "    X_train = X[label_index]\n",
    "    y_train = y[label_index]\n",
    "\n",
    "    # creating subsamples (index)\n",
    "    index_0 = (y <= threshold)\n",
    "    index_1 = (y > threshold)\n",
    "    \n",
    "\n",
    "    # test samples to be used in all of testes, including to calculate prediction value of the target model\n",
    "    test_index_0 = np.random.choice(np.where(index_0)[0], 500, replace=False)\n",
    "    test_index_1 = np.random.choice(np.where(index_1)[0], 500, replace=False)\n",
    "    test_sample = np.concatenate((test_index_0, test_index_1))\n",
    "    np.random.shuffle(test_sample)\n",
    "    X_test = X_train.iloc[test_sample]\n",
    "    y_test = y_train.iloc[test_sample]\n",
    "\n",
    "    # Removing test data from original dataset, avoiding data leak\n",
    "    X_train = X_train.drop(test_sample, axis=0).reset_index(drop=True)\n",
    "    y_train = y_train.drop(test_sample, axis=0).reset_index(drop=True)\n",
    "\n",
    "    # re-calculating the index without the test samples\n",
    "    index_0 = (y_train <= threshold)   # index_0 - index_0(test)\n",
    "    index_1 = (y_train > threshold)    # index_1 - index_1(test)\n",
    "\n",
    "    # target prediction value -> This value could be added to the query number, or we could consider that attacks were done before, out of attack context.\n",
    "    # if hard:\n",
    "    target_classify = classify_api(target, X_test, threshold=0.5, type='hard')\n",
    "    detection_rate = np.mean(target_classify.to_numpy() == y_test.to_numpy())\n",
    "    # if soft, comment line above and uncomment line below\n",
    "    #detection_rate = np.mean(classify_api(target, X_test, threshold=0.5, type='hard')[:,0] == y_test)\n",
    "    \n",
    "    for iter_count in tqdm.tqdm(range(max_iter), desc='Substitute model training'):\n",
    "        print(f'Iteration {iter_count + 1}: Number of samples = {num_samples * 2}')\n",
    "        balanced_sample_0 = np.random.choice(np.where(index_0)[0], num_samples, replace=False)\n",
    "        balanced_sample_1 = np.random.choice(np.where(index_1)[0], num_samples, replace=False)\n",
    "        balanced_sample = np.concatenate((balanced_sample_0, balanced_sample_1))\n",
    "        np.random.shuffle(balanced_sample)\n",
    "\n",
    "        X_subsample = X_train.iloc[balanced_sample]\n",
    "        y_labels_subsample = y_train.iloc[balanced_sample]\n",
    "        y_subsample = classify_api(target, X_subsample, threshold=0.5, type='hard')\n",
    "\n",
    "        \n",
    "        # Training the substitute model\n",
    "        if surrogate_algo == 'RandomForestClassifier':\n",
    "            surrogate_model = RandomForestClassifier(random_state=38)\n",
    "        elif surrogate_algo == 'KNNClassifier':\n",
    "            surrogate_model = KNeighborsClassifier(n_neighbors = 5)\n",
    "        elif surrogate_algo == 'QDAClassifier':\n",
    "            surrogate_model = QuadraticDiscriminantAnalysis()\n",
    "        elif surrogate_algo == 'ADABoostClassifier':\n",
    "            surrogate_model = AdaBoostClassifier(random_state=38)\n",
    "        elif surrogate_algo == 'LogisticRegression':\n",
    "            surrogate_model = LogisticRegression(random_state=38)\n",
    "        elif surrogate_algo == 'MLPClassifier':\n",
    "            surrogate_model = MLPClassifier(hidden_layer_sizes=(154, 77), activation='relu', solver='sgd', learning_rate_init=0.0001, max_iter=20, random_state=38, verbose=True)\n",
    "        else:\n",
    "            print(f\"Option {surrogate_algo} isn't within algorithms options.\")\n",
    "            break\n",
    "        \n",
    "        surrogate_model.fit(X_subsample, y_subsample)\n",
    "        dump(surrogate_model, \"surrogate_model.joblib\")\n",
    "        \n",
    "        surrogate_model_detection_rate = np.mean(classify_api(\"surrogate_model.joblib\", X_test, threshold=0.5, type='hard').to_numpy() == y_test.to_numpy())\n",
    "        print(\"=======================================================================================================\")\n",
    "        print(f'Detection rate of target model: {detection_rate}')\n",
    "        print(f'Detection rate of surrogate model: {surrogate_model_detection_rate:.4f}')\n",
    "        print(\"=======================================================================================================\")\n",
    "        \n",
    "\n",
    "        temp_file = os.path.join(os.getcwd(), \"surrogate_model.joblib\")\n",
    "        if os.path.exists(temp_file):\n",
    "            os.remove(temp_file)\n",
    "\n",
    "        if abs(surrogate_model_detection_rate - detection_rate) <= detection_variation:\n",
    "            print(\"Saving substitute model...\")\n",
    " \n",
    "            dump(surrogate_model, substitute_model_path)\n",
    "            print(f'Convergence achieved after {iter_count +1}. Saving model at: {substitute_model_path}')\n",
    "            break\n",
    "\n",
    "        # if continuing, the number of samples is enlarged\n",
    "        num_samples += 2\n",
    "        if num_samples > len(index_0) or num_samples > len(index_1):\n",
    "            print(f\"Process finished after {iter_count +1}. The convergence wasn't achieved.\")\n",
    "            break\n",
    "    print('Calculating EBFA...')\n",
    "    # this phase is not black-box because it needs the train dataset of target model\n",
    "    ave_ebfa = ebfa(target, substitute_model_path, X_train, y_train, X_subsample, y_subsample, X_test, y_test, k=15, p=0.01) \n",
    "    #confusion_matrix_target = confusion_matrix(y_test, target_classify)\n",
    "    #confusion_matrix_surrogate = confusion_matrix(y_labels_subsample, y_subsample)\n",
    "    return (num_samples * 2, surrogate_model_detection_rate, ave_ebfa)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a5020bf-b10a-45b5-b836-d3b8e467ac49",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Attack - Data Augmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "110f3a4f-9aa1-46e3-af61-1b3bf3eaaff3",
   "metadata": {},
   "source": [
    "In this type of attacks, we are using some augmentation technique to enlarge number of data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60df151a-bddf-4eb7-b3f0-8720d61e108c",
   "metadata": {},
   "source": [
    "### Augmentation Techniques"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f24697c-ee6b-4ee9-ae5e-704c77d72df3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_interpolation(data, num_samples, lambda_):\n",
    "    synthetic_data = []\n",
    "\n",
    "    for _ in range(num_samples):\n",
    "        # choose random two numbers\n",
    "        idx1, idx2 = np.random.choice(len(data), 2, replace=False)\n",
    "        point1, point2 = data.iloc[idx1], data.iloc[idx2]\n",
    "\n",
    "        if (lambda_):\n",
    "            factor = lambda_\n",
    "        else:\n",
    "            factor = np.random.uniform(0,1)\n",
    "\n",
    "        new_point = factor * point1 + (1 - factor) * point2\n",
    "        synthetic_data.append(tuple(new_point))\n",
    "    new_data = np.array(list(set(synthetic_data))) # set for avoiding repetition\n",
    "\n",
    "    return new_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e68146db-f56b-40c4-8b09-5392304593d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gaussian_noise(data, noise_level=0.01):\n",
    "    noise = np.random.normal(0, noise_level * np.std(data, axis=0), data.shape)\n",
    "    new_data = data + noise\n",
    "\n",
    "    return new_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "023aa8f9-92e4-41d7-b9f4-561123f2257b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fake_gradient(model, data, noise_level):\n",
    "    # inject noise to data (any method. In this case we're using gaussian_noise function before)\n",
    "    noisy_data = gaussian_noise(data, noise_level)\n",
    "\n",
    "    original_pred = model.predict(data)\n",
    "    noisy_pred = model.predict(noisy_data)\n",
    "\n",
    "    # difference value\n",
    "    deltas = noisy_pred - original_pred \n",
    "\n",
    "    fake_gradient_direction = np.sign(deltas)\n",
    "    fake_gradient_direction_expanded = np.tile(fake_gradient_direction[:, np.newaxis], (1, data.shape[1]))\n",
    "\n",
    "    synthetic_point = data + fake_gradient_direction_expanded * noise_level\n",
    "\n",
    "    return np.array(synthetic_point)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7105e352-b48d-43d2-b230-a8d464a120df",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Data augmented attack - Baseline 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b8fdb03-b619-44c7-ade5-9fa80f18d2bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def substitute_model_a2(target, X, y, X_train_target, y_train_target, lambda_, augment_count=2, detection_variation=0.03, threshold=0.5, mode='li', surrogate_algo='RandomForestClassifier'):\n",
    "    '''\n",
    "\n",
    "    :Param X: Attack data\n",
    "    :Param y: Attack label data\n",
    "    :Param X_train: The data used to train the target model. This is not a black-box phase\n",
    "    :Param y_train: The labels of data used to train the target model.\n",
    "    '''\n",
    "\n",
    "    # config\n",
    "    max_iter = 600 # max iterations to avoid an eternal loop\n",
    "    num_samples = 10 #stating number of each class sample. eg: 5 benign and 5 malicious samples\n",
    "    file_name = 'surrogate_model_a2_'+target \n",
    "    substitute_model_path = os.path.join(os.getcwd(), file_name)\n",
    "    \n",
    "    # reset unpaired index and drop unlabeled rows\n",
    "    X = X.reset_index(drop=True)\n",
    "    y = y.reset_index(drop=True)\n",
    "    \n",
    "    label_index = (y != -1)\n",
    "    X_train = X[label_index]\n",
    "    y_train = y[label_index]\n",
    "    #classify_api(target, X_subsample, threshold=0.5, type='hard')\n",
    "\n",
    "    # creating subsamples (index)\n",
    "    index_0 = (y <= threshold)\n",
    "    index_1 = (y > threshold)\n",
    "    \n",
    "\n",
    "    # test samples to be used in all of testes, including to calculate prediction value of the target model\n",
    "    test_index_0 = np.random.choice(np.where(index_0)[0], 500, replace=False)\n",
    "    test_index_1 = np.random.choice(np.where(index_1)[0], 500, replace=False)\n",
    "    test_sample = np.concatenate((test_index_0, test_index_1))\n",
    "    np.random.shuffle(test_sample)\n",
    "    X_test = X_train.iloc[test_sample]\n",
    "    y_test = y_train.iloc[test_sample]\n",
    "\n",
    "    # Removing test data from original dataset, avoiding data leak\n",
    "    X_train = X_train.drop(test_sample, axis=0).reset_index(drop=True)\n",
    "    y_train = y_train.drop(test_sample, axis=0).reset_index(drop=True)\n",
    "\n",
    "    # re-calculating the index without the test samples\n",
    "    index_0 = (y_train <= threshold)   # index_0 - index_0(test)\n",
    "    index_1 = (y_train > threshold)    # index_1 - index_1(test)\n",
    "\n",
    "    # target prediction value -> This value could be added to the query number, or we could consider that attacks were done before, out of attack context.\n",
    "    # if hard:\n",
    "    target_classify = classify_api(target, X_test, threshold=0.5, type='hard')\n",
    "    detection_rate = np.mean(target_classify.to_numpy() == y_test.to_numpy())\n",
    "    # if soft, comment line above and uncomment line below\n",
    "    #detection_rate = np.mean(classify_api(target, X_test, threshold=0.5, type='hard')[:,0] == y_test)\n",
    "\n",
    "    for iter_count in tqdm.tqdm(range(max_iter), desc='Substitute model training'):\n",
    "        print(f'Iteration {iter_count + 1}: Number of samples = {num_samples * 2}')\n",
    "        balanced_sample_0 = np.random.choice(np.where(index_0)[0], num_samples, replace=False)\n",
    "        balanced_sample_1 = np.random.choice(np.where(index_1)[0], num_samples, replace=False)\n",
    "        balanced_sample = np.concatenate((balanced_sample_0, balanced_sample_1))\n",
    "        np.random.shuffle(balanced_sample)\n",
    "\n",
    "        #creating REAL-DATA subsampling\n",
    "        X_subsample_real = X_train.iloc[balanced_sample]\n",
    "        y_subsample_real = classify_api(target, X_subsample_real, threshold=0.5, type='hard')\n",
    "        #y_subsample_real = y_train.iloc[balanced_sample]\n",
    "        \n",
    "        #creating synthetic-data subsampling\n",
    "        if mode == 'li': #linear interpolation\n",
    "            synthetic_data_0 = linear_interpolation(X_train.iloc[balanced_sample_0], (int(round(num_samples * augment_count))), lambda_)\n",
    "            synthetic_data_1 = linear_interpolation(X_train.iloc[balanced_sample_1], (int(round(num_samples * augment_count))), lambda_)\n",
    "            print(f'Creating {int(len(synthetic_data_0)) + int(len(synthetic_data_1))} synthetic samples using Linear Interpolation')\n",
    "\n",
    "        elif mode == 'gn': # gaussian noise\n",
    "            synthetic_data_0 = gaussian_noise(X_train.iloc[balanced_sample_0], noise_level=lambda_)\n",
    "            synthetic_data_1 = gaussian_noise(X_train.iloc[balanced_sample_1], noise_level=lambda_)\n",
    "            print(f'Creating {int(len(synthetic_data_0)) + int(len(synthetic_data_1))} syntetic samples using Gaussian noise')\n",
    "\n",
    "        elif mode == 'ag': # Fake gradient or Approximated Gradient\n",
    "\n",
    "            sub_substitute_model = RandomForestClassifier(random_state=20)\n",
    "            sub_substitute_model.fit(X_subsample_real, y_subsample_real)\n",
    "            synthetic_data_0 = fake_gradient(sub_substitute_model, X_train.iloc[balanced_sample_0], lambda_)\n",
    "            synthetic_data_1 = fake_gradient(sub_substitute_model, X_train.iloc[balanced_sample_1], lambda_)\n",
    "            print(f'Creating {int(len(synthetic_data_0)) + int(len(synthetic_data_1))} synthetic samples using fake gradient') \n",
    "\n",
    "        else:\n",
    "            print(\"Incorrect option.\")\n",
    "            break\n",
    "        \n",
    "        # Concatenating real samples with synthetic samples\n",
    "        X_subsample = np.concatenate((X_subsample_real, synthetic_data_0, synthetic_data_1))\n",
    "        X_subsample_df = pd.DataFrame(X_subsample, columns=X.columns) # changuing to pd dataframe\n",
    "\n",
    "        synthetic_y_0 = np.array([0] * len(synthetic_data_0))\n",
    "        synthetic_y_1 = np.array([0] * len(synthetic_data_1))\n",
    "        y_subsample = np.concatenate((y_subsample_real, synthetic_y_0, synthetic_y_1))\n",
    "        y_subsample_df = pd.Series(y_subsample)\n",
    "\n",
    "        \n",
    "        print(f'Total of {len(X_subsample_df)} samples')\n",
    "                                 \n",
    "        # Training the substitute model. We can add another algorithms to be used as surrogate model\n",
    "        if surrogate_algo == 'RandomForestClassifier':\n",
    "            surrogate_model = RandomForestClassifier(random_state=38)\n",
    "        elif surrogate_algo == 'KNNClassifier':\n",
    "            surrogate_model = KNeighborsClassifier(n_neighbors = 5)\n",
    "        elif surrogate_algo == 'QDAClassifier':\n",
    "            surrogate_model = QuadraticDiscriminantAnalysis()\n",
    "        elif surrogate_algo == 'ADABoostClassifier':\n",
    "            surrogate_model = AdaBoostClassifier(random_state=38)\n",
    "        elif surrogate_algo == 'LogisticRegression':\n",
    "            surrogate_model = LogisticRegression(random_state=38)\n",
    "        elif surrogate_algo == 'MLPClassifier':\n",
    "            surrogate_model = MLPClassifier(hidden_layer_sizes=(154, 77), activation='relu', solver='sgd', learning_rate_init=0.001, max_iter=20, random_state=38, verbose=True)\n",
    "        else:\n",
    "            print(f\"Option {surrogate_algo} isn't within algorithms options.\")\n",
    "            break\n",
    "            \n",
    "        surrogate_model.fit(X_subsample_df, y_subsample_df)\n",
    "        dump(surrogate_model, \"surrogate_model_intermediate_a2.joblib\")\n",
    "\n",
    "        surrogate_classify = classify_api(\"surrogate_model_intermediate_a2.joblib\", X_test, threshold=0.5, type='hard')\n",
    "        surrogate_model_detection_rate = np.mean(surrogate_classify.to_numpy() == y_test.to_numpy())\n",
    "        print(\"=======================================================================================================\")\n",
    "        print(f'Detection rate of target model: {detection_rate:.4f}')\n",
    "        print(f'Detection rate of surrogate model: {surrogate_model_detection_rate:.4f}')\n",
    "        print(\"=======================================================================================================\")\n",
    "        \n",
    "\n",
    "        temp_file = os.path.join(os.getcwd(), \"surrogate_model_intermediate_a2.joblib\")\n",
    "        if os.path.exists(temp_file):\n",
    "            os.remove(temp_file)\n",
    "\n",
    "        if abs(surrogate_model_detection_rate - detection_rate) <= detection_variation:\n",
    "            print(\"Saving substitute model\")\n",
    "            dump(surrogate_model, substitute_model_path)\n",
    "            \n",
    "            print(f'Convergence achieved after {iter_count +1}. Saving model at: {substitute_model_path}')\n",
    "            break\n",
    "\n",
    "        # if continuing, the number of samples is enlarged\n",
    "        num_samples += 2\n",
    "        if num_samples > len(index_0) or num_samples > len(index_1):\n",
    "            print(f\"Process finished after {iter_count +1}. The convergence wasn't achieved.\")\n",
    "            break\n",
    "            \n",
    "    print('Calculating EBFA...')\n",
    "    # this phase is not black-box because it needs the train dataset of target model\n",
    "    ave_ebfa = ebfa(target, substitute_model_path, X_train_target, y_train_target, X_subsample_df, y_subsample_df, X_test, y_test, k=25, p=0.0001) \n",
    "    #confusion_matrix_target = confusion_matrix(y_test, target_classify)\n",
    "    #confusion_matrix_surrogate = confusion_matrix(y_test, surrogate_classify)\n",
    "    return (num_samples * 2, surrogate_model_detection_rate, ave_ebfa)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "689433da-ea3d-4911-81b2-8ef345f58069",
   "metadata": {},
   "outputs": [],
   "source": [
    "def substitute_model_our_attack_v2(target, X, y, augment_count=2, string='aa', lambda_=0.5, ebfa_limit=0.6, mode='gn', surrogate_algo='RandomForestClassifier', threshold=0.5):\n",
    "    '''\n",
    "\n",
    "    :Param X: Attack data\n",
    "    :Param y: Attack label-data\n",
    "    :Param lambda_:\n",
    "    :Param augment_count:\n",
    "    :Param detection_variation:\n",
    "    '''\n",
    "\n",
    "    # config\n",
    "    max_iter = 100 # max iterations to avoid an eternal loop\n",
    "    num_samples = 200 #stating number of each class sample. eg: 5 benign and 5 malicious samples\n",
    "    file_name = string+'ebfa_limit_'+str(ebfa_limit)+'_surrogate_model_our_attack_'+surrogate_algo+'_'+target+'_'+mode \n",
    "    substitute_model_path = os.path.join(os.getcwd(), file_name)\n",
    "    target_model = load(target)\n",
    "    if surrogate_algo == 'RandomForestClassifier':\n",
    "        surrogate_model = RandomForestClassifier(random_state=38)\n",
    "    elif surrogate_algo == 'DecisionTreeClassifier':\n",
    "        surrogate_model = DecisionTreeClassifier(criterion='log_loss')\n",
    "    elif surrogate_algo == 'KNNClassifier':\n",
    "        surrogate_model = KNeighborsClassifier(n_neighbors = 5)\n",
    "    elif surrogate_algo == 'QDAClassifier':\n",
    "        surrogate_model = QuadraticDiscriminantAnalysis()\n",
    "    elif surrogate_algo == 'ADABoostClassifier':\n",
    "        surrogate_model = AdaBoostClassifier(random_state=38)\n",
    "    elif surrogate_algo == 'LogisticRegression':\n",
    "        surrogate_model = LogisticRegression(random_state=38)\n",
    "    elif surrogate_algo == 'MLPClassifier':\n",
    "        surrogate_model = MLPClassifier(hidden_layer_sizes=(154, 154), activation='relu', solver='sgd', alpha=1e-4, learning_rate_init=0.001, max_iter=100, random_state=38, verbose=True)\n",
    "    else:\n",
    "        print(f\"Option {surrogate_algo} isn't within algorithms options.\")\n",
    "        return -1\n",
    "            \n",
    "    \n",
    "    # reset unpaired index and drop unlabeled rows\n",
    "    X = X.reset_index(drop=True)\n",
    "    y = y.reset_index(drop=True)\n",
    "    \n",
    "    label_index = (y != -1)\n",
    "    X_train = X[label_index]\n",
    "    y_train = y[label_index]\n",
    "\n",
    "    # creating subsamples (index)\n",
    "    index_0 = (y <= threshold)\n",
    "    index_1 = (y > threshold)\n",
    "    \n",
    "\n",
    "    # test samples to be used in all of the validations, including to calculate the prediction value of the target model\n",
    "    test_index_0 = np.random.choice(np.where(index_0)[0], 500, replace=False)\n",
    "    test_index_1 = np.random.choice(np.where(index_1)[0], 500, replace=False)\n",
    "    test_sample = np.concatenate((test_index_0, test_index_1))\n",
    "    np.random.shuffle(test_sample)\n",
    "    X_test = X_train.iloc[test_sample]\n",
    "    y_test = y_train.iloc[test_sample]\n",
    "\n",
    "    # Removing test data from original dataset, avoiding data leak\n",
    "    X_train = X_train.drop(test_sample, axis=0).reset_index(drop=True)\n",
    "    y_train = y_train.drop(test_sample, axis=0).reset_index(drop=True)\n",
    "\n",
    "    # re-calculating the index without the test samples\n",
    "    index_0 = (y_train <= threshold)   # index_0 - index_0(test)\n",
    "    index_1 = (y_train > threshold)    # index_1 - index_1(test)\n",
    "\n",
    "    # Shap config\n",
    "    X_train_sample, _, y_train_sample, _ = train_test_split(X_train, y_train, train_size=0.01, stratify=y_train)\n",
    "    _, X_test_sample, _, y_test_sample = train_test_split(X_test, y_test, test_size=0.1, stratify=y_test)\n",
    "    X_test_sample = pd.DataFrame(X_test_sample, columns=X.columns)\n",
    "    background_data = X_test_sample.map(lambda x: 0)\n",
    "\n",
    "    \n",
    "    # calculating shap-sampling to the target model\n",
    "    print('Calculating Shap values of the target model')\n",
    "    explainer_1 = shap.SamplingExplainer(target_model.predict, X_train_sample)\n",
    "    shap_values_1 = explainer_1.shap_values(X_test_sample) \n",
    "\n",
    "    for iter_count in tqdm.tqdm(range(max_iter), desc='Substitute model training'):\n",
    "        print(f'Iteration {iter_count + 1}: Number of samples = {num_samples * 2}')\n",
    "        balanced_sample_0 = np.random.choice(np.where(index_0)[0], num_samples, replace=False)\n",
    "        balanced_sample_1 = np.random.choice(np.where(index_1)[0], num_samples, replace=False)\n",
    "        balanced_sample = np.concatenate((balanced_sample_0, balanced_sample_1))\n",
    "        np.random.shuffle(balanced_sample)\n",
    "\n",
    "        #creating REAL-DATA subsampling\n",
    "        X_subsample_real = X_train.iloc[balanced_sample]\n",
    "        y_subsample_real = classify_api(target, X_subsample_real, threshold=0.5, type='hard')\n",
    "        \n",
    "        #creating synthetic-data subsampling\n",
    "        if mode == 'li': #linear interpolation\n",
    "            synthetic_data_0 = linear_interpolation(X_train.iloc[balanced_sample_0], (int(round(num_samples * augment_count))), lambda_)\n",
    "            synthetic_data_1 = linear_interpolation(X_train.iloc[balanced_sample_1], (int(round(num_samples * augment_count))), lambda_)\n",
    "            print(f'Creating {int(len(synthetic_data_0)) + int(len(synthetic_data_1))} synthetic samples using Linear Interpolation')\n",
    "\n",
    "        elif mode == 'gn': # gaussian noise\n",
    "            synthetic_data_0 = gaussian_noise(X_train.iloc[balanced_sample_0], noise_level=lambda_)\n",
    "            synthetic_data_1 = gaussian_noise(X_train.iloc[balanced_sample_1], noise_level=lambda_)\n",
    "            print(f'Creating {int(len(synthetic_data_0)) + int(len(synthetic_data_1))} syntetic samples using Gaussian noise')\n",
    "\n",
    "        else:\n",
    "            print(\"Incorrect option.\")\n",
    "            break\n",
    "        \n",
    "        # Concatenating real samples with synthetic samples\n",
    "        X_subsample = np.concatenate((X_subsample_real, synthetic_data_0, synthetic_data_1))\n",
    "        X_subsample_df = pd.DataFrame(X_subsample, columns=X.columns) # changing to pd dataframe\n",
    "\n",
    "        synthetic_y_0 = np.array([0] * len(synthetic_data_0))\n",
    "        synthetic_y_1 = np.array([0] * len(synthetic_data_1))\n",
    "        y_subsample = np.concatenate((y_subsample_real, synthetic_y_0, synthetic_y_1))\n",
    "        y_subsample_df = pd.Series(y_subsample)\n",
    "\n",
    "        \n",
    "        print(f'Total of {len(X_subsample_df)} samples')\n",
    "        surrogate_model.fit(X_subsample_df, y_subsample_df)\n",
    "        dump(surrogate_model, \"surrogate_model_intermediate_our_attack.joblib\")\n",
    "        print('Calculating EBFA...')\n",
    "        explainer_2 = shap.SamplingExplainer(surrogate_model.predict, X_train_sample)\n",
    "        shap_values_2 = explainer_2.shap_values(X_test_sample)\n",
    "        ebfa_models = shap_values_importance(shap_values_1, shap_values_2, k=15)\n",
    "        \n",
    "        print(\"=======================================================================================================\")\n",
    "        print(f'Queries: {(num_samples*2):.4f}')\n",
    "        print(f'Explainability-Based Feature Agreement: {ebfa_models:.4f}')\n",
    "        print(\"=======================================================================================================\")\n",
    "        \n",
    "        temp_file = os.path.join(os.getcwd(), \"surrogate_model_intermediate_our_attack.joblib\")\n",
    "        if os.path.exists(temp_file):\n",
    "            os.remove(temp_file)\n",
    "\n",
    "        if abs(ebfa_models) >= ebfa_limit:\n",
    "            print(\"Saving substitute model\")\n",
    "            dump(surrogate_model, substitute_model_path)\n",
    "            \n",
    "            print(f'Convergence achieved after {iter_count +1}. Saving model at: {substitute_model_path}')\n",
    "            return (num_samples * 2, ebfa_models)\n",
    "\n",
    "        # if continuing, the number of samples is enlarged\n",
    "        num_samples += 200\n",
    "        if num_samples > len(index_0) or num_samples > len(index_1):\n",
    "            print(f\"Process finished after {iter_count +1}. The convergence wasn't achieved.\")\n",
    "            return -1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc429438-9490-4602-b00e-d01660a368ad",
   "metadata": {},
   "source": [
    "# Threat Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe393f26-50e7-491e-8b54-6b0627bc1620",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Query-only attacks - Baseline 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba4b8b0-21c6-4c7d-9256-7659b9e079e8",
   "metadata": {},
   "source": [
    "Here we execute the attacks comparing the results between the methods."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d7ea2c-2c75-459f-849f-247239bb93f6",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccc62db1-232f-43b3-a3d8-653b26173a30",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.00\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a1(target,X_data_extraction, y_data_extraction, X_train, y_train, detection_variation=Detection_variation, threshold=0.5, surrogate_algo=surrogate_algo)\n",
    "    grades.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_0_a1 = dict(zip(target_list, grades))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_0_a1}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6177f4c5-f3c8-4415-8966-80d76fcd0833",
   "metadata": {},
   "outputs": [],
   "source": [
    "dados = [[key] + list(value) for key, value in grades_detection_variation_0_a1.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df = pd.DataFrame(dados, columns=columns)\n",
    "df\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "649e8229-21bc-46fd-a9e2-a38bad4dae3d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_surrogate_randomforest = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.00\n",
    "for target in target_list:\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a1(target,X_data_extraction, y_data_extraction, X_train, y_train, detection_variation=Detection_variation, threshold=0.5)\n",
    "    grades_surrogate_randomforest.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_0_a1_surrogate_randomforest = dict(zip(target_list, grades_surrogate_randomforest))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_0_a1_surrogate_randomforest}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e0b0587-51c0-4bc6-ae89-dde01bbe1b0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "grades_detection_variation_0_a1_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55ab6abc-374a-47bf-94f7-3db4b0d141ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "dados = [[key] + list(value) for key, value in grades_detection_variation_0_a1_surrogate_randomforest.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a1 = pd.DataFrame(dados, columns=columns)\n",
    "df_randomforest_a1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55dc3633-dbc0-4647-827b-5262c0a68212",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d16e5e0-c9b2-4d66-afb5-c43fd294c18f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_01_a1 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.01\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a1(target,X_data_extraction, y_data_extraction, X_train, y_train, detection_variation=Detection_variation, threshold=0.5, surrogate_algo=surrogate_algo)\n",
    "    grades_01_a1.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_01_a1 = dict(zip(target_list, grades_01_a1))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_01_a1}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fedd2ed4-db75-4fa3-8820-9564ce1d1b08",
   "metadata": {},
   "outputs": [],
   "source": [
    "dados = [[key] + list(value) for key, value in grades_detection_variation_01_a1.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a1_01 = pd.DataFrame(dados, columns=columns)\n",
    "df_randomforest_a1_01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25300279-8f56-4dd0-a9b2-af1a37a1099a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier technique\n",
    "grades_01_a1_surrogate_randomforest = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.01\n",
    "for target in target_list:\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a1(target,X_data_extraction, y_data_extraction, X_train, y_train, detection_variation=Detection_variation, threshold=0.5)\n",
    "    grades_01_a1_surrogate_randomforest.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_01_a1_surrogate_randomforest = dict(zip(target_list, grades_01_a1_surrogate_randomforest))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_01_a1_surrogate_randomforest}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d659ea4-9933-445a-8627-5aa5354be762",
   "metadata": {},
   "outputs": [],
   "source": [
    "dados = [[key] + list(value) for key, value in grades_detection_variation_01_a1_surrogate_randomforest.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a1_01 = pd.DataFrame(dados, columns=columns)\n",
    "df_randomforest_a1_01"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "943a8ead-d737-4b9c-9147-ab28a15ebb1f",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22c69ab3-6bf6-425e-a246-a9015ff45686",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_02_a1 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.02\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a1(target,X_data_extraction, y_data_extraction, X_train, y_train, detection_variation=Detection_variation, threshold=0.5, surrogate_algo=surrogate_algo)\n",
    "    grades_02_a1.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_02_a1 = dict(zip(target_list, grades_02_a1))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_02_a1}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96e373c8-84a8-4ee1-b795-00376739721d",
   "metadata": {},
   "outputs": [],
   "source": [
    "dados = [[key] + list(value) for key, value in grades_detection_variation_02_a1.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a1_02 = pd.DataFrame(dados, columns=columns)\n",
    "df_randomforest_a1_02"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da03f4fa-e0f8-40ba-a8ee-a278512d7db3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier technique\n",
    "grades_02_a1_surrogate_randomforest = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.02\n",
    "for target in target_list:\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a1(target,X_data_extraction, y_data_extraction, X_train, y_train, detection_variation=Detection_variation, threshold=0.5)\n",
    "    grades_02_a1_surrogate_randomforest.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_02_a1_surrogate_randomforest = dict(zip(target_list, grades_02_a1_surrogate_randomforest))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_02_a1_surrogate_randomforest}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7301319d-9bc0-4696-8796-81d89c6b6cac",
   "metadata": {},
   "outputs": [],
   "source": [
    "dados = [[key] + list(value) for key, value in grades_detection_variation_02_a1_surrogate_randomforest.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a1_03_surrogate_randomforest = pd.DataFrame(dados, columns=columns)\n",
    "df_randomforest_a1_03_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9059fbbe-5215-442e-94a9-f78124b03757",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20aac575-e5d7-4aa4-af0b-fe3eb346aaf0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_03_a1 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.03\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a1(target,X_data_extraction, y_data_extraction, X_train, y_train, detection_variation=Detection_variation, threshold=0.5, surrogate_algo=surrogate_algo)\n",
    "    grades_03_a1.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_03_a1 = dict(zip(target_list, grades_03_a1))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_03_a1}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed68d647-2527-4e0d-8187-57ef7ba4bbbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "dados = [[key] + list(value) for key, value in grades_detection_variation_03_a1.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a1_03_surrogate_randomforest = pd.DataFrame(dados, columns=columns)\n",
    "df_randomforest_a1_03_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b00b6c91-c41b-4465-a4c8-4c90d4bf9172",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier technique\n",
    "grades_03_a1_surrogate_randomforest = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.03\n",
    "for target in target_list:\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a1(target,X_data_extraction, y_data_extraction, X_train, y_train, detection_variation=Detection_variation, threshold=0.5)\n",
    "    grades_03_a1_surrogate_randomforest.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_03_a1_surrogate_randomforest = dict(zip(target_list, grades_03_a1_surrogate_randomforest))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_03_a1_surrogate_randomforest}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0011481-390e-46c8-9bd6-35e691a13a54",
   "metadata": {},
   "outputs": [],
   "source": [
    "dados = [[key] + list(value) for key, value in grades_detection_variation_03_a1_surrogate_randomforest.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a1_03_surrogate_randomforest = pd.DataFrame(dados, columns=columns)\n",
    "df_randomforest_a1_03_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82ccc488-eac8-416a-86b9-df48091c008a",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Augmented-data attacks - Baseline 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdb08beb-116a-4ba1-824b-343b18606017",
   "metadata": {},
   "source": [
    "First type of attack, injecting Gaussian Noise into the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "999521ed-0e0b-4f0a-9a5a-4859a7c3edf8",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Detection Variation: 0.0 and Gaussian Noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cc27013-a4ba-4df3-be17-78b770853e5a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_a2_gn_00 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.00\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='gn', surrogate_algo=surrogate_algo)\n",
    "\n",
    "    grades_a2_gn_00.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_0_a2_gn = dict(zip(target_list, grades_a2_gn_00))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_0_a2_gn}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_0_a2_gn.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_00_gn = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "621d18f4-e2c7-43f9-92cb-7659fb0a37a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_00_gn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9e5967f-78ad-45f9-8462-0cea623a97ff",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_a2_gn_00_surrogate_randomforestclassifier = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.00\n",
    "for target in target_list:\n",
    "    \n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='gn')\n",
    "\n",
    "    grades_a2_gn_00_surrogate_randomforestclassifier.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_0_a2_gn_surrogate_randomforestclassifier = dict(zip(target_list, grades_a2_gn_00_surrogate_randomforestclassifier))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_0_a2_gn_surrogate_randomforestclassifier}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_0_a2_gn_surrogate_randomforestclassifier.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_00_gn_surrogate_randomforest = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6300b050-4e31-449f-ae01-3dd55f25233c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_00_gn_surrogate_randomforest\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623cd81f-9104-4df6-977b-85c6c8cff86d",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.1 and Gaussian Noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ecfd500-2347-4681-8bbe-23f3c3edd915",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_a2_gn_01 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.01\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='gn', surrogate_algo=surrogate_algo)\n",
    "\n",
    "    grades_a2_gn_01.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_01_a2_gn = dict(zip(target_list, grades_a2_gn_01))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_01_a2_gn}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_01_a2_gn.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_01_gn = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f795a7f8-34b2-478b-b426-9bd9c930b38d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_01_gn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ca365c-40cc-487c-a52f-d7173b94976a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_a2_gn_01_surrogate_randomforestclassifier = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.01\n",
    "for target in target_list:\n",
    "    \n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='gn')\n",
    "\n",
    "    grades_a2_gn_01_surrogate_randomforestclassifier.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_01_a2_gn_surrogate_randomforestclassifier = dict(zip(target_list, grades_a2_gn_01_surrogate_randomforestclassifier))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_01_a2_gn_surrogate_randomforestclassifier}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_01_a2_gn_surrogate_randomforestclassifier.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_01_gn_surrogate_randomforest = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6c56658-add4-45ba-9415-20a906e648de",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_01_gn_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8477cdaa-5a20-4edd-a2ca-589cb9f427de",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.2 and Gaussian Noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97a4d09d-490f-4953-8d3b-2d332c1f07a0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_a2_gn_02 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.02\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='gn', surrogate_algo=surrogate_algo)\n",
    "\n",
    "    grades_a2_gn_02.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_02_a2_gn = dict(zip(target_list, grades_a2_gn_02))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_02_a2_gn}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_02_a2_gn.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_02_gn = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c85a5b4-969c-4515-9748-1223277490d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_02_gn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4186987-b3da-4d02-918b-1e063932a54f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_a2_gn_02_surrogate_randomforestclassifier = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.02\n",
    "for target in target_list:\n",
    "    \n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='gn')\n",
    "\n",
    "    grades_a2_gn_02_surrogate_randomforestclassifier.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_02_a2_gn_surrogate_randomforestclassifier = dict(zip(target_list, grades_a2_gn_02_surrogate_randomforestclassifier))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_02_a2_gn_surrogate_randomforestclassifier}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_02_a2_gn_surrogate_randomforestclassifier.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_02_gn_surrogate_randomforest = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f92e1689-9311-4b01-9fc9-9e23e018c8de",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_02_gn_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67d38bf9-f213-468b-83a5-d76cf157bf9f",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.3 and Gaussian Noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a028577-04a7-4a35-9419-4df15ce861ad",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_a2_gn_03 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.03\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='gn', surrogate_algo=surrogate_algo)\n",
    "\n",
    "    grades_a2_gn_03.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_03_a2_gn = dict(zip(target_list, grades_a2_gn_03))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_03_a2_gn}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_03_a2_gn.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_03_gn = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4787ee4e-b1d1-4825-8f39-07d06665938f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_03_gn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e29bc1d3-64b5-4c51-b4cd-6fcab9b3b2c4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_a2_gn_03_surrogate_randomforestclassifier = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.03\n",
    "for target in target_list:\n",
    "    \n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='gn')\n",
    "\n",
    "    grades_a2_gn_03_surrogate_randomforestclassifier.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_03_a2_gn_surrogate_randomforestclassifier = dict(zip(target_list, grades_a2_gn_03_surrogate_randomforestclassifier))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_03_a2_gn_surrogate_randomforestclassifier}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_03_a2_gn_surrogate_randomforestclassifier.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_03_gn_surrogate_randomforest = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98b59c26-696e-4efc-b21b-284600b47309",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_03_gn_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d0e15f6-249c-474f-8e3e-36b196145aae",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.0 and Approximated Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8b652e7-cfda-437c-a280-d0bc17ac3a4b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_a2_ag_00 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.00\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='ag', surrogate_algo=surrogate_algo)\n",
    "\n",
    "    grades_a2_ag_00.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_00_a2_ag = dict(zip(target_list, grades_a2_ag_00))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_00_a2_ag}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_00_a2_ag.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_00_ag = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ab18141-292c-404a-ba42-847608a1041a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_00_ag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45503d07-23ba-4a2d-8905-3842fe9c6227",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_a2_ag_00_surrogate_randomforestclassifier = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.03\n",
    "for target in target_list:\n",
    "    \n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='ag')\n",
    "\n",
    "    grades_a2_ag_00_surrogate_randomforestclassifier.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_00_a2_ag_surrogate_randomforestclassifier = dict(zip(target_list, grades_a2_ag_00_surrogate_randomforestclassifier))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_00_a2_ag_surrogate_randomforestclassifier}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_00_a2_ag_surrogate_randomforestclassifier.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_00_ag_surrogate_randomforest = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30d9f450-d9c5-4955-b774-4d5bbaa13c2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_00_ag_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa9609fe-c804-4e05-b13f-5afdad0b300b",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.1 and Approximated Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc3f6fb2-15c9-4d0b-bc6e-13a5801685df",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_a2_ag_01 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.01\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='ag', surrogate_algo=surrogate_algo)\n",
    "\n",
    "    grades_a2_ag_01.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_01_a2_ag = dict(zip(target_list, grades_a2_ag_01))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_01_a2_ag}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_01_a2_ag.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_01_ag = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db9fb85e-450e-448f-94b4-2fa835a6b3c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_01_ag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f8dba18-ae29-45cf-abdb-c795b7accdc7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_a2_ag_01_surrogate_randomforestclassifier = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.01\n",
    "for target in target_list:\n",
    "    \n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='ag')\n",
    "\n",
    "    grades_a2_ag_01_surrogate_randomforestclassifier.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_01_a2_ag_surrogate_randomforestclassifier = dict(zip(target_list, grades_a2_ag_01_surrogate_randomforestclassifier))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_01_a2_ag_surrogate_randomforestclassifier}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_01_a2_ag_surrogate_randomforestclassifier.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_01_ag_surrogate_randomforest = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5ae3ee1-c019-4ced-a18d-a5bc27f572a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_01_ag_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26c88805-1b4a-41ff-9118-5c9cc6fd3f3f",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.2 and Approximated Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37cf7340-c817-4aa3-b062-2c754393c093",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_a2_ag_02 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.02\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='ag', surrogate_algo=surrogate_algo)\n",
    "\n",
    "    grades_a2_ag_02.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_02_a2_ag = dict(zip(target_list, grades_a2_ag_02))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_02_a2_ag}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_02_a2_ag.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_02_ag = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3ba941d-870b-4fe3-9d25-524dbbc98cb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_02_ag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5b09ea2-be40-4c94-ba09-5b7220d37134",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_a2_ag_02_surrogate_randomforestclassifier = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.02\n",
    "for target in target_list:\n",
    "    \n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='ag')\n",
    "\n",
    "    grades_a2_ag_02_surrogate_randomforestclassifier.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_02_a2_ag_surrogate_randomforestclassifier = dict(zip(target_list, grades_a2_ag_02_surrogate_randomforestclassifier))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_02_a2_ag_surrogate_randomforestclassifier}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_02_a2_ag_surrogate_randomforestclassifier.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_02_ag_surrogate_randomforest = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20c9b77e-93c8-4ad3-b4e1-7026ff1b41c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_02_ag_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4264edb-72b7-4d50-88b9-8d33d2a68748",
   "metadata": {},
   "source": [
    "### Detection Variation: 0.3 and Approximated Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c65c652-3814-49a7-9662-a7e0c22b1257",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the same technique of main model\n",
    "grades_a2_ag_03 = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.03\n",
    "for target in target_list:\n",
    "    surrogate_algo = target.split('.')[0]\n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='ag', surrogate_algo=surrogate_algo)\n",
    "\n",
    "    grades_a2_ag_03.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: {surrogate_algo} and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_03_a2_ag = dict(zip(target_list, grades_a2_ag_03))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_03_a2_ag}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_03_a2_ag.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_03_ag = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3446bbc0-f2b5-4427-84cb-93af56c0ce07",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_03_ag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bbb7248-33b9-4151-abea-1e6e4f4bd864",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Attacking models and using as surrogate model the RandomForestClassifier\n",
    "grades_a2_ag_03_surrogate_randomforestclassifier = []\n",
    "target_list = ['ADABoostClassifier.joblib', 'KNNClassifier.joblib', 'LogisticRegression.joblib', 'MLPClassifier.joblib', 'QDAClassifier.joblib', 'RandomForestClassifier.joblib']\n",
    "Detection_variation = 0.03\n",
    "for target in target_list:\n",
    "    \n",
    "    print(\"\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")\n",
    "    print(\"######################################################################\")\n",
    "    print(f'Starting attack at the target: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    grade = substitute_model_a2(target,X_data_extraction, y_data_extraction, X_train, y_train, lambda_=0.2, augment_count=2, detection_variation=Detection_variation, threshold=0.5, mode='ag')\n",
    "\n",
    "    grades_a2_ag_03_surrogate_randomforestclassifier.append(grade)\n",
    "    print(f'Targe: {target} and surrogate model: RandomForestClassifier and Detection Variation: {Detection_variation}\\n')\n",
    "    print(\"######################################################################\")\n",
    "    print(f'grade {grade}\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')\n",
    "    grades_detection_variation_03_a2_ag_surrogate_randomforestclassifier = dict(zip(target_list, grades_a2_ag_03_surrogate_randomforestclassifier))\n",
    "        \n",
    "print(f'Array of grades: {grades_detection_variation_03_a2_ag_surrogate_randomforestclassifier}')\n",
    "dados = [[key] + list(value) for key, value in grades_detection_variation_03_a2_ag_surrogate_randomforestclassifier.items()]\n",
    "columns = ['Model', 'Query', 'Detection Rate', 'EBFA']\n",
    "df_randomforest_a2_03_ag_surrogate_randomforest = pd.DataFrame(dados, columns=columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e6450ee-2507-454b-b909-b93b07e7fbd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_randomforest_a2_03_ag_surrogate_randomforest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68e7c29e-4fd8-41a3-8e54-8d61e23cb976",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Feito!!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fd12b30-38f5-4010-9035-9feaad404ca1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6809b88f-5f11-4cd6-bd8e-d62116c08fa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def features_importance(model_file, n):\n",
    "    '''\n",
    "    Return top n features of a Forest model\n",
    "    :param model_file: name of the saved model\n",
    "    :param n: top n features\n",
    "    '''\n",
    "    \n",
    "    a = load(model_file)\n",
    "    a_importances = a.feature_importances_\n",
    "    a_features_df = pd.DataFrame({\n",
    "        'Feature': (X_test.columns), \n",
    "        'Importance': a_importances\n",
    "    })\n",
    "    a_features_df = a_features_df.sort_values(by='Importance', ascending=False)\n",
    "    \n",
    "    return a_features_df.head(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ab4f8a8-c97e-4e1f-93af-5942c462638d",
   "metadata": {},
   "outputs": [],
   "source": [
    "features_importance('RandomForestClassifier.joblib',15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc8c2b85-b87b-4512-b60a-7faa5d20dd00",
   "metadata": {},
   "outputs": [],
   "source": [
    "features_importance('surrogate_model_our_attack_RandomForestClassifier.joblib', 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c5a2bc4-6e6e-453b-918f-a13faca24a80",
   "metadata": {},
   "source": [
    "## Final Augmented Attack - Our Attack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcbef049-24f3-4410-9c67-837bad9b93e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "''' Creating clusters to use as background data in our attack ''' \n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "def process_and_scale_data(X, y, n_clusters, n_init=10):\n",
    "    ''' Generate normalized centroids'''\n",
    "\n",
    "    df = X.copy()\n",
    "    df['label'] = y\n",
    "    \n",
    "    # Dividindo o DataFrame com base nas labels\n",
    "    df_0 = df[df['label'] == 0].drop('label', axis=1)\n",
    "    df_1 = df[df['label'] == 1].drop('label', axis=1)\n",
    "\n",
    "    # Treinando modelos KMeans\n",
    "    kmeans_0 = KMeans(n_clusters=n_clusters, n_init=n_init, random_state=25).fit(df_0)\n",
    "    kmeans_1 = KMeans(n_clusters=n_clusters, n_init=n_init, random_state=25).fit(df_1)\n",
    "\n",
    "    # Extraindo centróides\n",
    "    centroids_0 = kmeans_0.cluster_centers_\n",
    "    centroids_1 = kmeans_1.cluster_centers_\n",
    "\n",
    "    # Criando DataFrames para os centróides\n",
    "    centroids_df_0 = pd.DataFrame(centroids_0, columns=df_0.columns)\n",
    "    centroids_df_1 = pd.DataFrame(centroids_1, columns=df_1.columns)\n",
    "\n",
    "    # Normalizando os dados\n",
    "    scaler = MinMaxScaler()\n",
    "    scaled_df_0 = pd.DataFrame(scaler.fit_transform(centroids_df_0), columns=centroids_df_0.columns)\n",
    "    scaled_df_1 = pd.DataFrame(scaler.fit_transform(centroids_df_1), columns=centroids_df_1.columns)\n",
    "\n",
    "    return scaled_df_0, scaled_df_1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6402614-8fbb-4622-bb9a-abcd80ef38e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This functions is used with our attack\n",
    "def shap_values_importance(shap_values_1, shap_values_2, k=15): \n",
    "    # Extract SHAP values for the instance\n",
    "    importance_1 = ( np.sum(np.abs(shap_values_1), axis=1) / shap_values_1.shape[0] )\n",
    "    importance_2 = ( np.sum(np.abs(shap_values_2), axis=1) / shap_values_2.shape[0] )\n",
    "\n",
    "    # Determine top k features and convert indices to tuples\n",
    "    top_feats_1 = set(tuple(np.argsort(-importance_1))[:k])\n",
    "    top_feats_2 = set(tuple(np.argsort(-importance_2))[:k])\n",
    "\n",
    "    # Calculate intersection and EBFA score\n",
    "    intersection = top_feats_1.intersection(top_feats_2)\n",
    "    ebfa_score = len(intersection) / k\n",
    "\n",
    "    return ebfa_score\n",
    "    # we can return the intersection feature list, return (ebfa_score, intersection). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d425582c-be11-43c6-a280-7ab4f2b67155",
   "metadata": {},
   "outputs": [],
   "source": [
    "Ascaler = MinMaxScaler()\n",
    "X_scaled = Ascaler.fit_transform(X_data_extraction)\n",
    "X_scaled = pd.DataFrame(X_scaled, columns=list(X_data_extraction.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1776be25-b634-4567-b459-f8b17de504b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "def process_and_scale_data(X, y, n_clusters, n_init=10):\n",
    "    ''' Generate normalized centroids'''\n",
    "\n",
    "    df = X.copy()\n",
    "    df['label'] = y\n",
    "    \n",
    "    # Dividindo o DataFrame com base nas labels\n",
    "    df_0 = df[df['label'] == 0].drop('label', axis=1)\n",
    "    df_1 = df[df['label'] == 1].drop('label', axis=1)\n",
    "\n",
    "    # Treinando modelos KMeans\n",
    "    kmeans_0 = KMeans(n_clusters=n_clusters, n_init=n_init, random_state=25).fit(df_0)\n",
    "    kmeans_1 = KMeans(n_clusters=n_clusters, n_init=n_init, random_state=25).fit(df_1)\n",
    "\n",
    "    # Extraindo centróides\n",
    "    centroids_0 = kmeans_0.cluster_centers_\n",
    "    centroids_1 = kmeans_1.cluster_centers_\n",
    "\n",
    "    # Criando DataFrames para os centróides\n",
    "    centroids_df_0 = pd.DataFrame(centroids_0, columns=df_0.columns)\n",
    "    centroids_df_1 = pd.DataFrame(centroids_1, columns=df_1.columns)\n",
    "\n",
    "    # Normalizando os dados\n",
    "    scaler = MinMaxScaler()\n",
    "    scaled_df_0 = pd.DataFrame(scaler.fit_transform(centroids_df_0), columns=centroids_df_0.columns)\n",
    "    scaled_df_1 = pd.DataFrame(scaler.fit_transform(centroids_df_1), columns=centroids_df_1.columns)\n",
    "\n",
    "    return scaled_df_0, scaled_df_1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3aa27fa5-e248-4d26-87a8-6496bf743bdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def substitute_model_our_attack(target, X, y, augment_count=2, string='aa', lambda_=0.7, ebfa_limit=0.6, mode='li', surrogate_algo='RandomForestClassifier', threshold=0.5):\n",
    "    '''\n",
    "\n",
    "    :Param X: Attack data\n",
    "    :Param y: Attack label-data\n",
    "    :Param lambda_:\n",
    "    :Param augment_count:\n",
    "    :Param detection_variation:\n",
    "    '''\n",
    "\n",
    "    # config\n",
    "    max_iter = 20 # max iterations to avoid an eternal loop\n",
    "    num_samples = 200 #stating number of each class sample. eg: 5 benign and 5 malicious samples\n",
    "    file_name = string+'ebfa_limit_'+str(ebfa_limit)+'_surrogate_model_our_attack_'+surrogate_algo+'_'+target+'_'+mode \n",
    "    substitute_model_path = os.path.join(os.getcwd(), file_name)\n",
    "    target_model = load(target)\n",
    "    if surrogate_algo == 'RandomForestClassifier':\n",
    "        surrogate_model = RandomForestClassifier(random_state=38)\n",
    "    elif surrogate_algo == 'DecisionTreeClassifier':\n",
    "        surrogate_model = DecisionTreeClassifier(criterion='log_loss')\n",
    "    elif surrogate_algo == 'KNNClassifier':\n",
    "        surrogate_model = KNeighborsClassifier(n_neighbors = 5)\n",
    "    elif surrogate_algo == 'QDAClassifier':\n",
    "        surrogate_model = QuadraticDiscriminantAnalysis()\n",
    "    elif surrogate_algo == 'ADABoostClassifier':\n",
    "        surrogate_model = AdaBoostClassifier(random_state=38)\n",
    "    elif surrogate_algo == 'LogisticRegression':\n",
    "        surrogate_model = LogisticRegression(random_state=38)\n",
    "    elif surrogate_algo == 'MLPClassifier':\n",
    "        surrogate_model = MLPClassifier(hidden_layer_sizes=(154, 154), activation='relu', solver='sgd', alpha=1e-4, learning_rate_init=0.001, max_iter=100, random_state=38, verbose=True)\n",
    "    else:\n",
    "        print(f\"Option {surrogate_algo} isn't within algorithms options.\")\n",
    "        return -1\n",
    "            \n",
    "    \n",
    "    # reset unpaired index and drop unlabeled rows\n",
    "    X = X.reset_index(drop=True)\n",
    "    y = y.reset_index(drop=True)\n",
    "    \n",
    "    label_index = (y != -1)\n",
    "    X_train = X[label_index]\n",
    "    y_train = y[label_index]\n",
    "\n",
    "    # creating subsamples (index)\n",
    "    index_0 = (y <= threshold)\n",
    "    index_1 = (y > threshold)\n",
    "    \n",
    "\n",
    "    # test samples to be used in all of the validations, including to calculate the prediction value of the target model\n",
    "    test_index_0 = np.random.choice(np.where(index_0)[0], 500, replace=False)\n",
    "    test_index_1 = np.random.choice(np.where(index_1)[0], 500, replace=False)\n",
    "    test_sample = np.concatenate((test_index_0, test_index_1))\n",
    "    np.random.shuffle(test_sample)\n",
    "    X_test = X_train.iloc[test_sample]\n",
    "    y_test = y_train.iloc[test_sample]\n",
    "\n",
    "    # Removing test data from original dataset, avoiding data leak\n",
    "    X_train = X_train.drop(test_sample, axis=0).reset_index(drop=True)\n",
    "    y_train = y_train.drop(test_sample, axis=0).reset_index(drop=True)\n",
    "\n",
    "    # re-calculating the index without the test samples\n",
    "    index_0 = (y_train <= threshold)   # index_0 - index_0(test)\n",
    "    index_1 = (y_train > threshold)    # index_1 - index_1(test)\n",
    "\n",
    "    # Shap config\n",
    "    X_train_sample, _, y_train_sample, _ = train_test_split(X_train, y_train, train_size=0.01, stratify=y_train)\n",
    "    _, X_test_sample, _, y_test_sample = train_test_split(X_test, y_test, test_size=0.1, stratify=y_test)\n",
    "    X_test_sample = pd.DataFrame(X_test_sample, columns=X.columns)\n",
    "    background_data = X_test_sample.map(lambda x: 0)\n",
    "\n",
    "    \n",
    "    # calculating shap-sampling to the target model\n",
    "    print('Calculating Shap values of the target model')\n",
    "    explainer_1 = shap.SamplingExplainer(target_model.predict, pd.concat([process_and_scale_data(X_train, y_train, n_clusters=10, n_init=10)], columns=X_train.columns))\n",
    "    shap_values_1 = explainer_1.shap_values(X_test_sample) \n",
    "\n",
    "    for iter_count in tqdm.tqdm(range(max_iter), desc='Substitute model training'):\n",
    "        print(f'Iteration {iter_count + 1}: Number of samples = {num_samples * 2}')\n",
    "        balanced_sample_0 = np.random.choice(np.where(index_0)[0], num_samples, replace=False)\n",
    "        balanced_sample_1 = np.random.choice(np.where(index_1)[0], num_samples, replace=False)\n",
    "        balanced_sample = np.concatenate((balanced_sample_0, balanced_sample_1))\n",
    "        np.random.shuffle(balanced_sample)\n",
    "\n",
    "        #creating REAL-DATA subsampling\n",
    "        X_subsample_real = X_train.iloc[balanced_sample]\n",
    "        y_subsample_real = classify_api(target, X_subsample_real, threshold=0.5, type='hard')\n",
    "        \n",
    "        #creating synthetic-data subsampling\n",
    "        if mode == 'li': #linear interpolation\n",
    "            synthetic_data_0 = linear_interpolation(X_train.iloc[balanced_sample_0], (int(round(num_samples * augment_count))), lambda_)\n",
    "            synthetic_data_1 = linear_interpolation(X_train.iloc[balanced_sample_1], (int(round(num_samples * augment_count))), lambda_)\n",
    "            print(f'Creating {int(len(synthetic_data_0)) + int(len(synthetic_data_1))} synthetic samples using Linear Interpolation')\n",
    "\n",
    "        elif mode == 'gn': # gaussian noise\n",
    "            synthetic_data_0 = gaussian_noise(X_train.iloc[balanced_sample_0], noise_level=lambda_)\n",
    "            synthetic_data_1 = gaussian_noise(X_train.iloc[balanced_sample_1], noise_level=lambda_)\n",
    "            print(f'Creating {int(len(synthetic_data_0)) + int(len(synthetic_data_1))} syntetic samples using Gaussian noise')\n",
    "\n",
    "        else:\n",
    "            print(\"Incorrect option.\")\n",
    "            break\n",
    "        \n",
    "        # Concatenating real samples with synthetic samples\n",
    "        X_subsample = np.concatenate((X_subsample_real, synthetic_data_0, synthetic_data_1))\n",
    "        X_subsample_df = pd.DataFrame(X_subsample, columns=X.columns) # changing to pd dataframe\n",
    "\n",
    "        synthetic_y_0 = np.array([0] * len(synthetic_data_0))\n",
    "        synthetic_y_1 = np.array([0] * len(synthetic_data_1))\n",
    "        y_subsample = np.concatenate((y_subsample_real, synthetic_y_0, synthetic_y_1))\n",
    "        y_subsample_df = pd.Series(y_subsample)\n",
    "\n",
    "        \n",
    "        print(f'Total of {len(X_subsample_df)} samples')\n",
    "        surrogate_model.fit(X_subsample_df, y_subsample_df)\n",
    "        dump(surrogate_model, \"surrogate_model_intermediate_our_attack.joblib\")\n",
    "        print('Calculating EBFA...')\n",
    "        explainer_2 = shap.SamplingExplainer(target_model.predict,  pd.concat([process_and_scale_data(X_train_sample, y_train_sample, n_clusters=10, n_init=10)], columns=X_train.columns))\n",
    "        shap_values_2 = explainer_2.shap_values(X_test_sample)\n",
    "        ebfa_models = shap_values_importance(shap_values_1, shap_values_2, k=15)\n",
    "        \n",
    "        print(\"=======================================================================================================\")\n",
    "        print(f'Queries: {(num_samples*2):.4f}')\n",
    "        print(f'Explainability-Based Feature Agreement: {ebfa_models:.4f}')\n",
    "        print(\"=======================================================================================================\")\n",
    "        \n",
    "        temp_file = os.path.join(os.getcwd(), \"surrogate_model_intermediate_our_attack.joblib\")\n",
    "        if os.path.exists(temp_file):\n",
    "            os.remove(temp_file)\n",
    "\n",
    "        if abs(ebfa_models) >= ebfa_limit:\n",
    "            print(\"Saving substitute model\")\n",
    "            dump(surrogate_model, substitute_model_path)\n",
    "            \n",
    "            print(f'Convergence achieved after {iter_count +1}. Saving model at: {substitute_model_path}')\n",
    "            return (num_samples * 2, ebfa_models)\n",
    "        # if continuing, the number of samples is enlarged\n",
    "        num_samples += 200\n",
    "        if num_samples > len(index_0) or num_samples > len(index_1):\n",
    "            print(f\"Process finished after {iter_count +1}. The convergence wasn't achieved.\")\n",
    "            return -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d0b29e9-88ae-4f0b-954e-da7d66ef7630",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack(\"RandomForestClassifier.joblib\", X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.6, mode='li', surrogate_algo='DecisionTreeClassifier')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc6b8ca0-5bc9-4647-85fc-87c92b442660",
   "metadata": {},
   "outputs": [],
   "source": [
    "features_importance(\"RandomForestClassifier.joblib\", 15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "579c4726-746a-49b4-9ed7-e4df0604e269",
   "metadata": {},
   "outputs": [],
   "source": [
    "features_importance(\"aaebfa_limit_0.6_surrogate_model_our_attack_DecisionTreeClassifier_RandomForestClassifier.joblib_li\", 15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c30af5fb-c560-4548-92e9-50fc67db98ce",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack(\"RandomForestClassifier.joblib\", X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.8, mode='li', surrogate_algo='DecisionTreeClassifier')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcbc1f6d-1e4e-47a9-b9ad-ab6a33955304",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Our Attack against all attacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9f76d84-1c9d-436d-a219-a6d3ddcd7abc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack(\"ADABoostClassifier.joblib\", X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.8, mode='li', surrogate_algo='DecisionTreeClassifier')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee696a1a-5610-475d-b4fc-77ed483f5e88",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack('LogisticRegression.joblib', X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.8, mode='li', surrogate_algo='DecisionTreeClassifier')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74581adf-7317-4ccb-bca4-9f7244f1140e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack('MLPClassifier.joblib', X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.8, mode='li', surrogate_algo='DecisionTreeClassifier')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77f01fdf-4cf6-4a32-8a86-c9e707ab2e3e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack('QDAClassifier.joblib', X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.8, mode='li', surrogate_algo='DecisionTreeClassifier')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38fdc628-7f3a-4c4c-9cbf-10d5f6e3d43e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack('RandomForestClassifier.joblib', X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.6, mode='li', surrogate_algo='DecisionTreeClassifier')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cea9bcf-d30c-4a1d-8327-8b46ee2ca3eb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack('RandomForestClassifier.joblib', X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.6, mode='li', surrogate_algo='DecisionTreeClassifier')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "465a358b-a477-40c8-b9cb-5e872be5ebb5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack('RandomForestClassifier.joblib', X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.6, mode='li', surrogate_algo='RandomForestClassifier')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68044c9c-04b7-40c1-928c-ee1c7ddccc55",
   "metadata": {},
   "outputs": [],
   "source": [
    "features_importance('RandomForestClassifier.joblib', 15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ebc11e9-153d-4c3e-858f-2e980b49eb51",
   "metadata": {},
   "outputs": [],
   "source": [
    "features_importance('aaebfa_limit_0.6_surrogate_model_our_attack_RandomForestClassifier_RandomForestClassifier.joblib_li', 15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "972a5060-92aa-4c73-83c2-c29501954f69",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "substitute_model_our_attack('KNNClassifier.joblib', X_scaled, y_data_extraction, lambda_=0.6, ebfa_limit=0.8, mode='li', surrogate_algo='DecisionTreeClassifier')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}