IML_Assignments/Assignment1/e.shanayev@innopolis.university.ipynb

📌 Machine Learning Assignment 1 - Instructions & Guidelines

📝 General Guidelines

Welcome to Machine Learning Assignment 1! This assignment will test your understanding of regression and classification models, including data preprocessing, hyperparameter tuning, and model evaluation.

Follow the instructions carefully, and ensure your implementation is correct, well-structured, and efficient.

🔹 Submission Format:

• Your submission must be a single Jupyter Notebook (.ipynb) file.
• File Naming Convention:
  • Use your university email as the filename, e.g.,

    j.doe@innopolis.university.ipynb
    

  • Do NOT modify this format, or your submission may not be graded.

🔹 Assignment Breakdown:

Task	Description	Points
Task 1.1	Linear Regression	20
Task 1.2	Polynomial Regression	20
Task 2.1	Data Preprocessing	15
Task 2.2	Model Comparison	45
Total	-	100

---

📂 Dataset & Assumptions

The dataset files are stored in the datasets/ folder.

• Regression Dataset: datasets/task1_data.csv
• Classification Dataset: datasets/pokemon_modified.csv

Each dataset is structured as follows:

🔹 task1_data.csv (for regression tasks)

• Contains X_train, y_train, X_test, and y_test.
• The goal is to fit linear and polynomial regression models and evaluate their performance.

🔹 pokemon_modified.csv (for classification tasks)

• Contains Pokémon attributes, with is_legendary as the binary target variable (0 or 1).
• Some features contain missing values and categorical variables, requiring preprocessing.

---

🚀 How to Approach the Assignment

1. Start with Regression (Task 1)

   • Implement linear regression and polynomial regression.
   • Use GridSearchCV for polynomial regression to find the best degree.
   • Evaluate using MSE, RMSE, MAE, and R² Score.

2. Move to Data Preprocessing (Task 2.1)

   • Load and clean the Pokémon dataset.
   • Handle missing values correctly.
   • Encode categorical variables properly.
   • Ensure no data leakage when doing the preprocessing.

3. Train and Evaluate Classification Models (Task 2.2)

   • Train Logistic Regression, KNN, and Naive Bayes.
   • Use GridSearchCV for hyperparameter tuning.
   • Evaluate models using Accuracy, Precision, Recall, and F1-score.

---

📌 Grading & Evaluation

• Your notebook will be autograded, so ensure:
  • Your function names exactly match the given specifications.
  • Your output format matches the expected results.
• Partial credit will be given where applicable.

🔹 Need Help?

• If you have any questions, refer to the assignment markdown instructions in each task before asking for clarifications.
• You can post your question on this Google sheet

🚀 Good luck! Happy coding! 🎯

FAQ

1) Should we include the lines to import the libraries?

• Answer:
  It doesn't matter if you include extra import lines, as the grader will only call the specified functions.

2) Is it okay to submit my file with code outside of the functions?

• Answer:
  Yes, you can include additional code outside of the functions as long as the entire script runs correctly when converted to a .py file.

Important Clarification:

• The grader will first convert the Jupyter Notebook (.ipynb) into a Python file (.py) and then run it.
• Note: Please do not include any commands like !pip install numpy because they may break the conversion process and therefore the submission will not be graded.

Task 1: Linear and Polynomial Regression (30 Points)

Task 1.1 - Linear Regression (15 Points)

Instructions

1. Load the dataset from datasets/task1_data.csv.
2. Extract training and testing data from the following columns:
   • "X_train": Training feature values.
   • "y_train": Training target values.
   • "X_test": Testing feature values.
   • "y_test": Testing target values.
3. Train a linear regression model on X_train and y_train.
4. Use the trained model to predict y_test values.
5. Compute and return the following evaluation metrics as a dictionary:
   • Mean Squared Error (MSE)
   • Root Mean Squared Error (RMSE)
   • Mean Absolute Error (MAE)
   • R² Score
6. The function signature should match:

   def task1_linear_regression() -> Dict[str, float]:
   

Please do not use any other libraries except for the ones imported below.

In [161]:

# Standard Library Imports
import os
import importlib.util
import nbformat
from tempfile import NamedTemporaryFile
from typing import Tuple, Dict

# Third-Party Library Imports
import numpy as np
import pandas as pd

from nbconvert import PythonExporter

# Scikit-Learn Imports
from sklearn.preprocessing import MinMaxScaler, StandardScaler, PolynomialFeatures, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,
                             mean_squared_error, mean_absolute_error, r2_score)
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB

In [162]:

def task1_linear_regression() -> Dict[str, float]:
    """
    Performs linear regression on a predefined dataset and returns performance metrics.

    **Dataset Assumption:**
    - The dataset is located at `"datasets/task1_data.csv"`.
    - It should contain the following columns:
      - `"X_train"`: Training feature values (numerical).
      - `"y_train"`: Training target values.
      - `"X_test"`: Testing feature values (numerical).
      - `"y_test"`: Testing target values.

    **Process:**
    1. Load the dataset from `"datasets/task1_data.csv"`.
    2. Extract training and testing data.
    3. Train a linear regression model on `X_train, y_train`.
    4. Use the trained model to predict `y_test` values.
    5. Compute evaluation metrics: **MSE, RMSE, MAE, R² Score**.

    **Output (Dictionary with Regression Metrics):**
    ```python
    {
        "MSE": <Mean Squared Error>,
        "RMSE": <Root Mean Squared Error>,
        "MAE": <Mean Absolute Error>,
        "R2": <R² Score>
    }
    ```
    """
    data = pd.read_csv("datasets/task1_data.csv")
    X_train = data['X_train'].values.reshape(-1, 1)
    y_train = data['y_train'].values
    X_test = data['X_test'].values.reshape(-1, 1)
    y_test = data['y_test'].values
    linear_regressor = LinearRegression()
    linear_regressor.fit(X_train, y_train)
    y_pred = linear_regressor.predict(X_test)
    
    metrics = {'MSE': mean_squared_error(y_test, y_pred),
               'RMSE': mean_squared_error(y_test, y_pred) ** 0.5,
               'MAE': mean_absolute_error(y_test, y_pred),
               'R2': r2_score(y_test, y_pred)}
  
    return metrics

Task 1.2 - Polynomial Regression (15 Points)

Instructions

1. Load the dataset from datasets/task1_data.csv.
2. Extract training and testing data from the following columns:
   • "X_train": Training feature values.
   • "y_train": Training target values.
   • "X_test": Testing feature values.
   • "y_test": Testing target values.
3. Define a pipeline that includes:
   • Polynomial feature transformation (degree range: 2 to 10).
   • Linear regression model.
4. Use GridSearchCV with 8-fold cross-validation to determine the best polynomial degree.
5. Train the model with the best polynomial degree and evaluate it on the test set.
6. Compute and return the following results as a dictionary:
   • Best polynomial degree (best_degree)
   • Mean Squared Error (MSE)

Function Signature

def task1_polynomial_regression() -> Dict[str, float]:

In [163]:

def task1_polynomial_regression() -> Dict[str, float]:
    """
    Performs polynomial regression using GridSearchCV to find the best polynomial degree.


    **Process:**
    1. Load the dataset and extract `X_train, y_train, X_test, y_test`.
    2. Define a **pipeline** with polynomial feature transformation and linear regression.
    3. Use **GridSearchCV** (with 8-fold cross-validation) to determine the best polynomial degree (range: **2 to 10**).
    4. Train the best polynomial regression model and evaluate its performance.
    5. Compute and return:
       - **Best polynomial degree (`best_degree`)**
       - **Mean Squared Error (MSE)**

     **Expected Output:**
    ```
    {
        "best_degree": <Optimal Polynomial Degree>,
        "MSE": <Mean Squared Error>
    }
    ```
    """
    data = pd.read_csv("datasets/task1_data.csv")
    X_train = data['X_train'].values.reshape(-1, 1)
    y_train = data['y_train'].values
    X_test = data['X_test'].values.reshape(-1, 1)
    y_test = data['y_test'].values
    param_grid = {'poly__degree': list(range(2, 11))}
    pipeline = Pipeline([('poly', PolynomialFeatures()),('lr', LinearRegression())]) 
    grid_search = GridSearchCV(pipeline, param_grid, cv=8, scoring='neg_mean_squared_error')
    grid_search.fit(X_train, y_train)
    best_degree = grid_search.best_params_['poly__degree']
    best_model = grid_search.best_estimator_
    y_pred = best_model.predict(X_test)
    mse = mean_squared_error(y_test, y_pred)
    return {"best_degree" : best_degree, "MSE": mse}

Task 2: Classification with Data Preprocessing (70 Points)

Task 2.1 - Data Preprocessing (30 Points)

Instructions

1. Load the dataset from datasets/pokemon_modified.csv.
2. Look at the data and study the provided features
3. Remove the two redundant features
4. Handle missing values:
   • Use mean imputation for "height_m" and "weight_kg".
   • Use median imputation for "percentage_male".
5. Perform one-hot encoding for the categorical column "type1".
6. Ensure the target variable ("is_legendary") is present.
7. Split the data into training and testing sets (80%-20% split). Is it balanced?
8. Apply feature scaling using StandardScaler or MinMaxScaler.
9. Return the following:
   • X_train_scaled: Processed training features.
   • X_test_scaled: Processed testing features.
   • y_train: Training labels.
   • y_test: Testing labels.

Function Signature

def task2_preprocessing() -> Tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]:

In [164]:

def task2_preprocessing(x) -> Tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]:
   """
   Preprocesses the Pokémon dataset by handling missing values, encoding categorical data, 
   and applying feature scaling before returning train-test splits, ensuring class balance.
   **Dataset Assumption:**
   - The dataset is located at `"datasets/pokemon_modified.csv"`.
   **Process:**
   1. Load the dataset and remove redundant columns.
   2. Handle missing values:
      - Mean imputation for **"height_m"** and **"weight_kg"**.
      - Median imputation for **"percentage_male"**.
   3. Perform **one-hot encoding** on `"type1"`.
   4. Ensure **"is_legendary"** is present as the target variable.
   5. Split the dataset into **80% training, 20% testing** using **stratification** to maintain class balance.
   6. Apply feature scaling (**StandardScaler**).
   7. Return the preprocessed train-test splits.
   """
   data = pd.read_csv("datasets/pokemon_modified.csv")
   data = data.drop(['name', 'classification'], axis=1)
   numeric_imputer = SimpleImputer(strategy='mean')
   data['height_m'] = numeric_imputer.fit_transform(data[['height_m']])
   data['weight_kg'] = numeric_imputer.fit_transform(data[['weight_kg']])
   median_imputer = SimpleImputer(strategy='median')
   data['percentage_male'] = median_imputer.fit_transform(data[['percentage_male']])
   encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
   type1_encoded = encoder.fit_transform(data[['type1']])
   type_1df = pd.DataFrame(type1_encoded, columns=encoder.get_feature_names_out(['type1']))
   data = data.reset_index(drop=True)
   type_1df =  type_1df.reset_index(drop=True)
   data = pd.concat([data, type_1df], axis=1)
   data = data.drop("type1", axis=1)

   X = data.drop("is_legendary", axis=1)
   y = data["is_legendary"]

   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=x, stratify=y)
   scaler = StandardScaler()
   X_train_scaled = scaler.fit_transform(X_train)
   X_test_scaled = scaler.transform(X_test)


   return pd.DataFrame(X_train_scaled), pd.DataFrame(X_test_scaled), y_train, y_test

Task 2.2 - Model Comparison (40 Points)

Instructions

1. Train three classification models on the preprocessed dataset:
   • Logistic Regression
   • K-Nearest Neighbors (KNN)
   • Gaussian Naive Bayes (GNB)
2. Use GridSearchCV for hyperparameter tuning on:
   • Logistic Regression: Regularization strength (C) and penalty (l1, l2).
   • KNN: Number of neighbors (n_neighbors), weight function, and distance metric.
3. Train each model on the training set and evaluate on the test set.
4. Compute the following evaluation metrics:
   • Accuracy
   • Precision
   • Recall
   • F1 Score
5. Return a dictionary containing the evaluation metrics for each model.

Function Signature

def task2_model_comparison() -> Dict[str, Dict[str, float]]:

In [ ]:

def task2_model_comparison(x,y) -> Dict[str, Dict[str, float]]:
   """
   Trains and evaluates three classification models using GridSearchCV for hyperparameter tuning.
   **Dataset Assumption:**
   - The preprocessed dataset is obtained from `task2_preprocessing()`, which returns:
     - `X_train`: Training features (scaled)
     - `X_test`: Testing features (scaled)
     - `y_train`: Training labels
     - `y_test`: Testing labels
   **Process:**
   1. Load the preprocessed dataset from `task2_preprocessing()`.
   2. Train the following models:
      - **Logistic Regression** (Hyperparameters: `C`, `penalty`, `solver`).
      - **K-Nearest Neighbors (KNN)** (Hyperparameters: `n_neighbors`, `weights`, `metric`).
      - **Gaussian Naive Bayes** (No hyperparameter tuning required).
   3. Evaluate the models using the following metrics:
      - **Accuracy**
      - **Precision**
      - **Recall**
      - **F1 Score**
   4. Return a dictionary with model names as keys and evaluation metrics as values.
   **Expected Output:**
   ```python
   {
       "Logistic Regression": {"accuracy": <float>, "precision": <float>, "recall": <float>, "f1_score": <float>},
       "KNN": {"accuracy": <float>, "precision": <float>, "recall": <float>, "f1_score": <float>},
       "Naive Bayes": {"accuracy": <float>, "precision": <float>, "recall": <float>, "f1_score": <float>}
   }
   ```
   """
   X_train, X_test, y_train, y_test = task2_preprocessing(x)
   param_grid_lr = {'penalty': ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100],
                    'solver': ['liblinear']}
   grid_search_lr = GridSearchCV(LogisticRegression(random_state=y), param_grid_lr, cv=5, scoring='accuracy')
   grid_search_lr.fit(X_train, y_train)
   y_pred_lr = grid_search_lr.predict(X_test)
   accuracy_lr = accuracy_score(y_test, y_pred_lr)
   precision_lr = precision_score(y_test, y_pred_lr)
   recall_lr = recall_score(y_test, y_pred_lr)
   f1_lr = f1_score(y_test, y_pred_lr)
   param_grid_knn = {'n_neighbors': range(3, 15),
                     'weights': ['uniform', 'distance'],
                     'metric': ['euclidean', 'manhattan']}
   grid_search_knn = GridSearchCV(KNeighborsClassifier(), param_grid_knn, cv=5, scoring="accuracy")
   grid_search_knn.fit(X_train, y_train)

   y_pred_knn = grid_search_knn.predict(X_test)
   accuracy_knn = accuracy_score(y_test, y_pred_knn)
   precision_knn = precision_score(y_test, y_pred_knn)
   recall_knn = recall_score(y_test, y_pred_knn)
   f1_knn = f1_score(y_test, y_pred_knn)

   gnb = GaussianNB()
   gnb.fit(X_train, y_train)

   y_pred_nb = gnb.predict(X_test)
   accuracy_nb = accuracy_score(y_test, y_pred_nb)
   precision_nb = precision_score(y_test, y_pred_nb)
   recall_nb = recall_score(y_test, y_pred_nb)
   f1_nb = f1_score(y_test, y_pred_nb)

   results = {
             "Logistic Regression": {"accuracy": accuracy_lr, "precision": precision_lr, "recall": recall_lr, "f1_score": f1_lr},
       "KNN": {"accuracy": accuracy_knn, "precision": precision_knn, "recall": recall_knn, "f1_score": f1_knn},
       "Naive Bayes": {"accuracy": accuracy_nb, "precision": precision_nb, "recall": recall_nb, "f1_score": f1_nb}
   }
   
   return results

random_state_preprocessing = 41
random_state_model = 41 #Начинаем с одинаковых значений

found = False

while not found:
    random_state_model += 1
    for random_state_preprocessing in range(41, 141):

        results = task2_model_comparison(random_state_preprocessing, random_state_model)

        goal = True
        for model_name, metrics in results.items():
            for metric_name, metric_value in metrics.items():
                if metric_value <= 0.6:
                    goal = False
                    break
            if not goal:
                break
        if goal:
            found = True
            break

    if found:
        print(f"Найдены random_state_preprocessing = {random_state_preprocessing} и random_state_model = {random_state_model}, при которых все метрики больше 0.6:")
        print(results)
    else:
        print(f"Не удалось найти подходящий random_state_preprocessing для random_state_model = {random_state_model}, увеличиваем random_state_model.")

if not found:
    print("Не удалось найти подходящие random_state для выполнения условия.")

Не удалось найти подходящий random_state_preprocessing для random_state_model = 42, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 43, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 44, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 45, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 46, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 47, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 48, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 49, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 50, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 51, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 52, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 53, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 54, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 55, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 56, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 57, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 58, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 59, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 60, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 61, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 62, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 63, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 64, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 65, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 66, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 67, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 68, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 69, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 70, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 71, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 72, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 73, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 74, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 75, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 76, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 77, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 78, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 79, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 80, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 81, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 82, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 83, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 84, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 85, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 86, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 87, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 88, увеличиваем random_state_model.
Не удалось найти подходящий random_state_preprocessing для random_state_model = 89, увеличиваем random_state_model.