{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "FiloVlKib1r5" }, "source": [ "# 📌 Machine Learning Assignment 1 - Instructions & Guidelines\n", "\n", "### **📝 General Guidelines**\n", "Welcome to Machine Learning Assignment 1! This assignment will test your understanding of **regression and classification models**, including **data preprocessing, hyperparameter tuning, and model evaluation**.\n", "\n", "Follow the instructions carefully, and ensure your implementation is **correct, well-structured, and efficient**.\n", "\n", "🔹 **Submission Format:** \n", "- Your submission **must be a single Jupyter Notebook (.ipynb)** file. \n", "- **File Naming Convention:** \n", " - Use **your university email as the filename**, e.g., \n", " ```\n", " j.doe@innopolis.university.ipynb\n", " ```\n", " - **Do NOT modify this format**, or your submission may not be graded.\n", "\n", "🔹 **Assignment Breakdown:**\n", "| Task | Description | Points |\n", "|------|------------|--------|\n", "| **Task 1.1** | Linear Regression | 20 |\n", "| **Task 1.2** | Polynomial Regression | 20 |\n", "| **Task 2.1** | Data Preprocessing | 15 |\n", "| **Task 2.2** | Model Comparison | 45 |\n", "| **Total** | - | **100** |\n", "\n", "---\n", "\n", "### **📂 Dataset & Assumptions**\n", "The dataset files are stored in the `datasets/` folder. \n", "- **Regression Dataset:** `datasets/task1_data.csv`\n", "- **Classification Dataset:** `datasets/pokemon_modified.csv`\n", "\n", "Each dataset is structured as follows:\n", "\n", "🔹 **`task1_data.csv` (for regression tasks)** \n", "- Contains `X_train`, `y_train`, `X_test`, and `y_test`. \n", "- The goal is to fit **linear and polynomial regression models** and evaluate their performance. \n", "\n", "🔹 **`pokemon_modified.csv` (for classification tasks)** \n", "- Contains Pokémon attributes, with `is_legendary` as the **binary target variable (0 or 1)**. \n", "- Some features contain **missing values** and **categorical variables**, requiring preprocessing.\n", "\n", "---\n", "\n", "### **🚀 How to Approach the Assignment**\n", "1. **Start with Regression (Task 1)**\n", " - Implement **linear regression** and **polynomial regression**.\n", " - Use **GridSearchCV** for polynomial regression to find the best degree.\n", " - Evaluate using **MSE, RMSE, MAE, and R² Score**.\n", "\n", "2. **Move to Data Preprocessing (Task 2.1)**\n", " - Load and clean the Pokémon dataset.\n", " - Handle **missing values** correctly.\n", " - Encode categorical variables properly.\n", " - Ensure **no data leakage** when doing the preprocessing.\n", "\n", "3. **Train and Evaluate Classification Models (Task 2.2)**\n", " - Train **Logistic Regression, KNN, and Naive Bayes**.\n", " - Use **GridSearchCV** for hyperparameter tuning.\n", " - Evaluate models using **Accuracy, Precision, Recall, and F1-score**.\n", "\n", "---\n", "\n", "### **📌 Grading & Evaluation**\n", "- Your notebook will be **autograded**, so ensure:\n", " - Your function names **exactly match** the given specifications.\n", " - Your output format matches the expected results.\n", "- Partial credit will be given where applicable.\n", "\n", "🔹 **Need Help?** \n", "- If you have any questions, refer to the **assignment markdown instructions** in each task before asking for clarifications.\n", "- You can post your question on this [Google sheet](https://docs.google.com/spreadsheets/d/1oyrqXDjT2CeGYx12aZhZ-oDKcQQ-PCgT91wHPhTlBCY/edit?usp=sharing)\n", "\n", "🚀 **Good luck! Happy coding!** 🎯" ] }, { "cell_type": "markdown", "metadata": { "id": "1pVczyKvb1r_" }, "source": [ "### FAQ\n", "\n", "**1) Should we include the lines to import the libraries?**\n", "\n", "- **Answer:** \n", " It doesn't matter if you include extra import lines, as the grader will only call the specified functions.\n", "\n", "**2) Is it okay to submit my file with code outside of the functions?**\n", "\n", "- **Answer:** \n", " Yes, you can include additional code outside of the functions as long as the entire script runs correctly when converted to a `.py` file.\n", "\n", "**Important Clarification:**\n", "\n", "- The grader will first convert the Jupyter Notebook (.ipynb) into a Python file (.py) and then run it.\n", "- **Note:** Please do not include any commands like `!pip install numpy` because they may break the conversion process and therefore the submission will not be graded." ] }, { "cell_type": "markdown", "metadata": { "id": "N6r5oSCAb1sA" }, "source": [ "## Task 1: Linear and Polynomial Regression (30 Points)\n", "\n", "### Task 1.1 - Linear Regression (15 Points)\n", "#### **Instructions**\n", "1. Load the dataset from **`datasets/task1_data.csv`**.\n", "2. Extract training and testing data from the following columns:\n", " - `\"X_train\"`: Training feature values.\n", " - `\"y_train\"`: Training target values.\n", " - `\"X_test\"`: Testing feature values.\n", " - `\"y_test\"`: Testing target values.\n", "3. Train a **linear regression model** on `X_train` and `y_train`.\n", "4. Use the trained model to predict `y_test` values.\n", "5. Compute and return the following **evaluation metrics** as a dictionary:\n", " - **Mean Squared Error (MSE)**\n", " - **Root Mean Squared Error (RMSE)**\n", " - **Mean Absolute Error (MAE)**\n", " - **R² Score**\n", "6. The function signature should match:\n", " ```python\n", " def task1_linear_regression() -> Dict[str, float]:" ] }, { "cell_type": "markdown", "metadata": { "id": "okI6tauab1sB" }, "source": [ "Please do not use any other libraries except for the ones imported below." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "id": "HEpdM53cb1sB" }, "outputs": [], "source": [ "# Standard Library Imports\n", "import os\n", "import importlib.util\n", "import nbformat\n", "from tempfile import NamedTemporaryFile\n", "from typing import Tuple, Dict\n", "\n", "# Third-Party Library Imports\n", "import numpy as np\n", "import pandas as pd\n", "\n", "from nbconvert import PythonExporter\n", "\n", "# Scikit-Learn Imports\n", "from sklearn.preprocessing import MinMaxScaler, StandardScaler, PolynomialFeatures, OneHotEncoder\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,\n", " mean_squared_error, mean_absolute_error, r2_score)\n", "from sklearn.model_selection import train_test_split, GridSearchCV\n", "from sklearn.linear_model import LinearRegression, LogisticRegression\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.naive_bayes import GaussianNB\n" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "id": "arGVjHlib1sD" }, "outputs": [], "source": [ "def task1_linear_regression() -> Dict[str, float]:\n", " \"\"\"\n", " Performs linear regression on a predefined dataset and returns performance metrics.\n", "\n", " **Dataset Assumption:**\n", " - The dataset is located at `\"datasets/task1_data.csv\"`.\n", " - It should contain the following columns:\n", " - `\"X_train\"`: Training feature values (numerical).\n", " - `\"y_train\"`: Training target values.\n", " - `\"X_test\"`: Testing feature values (numerical).\n", " - `\"y_test\"`: Testing target values.\n", "\n", " **Process:**\n", " 1. Load the dataset from `\"datasets/task1_data.csv\"`.\n", " 2. Extract training and testing data.\n", " 3. Train a linear regression model on `X_train, y_train`.\n", " 4. Use the trained model to predict `y_test` values.\n", " 5. Compute evaluation metrics: **MSE, RMSE, MAE, R² Score**.\n", "\n", " **Output (Dictionary with Regression Metrics):**\n", " ```python\n", " {\n", " \"MSE\": ,\n", " \"RMSE\": ,\n", " \"MAE\": ,\n", " \"R2\": \n", " }\n", " ```\n", " \"\"\"\n", " df = pd.read_csv('datasets/task1_data.csv')\n", " X_train, y_train, X_test, y_test = df['X_train'].to_numpy().reshape((-1, 1)), df['y_train'], df['X_test'].to_numpy().reshape((-1, 1)), df['y_test']\n", " model = LinearRegression()\n", " model.fit(X_train, y_train)\n", " y_pred = model.predict(X_test)\n", " return {\n", " 'MSE': mean_squared_error(y_test, y_pred),\n", " 'RMSE': mean_squared_error(y_test, y_pred) ** 0.5,\n", " 'MAE': mean_absolute_error(y_test, y_pred),\n", " 'R2': r2_score(y_test, y_pred),\n", " }" ] }, { "cell_type": "markdown", "metadata": { "id": "AjCZsK2Sb1sD" }, "source": [ "### Task 1.2 - Polynomial Regression (15 Points)\n", "\n", "#### **Instructions**\n", "1. Load the dataset from **`datasets/task1_data.csv`**.\n", "2. Extract training and testing data from the following columns:\n", " - `\"X_train\"`: Training feature values.\n", " - `\"y_train\"`: Training target values.\n", " - `\"X_test\"`: Testing feature values.\n", " - `\"y_test\"`: Testing target values.\n", "3. Define a **pipeline** that includes:\n", " - **Polynomial feature transformation** (degree range: **2 to 10**).\n", " - **Linear regression model**.\n", "4. Use **GridSearchCV** with **8-fold cross-validation** to determine the best polynomial degree.\n", "5. Train the model with the best polynomial degree and **evaluate it on the test set**.\n", "6. Compute and return the following results as a dictionary:\n", " - **Best polynomial degree** (`best_degree`)\n", " - **Mean Squared Error (MSE)**\n", "\n", "#### **Function Signature**\n", "```python\n", "def task1_polynomial_regression() -> Dict[str, float]:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "id": "AhGlefpqb1sE" }, "outputs": [], "source": [ "def task1_polynomial_regression() -> Dict[str, float]:\n", " \"\"\"\n", " Performs polynomial regression using GridSearchCV to find the best polynomial degree.\n", "\n", "\n", " **Process:**\n", " 1. Load the dataset and extract `X_train, y_train, X_test, y_test`.\n", " 2. Define a **pipeline** with polynomial feature transformation and linear regression.\n", " 3. Use **GridSearchCV** (with 8-fold cross-validation) to determine the best polynomial degree (range: **2 to 10**).\n", " 4. Train the best polynomial regression model and evaluate its performance.\n", " 5. Compute and return:\n", " - **Best polynomial degree (`best_degree`)**\n", " - **Mean Squared Error (MSE)**\n", "\n", " **Expected Output:**\n", " ```\n", " {\n", " \"best_degree\": ,\n", " \"MSE\": \n", " }\n", " ```\n", " \"\"\"\n", " df = pd.read_csv('datasets/task1_data.csv')\n", " X_train, y_train, X_test, y_test = df['X_train'].to_numpy().reshape((-1, 1)), df['y_train'], df['X_test'].to_numpy().reshape((-1, 1)), df['y_test']\n", " pipeline = Pipeline([\n", " ('polynomial', PolynomialFeatures()),\n", " ('linear', LinearRegression())\n", " ])\n", " grid_search = GridSearchCV(pipeline, {\n", " 'polynomial__degree': range(2, 11)\n", " }, cv=8)\n", " grid_search.fit(X_train, y_train)\n", "\n", "\n", " return {\n", " 'best_degree': grid_search.best_params_['polynomial__degree'],\n", " 'MSE': mean_squared_error(y_test, grid_search.best_estimator_.predict(X_test)),\n", " }" ] }, { "cell_type": "markdown", "metadata": { "id": "7awwgLy3b1sE" }, "source": [ "## Task 2: Classification with Data Preprocessing (70 Points)\n", "\n", "### Task 2.1 - Data Preprocessing (30 Points)\n", "\n", "#### **Instructions**\n", "1. Load the dataset from **`datasets/pokemon_modified.csv`**.\n", "2. Look at the data and study the provided features\n", "3. Remove the **two redundant features**\n", "4. Handle **missing values**:\n", " - Use **mean imputation** for **\"height_m\"** and **\"weight_kg\"**.\n", " - Use **median imputation** for **\"percentage_male\"**.\n", "5. Perform **one-hot encoding** for the categorical column **\"type1\"**.\n", "6. Ensure the **target variable** (`\"is_legendary\"`) is present.\n", "7. **Split the data into training and testing sets** (`80%-20%` split). Is it balanced?\n", "8. **Apply feature scaling** using **StandardScaler** or **MinMaxScaler**.\n", "9. Return the following:\n", " - `X_train_scaled`: Processed training features.\n", " - `X_test_scaled`: Processed testing features.\n", " - `y_train`: Training labels.\n", " - `y_test`: Testing labels.\n", "\n", "#### **Function Signature**\n", "```python\n", "def task2_preprocessing() -> Tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "id": "9nrjr1U8b1sE" }, "outputs": [], "source": [ "def task2_preprocessing() -> Tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]:\n", " \"\"\"\n", " Preprocesses the Pokémon dataset by handling missing values, encoding categorical data,\n", " and applying feature scaling before returning train-test splits, ensuring class balance.\n", "\n", " **Dataset Assumption:**\n", " - The dataset is located at `\"datasets/pokemon_modified.csv\"`.\n", "\n", " **Process:**\n", " 1. Load the dataset and remove redundant columns.\n", " 2. Handle missing values:\n", " - Mean imputation for **\"height_m\"** and **\"weight_kg\"**.\n", " - Median imputation for **\"percentage_male\"**.\n", " 3. Perform **one-hot encoding** on `\"type1\"`.\n", " 4. Ensure **\"is_legendary\"** is present as the target variable.\n", " 5. Split the dataset into **80% training, 20% testing** using **stratification** to maintain class balance.\n", " 6. Apply feature scaling (**StandardScaler**).\n", " 7. Return the preprocessed train-test splits.\n", " \"\"\"\n", " df2 = pd.read_csv('datasets/pokemon_modified.csv')\n", " y = df2.is_legendary\n", " df2 = df2.drop(['is_legendary', 'name', 'classification'], axis=1)\n", "\n", " mean_imputation = SimpleImputer(strategy='mean')\n", " mean_imputation.fit(df2[['height_m', 'weight_kg']])\n", " df2[['height_m', 'weight_kg']] = mean_imputation.transform(df2[['height_m', 'weight_kg']])\n", "\n", " median_imputation = SimpleImputer(strategy='median')\n", " median_imputation.fit(df2[['percentage_male']])\n", " df2[['percentage_male']] = median_imputation.transform(df2[['percentage_male']])\n", "\n", " ohe = OneHotEncoder(sparse_output=False, drop='first')\n", " ohe.fit(df2[['type1']])\n", " df2 = np.concatenate([df2.drop('type1', axis=1), ohe.transform(df2[['type1']])], axis=1)\n", " X_train, X_test, y_train, y_test = train_test_split(df2, y, test_size=0.2, stratify=y)\n", "\n", " scaler = StandardScaler()\n", " X_train = scaler.fit_transform(X_train)\n", " X_test = scaler.transform(X_test)\n", "\n", "\n", " return pd.DataFrame(X_train), pd.DataFrame(X_test), y_train, y_test" ] }, { "cell_type": "markdown", "metadata": { "id": "YwaFciWWb1sE" }, "source": [ "### Task 2.2 - Model Comparison (40 Points)\n", "\n", "#### **Instructions**\n", "1. **Train three classification models** on the preprocessed dataset:\n", " - **Logistic Regression**\n", " - **K-Nearest Neighbors (KNN)**\n", " - **Gaussian Naive Bayes (GNB)**\n", "2. Use **GridSearchCV** for **hyperparameter tuning** on:\n", " - **Logistic Regression**: Regularization strength (`C`) and penalty (`l1`, `l2`).\n", " - **KNN**: Number of neighbors (`n_neighbors`), weight function, and distance metric.\n", "3. Train each model on the **training set** and evaluate on the **test set**.\n", "4. Compute the following **evaluation metrics**:\n", " - **Accuracy**\n", " - **Precision**\n", " - **Recall**\n", " - **F1 Score**\n", "5. Return a dictionary containing the evaluation metrics for each model.\n", "\n", "#### **Function Signature**\n", "```python\n", "def task2_model_comparison() -> Dict[str, Dict[str, float]]:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "id": "U_BFi3mJb1sF" }, "outputs": [ { "data": { "text/plain": [ "{'Logistic Regression': {'accuracy': 0.9937888198757764,\n", " 'precision': 1.0,\n", " 'recall': 0.9285714285714286,\n", " 'f1_score': 0.9629629629629629},\n", " 'KNN': {'accuracy': 0.9813664596273292,\n", " 'precision': 0.9230769230769231,\n", " 'recall': 0.8571428571428571,\n", " 'f1_score': 0.8888888888888888},\n", " 'Naive Bayes': {'accuracy': 0.8385093167701864,\n", " 'precision': 0.34210526315789475,\n", " 'recall': 0.9285714285714286,\n", " 'f1_score': 0.5}}" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def task2_model_comparison() -> Dict[str, Dict[str, float]]:\n", " \"\"\"\n", " Trains and evaluates three classification models using GridSearchCV for hyperparameter tuning.\n", "\n", " **Dataset Assumption:**\n", " - The preprocessed dataset is obtained from `task2_preprocessing()`, which returns:\n", " - `X_train`: Training features (scaled)\n", " - `X_test`: Testing features (scaled)\n", " - `y_train`: Training labels\n", " - `y_test`: Testing labels\n", "\n", " **Process:**\n", " 1. Load the preprocessed dataset from `task2_preprocessing()`.\n", " 2. Train the following models:\n", " - **Logistic Regression** (Hyperparameters: `C`, `penalty`, `solver`).\n", " - **K-Nearest Neighbors (KNN)** (Hyperparameters: `n_neighbors`, `weights`, `metric`).\n", " - **Gaussian Naive Bayes** (No hyperparameter tuning required).\n", " 3. Evaluate the models using the following metrics:\n", " - **Accuracy**\n", " - **Precision**\n", " - **Recall**\n", " - **F1 Score**\n", " 4. Return a dictionary with model names as keys and evaluation metrics as values.\n", "\n", " **Expected Output:**\n", " ```python\n", " {\n", " \"Logistic Regression\": {\"accuracy\": , \"precision\": , \"recall\": , \"f1_score\": },\n", " \"KNN\": {\"accuracy\": , \"precision\": , \"recall\": , \"f1_score\": },\n", " \"Naive Bayes\": {\"accuracy\": , \"precision\": , \"recall\": , \"f1_score\": }\n", " }\n", " ```\n", " \"\"\"\n", " X_train, X_test, y_train, y_test = task2_preprocessing()\n", "\n", " log_reg = GridSearchCV(LogisticRegression(), {\n", " 'C': [10 ** i for i in range(-10, 11)],\n", " 'penalty': ['l1', 'l2'],\n", " 'solver': ['liblinear'],\n", " # 'solver': ['lbfgs', 'liblinear', 'newton-cg', 'newton-cholesky', 'sag', 'saga']\n", " })\n", " log_reg.fit(X_train, y_train)\n", "\n", " knn = GridSearchCV(KNeighborsClassifier(), {\n", " 'n_neighbors': range(1, 15),\n", " 'weights': ['uniform', 'distance'],\n", " 'metric': ['minkowski', 'euclidean', 'manhattan', 'cosine', 'chebyshev']\n", " })\n", " knn.fit(X_train, y_train)\n", "\n", " gnb = GaussianNB()\n", " gnb.fit(X_train, y_train)\n", "\n", " return {\n", " \"Logistic Regression\": {\"accuracy\": accuracy_score(y_test, log_reg.predict(X_test)), \"precision\": precision_score(y_test, log_reg.predict(X_test)), \"recall\": recall_score(y_test, log_reg.predict(X_test)), \"f1_score\": f1_score(y_test, log_reg.predict(X_test))},\n", " \"KNN\": {\"accuracy\": accuracy_score(y_test, knn.predict(X_test)), \"precision\": precision_score(y_test, knn.predict(X_test)), \"recall\": recall_score(y_test, knn.predict(X_test)), \"f1_score\": f1_score(y_test, knn.predict(X_test))},\n", " \"Naive Bayes\": {\"accuracy\": accuracy_score(y_test, gnb.predict(X_test)), \"precision\": precision_score(y_test, gnb.predict(X_test)), \"recall\": recall_score(y_test, gnb.predict(X_test)), \"f1_score\": f1_score(y_test, gnb.predict(X_test))}\n", " }\n", "task2_model_comparison()" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 0 }