10_11_Transformer/MLTS_Exercise_10_Transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>MLTS Exercise 10 - Transformer Training</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your task is to train a Transformer network on the electic power consumption dataset. The goal is to forcast values for the next day based on the mesurement values from a fixed window over the data from previous days.\n",
    "\n",
    "The dataset can be downloaded from [Individual Household Electric Power Consumption](https://archive.ics.uci.edu/dataset/235/individual+household+electric+power+consumption)\n",
    "\n",
    "It contains \"Measurements of electric power consumption in one household with a one-minute sampling rate over a period of almost 4 years. Different electrical quantities and some sub-metering values are available.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dataset Reference**  \n",
    "Hebrail, G. & Berard, A. (2006). Individual Household Electric Power Consumption [Dataset]. UCI Machine Learning Repository. https://doi.org/10.24432/C58K54."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch.utils.data import DataLoader, TensorDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set random seed for reproducibility\n",
    "seed = 42\n",
    "np.random.seed(seed)\n",
    "torch.manual_seed(seed)\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.manual_seed_all(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing dataset\n",
    "path = '../08_09_RNN_LSTM/data/household_power_consumption.txt'\n",
    "\n",
    "Household_consumption = pd.read_csv(path, sep=';', low_memory=False)\n",
    "# Household_consumption"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess Data for training\n",
    "\n",
    "For our model, we just use the `Global_active_power` as a timeseries which we want to forcast. Therefore, we drop the other columns beforehand.\n",
    "\n",
    "Tasks:\n",
    "* Drop all columns except `Date`, `Time` and `Global_active_power`\n",
    "* Convert seperate date and time columns into datetime column\n",
    "* Convert numeric columns to correct type\n",
    "* Find and replace missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop unneeded columns\n",
    "Household_consumption.drop(columns=[\n",
    "    'Global_reactive_power', 'Voltage',\n",
    "    'Global_intensity', 'Sub_metering_1', 'Sub_metering_2', 'Sub_metering_3'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parsing date and time into a single datetime column\n",
    "Household_consumption['Datetime'] = pd.to_datetime(\n",
    "    Household_consumption['Date'] + ' ' + Household_consumption['Time'], \n",
    "    format='%d/%m/%Y %H:%M:%S',\n",
    "    errors='coerce'\n",
    ")\n",
    "# Drop date and time column\n",
    "Household_consumption.drop(columns=['Date', 'Time'], inplace=True)\n",
    "# Drop rows with missing datetime\n",
    "Household_consumption.dropna(subset=['Datetime'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert numerical columns to numeric type\n",
    "numeric_column = 'Global_active_power'\n",
    "Household_consumption[numeric_column] = pd.to_numeric(Household_consumption[numeric_column], errors='coerce')\n",
    "\n",
    "# Household_consumption.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find missing values\n",
    "missing_values = Household_consumption.isnull().sum()\n",
    "print(\"Missing values per column:\\n\", missing_values)\n",
    "\n",
    "# Fill missing values with median for simplicity\n",
    "Household_consumption[numeric_column] = Household_consumption[numeric_column].fillna(\n",
    "    Household_consumption[numeric_column].median()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the datasets for training\n",
    "\n",
    "We want to forcased the mean value of the next day given the previous mean values. Therefore we need to resample our data to only contain one value per day.  \n",
    "\n",
    "This will make training fast and it should therefore also work on your laptops CPU. Additionally, you can train the model on Google Colab or on in the CIP pool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resample data to dayly intervals, calculating the mean for each day\n",
    "\n",
    "Household_consumption_daily = (\n",
    "    Household_consumption\n",
    "    .resample('D', on='Datetime')  # Resample by day using the datetime column\n",
    "    .mean(numeric_only=True)       # Ensure only numeric columns are aggregated\n",
    ")\n",
    "\n",
    "# Calculate a 7-day rolling average\n",
    "# Household_consumption_daily['Global_active_power'] = Household_consumption_daily['Global_active_power'].rolling(window=7).mean()\n",
    "\n",
    "#check for missing values\n",
    "missing_values_daily = Household_consumption_daily.isnull().sum()\n",
    "print(\"Missing values per column:\\n\", missing_values_daily)\n",
    "# Drop rows with missing values after resampling, if necessary\n",
    "Household_consumption_daily.dropna(inplace=True)\n",
    "\n",
    "Household_consumption_daily.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daily_data_week = Household_consumption_daily['Global_active_power']\n",
    "\n",
    "# Plot daily trends and rolling average\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.plot(daily_data_week.index, daily_data_week, label=\"Daily Global Active Power\", color='blue', alpha=0.6)\n",
    "plt.title(\"Daily Global Active Power Consumption\", fontsize=16)\n",
    "plt.xlabel(\"Date\", fontsize=12)\n",
    "plt.ylabel(\"Energy Consumption (kW)\", fontsize=12)\n",
    "plt.grid(alpha=0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scale the data between 0 and 1\n",
    "\n",
    "Normalizing or scaling the data is an important step before using the data for training as high raw values can often cause exploding gradients during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "# Apply the scaler to the selected features\n",
    "Household_consumption_daily['Global_active_power_transformed'] = scaler.fit_transform(\n",
    "    Household_consumption_daily[['Global_active_power']]\n",
    ")\n",
    "\n",
    "# Print a preview of the normalized data\n",
    "Household_consumption_daily.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert Time Series to Supervised Format\n",
    "\n",
    "Time series data is sequential, but LSTMs require input-output pairs to learn patterns. By converting it into supervised format, we prepare the data so the model can learn from past observations to predict the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert Time Series to Supervised Learning Format\n",
    "def create_supervised_data(data: np.ndarray, lag: int) -> tuple[np.ndarray, np.ndarray]:\n",
    "    \"\"\"\n",
    "    Convert time series data into supervised learning format.\n",
    "    `lag` determines how many previous time steps are used.\n",
    "    \"\"\"\n",
    "    X, y = [], []\n",
    "    for i in range(lag, len(data)):\n",
    "        X.append(data[i - lag:i])  # Previous `lag` steps\n",
    "        y.append(data[i])       # Current step\n",
    "    return np.array(X), np.array(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Target column (e.g., 'Global_active_power')\n",
    "target_column = Household_consumption_daily['Global_active_power_transformed'].values\n",
    "\n",
    "# Use a lag of 30 (e.g., previous 30 time steps to predict the next step)\n",
    "lag = 30\n",
    "X, y = create_supervised_data(target_column, lag)\n",
    "\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Devide data into train and test set\n",
    "\n",
    "We do not split train and test set randomly but by year. This will give us a better estimate at how the model could perform over a longer period of time that has not been seen in the trainings set.\n",
    "\n",
    "For the test set, we will split away the last year of data, between '2010-01-01' and '2010-11-26'.  \n",
    "Therefore, the train set will be all the data between '2006-12-16' and '2010-01-01'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the cutoff for the test set (e.g., one year)\n",
    "test_start_date = '2010-01-01'  # Start of the test year\n",
    "test_end_date = '2010-11-26'    # End of the test year\n",
    "\n",
    "# Split data based on the dates\n",
    "train_data = Household_consumption_daily.loc[:test_start_date].iloc[:-1]  # Up to the start of the test year\n",
    "test_data = Household_consumption_daily.loc[test_start_date:test_end_date]  # The whole test year\n",
    "\n",
    "# Extract the target column for both sets\n",
    "train_target = train_data['Global_active_power_transformed'].values\n",
    "test_target = test_data['Global_active_power_transformed'].values\n",
    "\n",
    "# Create supervised learning format\n",
    "X_train, y_train = create_supervised_data(train_target, lag)\n",
    "X_test, y_test = create_supervised_data(test_target, lag)\n",
    "\n",
    "# Reshape for LSTM input\n",
    "X_train = X_train.reshape((X_train.shape[0], X_train.shape[1], 1))\n",
    "X_test = X_test.reshape((X_test.shape[0], X_test.shape[1], 1))\n",
    "\n",
    "# Print shapes for confirmation\n",
    "print(f\"X_train shape: {X_train.shape}\")\n",
    "print(f\"y_train shape: {y_train.shape}\")\n",
    "print(f\"X_test shape: {X_test.shape}\")\n",
    "print(f\"y_test shape: {y_test.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task: Setup Transformer Model\n",
    "\n",
    "Create your Transformer model here. You can either code it completely from scratch or use the already implemented models by pytorch.  \n",
    "\n",
    "Nevertheless, think about how your inputs and outputs will look like and how the data is processed through the dataset. How do you need to implement the Transformer model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Transformer(torch.nn.Module):\n",
    "\n",
    "    pass  # TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize Transformer model, the optimizer and the loss function. \n",
    "\n",
    "Loss function: MSELoss  \n",
    "Optimizer: Adam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters\n",
    "# TODO\n",
    "batch_size = 16\n",
    "lr = 0.001\n",
    "\n",
    "# Instantiate model\n",
    "model = None  # TODO\n",
    "\n",
    "# Define the loss function and optimizer\n",
    "loss_fn = torch.nn.MSELoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "\n",
    "# Print model summary\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for GPU availability\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Move model to device\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setup the dataloaders for the model\n",
    "\n",
    "Convert the train and test set to torch, create a TorchDataset and a DataLoader for both"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert NumPy arrays to PyTorch tensors\n",
    "X_train_tensor = torch.tensor(X_train, dtype=torch.float32)\n",
    "y_train_tensor = torch.tensor(y_train, dtype=torch.float32)\n",
    "\n",
    "X_test_tensor = torch.tensor(X_test, dtype=torch.float32)\n",
    "y_test_tensor = torch.tensor(y_test, dtype=torch.float32)\n",
    "\n",
    "# Print shapes to confirm\n",
    "print(f\"X_train_tensor shape: {X_train_tensor.shape}\")\n",
    "print(f\"y_train_tensor shape: {y_train_tensor.shape}\")\n",
    "print(f\"X_test_tensor shape: {X_test_tensor.shape}\")\n",
    "print(f\"y_test_tensor shape: {y_test_tensor.shape}\")\n",
    "\n",
    "# Create TensorDataset for train and test sets\n",
    "train_dataset = TensorDataset(X_train_tensor, y_train_tensor)\n",
    "test_dataset = TensorDataset(X_test_tensor, y_test_tensor)\n",
    "\n",
    "# Create DataLoader for batch processing\n",
    "train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training/Testing Loop\n",
    "\n",
    "Setup the training and test loop. Therefore, go through N epochs and in each epoch go through the data of your dataloader, pass the data to the model, calculate the loss and optimize the network. After each epoch, test the model on the test set by passing the data through the model and computing the loss. Save both test and train loss for later inspection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training parameters\n",
    "num_epochs = 35  # Number of epochs for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training loop\n",
    "# Lists to track loss values\n",
    "train_losses = []\n",
    "test_losses = []\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    model.train()  # Set the model to training mode\n",
    "    train_loss = 0.0\n",
    "\n",
    "    for X_batch, y_batch in train_loader:\n",
    "        X_batch, y_batch = X_batch.to(device), y_batch.to(device)  # Move to device\n",
    "        optimizer.zero_grad()\n",
    "        y_pred = model(X_batch)\n",
    "        loss = loss_fn(y_pred.squeeze(), y_batch)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        train_loss += loss.item()\n",
    "\n",
    "    train_loss /= len(train_loader)\n",
    "    train_losses.append(train_loss)  # Record train loss\n",
    "\n",
    "    # Validation step\n",
    "    model.eval()  # Set the model to evaluation mode\n",
    "    test_loss = 0.0\n",
    "    with torch.no_grad():\n",
    "        for X_batch, y_batch in test_loader:\n",
    "            X_batch, y_batch = X_batch.to(device), y_batch.to(device)  # Move to device\n",
    "            y_pred = model(X_batch)\n",
    "            loss = loss_fn(y_pred.squeeze(), y_batch)\n",
    "            test_loss += loss.item()\n",
    "\n",
    "    test_loss /= len(test_loader)\n",
    "    test_losses.append(test_loss)  # Record test loss\n",
    "\n",
    "    print(f\"Epoch {epoch+1}/{num_epochs}, Train Loss: {train_loss:.4f}, Test Loss: {test_loss:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display loss curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the loss curves\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(range(1, num_epochs + 1), train_losses, label=\"Train Loss\", marker='o')\n",
    "plt.plot(range(1, num_epochs + 1), test_losses, label=\"Test Loss\", marker='o')\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.title(\"Loss Curves During Training\")\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Evaluate your trained model by comparing the ground truth data against the predicted time series values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ground Truth vs. Predicted Time Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Switch model to evaluation mode\n",
    "model.eval()\n",
    "\n",
    "# Generate predictions\n",
    "with torch.no_grad():\n",
    "    # Pass test inputs through the model\n",
    "    y_pred_test = []\n",
    "\n",
    "    for X_batch, y_batch in test_loader:\n",
    "            X_batch, y_batch = X_batch.to(device), y_batch.to(device)  # Move to device\n",
    "            y_pred = model(X_batch).squeeze()\n",
    "            y_pred_test.append(y_pred.cpu())\n",
    "\n",
    "    y_pred_test = torch.concat(y_pred_test, dim=0).numpy()\n",
    "\n",
    "# rescale predictions\n",
    "rescaled_y_test = scaler.inverse_transform(y_test_tensor.numpy().reshape(-1, 1))\n",
    "rescaled_y_pred = scaler.inverse_transform(y_pred_test.reshape(-1, 1))\n",
    "\n",
    "# Plot Ground Truth vs Predictions\n",
    "plt.figure(figsize=(12, 5))\n",
    "plt.plot(rescaled_y_test, label=\"Ground Truth\", color='blue', alpha=0.7)\n",
    "plt.plot(rescaled_y_pred, label=\"Predicted\", color='red', linestyle='--', alpha=0.7)\n",
    "plt.title(\"Ground Truth vs Predicted Values\", fontsize=16)\n",
    "plt.xlabel(\"Time Steps\", fontsize=12)\n",
    "plt.ylabel(\"Global Active Power\", fontsize=12)\n",
    "plt.legend(fontsize=12)\n",
    "plt.grid(alpha=0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Long term forcasting\n",
    "\n",
    "Additionally, we want to test if our model can do long term forcasting and predict correct values based on its previous predictions. Of course, our model is not designed and trained to do that specifically, but it is a good test how errors propagate over time and maybe gives us more ideas on how we could improve the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function for iterative prediction\n",
    "def iterative_prediction(model, initial_input, num_steps):\n",
    "    \"\"\"\n",
    "    Iteratively predict the future time steps using the Transformer model.\n",
    "    \n",
    "    Parameters:\n",
    "    - model: Trained Transformer model.\n",
    "    - initial_input: The input sequence for the first prediction (shape: [1, timesteps, features]).\n",
    "    - num_steps: Number of future time steps to predict.\n",
    "    \n",
    "    Returns:\n",
    "    - predictions: List of predictions for the future time steps.\n",
    "    \"\"\"\n",
    "    model.eval()  # Set model to evaluation mode\n",
    "    input_sequence = initial_input.clone()  # Clone to avoid modifying the original input\n",
    "    predictions = []\n",
    "\n",
    "    for _ in range(num_steps):\n",
    "        with torch.no_grad():\n",
    "            # Predict the next step\n",
    "            next_step = model(input_sequence)\n",
    "            \n",
    "            # Append the prediction to the list\n",
    "            predictions.append(next_step.item())\n",
    "            \n",
    "            # Prepare the next input\n",
    "            next_input = torch.cat((input_sequence[:, 1:, :], next_step.unsqueeze(1)), dim=1)\n",
    "            input_sequence = next_input\n",
    "\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters for visualization\n",
    "initial_input = X_test_tensor[0].unsqueeze(0)  # Start with the first test sequence\n",
    "num_steps = 100  # Number of steps to predict iteratively\n",
    "\n",
    "# Get iterative predictions\n",
    "iterative_predictions = iterative_prediction(model, initial_input, num_steps)\n",
    "\n",
    "# Create a timeline for visualization\n",
    "timeline = list(range(len(y_test_tensor[:num_steps])))\n",
    "\n",
    "# Ground truth values for comparison\n",
    "ground_truth = y_test_tensor[:num_steps].numpy()\n",
    "\n",
    "# Plot the results\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.plot(list(range(-lag, 0, 1)), initial_input[0], label=\"Initial Input\", color='lightblue')\n",
    "plt.vlines(0, min(ground_truth), max(ground_truth), colors='black')\n",
    "plt.plot(timeline, ground_truth, label=\"Ground Truth\", color='blue')\n",
    "plt.plot(timeline, iterative_predictions, label=\"Iterative Predictions\", color='red')\n",
    "plt.title(\"Iterative Prediction vs. Ground Truth\", fontsize=16)\n",
    "plt.xlabel(\"Time Steps\", fontsize=12)\n",
    "plt.ylabel(\"Normalized Energy Consumption\", fontsize=12)\n",
    "plt.legend(fontsize=12)\n",
    "plt.grid(alpha=0.3)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions:\n",
    "\n",
    "* How well can the time series be predicted?\n",
    "* How could the training be improved / changed?\n",
    "* Can the data be modified to recieve better results?\n",
    "\n",
    "Feel free to test out more ideas as you like!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}