DDPG_Collaboration_and_Competition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Udacity Deep RL NanoDegree - Project 2 - Continuous Control\n",
    "---\n",
    "In this notebook, we will implement a Deep Deterministec Policy Gradient (DDPG) agent to solve the Unity ML-Agents environment about two agents controlling rackets to bounce a ball over a net, somewhat like playing Ping-Pong.\n",
    "\n",
    "### 1. Import the Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "import random\n",
    "import torch\n",
    "from collections import deque\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Instantiate the Environment and Agent\n",
    "\n",
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Tennis.app\"`\n",
    "- **Windows** (x86): `\"path/to/Tennis_Windows_x86/Tennis.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Tennis_Windows_x86_64/Tennis.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Tennis_Linux/Tennis.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Tennis_Linux/Tennis.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Tennis.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Tennis.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='./Tennis_Linux/Tennis.x86_64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, two agents control rackets to bounce a ball over a net. If an agent hits the ball over the net, it receives a reward of **`+0.1`**.  If an agent lets a ball hit the ground or hits the ball out of bounds, it receives a reward of **`-0.01`**.  Thus, the goal of each agent is to keep the ball in play.\n",
    "\n",
    "The observation space consists of **`8`** variables corresponding to the position and velocity of the ball and racket, the environment returns 3 stacked observation spaces at each timestep, son the returned variable has dimension **`24`**. Two continuous actions are available, corresponding to movement toward (or away from) the net, and jumping. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Size of each action: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.         -6.65278625 -1.5\n",
      " -0.          0.          6.83172083  6.         -0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents \n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, Instantiate the agent and load the Neural Network to memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the DDPG agent\n",
    "from ddpg_agent import Agent\n",
    "\n",
    "# Instantiate the Agent\n",
    "agent = Agent(state_size=24, action_size=2, random_seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Select which racket to print information about. 0: Right Racket, 1: Left Racket.\n",
    "racket = 0\n",
    "# The observation space has 3 stacks of 8x1 vectors of observation. Choose which of them you want to print.\n",
    "stack = 0\n",
    "\n",
    "# Run 10 episodes and choose random actions\n",
    "for i in range(50):                                      # play game for 5 episodes\n",
    "    env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "    states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "    scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "    while True:\n",
    "        actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "        actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "        env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "        next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "        rewards = env_info.rewards                         # get reward (for each agent)\n",
    "        dones = env_info.local_done                        # see if episode finished\n",
    "        scores += env_info.rewards                         # update the score (for each agent)\n",
    "        states = next_states                               # roll over states to next time step\n",
    "        if np.any(dones):                                  # exit loop if episode finished\n",
    "            break\n",
    "            \n",
    "        # Print the observation space as (Racket X position, Racket Y position, Racket X velocity, Racket Y velocity, Ball X position, Ball Y position)\n",
    "        print('\\rx:{:5.3}\\ty:{:5.3}\\tvx:{:5.3}\\tvy:{:5.3}\\tbx:{:5.3}\\tby:{:5.3}               '.format(states[racket][stack*8 + 0], states[racket][stack*8 + 1], states[racket][stack*8 + 2], states[racket][stack*8 + 3], states[racket][stack*8 + 4], states[racket][stack*8 + 5]), end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Train the Agent with DDPG\n",
    "\n",
    "Run the code cell below to train the agent from scratch.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/said/.anaconda3/envs/drlnd/lib/python3.6/site-packages/torch/nn/functional.py:1340: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 100 (0.21s)\tScore: 0.00\tMoving Average Score: 0.00       \n",
      "Episode: 200 (0.22s)\tScore: 0.00\tMoving Average Score: 0.00       \n",
      "Episode: 300 (0.22s)\tScore: 0.00\tMoving Average Score: 0.00       \n",
      "Episode: 400 (0.24s)\tScore: 0.00\tMoving Average Score: 0.01       \n",
      "Episode: 500 (0.54s)\tScore: 0.00\tMoving Average Score: 0.02       \n",
      "Episode: 600 (0.82s)\tScore: 0.10\tMoving Average Score: 0.04       \n",
      "Episode: 700 (0.47s)\tScore: 0.09\tMoving Average Score: 0.08       \n",
      "Episode: 800 (1.52s)\tScore: 0.20\tMoving Average Score: 0.15        \n",
      "Episode: 900 (0.27s)\tScore: 0.00\tMoving Average Score: 0.18        \n",
      "Episode: 954 (10.53s)\tScore: 1.69\tMoving Average Score: 0.51       \n",
      "Environment solved in 954 episodes! Max Average Score over 100 episodes: 0.51\n",
      "Episode: 1000 (5.41s)\tScore: 0.80\tMoving Average Score: 1.01       \n",
      "\n",
      "\n",
      "Training finished:\n",
      "\t[+] Total nuber of episodes:    1000\n",
      "\t[+] Best 100-average score:     1.01\n",
      "\t[+] Best individual game score: 2.70\n",
      "Task solved in 954 Episodes!\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXwV1d348c/3LtlXQBAIEFBEESK7UBFTUVGrKNaKS9XqY7G1fdQ+bW31qQu2/T3WWmttq9aqVSsFUdxXVIyICwqKgCwCEiQQtpCE3Kx3Ob8/ZnK5ubnZbnKz3e/79cord2bOzJxzJ5nvzDlnzogxBqWUUvHL0dUZUEop1bU0ECilVJzTQKCUUnFOA4FSSsU5DQRKKRXnNBAopVSc00Cg2kREXheRKzs6rYpPIlIoIqd1dT7inaurM6BiT0Q8IZMpQC3gt6evNcYsaO22jDFnxSKtUs0RkXzgKWNMTlfnpTfSQBAHjDFp9Z9FpBC4xhjzdng6EXEZY3ydmbfOICJOY4y/5ZSt2lav/I5CxUMZVUNaNRTHRCRfRIpE5Fcisgf4l4hki8grIrJfRErtzzkh6xSIyDX25x+IyAoRucdOu11Ezooy7XARWS4iFSLytoj8XUSeaiHft4jIAbt64bKQ5Y+LyIMi8pqIVALfFpHj7PyUiciXIjI7JH1fEXlZRA6JyKci8jsRWRGy3IjIT0RkC7DFnneOiKyxt/ehiOSFpP+ViOyyy7JZRGba86eIyCp7P3tF5N4oj5uIyJ9FZJ+9rXUiMsZeliwifxKRHSJSbn/nyfay2XbZy+zv4riQbRba+V4LVIqIS0Sm2mUrE5Ev7Kvy+vQ/EJGv7TJuD/3+w/J6h4g8KyJP22k/E5ETmkibKCL3ichu++c+e14q8DowSEQ89s+gaL471QRjjP7E0Q9QCJxmf84HfMAfgEQgGegLfBerCikdeAZ4IWT9Aqw7CoAfAF7gh4AT+DGwG5Ao0n4E3AMkANOBQ1hVAZHKUJ/ve+18nwJUAqPs5Y8D5cBJWBc76cBW4BZ7+6cCFSHpF9k/KcBoYCewImR/BngL6GN/R+OBfcCJdlmutL/XRGCUvf4ge91c4KiQMl5uf04DpkZ5DGcBq4EsQIDjgIH2sr/b3/tgO2/fsvN1jP0dnQ64gZvs7yQh5O9iDTDELuNgoAQ42/4OT7enjwBS7eNT//0NBI5vIq932Mf9Qnu/vwC2A+4If493Ah8D/e39fAj8NuSYF3X1/09v/enyDOhPJx/wxoGgDkhqJv04oDRkuoCGJ/etIctS7JPmkW1JCwzFOrGnhCx/ipYDQWrIvMXArfbnx4EnQ5adDOwBHCHzFtonKad9ohoVsux3NA4Ep4ZMP1h/ggqZtxkrIB2NFSROqz/ZhaRZDswH+rXzGJ4KfAVMDSuTA6gGToiwzq3A4rC0u4D8kL+Lq0OW/wr4d9g23sQKeqlAGdYFQ3ILeb0D+Dhsv8XAyRH+HrcBZ4eknQUUhhxzDQQx+tGqIbXfGFNTPyEiKSLyD7tq4RDWyStLRJxNrL+n/oMxpsr+mNbGtIOAgyHzwLqqbk6pMaYyZHqHvZ1I6w8CdhpjAmHpB2NdebrC0kfad+i8YcDP7SqTMhEpw7qSHmSM2QrciHUC3Ccii0KqMf4L68p8k10FdU6kgonV26q+CqRRlYsxZhnwN6yr/30i8rCIZAD9gCSsE2q4QXaZ67cRsMs0uJkyfi+sjNOx7jwqgbnAj4BiEXlVRI6NVJbw7dr7LaLhsYqYRxofUxUjGghU+PCzP8eq3jjRGJMBzLDnSwzzUAz0EZGUkHlDWlgn2647rjcUq6qpXmi5dgNDRMQRln4XsB/r7iK0N0qkfYdubyfwe2NMVshPijFmIYAx5j/GmOlYJ1ODVfWGMWaLMeYSrKqPPwDPhpUBO91Zxpg0+ydijy5jzP3GmIlYVVnHAL8EDgA1wFERVtlt5wew2hnscu5qpoz/DitjqjHmLnv/bxpjTseqFtoE/DNSPm3B79M+Bjk0PFYR80jDY6rDJMeQBgIVLh2reqFMRPoAt8d6h8aYHcAq4A4RSRCRacC5rVh1vp3+ZOAcrPaMSFYCVcBNIuK2Gz3PBRYZqzfRc/a+U+wr2yta2O8/gR+JyIl2w22qiHxHRNJFZJSInCoiiVgn5WogACAi3xeRI+yr4jJ7W4HIu2iaiEy29+3GqvevAQL2dh8D7hWRQSLiFJFpdl4WA98RkZn2ej/H6kb8YRO7eQo4V0Rm2dtJEquRPkdEBojIeXYQqwU8LZRjoohcICIurLulWqy2gHALgd+IyBEi0g+4zc4HwF6gr4hktvqLUq2mgUCFuw+rsfAA1j/rG52038uAaVgNkr8DnsY6YTRlD1CKdcW4APiRMWZTpITGmDqsE/9ZWOV6ALgiJP1PgUx7m//GOiE1uW9jzCqsRu+/2XnYitUGAlbD7F32fvZgXf3fbC87E/hSrOc6/gJcbIypbqaMTcnACkalWNUnJcAf7WW/ANYBnwIHse48HMaYzcD3gb/aeTsXONf+biKVcSdwHlYD+36sO4RfYp0zHMD/YH33B7HaRn7cTH5fxKpKKgUuBy4wxngjpPsd1gXBWrsMn9nzsI/VQuBru6pKq4w6UH2PDaW6FRF5GthkjGl0RyIxfrhIRP6A1eCtT0W3k4jcARxtjPl+V+dFNU3vCFS3YFd3HCUiDhE5E+tq9IVO2vexIpJnV/NMwWrUfb4z9q1Ud6BPFqvu4kisuvq+WL1KfmyM+byT9p2OVe0wCKsu+k9Y1RlKxQWtGlJKqTinVUNKKRXnelzVUL9+/Uxubm5U61ZWVpKa2qjbdq+mZY4PWub40J4yr169+oAx5ohIy3pcIMjNzWXVqlVRrVtQUEB+fn7HZqib0zLHBy1zfGhPmUVkR1PLtGpIKaXinAYCpZSKcxoIlFIqzvW4NoJIvF4vRUVF1NTUNJsuMzOTjRs3dlKuugct82FJSUnk5OTgdru7IFdKdV+9IhAUFRWRnp5Obm4u1qCKkVVUVJCent6JOet6WmaLMYaSkhKKiooYPnx4F+VMqe6pV1QN1dTU0Ldv32aDgIpvIkLfvn1bvGtUKh71ikAAaBBQLdK/EaUi6zWBQCmluto3e33sLfW3aZ2augDrt1ujge8r9fPNXl8sstYsDQQdxOl0Mm7cuODPXXfd1Wz6hx56iCeffLLd+83NzeXAgQOtTv/KK68wfvx4TjjhBEaPHs0//vGPdudBKWX51xsVPPTSoTat8+x7VSxZXkmZx8+DLx3iX29UxCh3TesVjcXdQXJyMmvWrGl1+h/96EcxzE1kXq+XefPm8cknn5CTk0NtbS2FhYXt2mbw5dcOvaZQKhr7y9t2BxEL+t8bY7m5udx0002MHTuWKVOmsHXrVgDuuOMO7rnnHgDuv/9+Ro8eTV5eHhdffDEABw8e5PzzzycvL4+pU6eydu1aAEpKSjjjjDM4/vjjueaaawgdPfapp55iypQpjBs3jmuvvRa/v+EfWEVFBT6fj759+wKQmJjIqFGjANi7dy9z5szhhBNO4IQTTuDDD603GN57772MGTOGMWPGcN999wFQWFjIqFGjuOKKKxgzZgw7d+7kj3/8I5MnTyYvL4/bb4/52y2V6jW8dk2Qy9l1bVi97o7gjU+qmqyjq6mpISmp7dsckO3kzCkpzaaprq5m3Lhxwembb76ZuXPnAla/9nXr1vHkk09y44038sorrzRY96677mL79u0kJiZSVma9yvb2229n/PjxvPDCCyxbtowrrriCNWvWMH/+fKZPn85tt93Gq6++yqOPPgrAxo0befrpp/nggw9wu91cd911LFiwgDlz5gT306dPH2bPns2wYcOYOXMm55xzDpdccgkOh4Prr7+eU045heeffx6/34/H42H16tX861//YuXKlRhjOPHEEznllFPIzs5my5YtPPHEE0ydOpWlS5eyZcsWPvnkE4wxzJ49m+XLlzNjxgyUUs3z+rr+VQC9LhB0leaqhi655JLg75/97GeNlufl5XHZZZdx/vnnc/755wOwYsUKlixZAsCpp55KSUkJhw4dYvny5Tz33HMAfOc73yE7OxuAd955h9WrVzN58mTACkz9+/dvtK9HHnmEdevW8fbbb3PPPffw1ltv8fjjj7Ns2bJgm4XT6SQzM5MVK1YwZ86c4GiHF1xwAe+//34wmEydOhWApUuXsnTpUsaPHw+Ax+Nhy5YtGghUXIn23S4+vwaCDtfclXtFBV3ycFVot8VIXRhfffVVli9fzssvv8zvf/971q1b1+Z9GGO48sor+b//+78G8ysqGjc8jR07lrFjx3L55ZczfPhwHn/88TbvL3QoXGMMN998M9dee22bt6NUb+Hr+qr+qGkbQSd4+umng7+nTZvWYFkgEGDnzp18+9vf5g9/+APl5eV4PB5OPvlkFixYAFhDz/br14+MjAxmzJjBf/7zHwBef/11SktLAZg5cybPPvss+/btA6w2hh07Go466/F4KCgoCE6vWbOGYcOGBdd/8MEHAfD7/ZSXl3PyySfzwgsvUFVVRWVlJc8//zwnn3xyo/LNmjWLxx57DI/HA8CuXbuC+VCqPaprA12dhVarqjl8ZV9dG8DrM3h9hto6g9++6q+tM3iqA8E0tXWH16nzHv5cW2c69U6h190RdJXwNoIzzzwz2IW0tLSUvLw8EhMTWbhwYYP1/H4/3//+9ykvL8cYw/XXX09WVhZ33HEHV199NXl5eaSkpPDEE08AVtvBJZdcwvHHH8+3vvUthg4dCsDo0aP53e9+xxlnnEEgEMDtdvP3v/+dPn36BPdljOHuu+/m2muvJTk5mdTU1ODdwF/+8hfmzZvHo48+itPp5MEHH2TatGn84Ac/YMqUKQBcc801jB8/vlFPozPOOIONGzcGg1xaWhpPPfVUxKoppVpr/fY6liyv5JrvpDO4X/c/Vd23pDz4+e5F1ucEt1DnNRyT4+aSmWnctdBqA5z9rRRe+rCqwfp/ff5wt9O7FpbRN8PJT+dkdELOe+A7iydNmmTCX0yzceNGjjvuuBbX7Ypxd+pfpNOvX79O3W89HWuoodb+rfQ0vfElLS9/WMVnW2o5Z1oKE49JbLS8u5V5/hOlzS6//crsYJrhA91sL/a2uM3br8xuMN3OF9OsNsZMirRMq4aUUirOdf/7rR6uvQ9sKaVUrOkdgVJKdbLuViUfs0AgIkNE5F0R2SAiX4rIDRHS5ItIuYissX9ui1V+lFJKRRbLqiEf8HNjzGcikg6sFpG3jDEbwtK9b4w5J4b5UEop1YyY3REYY4qNMZ/ZnyuAjcDgWO1PKaVUdDqlsVhEcoHxwMoIi6eJyBfAbuAXxpgvI6w/D5gHMGDAgAYPRYE1lk+kJ2jD+f3+VqWLRkZGBhdddBGPPPIIAD6fj5EjRzJp0iSeeeaZNm/v0UcfJTk5mUsvvbRd+fL7/dxzzz088MADbN++ne3btwcHnTPGcNNNN7F06VJSUlJ48MEHg89CLFiwgD/+8Y8A/PKXv+Syyy5rch8nnXQSI0eOjOoJ5Vho7jjX1NQ0+vvpDcIfFuwN1n2VTGFxIqsTqqjYXddoeXcrc2FhVrPLCwq+CKapKfOxp6zl029BwRcNpmNV5pgHAhFJA5YANxpjwgfq/gwYZozxiMjZwAvAyPBtGGMeBh4G6zmC8H60GzdubFVf+Vj2qU9NTWXz5s24XC6Sk5N5/fXXycnJweVyRbXPG2+8sUPyVVFRwcyZM7nwwgvJz88nLS0tmJ/XXnuNHTt2sG3bNlauXMkNN9zAypUrOXjwIHfffTerVq1CRJg4cSJz584NjmsUauPGjRhj+Pjjj3E4HA2GnoiGz+fD5Wrfn2VzxzkpKSk4JlJv0t361HeEioQqvIm1TJzYM54jeG9H888R5OePC6YZNsBFUiteQJOfP67BdKzKHNNeQyLixgoCC4wxz4UvN8YcMsZ47M+vAW4R6ZonrzrA2WefzauvvgrAwoULg4PNQeRhpQOBALm5ucERRwFGjhzJ3r17GwxTnZ+fz69+9SumTJnCMcccw/vvvw9AVVUVF110EaNHj2bOnDmceOKJhD9sBzB+/Hhyc3MbzX/xxRe54oorEBGmTp1KWVkZxcXFvPnmm5x++un06dOH7OxsTj/9dN54442IZV64cCGXX345Z5xxBi+++CIAU6dO5csvD9/Y5efns2rVKiorK7n66quZMmUK48ePD6Z//PHHmT17NqeeeiozZ87E4/Ewc+ZMJkyYwNixY4PpAH77298yatQopk+fziWXXBL8jrZt28aZZ57JxIkTmTVrFps2bWr5gCmlgBjeEYg1utqjwEZjzL1NpDkS2GuMMSIyBSswlbRnv74Xf47ZtTbiMrffh9fZ9iLL4Dxc5/2pxXQXX3wxd955J+eccw5r167l6quvDp60mxpW+rzzzuP555/nqquuYuXKlQwbNowBAwY0LpfPxyeffMJrr73G/Pnzefvtt3nggQfIzs5mw4YNrF+/vsEQF62xa9cuhgwZEpzOyclh165dTc6P5Omnn+att95i06ZN/PWvf+XSSy9l7ty5LF68mPnz51NcXExxcTGTJk3illtu4dRTT+Wxxx6jrKyMKVOmcNpppwHw2WefsXbtWvr06YPP5+P5558nIyODAwcOMHXqVGbPns2qVatYsmQJX3zxBV6vlwkTJjBx4kQA5s2bx0MPPcTIkSNZtmwZ1113HcuWLWvT96FUvIpl1dBJwOXAOhGpH5/5FmAogDHmIeBC4Mci4gOqgYtNd+tg2wZ5eXkUFhaycOFCzj777AbLmhpWeu7cudx5551cddVVLFq0KPgOg3AXXHABABMnTgw+pLZixQpuuMHqlTtmzBjy8vJiVLLI6ofOGDp0KIMHD+bqq6/m4MGDXHTRRZxxxhnMnz+fxYsXc+GFFwLWcNUvvfRS8Cq+pqaGb775BiB4BwJW28Utt9zC8uXLcTgc7Nq1i7179/LBBx9w3nnnkZSURFJSEueeey5g1Zt++OGHfO973wOsgfy83pYf31dKWWIWCIwxK4BmX7ljjPkb8LeO3G9zV+41FRUkxXjcndmzZ/OLX/yCgoICSkpavrmZNm0aW7duZf/+/bzwwgv85je/iZguMdGqI3U6nfh8HfNy68GDB7Nz587gdFFREYMHD2bw4MENGqSKiorIz8/n+eefZ/78+YD1XoOFCxeyadOmYLXToUOHWLJkCT/84Q/p27cva9eu5emnn+ahhx4CrBP8kiVLgm9Fq7dy5coGbQsLFixg//79rF69GrfbTW5uLjU1NU2WIxAIkJWVFXwfRDyOr6R6lu52uatPFnewq6++mttvv52xY8c2mN/UsNIiwpw5c/if//kfjjvuuGCPntY46aSTWLx4MQAbNmxo83sMZs+ezZNPPhls7M3MzGTgwIHMmjWLpUuXUlpaSmlpKUuXLmXWrFnMmTOHNWvWsGbNGiZMmMDixYtZt24dhYWFFBYW8uKLLwZHV507dy5333035eXlwTuVWbNm8de//jX4VOXnn38eMV/l5eX0798ft9vNu+++GxxO+6STTuLll1+mpqYGj8cTfNNbRkYGw4cPD/bOMsbwxRdfRNy2UqoxDQQdLCcnh+uvv77R/DvuuIPVq1eTl5fHr3/96+Cw0mCdNJ966qkmq4Wact1117F//35Gjx7Nb37zG44//ngyMzMbpbv//vvJycmhqKiIvLw8rrnmGsBq3B4xYgRHH300P/zhD3nggQcA65WWt956K5MnT2by5MncdtttDYazBnj//fcZPHgwgwYNCs6bMWMGGzZsoLi4mAsvvJBFixZx0UUXBZffeuuteL1e8vLyOP7447n11lsjluuyyy5j1apVjB07lieffJJjjz0WgMmTJzN79mzy8vI466yzGDt2bLC8CxYs4NFHH+WEE05gypQpDRqYlVLN02GoezC/34/X6yUpKYlt27Zx2mmnsXnzZhISEoJpeluZPR4PaWlpVFVVMWPGDB5++GEmTJjQII0OQ9079OZhqIf0d7FzX8tVvJ01DLWOPtqDVVVV8e1vfxuv14sxhgceeKBBEOiN5s2bx4YNG6ipqeHKK69sFASUUm2ngaAHS09Pj/jcQG9W/5pOpVTH6TVtBD2tikt1Pv0bUSqyXhEIkpKSKCkp0X901SRjDCUlJSQlJXV1VpTqdnpF1VB9j5j9+/c3m66mpibuTgRa5sOSkpLIycnpghwp1VB3u2btFYHA7XYzfPjwFtMVFBT0ygHHmqNlVkq1pFdUDSmllIqeBgKllIpzGgiUUqqTdbc2Ag0ESikV5zQQKKVUnNNAoJRSnUyrhpRSSnUrGgiUUirOaSBQSqk4p4FAKaXinAYCpZTqZN2srVgDgVJKxTsNBEopFec0ECilVCcrLmn5fcWdSQOBUkrFOQ0ESikV5zQQKKVUnNNAoJRScS5mgUBEhojIuyKyQUS+FJEbIqQREblfRLaKyFoRmRCr/CillIoslu8s9gE/N8Z8JiLpwGoRecsYsyEkzVnASPvnROBB+7dSSqlOErM7AmNMsTHmM/tzBbARGByW7DzgSWP5GMgSkYGxypNSSqnGYnlHECQiucB4YGXYosHAzpDpIntecdj684B5AAMGDKCgoCCqfHg8nqjX7am0zPGhN5Z53VfJFBYnsjqhiorddY2Wd7cyFxZmNbu8oOCLFtNEWidUrMoc80AgImnAEuBGY8yhaLZhjHkYeBhg0qRJJj8/P6q8FBQUEO26PZWWOT70xjJXJFThTaxl4sQUJh6T2Gh5dyqzMYb3dpQ1myY/fxzv7Sht03bz88c1mI5VmWPaa0hE3FhBYIEx5rkISXYBQ0Kmc+x5SimlOkksew0J8Ciw0RhzbxPJXgKusHsPTQXKjTHFTaRVSqluqbu9erKtYlk1dBJwObBORNbY824BhgIYYx4CXgPOBrYCVcBVMcyPUkrFhAaCJhhjVgDSQhoD/CRWeVBKKdUyfbJYKaXaqYffEGggUEqp9opV1dBjr1fEZsNhNBAopVQ3tXNf57y3QAOBUkq1U09vLNZAoJRScU4DgVJKxTkNBEopFec0ECilVDtpG4FSSsWp0go/n26q7epstJsGAqVU3KusDnDfs+XsK/W3ab0n3vTw2soq6nw9+5ZAA4FSKu5t2eWlvDLAh1/WtGm96jorAGjVkFJKxTkNBEoppXo0DQRKKRXnNBAopVQnMN24/kgDgVJKtUFFVYDq2gBw+IUrpocPRK2BQCml2uDeZ8q5Z/EhoOe/h6CeBgKllGqjQKC3hACLBgKllIpSsGqoFXEh2iaCzmhb0ECglFJxTgOBUkq1Uywv2jujs5EGAqWUaicNBEoppWKmM5qlNRAopVQUnn+/klqvPehcK9JH31gc3XptoYFAKRX3RKTlRGHWfl0X/KxVQ0op1cN15+EfNBAopVQPENM7gk5oJdBAoJSKe9FUDYWK5am6R98RiMhjIrJPRNY3sTxfRMpFZI39c1us8qKUUs3pjKqhaPfQGYHAFcNtPw78DXiymTTvG2POiWEelFIq5mIZSHp0IDDGLBeR3FhtXymlOkp7q4ZiyQC7D/jYU+qP2T5aHQhEJBkYaozZ3IH7nyYiXwC7gV8YY75sYt/zgHkAAwYMoKCgIKqdeTyeqNftqbTM8aE3lnndV8kUFieyOqGKit11jZZ3ZJm37XVTWJiKs7qOLF9Vs2kLC7MazfvggwoKC9ObXe+9976IuG5Lli8v55mPMwH47sTYHOdWBQIRORe4B0gAhovIOOBOY8zsduz7M2CYMcYjImcDLwAjIyU0xjwMPAwwadIkk5+fH9UOCwoKiHbdnkrLHB96Y5krEqrwJtYycWIKE49JbLS8I8ucva2OoppKxhyVQP701GbTvrejtNG8adPSWX+wotn1TjnlBFbsLGtz3qZPz+TTPeUApKWVxeQ4t7ax+A5gClAGYIxZAwxvz46NMYeMMR7782uAW0T6tWebSinVFeLlyWKvMaY8bF67siciR4pdMSciU+y8lLRnm0op1dt0xnMErW0j+FJELgWcIjISuB74sLkVRGQhkA/0E5Ei4HbADWCMeQi4EPixiPiAauBi050f71NKqSb09CEmWhsI/hv4X6AW+A/wJvC75lYwxlzSwvK/YXUvVUqpHq3XBwIRcQKvGmO+jRUMlFJKdZJYdhut12IbgTHGDwREJDPmuVFKqR4olu8sfqagMroV26C1VUMeYJ2IvAUEc2WMuT4muVJKqS4Qr62UrQ0Ez9k/SimlwvT0+NGqQGCMeUJEEoBj7FmbjTHe2GVLKaU6X7QjTfT0Do+tfbI4H3gCKAQEGCIiVxpjlscua0op1bm680NfsdxPa6uG/gScUT/OkIgcAywEJsYmW0op1bt055uG1j5Z7A4dbM4Y8xX2w2FKKdVbRF811LH5aHI/Mdpua+8IVonII8BT9vRlwKrYZEkppbpG1FVDHZuNpvfTxVVDPwZ+gjW0BMD7wAMxyZFSSqlO1dpA4AL+Yoy5F4JPGzceF1YppXqwbl81FKP9tLaN4B0gOWQ6GXi747OjlFJdJ6ZjBnVABVKsstfaQJBU/+4AAPtzSmyypJRSPUt37hHUGq0NBJUiMqF+QkQmYQ0drZRScS9eGotvBJ4Rkd329EBgbmyypJRSKrIoGzFa0OwdgYhMFpEjjTGfAscCTwNe4A1ge0xypJRSPUxvbyz+B1Bnf54G3AL8HSjFfpm8UkrFu9aMNdQRJ/Guaix2GmMO2p/nAg8bY5YYY24Fjo5RnpRSKu5kpTkYMbD5ARu66o7AKSL17QgzgWUhy1rbvqCUUr1aT+811NLJfCHwnogcwOol9D6AiBwNlMc4b0op1SN0RByQVjzN1iW9howxvxeRd7B6CS01hyvCHFgvtFdKKdUK3bmNoMXqHWPMxxHmfRWb7CilVM/T06uGWvtAmVJKqSb09u6jSimluo0ueKBMKaVUy/SOQCml4lxrzs8tncRbMwR2V48+qpRSqovpHYFSSnVT2muoCSLymIjsE5H1TSwXEblfRLaKyNrQYa6VUqonac1YQx2zn9hsN5Z3BI8DZ3nDgo4AACAASURBVDaz/CxgpP0zD3gwhnlRSqkebdaumzjiwEcx2XbMAoExZjlwsJkk5wFPGsvHQJaIDIxVfpRSPdMrH1Wx+qvars5Gs1pzpd6ei/mhnhVMOPgkqdW7W04cha4cOG4wsDNkusieVxyeUETmYd01MGDAAAoKCqLaocfjiXrdnkrLHB96Y5nXfZVMYXEiAE/t91MxqaLB8o4s87a9bgoLU3FU15Htr2o2bWFhVqN5qxKqKCxs/u29779fTmFhJiMG1PH13oRGyw8m+0lNClBc2ngE0tOrH+KQZLMh81TKYnCce8QIosaYh7HffzBp0iSTn58f1XYKCgqIdt2eSsscH3pjmSsSqvAmWncCR2Q5yc/PaLC8I8ucva2OoppKxoxIIP/k1GbTvrejtNG8iZNS2F3XfAA5+eRMPiku58wTU3htZeO0fTKcZKU6SCz2Nph/1KG3mLTjbdZkX0pSeh/y809uRYnapit7De0ChoRM59jzlFKqR4lVI+6xZS9y0Y7LAfio/w09srG4JS8BV9i9h6YC5caYRtVCSinV3bXlBN3Ug2Phs1O9+5iz81oAHj/qNcoShnXd6KPREpGFQD7QT0SKgNsBN4Ax5iHgNeBsYCtQBVwVq7wopVRXa+vV/HHlLwHw3NB/Upxi967vivcRtIcx5pIWlhvgJ7Hav1JKdZZYVNkMq1xBqXsomzPPPbyfjt8NoE8WK6Xi0P4yP2+tru6wB8Fas5W1X9e1ensJ/gqGeVZQmDaj4X56YRuBUkp1iafe9vDh+ho81Z03NsTbq6tbnfaUvXeRGPCwpu/lDeYHjA5DrZRSHSIQ6NjtdURjcejGxpQ+w/a0GexJPqHBIl8H57ueBgKllGqnjqhiqg8Qxx56maTAIbakz2qUxu/XOwKllOoQrRn7v6t8a999AGzMnN1omVcDgVJKdU8d1dKQULufATUbKEk8mir3EY2Wa9WQUkr1As3djQzfvQiAggG3RFyuVUNKKdXBOqo7ZkdtZ9CBtwDYmTo14nKtGlJKqW6qowJBWtXXrMu6kGpXn4jL/TGqGuoRo48qpVR31hGBwO2rIKW2mJKsYyIuzz3SxSBXTft3FIHeESil4lb9Cby13T9j+UrK7JotABxIHBlxeUaKA7czNvvWQKCUilsmrL9PtOf5tqzWVC1/VtVXAJQkRQ4EsaSBQCkVd+p77nSnxuLsqs34xU1pQm77N9ZGGgiUUnHrcNWQ9bulB81iWDPEkJKlHMg6ESOd33SrgUApFbfCT+xRVw21M0C4AtVkVW9hf5+T2rehKGkgUEr1eAve9jD/icbvEo7EUx3gUGXr+mH+85UK7l5U3p6stUq/ms0IhvLUyD2GYk27jyqlerytu7wtJ7L9afHhE7sJ+x1eNbS7xNdguqkr/7b0JopU/TS08kMA9medCK2LZx1K7wiUUnGrw6qG2pmPXM/7lCWPpDZpYDu3FB0NBEqpuNUdeg1l1RZylOdddmdN75jMREEDgVIqbgWrhlrbaygGeTi+7DkAtvc7LwZbbx0NBEqpuBVet98VvYaOqNlAmXsIe7JOitnL6VuigUApFbe6ompIQm47XIFqRlYsZXvaKR2TkShpIFBKxa22BoKm0m/e2fpeS6EmlDyOy9RRmD4jqvU7igYCpVTcCm8jiNaBcn+b13EYLzP3zAegOOwl9Z1NA4FSSnWyRH85122aDMDSgb+nPGEY0PSAdLGmgUApFbc6qmqorY459Drpvj3sSzqOL7MuAKweS13VWKxPFiul4lb4oHOdwek9xJm7bsInifx7xEvUOdM7b+dNiOkdgYicKSKbRWSriPw6wvIfiMh+EVlj/1wTy/wopVSozgwA9dK+fhmXqeOFIQ91iyAAMbwjEBEn8HfgdKAI+FREXjLGbAhL+rQx5qexyodSSjWlrXGgIwJH6s5leFxHsCXjzPZvrIPE8o5gCrDVGPO1MaYOWAR03aNzSqm481WRF5+/6bN3/QNlnXVj4AjUkVq0jMK0GREfY+6qxuJYthEMBnaGTBcBJ0ZI910RmQF8BfzMGLMzPIGIzAPmAQwYMICCgoKoMuTxeKJet6fSMseH3ljmdV8lU1icCEBZqp+CgooGy0PLXFiYBUBBwRfB5XvLnbyxJp3jcmqZclR1cH59WoCPPvJQmO1j064ECgtTkKo6+gSqGqWt326dr+H6bZXnW46r+gAFfIvCwh0NlpUf8JPkClBc5o64rqO6jvE5sTnOXd1Y/DKw0BhTKyLXAk8Ap4YnMsY8DDwMMGnSJJOfnx/VzgoKCoh23Z5KyxwfemOZKxKq8CbWAnBElpP8/IwGy0PL/N4Oa+zm/PxxweWbvqljU1klQ4e4yc9PC86vTwtw4olpjBjkJmVTLXt9VYwZnkD+jNRGaeu3W1MX4KNd0b+fIG/vHqiGmuHnk+tMbbBsQLaTlCQHicWRH04bMyKBNH9VTI5zLKuGdgFDQqZz7HlBxpgSY0ytPfkIMDGG+VFK9XJteS8AdNwDZS3vyOAM1DK2dDE1R4zDGxYEulos7wg+BUaKyHCsAHAxcGloAhEZaIwptidnAxtjmB+lVC9nTMsjiHa26Xv/yMn7/hScPjD8SqhunK4r8x2zQGCM8YnIT4E3ASfwmDHmSxG5E1hljHkJuF5EZgM+4CDwg1jlRynV+8X6AbG2pncFqhsEAYCqQSfBtsjpe2NjMcaY14DXwubdFvL5ZuDmWOZBKRU/ou0O2tYqpdZI9e5luh0Enh/yDzZlnUeCv4JzBh0J2yoj56fDc9E6Xd1YrJRSrRIImEYvkGnr+wQCgYYJAsaaF4jBGXjKgX8w4eCTlCbkWt1FgTpnetRVQLGsOtJAoJTq9vaX+fntv8sazDtlGNy9qJykhMNnyJYCQfg2Fi3ztCkfbblxGFj9OeXuwfxzZAF+R1Kb9tPZdNA5pVSPZAzU1BnKPIEG87qDzLod5FR+ytb0M7p9EAC9I1BK9VCRzvkGQ9c1uVoS/BVct9l6drYwrfUvpJcu7DakdwRKqR4p0tV/c3cE4e0DHbXPcEMqVwLwVcaZfJVxVrv32Rk0ECileqS2BgKvL3Z5CXV0xVsAvJLzF5DGp9ju9pwDaCBQSvVQkS7wmw0EzQw+11FGlz3HhINPUJJ4NLXOzJjvr6NoIFBK9UjGNL60bu5UX+ftgKqhFpZPPvBPKl39WJS7qN376kzaWKyU6pEi3RG8v66GsooADgds3mnVBW3e6WXZ59VkpkZ/3WuMabExV4yf/jUbWN33Kg4l5ES9r66ggUAp1SNFqgb6ZGNt45nA+2trmD42tt04s+sKcZla9icd22w6bSNQSqkO0taKnjpf9FVDrXm38YSSfwFwIHFU1PsJNWpI5PcSxIIGAqVUjxSpjaA5vnb0GmpNt9FhlR8AsD+pYwJBZ9KqIaVUj9TWxwJqW2gsPqb8NY4sdmH25EH/UYjj8HVyS7sS4ye7tpBP+l6Lz5HStozVbyOqtTqGBgKlVI/kD7ScJlRVbeTTuRgf4w4u4Mzdv4JvwPspkHoErmteBEYATVcNuQOVDK5azfiSJ3CbavYkj21bpprRmcNlaCBQSvVIfn/brqGrag5HjkFVq5m169ck+w+S6T384sSVUxfzraGl+Bdfi9n8FnBtk9tLr9vFd3b9jOGe5cF529NOaTEf3bCtWAOBUuown9+w6Rsvx+e6W+wuuW23l/5ZTtJTrCqULUVeBvV1kprctqbHrbu8HNnHSVqyg693e+mX6cQY+GxL5B5A9cqr27afyhqDw3iZdOCfzNxzJzWODHakTWdT5jmUJoxgX9Jx9Ok/HeeUVPxv/T/8b9zOnIxP2JUykbUfn0PKwFzKahMZUvkRfWu2cNK+P5Phs16wuHD4YgpTT45plyAdhlopFTOlFX4++LKWs6ck89bqaj7ZWEtqUhpbd/tYv72On5yXQYK74VnIGMNTb3nITndy/QUZ+PyG/7zjYWBfF/POSW/1vgMBw4K3PfTPcvLj8zL491vWsNADsp0trrtiUyq5uc2n6VezkezaQgZVf05yoJyjypeS4Stmb9JoXhj6MAcTj26QPtv+7br4EXzP/JhjD7zKsYdehRfuxC9uvk77Nt+vWBpMX5g6nSXD/kWds/VlBmuAuVi8DCdaGgiUinMvflDFjr0+8oYnUFZhVZ/U+eDD9TUAfLGtjsnHJjZYp75+vrTCD4DX7pq5v8zfpn3Xj/+zr8zfYFC4vaVt2049V6CK4RXvceyhVxhS+XGw2sePi1pnBnuTx/Jm37vYln4aRhoHm4BdLsdRM3Df+BGPPfYBIBxZvY7hnncZaQeB5f1/ySH3IDZkzWk0zPRPz8/gby8cajafIt1nyGzQQKBU3AvtfVN/gne2UOsSPlyDL7rzdoPxf6IeFM4YhlWuYNq++xlStRKXqaPamU1x8gms6fN9CtNOYX/SKLyO1BY3FQg5O0tSBkWpUwEoSj2RVf2uwRWoJqtuBwcSj4k4oFxrRarm6ZUvr1dK9SwG8LeyT2b4Sbv+jqCtF7nekIe8WvPAlzNQy+CqTzmiZjPjajdz1I59DK94D7epxuPqz+q+V7M9LZ/CtOkYafvpraWeSD5HMgdaeHK4JSLdr8FYA4FSCgC/3wSrRkJP0JGuVMNH8vRGe0fgC/0cORD0qd3GUYfeYoSngCGVH+M2VpWVDxdVgSNYmz2XQwmDWZt9MVWuI6LLiC3Qxi6p0XI4gCi/s1jQQKCUAqyTef0VcU1d81fn4VVDwTuCNt4SNKgaCjsxugOVnLz3j5x44CEAShOGsSnzHL5On0lRymTW7fIybPjwtu2wBa29I2ovvSNQKgK/fUJwOgWf32CMdXVWUR0gI8WBw2GdbJwOadSDpaP4/Aax81DP2qfV9dDlBId9eex2QWlFgNRkIdEteH1WvboBXE6hzmulB+uJ1kS34HA0zHed11DrNTjEqmNPTRYcYl0li1j11Q4R63PAOkmlJFn10hVVAVISre15qg3VdYLXZ6iqMaQkCU6Htc3676q6NoDLKVRUBUhwCy6ntc3qWmsdgN0lPg6UW2fjfWWHL433lfmD850O62Qf2ph7qDJAqd3IHAiY4L6cDqu6p7bOOuE7xGqPqP/tcsKeg4e3E/oZYOShpZx44CGqHZm8NPRBvk77doPbEyM72nB0W6e80lBZHcDpbDkYtkd3G3gu7gKBMYYtu3wcPcjV6B9TtV3hHqsPeFJC+4at+tMzhwgEDL++NIu/LDmEp/rwiSjnCBdF+w/XIcw7J52BfTv+T/f3T5VxZB8n156bgTGGddu9PP9+ZcS0eSMSWPt1HQBjhyewbntdcNm530rh5Q+rmHZ8ErV1hs+21JLgFm64IIOUJAdF+31kpTlY8Lan0ckvvKzhrvlOOimJwv3PHaJ/lpN9di+dwsJMPikua5T+v+dksHpLXbAHUHOWf3E4zccbDn/+dFMtn25quk//n58ttz4YQ7K/lCf+tZZE/yGS/WUk+ctxGC9CAFegBsEgBBBjwP58krF+H1hiOMV4SfRbPW7yShcRwMH9x60j4EhoMf8dobTCzz2Ly9u1jcQE4cg+zkbHNtSwAS427/Q2u50BfZx8VdR8mo4Sd4HgqyIfi5Z5mDkhOebD0vZ2Xp/hiTc95B7p4spZbetHHa669vAVZWgQADhU1XB6d4k/JoEADl+V7isLNBkEgGAQABoEAYCXP6wCoHCPj+IS66Re5zX8+y0P156bwaOvVZCZ6qC8snGFdHNBAKDME8Dvd9h5bLmS+cChQKuCAEBKkqPB07fnTEvhlY+ssowZnsCWrQcYUL2OVN9+Un0HOMq9FW/ZXlyBWhIClWR4ixo8pRuNAE5qnJm43YKk9qV8yOl8Z3oWTofw4foa9pX5uWRmGocqA6xLruSkb6Xh9xtEIC3ZQZnHulPy+sFTHUAEnA7hhRWVDBvg4rszUvlmn4/kBKGq1rBkedPHuLXGH53I+sI6vD7DqROSSUt2cMmpaZQc8hMw8FWRt8Hw2CLCd2ek8v8WNA7cofJPSGLyqERe/biqxaDRXnEXCCrsk0qpp5NahXqx+nri4pKOa/WK1HOks941Gyq8WqCpE3dz+mc5KC45PL3noD/4EFFbt1XPmPa/cvHmS7MIGIMgJLihzmtVdYnxc/Drjbz+xiaMCCfs3UNe0joo+wbzURHs/6rhhtzJ7HPm4pcE6hypJI2YzNvlP6TCPZBqVx+qXP3wSjI+RxIBnPzPpUficDoJGHjgJQ9llXDxqemMGJwACIjgdggJhuDdegowwN5d3ghrWOb6J549xV6OyWk4VHNOE23FY4e7g9s8Pvfw3cXoYW68Pqua6ndPNT4xnzQmiQ9aCKT9sx3k1rjYUuQNPgiXkeogw34Rzjf7Gv8Bu13CyBw3W5q54nc4hPQU4dihbg0Eqvtqz/juTW4zwt97U71JYim8MTQrre2BIFIAa29Qq/MdbntoDX9Y0HAGanDt+QyzbzNU7MFfvhtnbQX+vZswu78g3VvNRXbaQCGQ2g/pdxQy4FgKHHMoTh7H3PNGkZDdH1L68ui/D1ej/PecDD59vukHqZwpGdZvwO/wEXAEcCe6cboanoaaqj9vaciL5jRVDexwCInN1DqlJbe8T7dLmm38beoZC7ez+1RNayBQUauLwZV6pODi64SXjgfCeouE5yM1qW1tIE6nRLxyr/MZxPhI8pfjDlST7t1NureYDO9u0nx7cAeqcZo63IFq0rx78Yubalc2TuPFYfz086RhMgYxviSHxEAFyb5SEgMV+KqLOXLbIQLiotaRjsvU4Rc3R7yawjkH3aR7i8mq20GGdxe+L0MCWmIaJKYjfY/CMfUaao4YxzOfZxEQJ1dfPgHS+gdPwB8+UQqAe3AWEuHE6na1/cSWEMU6nak1xz203JF6TTX19+sOO/v22gfKRORM4C9YFwGPGGPuClueCDwJTARKgLnGmMJY5kl1nPqr5o48TbfmBeOxeDQ//KotPB8pSY3/S52BGvrVfhU8qSf7D+IOVJHqO8Ag7wYytxUzo64Wd6CKZH8ZDuPF/aWPXwci3+bXSTJeZyp+ScAniXhcA0gIVJFZvQufI5EATpJKakn85lXO9Ft1zj5JpNaZTqU/iWoZgsP4yarbgc+RhNN4cZd4GVZZQYV7ILtSJrM+8Xvkz5qMDBwN6UciyVkN8iDVAb7ZYl3lS3p2ozxC01fXblfbx9CJJnh0pkjHPZzb2fxJvMk7gm5U9pgFAhFxAn8HTgeKgE9F5CVjzIaQZP8FlBpjjhaRi4E/AHNjlScAf00FZ+z6DelmKIG00ZA+AEnKAneS/ZMMriRwOK1HyMXRrlvS3qyjqoZCTxy1tT7cgUqSfQdJCHjoU/s1fWu3khDwkOQvxxWoYVC1C9/n9p+uSMh/ofX5qN278e1/+vDy+hv3sHShn/0+w+m7vXglCf+yQaSXpzPEk4PL1OE0tRy1Yx+Zu7eS4S0iu66QBH8Fmd5dOCI8FRTAQUXyCCqSc/FIIj5HEtXObPzi5oRjUlm1FaqdWXgdKVS6+lOeMBiPeyA1jswWLwtPzksi1VXHBx/vpNqVjU+SQYTCwh3k5g5rlH7q6KQGPYAATj0h8gkewNmGaqdw0VR1hF8VdzeJreiq3PCOoPH/RHj1XP0hbm0g6IwxiWJ5GKYAW40xXwOIyCLgPCA0EJwH3GF/fhb4m4iIicGwfEUfvM5Rb/yMfv6dOI0PDoJvfevXD+DEiAODFRwC4gAc9ryQk43920Q6+RB69WxPhy1van1DE+la2P5Ir4/977gjrB9hX4BgQrZiQuYR8hdp/c4whh8Za/6B34jdJZDI65uwbYX9/u9AgGRfKc51Pn5BY35xU+tIp86RgqMKyndG2q71Odnno2KvM9hFsWGpI68DhuMNuANV+F+rYyjw/dAM7IAcScbjHsjBxBHUJo5iQ8Ic9iaPocrVD59YJ/s6Zyq1jnRGDE3n62Jfo5PA58ZB2YDoOyqs2lyHQ6AyYXCr0q/Z2vxQzuEc7bjocTqFRDfU1LWctv75huZr17tea07WCS4Jpot00djUV5qU0HCBy2ndXYSrf64llt3dYxkIBgM7Q6aLgBObSmOM8YlIOdAXOBCaSETmAfMABgwYQEFBQZsz4935DX1dORxMHMZO/1DWD5hLhm8vaf4SEgOVuEwtLlOHK2D9DvZ3xuAwfvukYfV/dmBNiwkE0zXIb/iJMCyuNXUyPDyfsPWaWN6K9f3ix2m/ci98eeNtNA4M4QHEhP1VV9c5SUowwTOtCfvHbm790GVev4OqxHQcCQnUBdxUu7I4UJvOgCHZbKw9hv3VqZR6nLhdhkHZzfegqKmpISmp7V2DyyqdJDoDpCdUk+o/SEZ1EWXeVDLSXYw8OoUPdw2mxuvEH4DkBMOgPl4K9yXgwFB0wG11YUyqYEjfg1Duw3cogUBA2Fnixu00DMjyUVppOHDIRWaKnwSXYd8hF/0zfKQmBiircuLzCzVeITMlQEayn9JKJ33T/FTVOXA7DVWV1neWnGDw+eFAhQunGI4+ogJTXUeiO8Cug26S3IaDlU4GZXsprXRS5xNGDayjX7qPgoKmG3eMgTR/EtmpfgoKGn7PozJdeGocFBQcPtPnpropq3SQ4DIUFNQxKsPB2+vSqKx1kOQO0D/ThwiMyamloODw3dOQBAc+3Hz2aW3UdeMejyeqc0FTjs92svabJHx+qPE5GJTlZcOaGqQqBRFwiOGgx0V6snXsBmb5KK9ysm1DDak+Ic2fyK6tNRRva7jdRK8gVcn4/EKiO8DXG2vYvdU6fpUH0xjRv44ar4MjqEX8ULY/jfG5NcHv3x+wjkliZS2emo4tc5AxJiY/wIVY7QL105cDfwtLsx7ICZneBvRrbrsTJ0400Xr33XejXren0jLHBy1zfGhPmYFVponzavseB23eLmBIyHSOPS9iGhFxAZlYjcZKKaU6SSwDwafASBEZLiIJwMXAS2FpXgKutD9fCCyzI5dSSqlOErM2AmPV+f8UeBOr++hjxpgvReROrFuUl4BHgX+LyFbgIFawUEop1Yli2nnLGPMa8FrYvNtCPtcA34tlHpRSSjUvllVDSimlegANBEopFec0ECilVJzTQKCUUnFOelpvTRHZD0T7jrp+hD21HAe0zPFByxwf2lPmYcaYiG9s6HGBoD1EZJUxZlJX56MzaZnjg5Y5PsSqzFo1pJRScU4DgVJKxbl4CwQPd3UGuoCWOT5omeNDTMocV20ESimlGou3OwKllFJhNBAopVSci5tAICJnishmEdkqIr/u6vx0BBEZIiLvisgGEflSRG6w5/cRkbdEZIv9O9ueLyJyv/0drBWRCV1bguiJiFNEPheRV+zp4SKy0i7b0/bQ54hIoj291V6e25X5jpaIZInIsyKySUQ2isi03n6cReRn9t/1ehFZKCJJve04i8hjIrJPRNaHzGvzcRWRK+30W0Tkykj7ak5cBAIRcQJ/B84CRgOXiMjors1Vh/ABPzfGjAamAj+xy/Vr4B1jzEjgHXsarPKPtH/mAQ92fpY7zA3AxpDpPwB/NsYcDZQC/2XP/y+g1J7/ZztdT/QX4A1jzLHACVhl77XHWUQGA9cDk4wxY7CGsr+Y3necHwfODJvXpuMqIn2A27FeBTwFuL0+eLRaU68u600/wDTgzZDpm4GbuzpfMSjni8DpwGZgoD1vILDZ/vwP4JKQ9MF0PekH62137wCnAq9gvRD5AOAKP95Y78OYZn922emkq8vQxvJmAtvD892bjzOH32fexz5urwCzeuNxBnKB9dEeV+AS4B8h8xuka81PXNwRcPiPql6RPa/XsG+FxwMrgQHGmGJ70R5ggP25t3wP9wE3AQF7ui9QZoypfyt7aLmCZbaXl9vpe5LhwH7gX3Z12CMikkovPs7GmF3APcA3QDHWcVtN7z7O9dp6XNt9vOMlEPRqIpIGLAFuNMYcCl1mrEuEXtNHWETOAfYZY1Z3dV46kQuYADxojBkPVHK4ugDolcc5GzgPKwgOAlJpXIXS63XWcY2XQLALGBIynWPP6/FExI0VBBYYY56zZ+8VkYH28oHAPnt+b/geTgJmi0ghsAireugvQJaI1L9xL7RcwTLbyzOBks7McAcoAoqMMSvt6WexAkNvPs6nAduNMfuNMV7gOaxj35uPc722Htd2H+94CQSfAiPtHgcJWI1OL3VxntpNRATrvc8bjTH3hix6CajvOXAlVttB/fwr7N4HU4HykFvQHsEYc7MxJscYk4t1HJcZYy4D3gUutJOFl7n+u7jQTt+jrpyNMXuAnSIyyp41E9hALz7OWFVCU0Ukxf47ry9zrz3OIdp6XN8EzhCRbPtO6gx7Xut1dUNJJzbInA18BWwD/rer89NBZZqOddu4Flhj/5yNVTf6DrAFeBvoY6cXrN5T24B1WD0yurwc7Sh/PvCK/XkE8AmwFXgGSLTnJ9nTW+3lI7o631GWdRywyj7WLwDZvf04A/OBTcB64N9AYm87zsBCrDYQL9ad339Fc1yBq+2ybwWuams+dIgJpZSKc/FSNaSUUqoJGgiUUirOaSBQSqk4p4FAKaXinAYCpZSKcxoIVNwQEb+IrAn5aXYUWhH5kYhc0QH7LRSRflGsN0tE5tujUb7e3nwo1RRXy0mU6jWqjTHjWpvYGPNQLDPTCidjPUB1MrCii/OiejG9I1Bxz75iv1tE1onIJyJytD3/DhH5hf35erHe+7BWRBbZ8/qIyAv2vI9FJM+e31dEltpj6T+C9SBQ/b6+b+9jjYj8wx4iPTw/c0VkDdYwzPcB/wSuEpEe/zS86p40EKh4khxWNTQ3ZFm5MWYs8Desk2+4XwPjjTF5wI/sefOBz+15twBP2vNvB1YYY44HngeGAojIccBc4CT7zsQPXBa+I2PM01gjya6387TO3vfs9hReqaZo1ZCKJ81VNF298wAAAWtJREFUDS0M+f3nCMvXAgtE5AWsIR7AGuLjuwDGmGX2nUAGMAO4wJ7/qoiU2ulnAhOBT63hc0jm8IBi4Y4BvrY/pxpjKlpRPqWiooFAKYtp4nO972Cd4M8F/ldExkaxDwGeMMbc3GwikVVAP8AlIhuAgXZV0X8bY96PYr9KNUurhpSyzA35/VHoAhFxAEOMMe8Cv8Ia4jgNeB+7akdE8oEDxnofxHLgUnv+WVgDxIE1kNiFItLfXtZHRIaFZ8QYMwl4FWs8/ruxBkkcp0FAxYreEah4kmxfWdd7wxhT34U0W0TWArVYr/4L5QSeEpFMrKv6+40xZSJyB/CYvV4Vh4cOng8sFJEvgQ+xhlTGGLNBRH4DLLWDixf4CbAjQl4nYDUWXwfcG2G5Uh1GRx9Vcc9+yc0kY8yBrs6LUl1Bq4aUUirO6R2BUkrFOb0jUEqpOKeBQCml4pwGAqWUinMaCJRSKs5pIFBKqTj3/wFiVM0MRiYbbQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def ddpg(n_episodes=1000, print_every=100):\n",
    "    \"\"\"Deep Deterministic Policy Gradient (DDPG)\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes  (int): maximum number of training episodes\n",
    "        print_every (int): How many episodes pass between each print.\n",
    "    \"\"\"\n",
    "    # Queue to store the last 100 max scores, the then plot a moving average.\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    # Full score list to return and later plot.\n",
    "    scores = []\n",
    "    # List of the mivong avergare score over 100 episodes, to plot later.\n",
    "    moving_avg_score = []\n",
    "    # Define a finish flag for the learning task, a best score tracker, \n",
    "    finished_learning = False\n",
    "    solved_in = -1\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):        \n",
    "        # Restart the Environment in train mode \n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        # Get the initial state\n",
    "        states = env_info.vector_observations\n",
    "        rr_state = states[0]                                    # Separate right racket state\n",
    "        lr_state = states[1]                                    # Separate Left racket state\n",
    "        # Reset the agents\n",
    "        agent.reset()\n",
    "        # Set the score of each agent to 0\n",
    "        score = np.zeros(num_agents)\n",
    "\n",
    "        # Start a timer to see how long the episode took:\n",
    "        t_start = time.time()\n",
    "            \n",
    "        while(True):\n",
    "            ## Select Actions per racket.\n",
    "            rr_action = agent.act(rr_state)                    # get the action of the left racket.   \n",
    "            lr_action = agent.act(lr_state)                    # get the action of the right racket.\n",
    "            # Join the actions into a single vector to pass to the environment\n",
    "            actions = np.vstack([rr_action, lr_action])\n",
    "            \n",
    "            # Step the Environment and gather all the information of the timestep\n",
    "            env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "            \n",
    "            # Separate the state for the Right and Left Racket.\n",
    "            next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "            rr_next_state = next_states[0]                     # Separate right racket state\n",
    "            lr_next_state = next_states[1]                     # Separate Left racket state\n",
    "        \n",
    "            # Separate rewards by racket \n",
    "            rewards = env_info.rewards                         # get reward (for each agent)\n",
    "            rr_reward = rewards[0]\n",
    "            lr_reward = rewards[1]\n",
    "            \n",
    "            # Separate dones by racket\n",
    "            dones = env_info.local_done                        # see if episode finished\n",
    "            rr_done = dones[0]\n",
    "            lr_done = dones[1]\n",
    "       \n",
    "            # Add the experience of each racket separate of each other\n",
    "            agent.step(rr_state, rr_action, rr_reward, rr_next_state, rr_done)\n",
    "            agent.step(lr_state, lr_action, lr_reward, lr_next_state, lr_done)\n",
    "            \n",
    "            # roll over states to next time step\n",
    "            rr_state = rr_next_state                           \n",
    "            lr_state = lr_next_state\n",
    "            \n",
    "            # update the score (for each agent)\n",
    "            score += rewards                                   \n",
    "\n",
    "            # Check if the Episode is over\n",
    "            if np.any(dones):\n",
    "                break\n",
    "         \n",
    "        # Stop the episode timer\n",
    "        episode_timer = time.time() - t_start\n",
    "        \n",
    "        # Save the max of the two scores between the two rackets.\n",
    "        scores_deque.append(np.max(score))\n",
    "        scores.append(np.max(score))\n",
    "        # Also save the current moving average score\n",
    "        moving_avg_score.append(np.mean(scores_deque))\n",
    "        \n",
    "        # Print the current status of the training\n",
    "        print('\\rEpisode: {} ({:4.2f}s)\\tScore: {:4.2f}\\tMoving Average Score: {:4.2f}       '.format(i_episode, episode_timer, scores[-1], np.mean(scores_deque)), end='')\n",
    "        torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "        torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "        if i_episode % print_every == 0:\n",
    "            print('\\rEpisode: {} ({:4.2f}s)\\tScore: {:4.2f}\\tMoving Average Score: {:4.2f}       '.format(i_episode, episode_timer, scores[-1], np.mean(scores_deque)))\n",
    "            \n",
    "        # Save the code if the training already succeded    \n",
    "        if np.mean(scores_deque)>=0.5 and finished_learning == False:\n",
    "            print('\\nEnvironment solved in {:d} episodes! Max Average Score over 100 episodes: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor_SUCCESS.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic_SUCCESS.pth')\n",
    "            finished_learning = True\n",
    "            # Save how many episodes it took to solve the task\n",
    "            solved_in = i_episode\n",
    "            \n",
    "        # Check if the current average is the best of all the averages, and keep updating the  \"Success weights\" the current best candidate\n",
    "        if len(moving_avg_score) > 2:\n",
    "            if moving_avg_score[-1] > np.max(moving_avg_score[:-1]) and moving_avg_score[-1] > 0.5:\n",
    "                torch.save(agent.actor_local.state_dict(), 'checkpoint_actor_SUCCESS.pth')\n",
    "                torch.save(agent.critic_local.state_dict(), 'checkpoint_critic_SUCCESS.pth')\n",
    "            \n",
    "    \n",
    "    ## Print the final results of the training:\n",
    "    print(\"\\n\\nTraining finished:\")\n",
    "    print(\"\\t[+] Total nuber of episodes:    {}\".format(n_episodes))\n",
    "    print(\"\\t[+] Best 100-average score:     {:4.2f}\".format(np.max(moving_avg_score)))\n",
    "    print(\"\\t[+] Best individual game score: {:4.2f}\".format(np.max(scores)))\n",
    "    \n",
    "    # Check if the task was actually solved\n",
    "    if solved_in != -1:\n",
    "        print(\"Task solved in {} Episodes!\".format(solved_in))\n",
    "    \n",
    "    \n",
    "    return scores, moving_avg_score\n",
    "\n",
    "\n",
    "# Run the training algorithm\n",
    "scores, moving_avg_score = ddpg()\n",
    "\n",
    "# Plot the scores and their averages.\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores, c='xkcd:blue', label = \"Episode Score\", alpha=0.5)\n",
    "plt.plot(np.arange(1, len(scores)+1), moving_avg_score, c='xkcd:orange', label = \"Moving 100-Average\")\n",
    "\n",
    "# Set the titles of the plots and axis\n",
    "plt.title('Training progress - scores plot')\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "\n",
    "# Add grid and legend\n",
    "plt.grid()\n",
    "plt.legend()\n",
    "\n",
    "# Show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Watch a Smart Agent!\n",
    "\n",
    "In the next code cell, you will load the trained weights from file to watch a smart agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/said/.anaconda3/envs/drlnd/lib/python3.6/site-packages/torch/nn/functional.py:1340: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score (Max over agents) of this episode: 1.2000000178813934\n",
      "Score (Max over agents) of this episode: 1.3000000193715096\n"
     ]
    }
   ],
   "source": [
    "## load the weights from file\n",
    "# Weights of the Actor\n",
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor_SUCCESS.pth'))\n",
    "# Weights of the Critic\n",
    "agent.critic_local.load_state_dict(torch.load('checkpoint_critic_SUCCESS.pth'))\n",
    "\n",
    "\n",
    "# Run 2 episodes one after the other.\n",
    "for i in range(2):  \n",
    "    # Restart the Environment\n",
    "    env_info = env.reset(train_mode=False)[brain_name]\n",
    "    # Get the initial state\n",
    "    states = env_info.vector_observations\n",
    "    # Reset the agents\n",
    "    agent.reset()\n",
    "    # Set the score of each agent to 0\n",
    "    score = np.zeros(num_agents)\n",
    "\n",
    "    ## Run the Agent\n",
    "    while(True):\n",
    "        # Separate the state by racket to feed it separatly to the NN\n",
    "        rr_state = states[0]                               # Separate right racket state\n",
    "        lr_state = states[1]                               # Separate Left racket state\n",
    "        ## Select Actions per racket.\n",
    "        rr_action = agent.act(rr_state, add_noise=False)   # get the action of the left racket.   \n",
    "        lr_action = agent.act(lr_state, add_noise=False)   # get the action of the right racket.\n",
    "        # Join the actions into a single vector to pass to the environment\n",
    "        actions = np.vstack([rr_action, lr_action])\n",
    "        # Step the Environment and gather all the information of the timestep\n",
    "        env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "        next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "        rewards = env_info.rewards                         # get reward (for each agent)\n",
    "        dones = env_info.local_done                        # see if episode finished\n",
    "        # Save rewards and state\n",
    "        states = next_states                               # roll over states to next time step\n",
    "        score += rewards                                   # update the score (for each agent)\n",
    "\n",
    "        # Check if the Episode if over\n",
    "        if np.any(dones):\n",
    "            break\n",
    "\n",
    "    ## Print the results\n",
    "    print('Score (Max over agents) of this episode: {}'.format(np.max(score)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}