sigmoid_attn.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import time\n",
    "import torch.nn as nn\n",
    "import matplotlib.pyplot as plt\n",
    "from torch.nn import functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Softmax Attention**\n",
    "\n",
    "In standard transformers, Softmax Attention is used to compute attention scores.\n",
    "   $$\n",
    "   \\text{SoftmaxAttn}(X) = \\text{Softmax}\\left( \\frac{QK^T}{\\sqrt{d_{qk}}} \\right) V\n",
    "   $$\n",
    "\n",
    "- **Q** (queries), **K** (keys), and **V** (values) are computed from the input \\(X\\) by applying linear transformations:\n",
    "  $$\n",
    "  Q = XW_q, \\quad K = XW_k, \\quad V = XW_v\n",
    "  $$\n",
    "  where $W_q$, $W_k$, and $W_v$ are learnable weight matrices.\n",
    "  \n",
    "- The **Softmax** operation normalizes the dot-product attention scores, ensuring the attention weights sum up to 1 for each token in the sequence. The softmax operation is performed over the dot products of queries and keys:\n",
    "  $$\n",
    "  \\frac{QK^T}{\\sqrt{d_{qk}}}\n",
    "  $$\n",
    "  where $d_{qk}$ is the dimensionality of the query/key vectors. The scaling by $sqrt{d_{qk}}$ prevents the values from growing too large as the dimensionality increases, which would make the softmax gradients too small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SoftmaxAttention(nn.Module):\n",
    "    def __init__(self, feature_dim, heads):\n",
    "        super(SoftmaxAttention, self).__init__()\n",
    "        self.feature_dim = feature_dim\n",
    "        self.heads = heads\n",
    "        self.query = nn.Linear(feature_dim, feature_dim)\n",
    "        self.key = nn.Linear(feature_dim, feature_dim)\n",
    "        self.value = nn.Linear(feature_dim, feature_dim)\n",
    "    \n",
    "    def forward(self, X):\n",
    "        Q = self.query(X)\n",
    "        K = self.key(X)\n",
    "        V = self.value(X)\n",
    "        \n",
    "        # Classic softmax attention: Eq. (2) in the paper\n",
    "        attention_scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(self.feature_dim)\n",
    "        attention_weights = torch.softmax(attention_scores, dim=-1)\n",
    "        output = torch.matmul(attention_weights, V)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Sigmoid Attention**\n",
    "\n",
    "Replaces the softmax normalization step with an element-wise sigmoid activation function.\n",
    "\n",
    "$$\n",
    "\\text{SigmoidAttn}(X) = \\sigma\\left( \\frac{QK^T}{\\sqrt{d_{qk}}} \\right) V\n",
    "$$\n",
    "\n",
    "- The same **Q**, **K**, and **V** matrices are computed from the input.\n",
    "- Has an element-wise sigmoid function $sigma(x)$ = $\\frac{1}{1 + e^{-x}}$ to the scaled dot products between queries and keys:\n",
    "  $$\n",
    "  \\sigma\\left( \\frac{QK^T}{\\sqrt{d_{qk}}} \\right)\n",
    "  $$\n",
    "  \n",
    "  The sigmoid function outputs values between 0 and 1, but unlike softmax, it does not enforce the attention weights to sum to 1 across the sequence. This introduces a different kind of regularization to the attention mechanism in which the output becomes:\n",
    "\n",
    "$$\n",
    "y_i = \\sum_{j=1}^{n} \\frac{\\exp(\\langle W_q x_i, W_k x_j \\rangle)}{\\exp(\\langle W_q x_i, W_k x_j \\rangle) + n} W_v x_j\n",
    "$$\n",
    "\n",
    "As $n \\to \\infty$, this equation converges to:\n",
    "\n",
    "$$\n",
    "y_i \\rightarrow \\int \\frac{\\exp(\\langle W_q x_i, W_k x \\rangle)}{\\exp(\\langle W_q x_i, W_k x \\rangle)} W_v x d\\mu(x)\n",
    "$$\n",
    "\n",
    "- $y_i$ represents the output for the $i$-th token.\n",
    "- $\\langle W_q x_i, W_k x_j \\rangle$ is the dot product between the query of token $i$ and the key of token $j$, which influences the attention weight assigned to each token.\n",
    "- The term $\\exp(\\langle W_q x_i, W_k x_j \\rangle)$ appears in both the numerator and the denominator, and as the sequence length $n$ becomes very large, this fraction tends towards a constant.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SigmoidAttention(nn.Module):\n",
    "    def __init__(self, feature_dim, heads):\n",
    "        super(SigmoidAttention, self).__init__()\n",
    "        self.feature_dim = feature_dim\n",
    "        self.heads = heads\n",
    "        self.query = nn.Linear(feature_dim, feature_dim)\n",
    "        self.key = nn.Linear(feature_dim, feature_dim)\n",
    "        self.value = nn.Linear(feature_dim, feature_dim)\n",
    "    \n",
    "    def forward(self, X):\n",
    "        Q = self.query(X)\n",
    "        K = self.key(X)\n",
    "        V = self.value(X)\n",
    "        \n",
    "        # Eq. (3) from the Sigmoid Attention Paper\n",
    "        attention_scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(self.feature_dim)\n",
    "        attention_weights = torch.sigmoid(attention_scores)  # Sigmoid replaces softmax\n",
    "        output = torch.matmul(attention_weights, V)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Toy Experiment (Sigmoid VS Softmax Attention)** \n",
    "\n",
    "Here we will train a simplified nanoGPT on shakespare dataset with sigmoid attention and compare performance with softmax for single and multi-head attentions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size: 65 (number of unique characters in dataset)\n",
      "Train data size: 1003854\n",
      "Test data size: 111540\n"
     ]
    }
   ],
   "source": [
    "# Preprocess data\n",
    "with open('data/input.txt', 'r') as f:\n",
    "    text = f.read()\n",
    "\n",
    "# Create a character-level vocabulary\n",
    "chars = sorted(list(set(text)))\n",
    "vocab_size = len(chars)\n",
    "\n",
    "# Create mappings from characters to integers and vice versa\n",
    "stoi = {ch:i for i,ch in enumerate(chars)}\n",
    "itos = {i:ch for i,ch in enumerate(chars)}\n",
    "\n",
    "# Encode the entire text\n",
    "data = torch.tensor([stoi[c] for c in text], dtype=torch.long)\n",
    "\n",
    "# Split into train and test sets (90% train, 10% test)\n",
    "n = int(0.9 * len(data))\n",
    "train_data = data[:n]\n",
    "test_data = data[n:]\n",
    "\n",
    "# Save binaries\n",
    "torch.save(train_data, 'data/train.bin')\n",
    "torch.save(test_data, 'data/test.bin')\n",
    "\n",
    "print(f\"Vocabulary size: {vocab_size} (number of unique characters in dataset)\")\n",
    "print(f\"Train data size: {len(train_data)}\")\n",
    "print(f\"Test data size: {len(test_data)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Simplified nanoGPT**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Builds on Kapathy's nanoGPT\n",
    "import math\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "class SingleHeadAttention(nn.Module):\n",
    "    def __init__(self, head_size, n_embd, block_size, dropout, use_sigmoid=False):\n",
    "        super().__init__()\n",
    "        self.key = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.query = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.value = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.use_sigmoid = use_sigmoid\n",
    "\n",
    "    def forward(self, x):\n",
    "        B,T,C = x.shape\n",
    "        k = self.key(x)   # (B,T,C)\n",
    "        q = self.query(x) # (B,T,C)\n",
    "        # compute attention scores (\"affinities\")\n",
    "        wei = q @ k.transpose(-2,-1) * C**-0.5 # (B, T, T)\n",
    "        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf')) # (B, T, T)\n",
    "        if self.use_sigmoid:\n",
    "            wei = torch.sigmoid(wei)\n",
    "        else:\n",
    "            wei = F.softmax(wei, dim=-1) # (B, T, T)\n",
    "        wei = self.dropout(wei)\n",
    "        # perform the weighted aggregation of the values\n",
    "        v = self.value(x) # (B,T,C)\n",
    "        out = wei @ v # (B, T, C)\n",
    "        return out\n",
    "\n",
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, num_heads, head_size, n_embd, block_size, dropout, use_sigmoid=False):\n",
    "        super().__init__()\n",
    "        self.heads = nn.ModuleList([SingleHeadAttention(head_size, n_embd, block_size, dropout, use_sigmoid) for _ in range(num_heads)])\n",
    "        self.proj = nn.Linear(n_embd, n_embd)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
    "        out = self.dropout(self.proj(out))\n",
    "        return out\n",
    "\n",
    "class FeedForward(nn.Module):\n",
    "    def __init__(self, n_embd, dropout):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(n_embd, 4 * n_embd),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(4 * n_embd, n_embd),\n",
    "            nn.Dropout(dropout),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.net(x)\n",
    "\n",
    "class Block(nn.Module):\n",
    "    def __init__(self, n_embd, n_head, block_size, dropout, use_sigmoid=False):\n",
    "        super().__init__()\n",
    "        head_size = n_embd // n_head\n",
    "        self.sa = MultiHeadAttention(n_head, head_size, n_embd, block_size, dropout, use_sigmoid)\n",
    "        self.ffwd = FeedForward(n_embd, dropout)\n",
    "        self.ln1 = nn.LayerNorm(n_embd)\n",
    "        self.ln2 = nn.LayerNorm(n_embd)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.sa(self.ln1(x))\n",
    "        x = x + self.ffwd(self.ln2(x))\n",
    "        return x\n",
    "\n",
    "class nanoGPT(nn.Module):\n",
    "    def __init__(self, vocab_size, n_embd, block_size, n_head, n_layer, dropout, use_sigmoid=False):\n",
    "        super().__init__()\n",
    "        self.block_size = block_size\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embd)\n",
    "        self.blocks = nn.Sequential(*[Block(n_embd, n_head, block_size, dropout, use_sigmoid) for _ in range(n_layer)])\n",
    "        self.ln_f = nn.LayerNorm(n_embd)\n",
    "        self.lm_head = nn.Linear(n_embd, vocab_size)\n",
    "\n",
    "        self.config = type('Config', (), {\n",
    "            'n_layer': n_layer,\n",
    "            'n_head': n_head,\n",
    "            'n_embd': n_embd,\n",
    "            'block_size': block_size,\n",
    "            'vocab_size': vocab_size,\n",
    "        })\n",
    "\n",
    "    def forward(self, idx, targets=None):\n",
    "        B, T = idx.shape\n",
    "\n",
    "        tok_emb = self.token_embedding_table(idx) # (B,T,C)\n",
    "        pos_emb = self.position_embedding_table(torch.arange(T, device=idx.device)) # (T,C)\n",
    "        x = tok_emb + pos_emb # (B,T,C)\n",
    "        x = self.blocks(x) # (B,T,C)\n",
    "        x = self.ln_f(x) # (B,T,C)\n",
    "        logits = self.lm_head(x) # (B,T,vocab_size)\n",
    "\n",
    "        if targets is None:\n",
    "            loss = None\n",
    "        else:\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B*T, C)\n",
    "            targets = targets.view(B*T)\n",
    "            loss = F.cross_entropy(logits, targets)\n",
    "\n",
    "        return logits, loss\n",
    "    \n",
    "    def estimate_mfu(self, fwdbwd_per_iter, dt):\n",
    "        \"\"\" estimate model flops utilization (MFU) in units of A100 bfloat16 peak FLOPS \"\"\"\n",
    "        # first estimate the number of flops we do per iteration.\n",
    "        # see PaLM paper Appendix B as ref: https://arxiv.org/abs/2204.02311\n",
    "        N = sum(p.numel() for p in self.parameters())\n",
    "        cfg = self.config\n",
    "        L, H, Q, T = cfg.n_layer, cfg.n_head, cfg.n_embd//cfg.n_head, cfg.block_size\n",
    "        flops_per_token = 6*N + 12*L*H*Q*T\n",
    "        flops_per_fwdbwd = flops_per_token * T\n",
    "        flops_per_iter = flops_per_fwdbwd * fwdbwd_per_iter\n",
    "        # express our flops throughput as ratio of A100 bfloat16 peak flops\n",
    "        flops_achieved = flops_per_iter * (1.0/dt) # per second\n",
    "        flops_promised = 312e12 # A100 GPU bfloat16 peak flops is 312 TFLOPS\n",
    "        mfu = flops_achieved / flops_promised\n",
    "        return mfu\n",
    "\n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        for _ in range(max_new_tokens):\n",
    "            idx_cond = idx[:, -self.block_size:]\n",
    "            logits, _ = self(idx_cond)\n",
    "            logits = logits[:, -1, :] # focus only on the last time step\n",
    "            probs = F.softmax(logits, dim=-1) # (B, C)\n",
    "            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)\n",
    "            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)\n",
    "        return idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Train**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparameters\n",
    "batch_size = 12\n",
    "block_size = 64\n",
    "max_iters = 4000\n",
    "eval_interval = 200\n",
    "learning_rate = 3e-4\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "eval_iters = 10\n",
    "n_embd = 128\n",
    "n_head = 4\n",
    "n_layer = 4\n",
    "dropout = 0\n",
    "\n",
    "# Load data\n",
    "train_data = torch.load('data/train.bin', weights_only=False)\n",
    "test_data = torch.load('data/test.bin', weights_only=False)\n",
    "\n",
    "def get_batch(split):\n",
    "    data = train_data if split == 'train' else test_data\n",
    "    ix = torch.randint(len(data) - block_size, (batch_size,))\n",
    "    x = torch.stack([data[i:i+block_size] for i in ix])\n",
    "    y = torch.stack([data[i+1:i+block_size+1] for i in ix])\n",
    "    return x.to(device), y.to(device)\n",
    "\n",
    "@torch.no_grad()\n",
    "def estimate_loss(model):\n",
    "    out = {}\n",
    "    model.eval()\n",
    "    for split in ['train', 'val']:\n",
    "        losses = torch.zeros(eval_iters)\n",
    "        for k in range(eval_iters):\n",
    "            X, Y = get_batch(split)\n",
    "            logits, loss = model(X, Y)\n",
    "            losses[k] = loss.item()\n",
    "        out[split] = losses.mean()\n",
    "    model.train()\n",
    "    return out\n",
    "\n",
    "def train_model(model, model_type):\n",
    "    optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n",
    "    \n",
    "    train_losses = []\n",
    "    val_losses = []\n",
    "    \n",
    "    # For calculating MFU (Model Flops Utilization)\n",
    "    t0 = time.time()\n",
    "    local_iter_num = 0\n",
    "    running_mfu = -1.0\n",
    "    \n",
    "    for iter in range(max_iters):\n",
    "        # every once in a while evaluate the loss on train and val sets\n",
    "        if iter % eval_interval == 0:\n",
    "            losses = estimate_loss(model)\n",
    "            print(f\"step {iter}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}\")\n",
    "            train_losses.append(losses['train'])\n",
    "            val_losses.append(losses['val'])\n",
    "\n",
    "        # sample a batch of data\n",
    "        xb, yb = get_batch('train')\n",
    "\n",
    "        # evaluate the loss\n",
    "        logits, loss = model(xb, yb)\n",
    "        optimizer.zero_grad(set_to_none=True)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        # timing and logging\n",
    "        t1 = time.time()\n",
    "        dt = t1 - t0\n",
    "        t0 = t1\n",
    "        if iter % 10 == 0:\n",
    "            # get loss as float. note: this is a CPU-GPU sync point\n",
    "            lossf = loss.item()\n",
    "            if local_iter_num >= 5: # let the training loop settle a bit\n",
    "                mfu = model.estimate_mfu(batch_size * block_size, dt)\n",
    "                running_mfu = mfu if running_mfu == -1.0 else 0.9*running_mfu + 0.1*mfu\n",
    "            print(f\"iter {iter}: loss {lossf:.4f}, time {dt*1000:.2f}ms, mfu {running_mfu*100:.2f}%\")\n",
    "        local_iter_num += 1\n",
    "        \n",
    "        if iter % 1000 == 0:\n",
    "            print(f\"saving checkpoint to out-{model_type}\")\n",
    "            torch.save(model.state_dict(), f\"checkpoints/out-{model_type}.pt\")\n",
    "\n",
    "    return train_losses, val_losses\n",
    "\n",
    "# Add this method to your GPTLanguageModel class\n",
    "def estimate_mfu(self, fwdbwd_per_iter, dt):\n",
    "    \"\"\" estimate model flops utilization (MFU) in units of A100 bfloat16 peak FLOPS \"\"\"\n",
    "    # first estimate the number of flops we do per iteration.\n",
    "    # see PaLM paper Appendix B as ref: https://arxiv.org/abs/2204.02311\n",
    "    N = sum(p.numel() for p in self.parameters())\n",
    "    cfg = self.config\n",
    "    L, H, Q, T = cfg.n_layer, cfg.n_head, cfg.n_embd//cfg.n_head, cfg.block_size\n",
    "    flops_per_token = 6*N + 12*L*H*Q*T\n",
    "    flops_per_fwdbwd = flops_per_token * T\n",
    "    flops_per_iter = flops_per_fwdbwd * fwdbwd_per_iter\n",
    "    # express our flops throughput as ratio of A100 bfloat16 peak flops\n",
    "    flops_achieved = flops_per_iter * (1.0/dt) # per second\n",
    "    flops_promised = 312e12 # A100 GPU bfloat16 peak flops is 312 TFLOPS\n",
    "    mfu = flops_achieved / flops_promised\n",
    "    return mfu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training With Softmax Attention\n",
      "step 0: train loss 4.3226, val loss 4.3094\n",
      "iter 0: loss 4.3172, time 262.85ms, mfu -100.00%\n",
      "saving checkpoint to out-Sigmoid\n",
      "iter 10: loss 3.3473, time 25.24ms, mfu 3.30%\n",
      "iter 20: loss 3.4084, time 28.34ms, mfu 3.27%\n",
      "iter 30: loss 3.3309, time 24.76ms, mfu 3.28%\n",
      "iter 40: loss 3.2824, time 26.12ms, mfu 3.27%\n",
      "iter 50: loss 3.2582, time 27.85ms, mfu 3.24%\n",
      "iter 60: loss 3.2619, time 25.72ms, mfu 3.24%\n",
      "iter 70: loss 3.3017, time 28.23ms, mfu 3.21%\n",
      "iter 80: loss 3.0144, time 28.18ms, mfu 3.19%\n",
      "iter 90: loss 3.0160, time 27.98ms, mfu 3.17%\n",
      "iter 100: loss 2.8799, time 25.85ms, mfu 3.17%\n",
      "iter 110: loss 2.7632, time 32.00ms, mfu 3.12%\n",
      "iter 120: loss 2.7708, time 25.46ms, mfu 3.13%\n",
      "iter 130: loss 2.7243, time 26.75ms, mfu 3.13%\n",
      "iter 140: loss 2.7691, time 28.18ms, mfu 3.11%\n",
      "iter 150: loss 2.6791, time 27.94ms, mfu 3.10%\n",
      "iter 160: loss 2.6737, time 24.35ms, mfu 3.13%\n",
      "iter 170: loss 2.6021, time 27.21ms, mfu 3.13%\n",
      "iter 180: loss 2.5726, time 28.32ms, mfu 3.11%\n",
      "iter 190: loss 2.5256, time 53.68ms, mfu 2.95%\n",
      "step 200: train loss 2.5902, val loss 2.5980\n",
      "iter 200: loss 2.5257, time 170.39ms, mfu 2.71%\n",
      "iter 210: loss 2.6218, time 25.57ms, mfu 2.76%\n",
      "iter 220: loss 2.5376, time 27.02ms, mfu 2.79%\n",
      "iter 230: loss 2.6263, time 31.38ms, mfu 2.78%\n",
      "iter 240: loss 2.6048, time 24.95ms, mfu 2.84%\n",
      "iter 250: loss 2.5839, time 25.07ms, mfu 2.89%\n",
      "iter 260: loss 2.5757, time 28.36ms, mfu 2.89%\n",
      "iter 270: loss 2.6142, time 26.76ms, mfu 2.91%\n",
      "iter 280: loss 2.5902, time 23.06ms, mfu 2.98%\n",
      "iter 290: loss 2.5937, time 22.74ms, mfu 3.05%\n",
      "iter 300: loss 2.5163, time 25.10ms, mfu 3.08%\n",
      "iter 310: loss 2.5262, time 23.63ms, mfu 3.12%\n",
      "iter 320: loss 2.5241, time 24.73ms, mfu 3.15%\n",
      "iter 330: loss 2.5010, time 23.88ms, mfu 3.18%\n",
      "iter 340: loss 2.5050, time 26.92ms, mfu 3.17%\n",
      "iter 350: loss 2.5954, time 27.49ms, mfu 3.16%\n",
      "iter 360: loss 2.5613, time 35.27ms, mfu 3.08%\n",
      "iter 370: loss 2.4880, time 24.37ms, mfu 3.12%\n",
      "iter 380: loss 2.5149, time 22.85ms, mfu 3.17%\n",
      "iter 390: loss 2.4694, time 25.68ms, mfu 3.18%\n",
      "step 400: train loss 2.4699, val loss 2.4795\n",
      "iter 400: loss 2.5280, time 158.10ms, mfu 2.91%\n",
      "iter 410: loss 2.4329, time 25.06ms, mfu 2.95%\n",
      "iter 420: loss 2.4130, time 24.84ms, mfu 2.99%\n",
      "iter 430: loss 2.4655, time 29.47ms, mfu 2.98%\n",
      "iter 440: loss 2.4629, time 26.73ms, mfu 2.99%\n",
      "iter 450: loss 2.4349, time 38.67ms, mfu 2.91%\n",
      "iter 460: loss 2.4804, time 26.03ms, mfu 2.94%\n",
      "iter 470: loss 2.5630, time 25.15ms, mfu 2.98%\n",
      "iter 480: loss 2.4614, time 25.41ms, mfu 3.01%\n",
      "iter 490: loss 2.4843, time 31.62ms, mfu 2.97%\n",
      "iter 500: loss 2.4021, time 42.23ms, mfu 2.87%\n",
      "iter 510: loss 2.4886, time 28.46ms, mfu 2.88%\n",
      "iter 520: loss 2.4664, time 22.17ms, mfu 2.96%\n",
      "iter 530: loss 2.3926, time 24.38ms, mfu 3.01%\n",
      "iter 540: loss 2.3807, time 30.04ms, mfu 2.99%\n",
      "iter 550: loss 2.4407, time 28.13ms, mfu 2.98%\n",
      "iter 560: loss 2.5629, time 25.53ms, mfu 3.01%\n",
      "iter 570: loss 2.4045, time 27.46ms, mfu 3.02%\n",
      "iter 580: loss 2.4285, time 28.32ms, mfu 3.01%\n",
      "iter 590: loss 2.3836, time 33.35ms, mfu 2.96%\n",
      "step 600: train loss 2.3777, val loss 2.3975\n",
      "iter 600: loss 2.3988, time 173.04ms, mfu 2.71%\n",
      "iter 610: loss 2.2855, time 25.12ms, mfu 2.77%\n",
      "iter 620: loss 2.3899, time 28.40ms, mfu 2.79%\n",
      "iter 630: loss 2.3603, time 51.20ms, mfu 2.67%\n",
      "iter 640: loss 2.4583, time 56.92ms, mfu 2.55%\n",
      "iter 650: loss 2.3647, time 27.00ms, mfu 2.60%\n",
      "iter 660: loss 2.3322, time 22.74ms, mfu 2.71%\n",
      "iter 670: loss 2.2891, time 23.09ms, mfu 2.80%\n",
      "iter 680: loss 2.3740, time 23.97ms, mfu 2.87%\n",
      "iter 690: loss 2.2897, time 30.01ms, mfu 2.86%\n",
      "iter 700: loss 2.5036, time 24.80ms, mfu 2.91%\n",
      "iter 710: loss 2.3511, time 27.89ms, mfu 2.92%\n",
      "iter 720: loss 2.3599, time 22.75ms, mfu 2.99%\n",
      "iter 730: loss 2.3270, time 24.30ms, mfu 3.04%\n",
      "iter 740: loss 2.4020, time 24.39ms, mfu 3.07%\n",
      "iter 750: loss 2.3568, time 28.11ms, mfu 3.06%\n",
      "iter 760: loss 2.3602, time 23.18ms, mfu 3.12%\n",
      "iter 770: loss 2.3207, time 22.51ms, mfu 3.18%\n",
      "iter 780: loss 2.3178, time 31.32ms, mfu 3.12%\n",
      "iter 790: loss 2.4127, time 22.64ms, mfu 3.18%\n",
      "step 800: train loss 2.3404, val loss 2.3276\n",
      "iter 800: loss 2.3808, time 148.55ms, mfu 2.92%\n",
      "iter 810: loss 2.3442, time 25.15ms, mfu 2.96%\n",
      "iter 820: loss 2.3759, time 25.68ms, mfu 2.99%\n",
      "iter 830: loss 2.2842, time 25.78ms, mfu 3.01%\n",
      "iter 840: loss 2.2858, time 23.08ms, mfu 3.07%\n",
      "iter 850: loss 2.2982, time 23.95ms, mfu 3.11%\n",
      "iter 860: loss 2.3719, time 22.49ms, mfu 3.17%\n",
      "iter 870: loss 2.3022, time 27.97ms, mfu 3.15%\n",
      "iter 880: loss 2.3601, time 26.41ms, mfu 3.15%\n",
      "iter 890: loss 2.2896, time 31.77ms, mfu 3.10%\n",
      "iter 900: loss 2.3511, time 30.49ms, mfu 3.06%\n",
      "iter 910: loss 2.2663, time 25.18ms, mfu 3.09%\n",
      "iter 920: loss 2.3333, time 25.15ms, mfu 3.11%\n",
      "iter 930: loss 2.3115, time 22.18ms, mfu 3.18%\n",
      "iter 940: loss 2.3354, time 26.52ms, mfu 3.17%\n",
      "iter 950: loss 2.3744, time 35.45ms, mfu 3.09%\n",
      "iter 960: loss 2.2907, time 27.27ms, mfu 3.09%\n",
      "iter 970: loss 2.3438, time 21.84ms, mfu 3.16%\n",
      "iter 980: loss 2.3191, time 23.47ms, mfu 3.20%\n",
      "iter 990: loss 2.3655, time 28.12ms, mfu 3.18%\n",
      "step 1000: train loss 2.2707, val loss 2.3235\n",
      "iter 1000: loss 2.2518, time 143.46ms, mfu 2.92%\n",
      "saving checkpoint to out-Sigmoid\n",
      "iter 1010: loss 2.2429, time 23.40ms, mfu 2.98%\n",
      "iter 1020: loss 2.2258, time 23.26ms, mfu 3.04%\n",
      "iter 1030: loss 2.3261, time 22.76ms, mfu 3.10%\n",
      "iter 1040: loss 2.3652, time 24.43ms, mfu 3.14%\n",
      "iter 1050: loss 2.2776, time 35.30ms, mfu 3.06%\n",
      "iter 1060: loss 2.2569, time 30.21ms, mfu 3.03%\n",
      "iter 1070: loss 2.3559, time 31.78ms, mfu 2.99%\n",
      "iter 1080: loss 2.2512, time 29.36ms, mfu 2.97%\n",
      "iter 1090: loss 2.1863, time 24.94ms, mfu 3.01%\n",
      "iter 1100: loss 2.1953, time 35.10ms, mfu 2.95%\n",
      "iter 1110: loss 2.2276, time 23.97ms, mfu 3.00%\n",
      "iter 1120: loss 2.2937, time 26.13ms, mfu 3.02%\n",
      "iter 1130: loss 2.1913, time 24.14ms, mfu 3.06%\n",
      "iter 1140: loss 2.3076, time 22.98ms, mfu 3.12%\n",
      "iter 1150: loss 2.1957, time 26.55ms, mfu 3.12%\n",
      "iter 1160: loss 2.2338, time 23.85ms, mfu 3.16%\n",
      "iter 1170: loss 2.3138, time 24.08ms, mfu 3.19%\n",
      "iter 1180: loss 2.3667, time 24.13ms, mfu 3.22%\n",
      "iter 1190: loss 2.2909, time 24.19ms, mfu 3.24%\n",
      "step 1200: train loss 2.2548, val loss 2.2703\n",
      "iter 1200: loss 2.2598, time 148.31ms, mfu 2.97%\n",
      "iter 1210: loss 2.2137, time 24.53ms, mfu 3.01%\n",
      "iter 1220: loss 2.2446, time 28.92ms, mfu 3.00%\n",
      "iter 1230: loss 2.2674, time 43.07ms, mfu 2.89%\n",
      "iter 1240: loss 2.2134, time 27.74ms, mfu 2.91%\n",
      "iter 1250: loss 2.3232, time 28.74ms, mfu 2.91%\n",
      "iter 1260: loss 2.3098, time 26.55ms, mfu 2.93%\n",
      "iter 1270: loss 2.2504, time 25.35ms, mfu 2.97%\n",
      "iter 1280: loss 2.2860, time 23.05ms, mfu 3.03%\n",
      "iter 1290: loss 2.2920, time 26.78ms, mfu 3.04%\n",
      "iter 1300: loss 2.2187, time 22.71ms, mfu 3.10%\n",
      "iter 1310: loss 2.2200, time 25.85ms, mfu 3.11%\n",
      "iter 1320: loss 2.2360, time 24.00ms, mfu 3.15%\n",
      "iter 1330: loss 2.2535, time 23.14ms, mfu 3.20%\n",
      "iter 1340: loss 2.2043, time 23.52ms, mfu 3.23%\n",
      "iter 1350: loss 2.2031, time 26.02ms, mfu 3.23%\n",
      "iter 1360: loss 2.1844, time 23.05ms, mfu 3.27%\n",
      "iter 1370: loss 2.2339, time 24.74ms, mfu 3.28%\n",
      "iter 1380: loss 2.1863, time 24.22ms, mfu 3.29%\n",
      "iter 1390: loss 2.2482, time 23.30ms, mfu 3.32%\n",
      "step 1400: train loss 2.2017, val loss 2.2763\n",
      "iter 1400: loss 2.3428, time 152.07ms, mfu 3.05%\n",
      "iter 1410: loss 2.2441, time 31.02ms, mfu 3.01%\n",
      "iter 1420: loss 2.2731, time 24.29ms, mfu 3.05%\n",
      "iter 1430: loss 2.2026, time 24.79ms, mfu 3.08%\n",
      "iter 1440: loss 2.1181, time 24.00ms, mfu 3.12%\n",
      "iter 1450: loss 2.3028, time 24.10ms, mfu 3.16%\n",
      "iter 1460: loss 2.2221, time 23.80ms, mfu 3.19%\n",
      "iter 1470: loss 2.2585, time 23.25ms, mfu 3.23%\n",
      "iter 1480: loss 2.2300, time 24.84ms, mfu 3.24%\n",
      "iter 1490: loss 2.2417, time 22.83ms, mfu 3.28%\n",
      "iter 1500: loss 2.2559, time 24.25ms, mfu 3.30%\n",
      "iter 1510: loss 2.2366, time 24.19ms, mfu 3.31%\n",
      "iter 1520: loss 2.2224, time 22.62ms, mfu 3.35%\n",
      "iter 1530: loss 2.2185, time 21.53ms, mfu 3.40%\n",
      "iter 1540: loss 2.2402, time 24.67ms, mfu 3.40%\n",
      "iter 1550: loss 2.2490, time 28.40ms, mfu 3.36%\n",
      "iter 1560: loss 2.3266, time 25.94ms, mfu 3.34%\n",
      "iter 1570: loss 2.1948, time 22.59ms, mfu 3.38%\n",
      "iter 1580: loss 2.1803, time 35.40ms, mfu 3.27%\n",
      "iter 1590: loss 2.1782, time 24.56ms, mfu 3.29%\n",
      "step 1600: train loss 2.1896, val loss 2.2046\n",
      "iter 1600: loss 2.1763, time 159.70ms, mfu 3.01%\n",
      "iter 1610: loss 2.1800, time 24.43ms, mfu 3.05%\n",
      "iter 1620: loss 2.1691, time 23.14ms, mfu 3.11%\n",
      "iter 1630: loss 2.1317, time 26.32ms, mfu 3.11%\n",
      "iter 1640: loss 2.1375, time 22.59ms, mfu 3.17%\n",
      "iter 1650: loss 2.2357, time 23.58ms, mfu 3.21%\n",
      "iter 1660: loss 2.1944, time 21.53ms, mfu 3.27%\n",
      "iter 1670: loss 2.2606, time 22.34ms, mfu 3.32%\n",
      "iter 1680: loss 2.1346, time 23.67ms, mfu 3.34%\n",
      "iter 1690: loss 2.1717, time 24.30ms, mfu 3.35%\n",
      "iter 1700: loss 2.1804, time 27.96ms, mfu 3.31%\n",
      "iter 1710: loss 2.1973, time 22.86ms, mfu 3.35%\n",
      "iter 1720: loss 2.1343, time 27.99ms, mfu 3.31%\n",
      "iter 1730: loss 2.1953, time 24.38ms, mfu 3.32%\n",
      "iter 1740: loss 2.1829, time 22.91ms, mfu 3.35%\n",
      "iter 1750: loss 2.1107, time 24.29ms, mfu 3.36%\n",
      "iter 1760: loss 2.1982, time 24.83ms, mfu 3.36%\n",
      "iter 1770: loss 2.1439, time 22.21ms, mfu 3.40%\n",
      "iter 1780: loss 2.1196, time 23.50ms, mfu 3.41%\n",
      "iter 1790: loss 2.1939, time 24.18ms, mfu 3.42%\n",
      "step 1800: train loss 2.1413, val loss 2.1559\n",
      "iter 1800: loss 2.1059, time 153.90ms, mfu 3.13%\n",
      "iter 1810: loss 2.2066, time 22.12ms, mfu 3.19%\n",
      "iter 1820: loss 2.1558, time 25.43ms, mfu 3.20%\n",
      "iter 1830: loss 2.2252, time 25.06ms, mfu 3.22%\n",
      "iter 1840: loss 2.0804, time 23.15ms, mfu 3.25%\n",
      "iter 1850: loss 2.0857, time 25.34ms, mfu 3.26%\n",
      "iter 1860: loss 2.0531, time 22.13ms, mfu 3.31%\n",
      "iter 1870: loss 2.1912, time 24.88ms, mfu 3.31%\n",
      "iter 1880: loss 2.1898, time 25.50ms, mfu 3.31%\n",
      "iter 1890: loss 2.0833, time 25.37ms, mfu 3.31%\n",
      "iter 1900: loss 2.2052, time 25.56ms, mfu 3.30%\n",
      "iter 1910: loss 2.1543, time 24.15ms, mfu 3.32%\n",
      "iter 1920: loss 2.2195, time 25.35ms, mfu 3.31%\n",
      "iter 1930: loss 2.1515, time 36.20ms, mfu 3.21%\n",
      "iter 1940: loss 2.1542, time 24.10ms, mfu 3.24%\n",
      "iter 1950: loss 2.0682, time 22.89ms, mfu 3.28%\n",
      "iter 1960: loss 2.1911, time 26.43ms, mfu 3.27%\n",
      "iter 1970: loss 2.1634, time 24.51ms, mfu 3.28%\n",
      "iter 1980: loss 2.1345, time 25.09ms, mfu 3.28%\n",
      "iter 1990: loss 2.1651, time 21.97ms, mfu 3.34%\n",
      "step 2000: train loss 2.1275, val loss 2.1425\n",
      "iter 2000: loss 2.1420, time 154.84ms, mfu 3.06%\n",
      "saving checkpoint to out-Sigmoid\n",
      "iter 2010: loss 2.0798, time 23.89ms, mfu 3.10%\n",
      "iter 2020: loss 2.1161, time 22.64ms, mfu 3.16%\n",
      "iter 2030: loss 1.9933, time 22.37ms, mfu 3.21%\n",
      "iter 2040: loss 2.1001, time 24.98ms, mfu 3.23%\n",
      "iter 2050: loss 2.0842, time 23.82ms, mfu 3.25%\n",
      "iter 2060: loss 2.1492, time 24.67ms, mfu 3.27%\n",
      "iter 2070: loss 2.1572, time 21.84ms, mfu 3.32%\n",
      "iter 2080: loss 2.0577, time 24.83ms, mfu 3.33%\n",
      "iter 2090: loss 2.0748, time 23.90ms, mfu 3.34%\n",
      "iter 2100: loss 2.0750, time 23.57ms, mfu 3.36%\n",
      "iter 2110: loss 2.0594, time 24.68ms, mfu 3.36%\n",
      "iter 2120: loss 2.1840, time 24.32ms, mfu 3.37%\n",
      "iter 2130: loss 2.0188, time 22.99ms, mfu 3.40%\n",
      "iter 2140: loss 2.1059, time 24.50ms, mfu 3.40%\n",
      "iter 2150: loss 1.9913, time 22.68ms, mfu 3.42%\n",
      "iter 2160: loss 2.0566, time 27.10ms, mfu 3.39%\n",
      "iter 2170: loss 2.0442, time 29.49ms, mfu 3.33%\n",
      "iter 2180: loss 2.0254, time 25.57ms, mfu 3.33%\n",
      "iter 2190: loss 2.0792, time 23.10ms, mfu 3.35%\n",
      "step 2200: train loss 2.0603, val loss 2.1254\n",
      "iter 2200: loss 1.9614, time 149.80ms, mfu 3.07%\n",
      "iter 2210: loss 2.0357, time 22.83ms, mfu 3.13%\n",
      "iter 2220: loss 2.2227, time 23.05ms, mfu 3.18%\n",
      "iter 2230: loss 2.0791, time 23.94ms, mfu 3.21%\n",
      "iter 2240: loss 1.9548, time 23.06ms, mfu 3.25%\n",
      "iter 2250: loss 2.1372, time 24.40ms, mfu 3.27%\n",
      "iter 2260: loss 1.9133, time 23.23ms, mfu 3.30%\n",
      "iter 2270: loss 2.0311, time 23.95ms, mfu 3.32%\n",
      "iter 2280: loss 2.1594, time 24.42ms, mfu 3.33%\n",
      "iter 2290: loss 2.0897, time 23.78ms, mfu 3.35%\n",
      "iter 2300: loss 1.9691, time 22.51ms, mfu 3.38%\n",
      "iter 2310: loss 2.0847, time 24.79ms, mfu 3.38%\n",
      "iter 2320: loss 2.1390, time 21.16ms, mfu 3.44%\n",
      "iter 2330: loss 2.0788, time 28.17ms, mfu 3.39%\n",
      "iter 2340: loss 2.0591, time 24.05ms, mfu 3.40%\n",
      "iter 2350: loss 1.9822, time 23.33ms, mfu 3.41%\n",
      "iter 2360: loss 2.0305, time 23.32ms, mfu 3.43%\n",
      "iter 2370: loss 2.0952, time 25.00ms, mfu 3.42%\n",
      "iter 2380: loss 1.9171, time 23.27ms, mfu 3.44%\n",
      "iter 2390: loss 2.1647, time 23.71ms, mfu 3.45%\n",
      "step 2400: train loss 2.0644, val loss 2.1127\n",
      "iter 2400: loss 1.9811, time 154.79ms, mfu 3.15%\n",
      "iter 2410: loss 2.0353, time 22.50ms, mfu 3.21%\n",
      "iter 2420: loss 2.0798, time 23.34ms, mfu 3.25%\n",
      "iter 2430: loss 1.9777, time 22.35ms, mfu 3.29%\n",
      "iter 2440: loss 2.0923, time 23.47ms, mfu 3.32%\n",
      "iter 2450: loss 2.1456, time 23.08ms, mfu 3.35%\n",
      "iter 2460: loss 2.0964, time 23.34ms, mfu 3.37%\n",
      "iter 2470: loss 2.0219, time 23.29ms, mfu 3.39%\n",
      "iter 2480: loss 2.0664, time 21.88ms, mfu 3.43%\n",
      "iter 2490: loss 2.0860, time 23.97ms, mfu 3.44%\n",
      "iter 2500: loss 2.0947, time 24.37ms, mfu 3.44%\n",
      "iter 2510: loss 2.0625, time 22.42ms, mfu 3.47%\n",
      "iter 2520: loss 2.0141, time 24.72ms, mfu 3.46%\n",
      "iter 2530: loss 2.1300, time 23.35ms, mfu 3.47%\n",
      "iter 2540: loss 2.0434, time 24.94ms, mfu 3.46%\n",
      "iter 2550: loss 2.0309, time 24.62ms, mfu 3.45%\n",
      "iter 2560: loss 1.9872, time 22.02ms, mfu 3.48%\n",
      "iter 2570: loss 1.9981, time 24.54ms, mfu 3.47%\n",
      "iter 2580: loss 1.9794, time 24.61ms, mfu 3.47%\n",
      "iter 2590: loss 2.0552, time 22.57ms, mfu 3.49%\n",
      "step 2600: train loss 2.0418, val loss 2.0649\n",
      "iter 2600: loss 2.0730, time 153.87ms, mfu 3.19%\n",
      "iter 2610: loss 1.9857, time 21.31ms, mfu 3.27%\n",
      "iter 2620: loss 1.9560, time 23.71ms, mfu 3.29%\n",
      "iter 2630: loss 2.0037, time 24.86ms, mfu 3.30%\n",
      "iter 2640: loss 1.8806, time 23.07ms, mfu 3.33%\n",
      "iter 2650: loss 2.0675, time 24.85ms, mfu 3.33%\n",
      "iter 2660: loss 1.9892, time 23.71ms, mfu 3.35%\n",
      "iter 2670: loss 2.0036, time 23.13ms, mfu 3.38%\n",
      "iter 2680: loss 2.0326, time 24.51ms, mfu 3.38%\n",
      "iter 2690: loss 1.9123, time 24.98ms, mfu 3.37%\n",
      "iter 2700: loss 2.0747, time 23.66ms, mfu 3.39%\n",
      "iter 2710: loss 2.0315, time 25.18ms, mfu 3.38%\n",
      "iter 2720: loss 2.0847, time 22.65ms, mfu 3.41%\n",
      "iter 2730: loss 2.0297, time 21.36ms, mfu 3.46%\n",
      "iter 2740: loss 1.9555, time 23.16ms, mfu 3.47%\n",
      "iter 2750: loss 1.9419, time 24.08ms, mfu 3.47%\n",
      "iter 2760: loss 1.8974, time 23.05ms, mfu 3.49%\n",
      "iter 2770: loss 1.9025, time 24.14ms, mfu 3.48%\n",
      "iter 2780: loss 1.9816, time 22.67ms, mfu 3.50%\n",
      "iter 2790: loss 1.9584, time 24.16ms, mfu 3.50%\n",
      "step 2800: train loss 1.9953, val loss 2.0530\n",
      "iter 2800: loss 1.9913, time 146.97ms, mfu 3.21%\n",
      "iter 2810: loss 2.0350, time 22.90ms, mfu 3.25%\n",
      "iter 2820: loss 2.0051, time 27.72ms, mfu 3.23%\n",
      "iter 2830: loss 2.0480, time 22.77ms, mfu 3.27%\n",
      "iter 2840: loss 2.0134, time 32.25ms, mfu 3.20%\n",
      "iter 2850: loss 1.8805, time 23.59ms, mfu 3.23%\n",
      "iter 2860: loss 1.8494, time 22.75ms, mfu 3.28%\n",
      "iter 2870: loss 1.8683, time 23.78ms, mfu 3.30%\n",
      "iter 2880: loss 1.9940, time 23.38ms, mfu 3.33%\n",
      "iter 2890: loss 1.9920, time 33.83ms, mfu 3.24%\n",
      "iter 2900: loss 2.1345, time 25.36ms, mfu 3.25%\n",
      "iter 2910: loss 1.9458, time 23.39ms, mfu 3.28%\n",
      "iter 2920: loss 2.0046, time 22.47ms, mfu 3.32%\n",
      "iter 2930: loss 1.9497, time 22.27ms, mfu 3.36%\n",
      "iter 2940: loss 1.9885, time 22.17ms, mfu 3.40%\n",
      "iter 2950: loss 1.9566, time 24.39ms, mfu 3.40%\n",
      "iter 2960: loss 2.0770, time 26.87ms, mfu 3.37%\n",
      "iter 2970: loss 1.9930, time 22.47ms, mfu 3.41%\n",
      "iter 2980: loss 1.9611, time 22.40ms, mfu 3.44%\n",
      "iter 2990: loss 1.8972, time 22.88ms, mfu 3.46%\n",
      "step 3000: train loss 1.9804, val loss 2.0570\n",
      "iter 3000: loss 1.9452, time 150.30ms, mfu 3.17%\n",
      "saving checkpoint to out-Sigmoid\n",
      "iter 3010: loss 1.9366, time 24.51ms, mfu 3.19%\n",
      "iter 3020: loss 1.9623, time 23.27ms, mfu 3.23%\n",
      "iter 3030: loss 1.9444, time 24.02ms, mfu 3.26%\n",
      "iter 3040: loss 1.9000, time 22.16ms, mfu 3.31%\n",
      "iter 3050: loss 1.9925, time 24.00ms, mfu 3.32%\n",
      "iter 3060: loss 1.9031, time 23.34ms, mfu 3.35%\n",
      "iter 3070: loss 1.9973, time 23.64ms, mfu 3.37%\n",
      "iter 3080: loss 1.8801, time 24.29ms, mfu 3.37%\n",
      "iter 3090: loss 1.9332, time 24.53ms, mfu 3.38%\n",
      "iter 3100: loss 1.8817, time 21.83ms, mfu 3.42%\n",
      "iter 3110: loss 1.8959, time 23.60ms, mfu 3.43%\n",
      "iter 3120: loss 1.8883, time 23.65ms, mfu 3.44%\n",
      "iter 3130: loss 1.9385, time 23.93ms, mfu 3.45%\n",
      "iter 3140: loss 1.9174, time 25.15ms, mfu 3.43%\n",
      "iter 3150: loss 1.9360, time 24.14ms, mfu 3.43%\n",
      "iter 3160: loss 1.9606, time 23.39ms, mfu 3.45%\n",
      "iter 3170: loss 1.8992, time 22.83ms, mfu 3.47%\n",
      "iter 3180: loss 2.0255, time 24.65ms, mfu 3.46%\n",
      "iter 3190: loss 2.0877, time 26.02ms, mfu 3.43%\n",
      "step 3200: train loss 1.9352, val loss 1.9947\n",
      "iter 3200: loss 1.9053, time 153.25ms, mfu 3.15%\n",
      "iter 3210: loss 1.9112, time 21.99ms, mfu 3.21%\n",
      "iter 3220: loss 1.9095, time 23.87ms, mfu 3.24%\n",
      "iter 3230: loss 1.8544, time 25.87ms, mfu 3.24%\n",
      "iter 3240: loss 1.9133, time 21.89ms, mfu 3.29%\n",
      "iter 3250: loss 1.8271, time 27.78ms, mfu 3.26%\n",
      "iter 3260: loss 1.8568, time 24.51ms, mfu 3.28%\n",
      "iter 3270: loss 2.0319, time 23.23ms, mfu 3.31%\n",
      "iter 3280: loss 1.8724, time 22.87ms, mfu 3.34%\n",
      "iter 3290: loss 1.8851, time 24.22ms, mfu 3.35%\n",
      "iter 3300: loss 1.8676, time 22.84ms, mfu 3.38%\n",
      "iter 3310: loss 1.9361, time 24.93ms, mfu 3.38%\n",
      "iter 3320: loss 1.9860, time 22.41ms, mfu 3.41%\n",
      "iter 3330: loss 1.9112, time 24.12ms, mfu 3.42%\n",
      "iter 3340: loss 1.8698, time 22.88ms, mfu 3.44%\n",
      "iter 3350: loss 1.9281, time 25.36ms, mfu 3.43%\n",
      "iter 3360: loss 1.9873, time 23.59ms, mfu 3.44%\n",
      "iter 3370: loss 2.0249, time 23.15ms, mfu 3.45%\n",
      "iter 3380: loss 1.9100, time 23.98ms, mfu 3.46%\n",
      "iter 3390: loss 1.8017, time 25.49ms, mfu 3.44%\n",
      "step 3400: train loss 1.8883, val loss 1.9893\n",
      "iter 3400: loss 1.8637, time 144.55ms, mfu 3.15%\n",
      "iter 3410: loss 1.9503, time 21.60ms, mfu 3.22%\n",
      "iter 3420: loss 1.9190, time 24.43ms, mfu 3.24%\n",
      "iter 3430: loss 1.8747, time 23.98ms, mfu 3.26%\n",
      "iter 3440: loss 1.9887, time 23.74ms, mfu 3.29%\n",
      "iter 3450: loss 1.7768, time 24.91ms, mfu 3.30%\n",
      "iter 3460: loss 1.8468, time 23.57ms, mfu 3.32%\n",
      "iter 3470: loss 1.8768, time 24.32ms, mfu 3.33%\n",
      "iter 3480: loss 1.8429, time 26.27ms, mfu 3.31%\n",
      "iter 3490: loss 1.9548, time 26.34ms, mfu 3.30%\n",
      "iter 3500: loss 2.0404, time 24.45ms, mfu 3.31%\n",
      "iter 3510: loss 1.9327, time 22.90ms, mfu 3.34%\n",
      "iter 3520: loss 1.9660, time 24.38ms, mfu 3.35%\n",
      "iter 3530: loss 1.8850, time 25.27ms, mfu 3.35%\n",
      "iter 3540: loss 1.8944, time 25.97ms, mfu 3.33%\n",
      "iter 3550: loss 1.7721, time 24.89ms, mfu 3.33%\n",
      "iter 3560: loss 1.8245, time 22.55ms, mfu 3.37%\n",
      "iter 3570: loss 1.9509, time 25.08ms, mfu 3.37%\n",
      "iter 3580: loss 1.9548, time 25.27ms, mfu 3.36%\n",
      "iter 3590: loss 1.9681, time 23.90ms, mfu 3.37%\n",
      "step 3600: train loss 1.9156, val loss 2.0159\n",
      "iter 3600: loss 1.8868, time 154.55ms, mfu 3.09%\n",
      "iter 3610: loss 1.8707, time 21.01ms, mfu 3.18%\n",
      "iter 3620: loss 1.9427, time 23.69ms, mfu 3.21%\n",
      "iter 3630: loss 2.0060, time 24.03ms, mfu 3.24%\n",
      "iter 3640: loss 1.8739, time 24.08ms, mfu 3.26%\n",
      "iter 3650: loss 1.8961, time 24.11ms, mfu 3.28%\n",
      "iter 3660: loss 1.8675, time 22.46ms, mfu 3.32%\n",
      "iter 3670: loss 1.8083, time 23.07ms, mfu 3.35%\n",
      "iter 3680: loss 1.8284, time 23.63ms, mfu 3.37%\n",
      "iter 3690: loss 1.9482, time 21.72ms, mfu 3.42%\n",
      "iter 3700: loss 1.7525, time 23.06ms, mfu 3.44%\n",
      "iter 3710: loss 1.8998, time 24.91ms, mfu 3.43%\n",
      "iter 3720: loss 1.8243, time 25.76ms, mfu 3.41%\n",
      "iter 3730: loss 1.9081, time 23.03ms, mfu 3.43%\n",
      "iter 3740: loss 1.8794, time 24.91ms, mfu 3.42%\n",
      "iter 3750: loss 1.8623, time 30.38ms, mfu 3.35%\n",
      "iter 3760: loss 1.8576, time 24.64ms, mfu 3.36%\n",
      "iter 3770: loss 1.8817, time 24.17ms, mfu 3.37%\n",
      "iter 3780: loss 1.8939, time 23.17ms, mfu 3.39%\n",
      "iter 3790: loss 1.8857, time 23.02ms, mfu 3.41%\n",
      "step 3800: train loss 1.8441, val loss 1.9590\n",
      "iter 3800: loss 1.8418, time 145.95ms, mfu 3.13%\n",
      "iter 3810: loss 1.8162, time 22.06ms, mfu 3.19%\n",
      "iter 3820: loss 1.7522, time 23.75ms, mfu 3.23%\n",
      "iter 3830: loss 1.8651, time 24.06ms, mfu 3.25%\n",
      "iter 3840: loss 1.7765, time 24.48ms, mfu 3.27%\n",
      "iter 3850: loss 1.8979, time 25.55ms, mfu 3.27%\n",
      "iter 3860: loss 1.8274, time 44.91ms, mfu 3.12%\n",
      "iter 3870: loss 1.8143, time 26.49ms, mfu 3.13%\n",
      "iter 3880: loss 1.8178, time 25.97ms, mfu 3.14%\n",
      "iter 3890: loss 1.8281, time 29.35ms, mfu 3.11%\n",
      "iter 3900: loss 1.8757, time 26.83ms, mfu 3.11%\n",
      "iter 3910: loss 1.7947, time 28.09ms, mfu 3.09%\n",
      "iter 3920: loss 1.8659, time 24.48ms, mfu 3.12%\n",
      "iter 3930: loss 1.8951, time 29.60ms, mfu 3.09%\n",
      "iter 3940: loss 1.8413, time 25.30ms, mfu 3.11%\n",
      "iter 3950: loss 1.8269, time 25.46ms, mfu 3.13%\n",
      "iter 3960: loss 1.9514, time 26.85ms, mfu 3.13%\n",
      "iter 3970: loss 1.8283, time 26.97ms, mfu 3.12%\n",
      "iter 3980: loss 1.8012, time 32.33ms, mfu 3.07%\n",
      "iter 3990: loss 1.7146, time 24.19ms, mfu 3.11%\n"
     ]
    }
   ],
   "source": [
    "# Train with sigmoid attention\n",
    "print(\"Training With Softmax Attention\")\n",
    "model_sigmoid = nanoGPT(vocab_size, n_embd, block_size, n_head, n_layer, dropout, use_sigmoid=True)\n",
    "model_sigmoid = model_sigmoid.to(device)\n",
    "train_losses_sigmoid, test_losses_sigmoid = train_model(model_sigmoid, \"Sigmoid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training With Softmax Attention\n",
      "step 0: train loss 4.3452, val loss 4.3566\n",
      "iter 0: loss 4.3652, time 168.84ms, mfu -100.00%\n",
      "saving checkpoint to out-Softmax\n",
      "iter 10: loss 3.4071, time 29.08ms, mfu 2.87%\n",
      "iter 20: loss 3.2465, time 27.62ms, mfu 2.88%\n",
      "iter 30: loss 3.2501, time 25.94ms, mfu 2.92%\n",
      "iter 40: loss 3.0229, time 26.67ms, mfu 2.94%\n",
      "iter 50: loss 3.0703, time 24.86ms, mfu 2.98%\n",
      "iter 60: loss 2.9342, time 24.09ms, mfu 3.03%\n",
      "iter 70: loss 2.8410, time 23.29ms, mfu 3.08%\n",
      "iter 80: loss 2.7101, time 24.52ms, mfu 3.11%\n",
      "iter 90: loss 2.6872, time 23.28ms, mfu 3.16%\n",
      "iter 100: loss 2.7477, time 27.88ms, mfu 3.14%\n",
      "iter 110: loss 2.7051, time 26.95ms, mfu 3.14%\n",
      "iter 120: loss 2.6606, time 27.49ms, mfu 3.13%\n",
      "iter 130: loss 2.6660, time 22.61ms, mfu 3.18%\n",
      "iter 140: loss 2.6813, time 23.28ms, mfu 3.22%\n",
      "iter 150: loss 2.5488, time 25.11ms, mfu 3.23%\n",
      "iter 160: loss 2.6517, time 23.95ms, mfu 3.26%\n",
      "iter 170: loss 2.5588, time 25.95ms, mfu 3.25%\n",
      "iter 180: loss 2.5452, time 35.26ms, mfu 3.17%\n",
      "iter 190: loss 2.6127, time 24.01ms, mfu 3.20%\n",
      "step 200: train loss 2.5312, val loss 2.5852\n",
      "iter 200: loss 2.6345, time 153.87ms, mfu 2.93%\n",
      "iter 210: loss 2.5674, time 23.70ms, mfu 2.99%\n",
      "iter 220: loss 2.5404, time 29.77ms, mfu 2.97%\n",
      "iter 230: loss 2.4539, time 32.33ms, mfu 2.93%\n",
      "iter 240: loss 2.4661, time 35.00ms, mfu 2.88%\n",
      "iter 250: loss 2.4813, time 27.01ms, mfu 2.90%\n",
      "iter 260: loss 2.5050, time 31.73ms, mfu 2.87%\n",
      "iter 270: loss 2.5557, time 34.71ms, mfu 2.82%\n",
      "iter 280: loss 2.5213, time 25.68ms, mfu 2.87%\n",
      "iter 290: loss 2.4468, time 23.11ms, mfu 2.94%\n",
      "iter 300: loss 2.4921, time 24.41ms, mfu 2.99%\n",
      "iter 310: loss 2.4428, time 22.73ms, mfu 3.06%\n",
      "iter 320: loss 2.5377, time 25.97ms, mfu 3.07%\n",
      "iter 330: loss 2.5080, time 26.00ms, mfu 3.09%\n",
      "iter 340: loss 2.3883, time 25.98ms, mfu 3.10%\n",
      "iter 350: loss 2.4642, time 23.59ms, mfu 3.14%\n",
      "iter 360: loss 2.5030, time 26.27ms, mfu 3.14%\n",
      "iter 370: loss 2.3991, time 26.04ms, mfu 3.15%\n",
      "iter 380: loss 2.3701, time 26.06ms, mfu 3.16%\n",
      "iter 390: loss 2.4639, time 26.35ms, mfu 3.16%\n",
      "step 400: train loss 2.4483, val loss 2.4924\n",
      "iter 400: loss 2.4656, time 160.92ms, mfu 2.89%\n",
      "iter 410: loss 2.4079, time 21.55ms, mfu 2.99%\n",
      "iter 420: loss 2.4600, time 24.69ms, mfu 3.03%\n",
      "iter 430: loss 2.4599, time 26.92ms, mfu 3.04%\n",
      "iter 440: loss 2.4294, time 22.07ms, mfu 3.11%\n",
      "iter 450: loss 2.4433, time 23.52ms, mfu 3.15%\n",
      "iter 460: loss 2.4865, time 25.03ms, mfu 3.17%\n",
      "iter 470: loss 2.4378, time 23.24ms, mfu 3.21%\n",
      "iter 480: loss 2.2942, time 23.84ms, mfu 3.24%\n",
      "iter 490: loss 2.3840, time 35.85ms, mfu 3.15%\n",
      "iter 500: loss 2.4053, time 25.57ms, mfu 3.16%\n",
      "iter 510: loss 2.3737, time 26.02ms, mfu 3.17%\n",
      "iter 520: loss 2.4238, time 26.40ms, mfu 3.17%\n",
      "iter 530: loss 2.4091, time 23.13ms, mfu 3.21%\n",
      "iter 540: loss 2.3395, time 22.28ms, mfu 3.26%\n",
      "iter 550: loss 2.3781, time 23.23ms, mfu 3.30%\n",
      "iter 560: loss 2.3571, time 28.41ms, mfu 3.26%\n",
      "iter 570: loss 2.3356, time 33.80ms, mfu 3.18%\n",
      "iter 580: loss 2.3470, time 27.46ms, mfu 3.17%\n",
      "iter 590: loss 2.3144, time 24.11ms, mfu 3.20%\n",
      "step 600: train loss 2.3720, val loss 2.4167\n",
      "iter 600: loss 2.4902, time 155.47ms, mfu 2.93%\n",
      "iter 610: loss 2.4283, time 23.24ms, mfu 3.00%\n",
      "iter 620: loss 2.3531, time 25.04ms, mfu 3.03%\n",
      "iter 630: loss 2.4155, time 22.34ms, mfu 3.10%\n",
      "iter 640: loss 2.3173, time 25.87ms, mfu 3.11%\n",
      "iter 650: loss 2.3473, time 26.61ms, mfu 3.11%\n",
      "iter 660: loss 2.3110, time 22.14ms, mfu 3.18%\n",
      "iter 670: loss 2.3637, time 24.95ms, mfu 3.20%\n",
      "iter 680: loss 2.3927, time 25.17ms, mfu 3.21%\n",
      "iter 690: loss 2.2851, time 23.21ms, mfu 3.25%\n",
      "iter 700: loss 2.3531, time 37.31ms, mfu 3.14%\n",
      "iter 710: loss 2.2860, time 25.81ms, mfu 3.15%\n",
      "iter 720: loss 2.3384, time 22.93ms, mfu 3.20%\n",
      "iter 730: loss 2.3517, time 24.00ms, mfu 3.23%\n",
      "iter 740: loss 2.2752, time 25.08ms, mfu 3.24%\n",
      "iter 750: loss 2.3010, time 25.91ms, mfu 3.24%\n",
      "iter 760: loss 2.3503, time 74.05ms, mfu 3.03%\n",
      "iter 770: loss 2.1962, time 25.14ms, mfu 3.05%\n",
      "iter 780: loss 2.3058, time 24.92ms, mfu 3.08%\n",
      "iter 790: loss 2.2517, time 23.40ms, mfu 3.13%\n",
      "step 800: train loss 2.3129, val loss 2.2951\n",
      "iter 800: loss 2.2768, time 157.53ms, mfu 2.87%\n",
      "iter 810: loss 2.2734, time 23.54ms, mfu 2.94%\n",
      "iter 820: loss 2.2183, time 25.06ms, mfu 2.98%\n",
      "iter 830: loss 2.2824, time 24.73ms, mfu 3.02%\n",
      "iter 840: loss 2.2373, time 25.10ms, mfu 3.05%\n",
      "iter 850: loss 2.2699, time 27.29ms, mfu 3.05%\n",
      "iter 860: loss 2.2685, time 29.58ms, mfu 3.03%\n",
      "iter 870: loss 2.3204, time 26.27ms, mfu 3.04%\n",
      "iter 880: loss 2.1984, time 24.94ms, mfu 3.07%\n",
      "iter 890: loss 2.2997, time 26.15ms, mfu 3.08%\n",
      "iter 900: loss 2.2485, time 23.93ms, mfu 3.12%\n",
      "iter 910: loss 2.3179, time 29.78ms, mfu 3.09%\n",
      "iter 920: loss 2.2675, time 26.47ms, mfu 3.10%\n",
      "iter 930: loss 2.2379, time 24.75ms, mfu 3.12%\n",
      "iter 940: loss 2.2516, time 25.99ms, mfu 3.13%\n",
      "iter 950: loss 2.1948, time 24.32ms, mfu 3.16%\n",
      "iter 960: loss 2.1995, time 21.58ms, mfu 3.23%\n",
      "iter 970: loss 2.2362, time 26.40ms, mfu 3.22%\n",
      "iter 980: loss 2.2340, time 23.71ms, mfu 3.25%\n",
      "iter 990: loss 2.2159, time 28.34ms, mfu 3.22%\n",
      "step 1000: train loss 2.1495, val loss 2.2203\n",
      "iter 1000: loss 2.2121, time 158.46ms, mfu 2.95%\n",
      "saving checkpoint to out-Softmax\n",
      "iter 1010: loss 2.2321, time 26.93ms, mfu 2.97%\n",
      "iter 1020: loss 2.1832, time 24.68ms, mfu 3.01%\n",
      "iter 1030: loss 2.2422, time 26.70ms, mfu 3.02%\n",
      "iter 1040: loss 2.1867, time 25.29ms, mfu 3.05%\n",
      "iter 1050: loss 2.1775, time 22.01ms, mfu 3.12%\n",
      "iter 1060: loss 2.1363, time 27.02ms, mfu 3.12%\n",
      "iter 1070: loss 2.1717, time 30.66ms, mfu 3.08%\n",
      "iter 1080: loss 2.2314, time 34.40ms, mfu 3.01%\n",
      "iter 1090: loss 2.2451, time 24.76ms, mfu 3.05%\n",
      "iter 1100: loss 2.2116, time 31.94ms, mfu 3.00%\n",
      "iter 1110: loss 2.1495, time 21.11ms, mfu 3.10%\n",
      "iter 1120: loss 2.1959, time 26.45ms, mfu 3.10%\n",
      "iter 1130: loss 2.2408, time 22.90ms, mfu 3.16%\n",
      "iter 1140: loss 2.2679, time 23.93ms, mfu 3.19%\n",
      "iter 1150: loss 2.1401, time 24.98ms, mfu 3.21%\n",
      "iter 1160: loss 2.2957, time 27.74ms, mfu 3.19%\n",
      "iter 1170: loss 2.1319, time 23.89ms, mfu 3.22%\n",
      "iter 1180: loss 2.0770, time 26.23ms, mfu 3.21%\n",
      "iter 1190: loss 2.1231, time 23.99ms, mfu 3.24%\n",
      "step 1200: train loss 2.1362, val loss 2.1705\n",
      "iter 1200: loss 2.0961, time 155.28ms, mfu 2.97%\n",
      "iter 1210: loss 2.2046, time 22.76ms, mfu 3.04%\n",
      "iter 1220: loss 2.2140, time 23.64ms, mfu 3.09%\n",
      "iter 1230: loss 2.0942, time 25.24ms, mfu 3.11%\n",
      "iter 1240: loss 2.1188, time 23.89ms, mfu 3.15%\n",
      "iter 1250: loss 2.2109, time 23.49ms, mfu 3.19%\n",
      "iter 1260: loss 2.0557, time 24.54ms, mfu 3.21%\n",
      "iter 1270: loss 2.1248, time 22.39ms, mfu 3.26%\n",
      "iter 1280: loss 2.0704, time 22.47ms, mfu 3.31%\n",
      "iter 1290: loss 2.1642, time 22.96ms, mfu 3.34%\n",
      "iter 1300: loss 2.1548, time 25.17ms, mfu 3.34%\n",
      "iter 1310: loss 2.1888, time 28.96ms, mfu 3.29%\n",
      "iter 1320: loss 2.0745, time 23.16ms, mfu 3.32%\n",
      "iter 1330: loss 2.1128, time 48.38ms, mfu 3.16%\n",
      "iter 1340: loss 2.1491, time 22.20ms, mfu 3.22%\n",
      "iter 1350: loss 2.1065, time 23.89ms, mfu 3.25%\n",
      "iter 1360: loss 2.0743, time 22.64ms, mfu 3.29%\n",
      "iter 1370: loss 2.0451, time 21.54ms, mfu 3.35%\n",
      "iter 1380: loss 2.1141, time 22.82ms, mfu 3.38%\n",
      "iter 1390: loss 2.1840, time 21.15ms, mfu 3.44%\n",
      "step 1400: train loss 2.1141, val loss 2.1201\n",
      "iter 1400: loss 2.0490, time 157.13ms, mfu 3.15%\n",
      "iter 1410: loss 2.0743, time 24.29ms, mfu 3.17%\n",
      "iter 1420: loss 2.0484, time 23.44ms, mfu 3.21%\n",
      "iter 1430: loss 2.0531, time 25.19ms, mfu 3.22%\n",
      "iter 1440: loss 2.0196, time 29.08ms, mfu 3.19%\n",
      "iter 1450: loss 2.0803, time 22.54ms, mfu 3.24%\n",
      "iter 1460: loss 2.1227, time 22.55ms, mfu 3.28%\n",
      "iter 1470: loss 2.0450, time 22.15ms, mfu 3.33%\n",
      "iter 1480: loss 2.1150, time 23.54ms, mfu 3.35%\n",
      "iter 1490: loss 2.0834, time 22.97ms, mfu 3.38%\n",
      "iter 1500: loss 2.1502, time 25.08ms, mfu 3.38%\n",
      "iter 1510: loss 2.0497, time 23.17ms, mfu 3.40%\n",
      "iter 1520: loss 2.0971, time 28.31ms, mfu 3.35%\n",
      "iter 1530: loss 2.0574, time 22.05ms, mfu 3.40%\n",
      "iter 1540: loss 2.0457, time 22.75ms, mfu 3.42%\n",
      "iter 1550: loss 2.0179, time 22.77ms, mfu 3.45%\n",
      "iter 1560: loss 2.0642, time 22.68ms, mfu 3.47%\n",
      "iter 1570: loss 2.1556, time 21.71ms, mfu 3.51%\n",
      "iter 1580: loss 2.0699, time 25.44ms, mfu 3.48%\n",
      "iter 1590: loss 1.9984, time 25.10ms, mfu 3.47%\n",
      "step 1600: train loss 1.9890, val loss 2.1229\n",
      "iter 1600: loss 1.9410, time 150.27ms, mfu 3.18%\n",
      "iter 1610: loss 2.0722, time 23.88ms, mfu 3.21%\n",
      "iter 1620: loss 2.0255, time 22.55ms, mfu 3.26%\n",
      "iter 1630: loss 2.0457, time 24.30ms, mfu 3.27%\n",
      "iter 1640: loss 2.0010, time 23.77ms, mfu 3.30%\n",
      "iter 1650: loss 1.9963, time 24.10ms, mfu 3.31%\n",
      "iter 1660: loss 2.0741, time 23.22ms, mfu 3.34%\n",
      "iter 1670: loss 2.0464, time 26.23ms, mfu 3.33%\n",
      "iter 1680: loss 2.0809, time 21.26ms, mfu 3.39%\n",
      "iter 1690: loss 2.0586, time 23.07ms, mfu 3.41%\n",
      "iter 1700: loss 2.0117, time 22.85ms, mfu 3.43%\n",
      "iter 1710: loss 2.0234, time 23.78ms, mfu 3.44%\n",
      "iter 1720: loss 2.0319, time 22.97ms, mfu 3.46%\n",
      "iter 1730: loss 1.9232, time 24.33ms, mfu 3.46%\n",
      "iter 1740: loss 1.9853, time 24.53ms, mfu 3.45%\n",
      "iter 1750: loss 2.0109, time 25.62ms, mfu 3.43%\n",
      "iter 1760: loss 1.9733, time 24.85ms, mfu 3.42%\n",
      "iter 1770: loss 1.9121, time 28.01ms, mfu 3.38%\n",
      "iter 1780: loss 2.0394, time 26.17ms, mfu 3.36%\n",
      "iter 1790: loss 1.9474, time 24.59ms, mfu 3.36%\n",
      "step 1800: train loss 1.9984, val loss 2.0513\n",
      "iter 1800: loss 1.9438, time 161.72ms, mfu 3.08%\n",
      "iter 1810: loss 2.0583, time 24.87ms, mfu 3.11%\n",
      "iter 1820: loss 1.8880, time 23.00ms, mfu 3.16%\n",
      "iter 1830: loss 2.0440, time 23.09ms, mfu 3.20%\n",
      "iter 1840: loss 2.0484, time 24.91ms, mfu 3.22%\n",
      "iter 1850: loss 2.0450, time 24.56ms, mfu 3.24%\n",
      "iter 1860: loss 1.9692, time 25.43ms, mfu 3.24%\n",
      "iter 1870: loss 1.9342, time 23.60ms, mfu 3.27%\n",
      "iter 1880: loss 1.8880, time 23.19ms, mfu 3.30%\n",
      "iter 1890: loss 1.9965, time 22.54ms, mfu 3.34%\n",
      "iter 1900: loss 1.9914, time 23.78ms, mfu 3.36%\n",
      "iter 1910: loss 2.0269, time 23.93ms, mfu 3.37%\n",
      "iter 1920: loss 2.0058, time 25.28ms, mfu 3.36%\n",
      "iter 1930: loss 1.9261, time 25.94ms, mfu 3.35%\n",
      "iter 1940: loss 1.9787, time 23.55ms, mfu 3.37%\n",
      "iter 1950: loss 2.0319, time 24.79ms, mfu 3.37%\n",
      "iter 1960: loss 1.9506, time 27.33ms, mfu 3.34%\n",
      "iter 1970: loss 1.9925, time 26.46ms, mfu 3.32%\n",
      "iter 1980: loss 1.9923, time 23.48ms, mfu 3.34%\n",
      "iter 1990: loss 1.9388, time 21.49ms, mfu 3.39%\n",
      "step 2000: train loss 1.9513, val loss 2.0193\n",
      "iter 2000: loss 1.9082, time 149.39ms, mfu 3.11%\n",
      "saving checkpoint to out-Softmax\n",
      "iter 2010: loss 1.9145, time 22.26ms, mfu 3.17%\n",
      "iter 2020: loss 1.8848, time 23.91ms, mfu 3.21%\n",
      "iter 2030: loss 2.0100, time 21.90ms, mfu 3.27%\n",
      "iter 2040: loss 2.0115, time 22.87ms, mfu 3.30%\n",
      "iter 2050: loss 1.8042, time 23.46ms, mfu 3.33%\n",
      "iter 2060: loss 2.0474, time 23.77ms, mfu 3.35%\n",
      "iter 2070: loss 2.0171, time 21.98ms, mfu 3.39%\n",
      "iter 2080: loss 1.8998, time 22.59ms, mfu 3.42%\n",
      "iter 2090: loss 1.8329, time 22.76ms, mfu 3.45%\n",
      "iter 2100: loss 1.8876, time 23.47ms, mfu 3.46%\n",
      "iter 2110: loss 1.9786, time 24.93ms, mfu 3.45%\n",
      "iter 2120: loss 1.8643, time 23.20ms, mfu 3.46%\n",
      "iter 2130: loss 1.8412, time 25.73ms, mfu 3.44%\n",
      "iter 2140: loss 1.9543, time 23.72ms, mfu 3.45%\n",
      "iter 2150: loss 1.8957, time 25.82ms, mfu 3.42%\n",
      "iter 2160: loss 1.8981, time 23.25ms, mfu 3.44%\n",
      "iter 2170: loss 1.9149, time 25.14ms, mfu 3.43%\n",
      "iter 2180: loss 2.0030, time 25.65ms, mfu 3.41%\n",
      "iter 2190: loss 1.9411, time 27.28ms, mfu 3.38%\n",
      "step 2200: train loss 1.9275, val loss 2.0025\n",
      "iter 2200: loss 1.8231, time 157.77ms, mfu 3.09%\n",
      "iter 2210: loss 1.9824, time 23.78ms, mfu 3.13%\n",
      "iter 2220: loss 1.8861, time 24.21ms, mfu 3.16%\n",
      "iter 2230: loss 1.9702, time 22.78ms, mfu 3.21%\n",
      "iter 2240: loss 1.9163, time 26.30ms, mfu 3.21%\n",
      "iter 2250: loss 1.9008, time 24.79ms, mfu 3.22%\n",
      "iter 2260: loss 1.9305, time 24.01ms, mfu 3.25%\n",
      "iter 2270: loss 1.8068, time 22.55ms, mfu 3.29%\n",
      "iter 2280: loss 1.9335, time 23.66ms, mfu 3.32%\n",
      "iter 2290: loss 1.9257, time 22.97ms, mfu 3.35%\n",
      "iter 2300: loss 1.8295, time 24.41ms, mfu 3.36%\n",
      "iter 2310: loss 1.9358, time 24.23ms, mfu 3.36%\n",
      "iter 2320: loss 1.8451, time 22.51ms, mfu 3.40%\n",
      "iter 2330: loss 1.9435, time 24.75ms, mfu 3.40%\n",
      "iter 2340: loss 1.9895, time 33.95ms, mfu 3.30%\n",
      "iter 2350: loss 1.8172, time 23.34ms, mfu 3.33%\n",
      "iter 2360: loss 1.8833, time 27.02ms, mfu 3.30%\n",
      "iter 2370: loss 1.8767, time 23.14ms, mfu 3.33%\n",
      "iter 2380: loss 1.8614, time 24.04ms, mfu 3.35%\n",
      "iter 2390: loss 1.9674, time 22.57ms, mfu 3.38%\n",
      "step 2400: train loss 1.9116, val loss 1.9549\n",
      "iter 2400: loss 1.8625, time 149.12ms, mfu 3.10%\n",
      "iter 2410: loss 1.8956, time 22.34ms, mfu 3.16%\n",
      "iter 2420: loss 1.8799, time 22.75ms, mfu 3.21%\n",
      "iter 2430: loss 1.8223, time 24.73ms, mfu 3.23%\n",
      "iter 2440: loss 1.9576, time 21.87ms, mfu 3.29%\n",
      "iter 2450: loss 1.9115, time 23.79ms, mfu 3.31%\n",
      "iter 2460: loss 1.8828, time 22.57ms, mfu 3.35%\n",
      "iter 2470: loss 1.8091, time 22.18ms, mfu 3.39%\n",
      "iter 2480: loss 1.9055, time 26.21ms, mfu 3.37%\n",
      "iter 2490: loss 1.8415, time 24.68ms, mfu 3.37%\n",
      "iter 2500: loss 1.9534, time 23.82ms, mfu 3.38%\n",
      "iter 2510: loss 1.8617, time 23.09ms, mfu 3.41%\n",
      "iter 2520: loss 1.8658, time 23.59ms, mfu 3.42%\n",
      "iter 2530: loss 1.7760, time 24.54ms, mfu 3.42%\n",
      "iter 2540: loss 1.8920, time 22.14ms, mfu 3.45%\n",
      "iter 2550: loss 1.8833, time 22.92ms, mfu 3.47%\n",
      "iter 2560: loss 1.9645, time 22.70ms, mfu 3.49%\n",
      "iter 2570: loss 1.9735, time 23.79ms, mfu 3.49%\n",
      "iter 2580: loss 1.9658, time 22.16ms, mfu 3.52%\n",
      "iter 2590: loss 1.9166, time 23.81ms, mfu 3.52%\n",
      "step 2600: train loss 1.8441, val loss 1.9817\n",
      "iter 2600: loss 1.8671, time 149.09ms, mfu 3.22%\n",
      "iter 2610: loss 1.8547, time 22.09ms, mfu 3.28%\n",
      "iter 2620: loss 1.9674, time 22.99ms, mfu 3.31%\n",
      "iter 2630: loss 1.8384, time 24.46ms, mfu 3.32%\n",
      "iter 2640: loss 1.9272, time 22.71ms, mfu 3.36%\n",
      "iter 2650: loss 1.8074, time 24.39ms, mfu 3.36%\n",
      "iter 2660: loss 1.8290, time 25.99ms, mfu 3.35%\n",
      "iter 2670: loss 1.8565, time 22.80ms, mfu 3.38%\n",
      "iter 2680: loss 1.9017, time 23.45ms, mfu 3.40%\n",
      "iter 2690: loss 1.8188, time 21.08ms, mfu 3.45%\n",
      "iter 2700: loss 1.9342, time 23.21ms, mfu 3.47%\n",
      "iter 2710: loss 1.8710, time 22.96ms, mfu 3.48%\n",
      "iter 2720: loss 1.9030, time 22.94ms, mfu 3.50%\n",
      "iter 2730: loss 1.7223, time 20.85ms, mfu 3.55%\n",
      "iter 2740: loss 1.9347, time 22.91ms, mfu 3.56%\n",
      "iter 2750: loss 1.8726, time 25.40ms, mfu 3.53%\n",
      "iter 2760: loss 1.8141, time 24.73ms, mfu 3.51%\n",
      "iter 2770: loss 1.8563, time 24.35ms, mfu 3.51%\n",
      "iter 2780: loss 1.9023, time 25.97ms, mfu 3.48%\n",
      "iter 2790: loss 1.8599, time 25.91ms, mfu 3.45%\n",
      "step 2800: train loss 1.8448, val loss 1.9667\n",
      "iter 2800: loss 1.8878, time 165.53ms, mfu 3.16%\n",
      "iter 2810: loss 1.8088, time 22.52ms, mfu 3.21%\n",
      "iter 2820: loss 1.8042, time 24.48ms, mfu 3.23%\n",
      "iter 2830: loss 1.8608, time 24.50ms, mfu 3.25%\n",
      "iter 2840: loss 1.8853, time 23.22ms, mfu 3.28%\n",
      "iter 2850: loss 1.8982, time 22.85ms, mfu 3.32%\n",
      "iter 2860: loss 1.7689, time 24.22ms, mfu 3.33%\n",
      "iter 2870: loss 1.9067, time 21.14ms, mfu 3.39%\n",
      "iter 2880: loss 1.8006, time 23.65ms, mfu 3.41%\n",
      "iter 2890: loss 1.7455, time 25.91ms, mfu 3.39%\n",
      "iter 2900: loss 1.8379, time 23.96ms, mfu 3.40%\n",
      "iter 2910: loss 1.8526, time 23.01ms, mfu 3.42%\n",
      "iter 2920: loss 1.9329, time 22.79ms, mfu 3.44%\n",
      "iter 2930: loss 1.6926, time 23.83ms, mfu 3.45%\n",
      "iter 2940: loss 1.7921, time 23.33ms, mfu 3.46%\n",
      "iter 2950: loss 1.8776, time 22.92ms, mfu 3.48%\n",
      "iter 2960: loss 1.7528, time 23.24ms, mfu 3.49%\n",
      "iter 2970: loss 1.9233, time 24.95ms, mfu 3.48%\n",
      "iter 2980: loss 1.8596, time 22.02ms, mfu 3.51%\n",
      "iter 2990: loss 1.7571, time 24.94ms, mfu 3.49%\n",
      "step 3000: train loss 1.7936, val loss 1.9306\n",
      "iter 3000: loss 1.8734, time 156.94ms, mfu 3.19%\n",
      "saving checkpoint to out-Softmax\n",
      "iter 3010: loss 1.8327, time 24.44ms, mfu 3.22%\n",
      "iter 3020: loss 1.8211, time 22.74ms, mfu 3.26%\n",
      "iter 3030: loss 1.9009, time 24.56ms, mfu 3.27%\n",
      "iter 3040: loss 1.9209, time 22.86ms, mfu 3.31%\n",
      "iter 3050: loss 1.8100, time 27.26ms, mfu 3.29%\n",
      "iter 3060: loss 1.8492, time 24.66ms, mfu 3.30%\n",
      "iter 3070: loss 1.8182, time 22.62ms, mfu 3.34%\n",
      "iter 3080: loss 1.7464, time 24.56ms, mfu 3.34%\n",
      "iter 3090: loss 1.9129, time 23.05ms, mfu 3.37%\n",
      "iter 3100: loss 1.8142, time 25.08ms, mfu 3.36%\n",
      "iter 3110: loss 1.7986, time 23.21ms, mfu 3.39%\n",
      "iter 3120: loss 1.7398, time 25.44ms, mfu 3.38%\n",
      "iter 3130: loss 1.7582, time 25.21ms, mfu 3.37%\n",
      "iter 3140: loss 1.8495, time 35.09ms, mfu 3.27%\n",
      "iter 3150: loss 1.9159, time 22.26ms, mfu 3.32%\n",
      "iter 3160: loss 1.6985, time 24.34ms, mfu 3.33%\n",
      "iter 3170: loss 1.9058, time 23.76ms, mfu 3.35%\n",
      "iter 3180: loss 1.8006, time 22.88ms, mfu 3.38%\n",
      "iter 3190: loss 1.8114, time 23.93ms, mfu 3.39%\n",
      "step 3200: train loss 1.7626, val loss 1.9296\n",
      "iter 3200: loss 1.8096, time 154.51ms, mfu 3.10%\n",
      "iter 3210: loss 1.8197, time 25.79ms, mfu 3.12%\n",
      "iter 3220: loss 1.7595, time 21.73ms, mfu 3.19%\n",
      "iter 3230: loss 1.7071, time 23.90ms, mfu 3.22%\n",
      "iter 3240: loss 1.8938, time 25.48ms, mfu 3.22%\n",
      "iter 3250: loss 1.7666, time 22.29ms, mfu 3.28%\n",
      "iter 3260: loss 1.8307, time 23.40ms, mfu 3.30%\n",
      "iter 3270: loss 1.6693, time 23.47ms, mfu 3.33%\n",
      "iter 3280: loss 1.7967, time 22.91ms, mfu 3.36%\n",
      "iter 3290: loss 1.8312, time 25.59ms, mfu 3.35%\n",
      "iter 3300: loss 1.7140, time 27.40ms, mfu 3.32%\n",
      "iter 3310: loss 1.7865, time 26.01ms, mfu 3.31%\n",
      "iter 3320: loss 1.7374, time 23.63ms, mfu 3.33%\n",
      "iter 3330: loss 1.7731, time 24.07ms, mfu 3.34%\n",
      "iter 3340: loss 1.7772, time 22.99ms, mfu 3.37%\n",
      "iter 3350: loss 1.9010, time 23.69ms, mfu 3.39%\n",
      "iter 3360: loss 1.8592, time 24.44ms, mfu 3.39%\n",
      "iter 3370: loss 1.8488, time 23.05ms, mfu 3.41%\n",
      "iter 3380: loss 1.8702, time 23.53ms, mfu 3.43%\n",
      "iter 3390: loss 1.7569, time 25.47ms, mfu 3.41%\n",
      "step 3400: train loss 1.7410, val loss 1.9078\n",
      "iter 3400: loss 1.7396, time 148.78ms, mfu 3.13%\n",
      "iter 3410: loss 1.8014, time 22.24ms, mfu 3.19%\n",
      "iter 3420: loss 1.7753, time 24.74ms, mfu 3.21%\n",
      "iter 3430: loss 1.7123, time 24.08ms, mfu 3.23%\n",
      "iter 3440: loss 1.8251, time 23.51ms, mfu 3.26%\n",
      "iter 3450: loss 1.8346, time 24.42ms, mfu 3.28%\n",
      "iter 3460: loss 1.7549, time 23.62ms, mfu 3.30%\n",
      "iter 3470: loss 1.8158, time 23.87ms, mfu 3.32%\n",
      "iter 3480: loss 1.7170, time 23.00ms, mfu 3.35%\n",
      "iter 3490: loss 1.7032, time 23.69ms, mfu 3.37%\n",
      "iter 3500: loss 1.8357, time 23.19ms, mfu 3.39%\n",
      "iter 3510: loss 1.8055, time 22.42ms, mfu 3.43%\n",
      "iter 3520: loss 1.8100, time 24.80ms, mfu 3.42%\n",
      "iter 3530: loss 1.6907, time 20.89ms, mfu 3.48%\n",
      "iter 3540: loss 1.7730, time 24.20ms, mfu 3.47%\n",
      "iter 3550: loss 1.7790, time 23.83ms, mfu 3.48%\n",
      "iter 3560: loss 1.6125, time 23.18ms, mfu 3.49%\n",
      "iter 3570: loss 1.7166, time 22.32ms, mfu 3.51%\n",
      "iter 3580: loss 1.8282, time 23.39ms, mfu 3.52%\n",
      "iter 3590: loss 1.7302, time 23.05ms, mfu 3.53%\n",
      "step 3600: train loss 1.8133, val loss 1.9161\n",
      "iter 3600: loss 1.7830, time 152.67ms, mfu 3.23%\n",
      "iter 3610: loss 1.7253, time 22.09ms, mfu 3.28%\n",
      "iter 3620: loss 1.7360, time 23.97ms, mfu 3.30%\n",
      "iter 3630: loss 1.9366, time 23.49ms, mfu 3.33%\n",
      "iter 3640: loss 1.7205, time 23.65ms, mfu 3.35%\n",
      "iter 3650: loss 1.8383, time 23.50ms, mfu 3.37%\n",
      "iter 3660: loss 1.7079, time 22.91ms, mfu 3.40%\n",
      "iter 3670: loss 1.6882, time 26.03ms, mfu 3.38%\n",
      "iter 3680: loss 1.7615, time 24.48ms, mfu 3.38%\n",
      "iter 3690: loss 1.8174, time 21.40ms, mfu 3.43%\n",
      "iter 3700: loss 1.7857, time 21.91ms, mfu 3.47%\n",
      "iter 3710: loss 1.8344, time 25.07ms, mfu 3.45%\n",
      "iter 3720: loss 1.7389, time 23.61ms, mfu 3.46%\n",
      "iter 3730: loss 1.6542, time 24.54ms, mfu 3.46%\n",
      "iter 3740: loss 1.8584, time 21.77ms, mfu 3.49%\n",
      "iter 3750: loss 1.6565, time 22.52ms, mfu 3.51%\n",
      "iter 3760: loss 1.7865, time 24.86ms, mfu 3.50%\n",
      "iter 3770: loss 1.6797, time 23.44ms, mfu 3.50%\n",
      "iter 3780: loss 1.7567, time 25.36ms, mfu 3.48%\n",
      "iter 3790: loss 1.5557, time 22.99ms, mfu 3.50%\n",
      "step 3800: train loss 1.7199, val loss 1.8627\n",
      "iter 3800: loss 1.6343, time 156.56ms, mfu 3.20%\n",
      "iter 3810: loss 1.7926, time 21.69ms, mfu 3.27%\n",
      "iter 3820: loss 1.7477, time 23.24ms, mfu 3.30%\n",
      "iter 3830: loss 1.7002, time 21.41ms, mfu 3.36%\n",
      "iter 3840: loss 1.6104, time 22.00ms, mfu 3.40%\n",
      "iter 3850: loss 1.6931, time 24.31ms, mfu 3.40%\n",
      "iter 3860: loss 1.7311, time 24.10ms, mfu 3.41%\n",
      "iter 3870: loss 1.7392, time 23.11ms, mfu 3.43%\n",
      "iter 3880: loss 1.7548, time 23.79ms, mfu 3.44%\n",
      "iter 3890: loss 1.8147, time 30.15ms, mfu 3.37%\n",
      "iter 3900: loss 1.8317, time 25.50ms, mfu 3.36%\n",
      "iter 3910: loss 1.6649, time 24.21ms, mfu 3.37%\n",
      "iter 3920: loss 1.5561, time 23.82ms, mfu 3.38%\n",
      "iter 3930: loss 1.6159, time 35.30ms, mfu 3.28%\n",
      "iter 3940: loss 1.8028, time 22.97ms, mfu 3.31%\n",
      "iter 3950: loss 1.7125, time 25.35ms, mfu 3.31%\n",
      "iter 3960: loss 1.7282, time 36.00ms, mfu 3.21%\n",
      "iter 3970: loss 1.6898, time 26.05ms, mfu 3.21%\n",
      "iter 3980: loss 1.7579, time 26.48ms, mfu 3.21%\n",
      "iter 3990: loss 1.7581, time 22.99ms, mfu 3.25%\n"
     ]
    }
   ],
   "source": [
    "# Train with softmax attention\n",
    "print(\"Training With Softmax Attention\")\n",
    "model_softmax = nanoGPT(vocab_size, n_embd, block_size, n_head, n_layer, dropout, use_sigmoid=False)\n",
    "model_softmax = model_softmax.to(device)\n",
    "train_losses_softmax, test_losses_softmax = train_model(model_softmax, \"Softmax\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Evaluate**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+kAAAIhCAYAAAAy8fsSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAAC1UklEQVR4nOzdd3xV9f3H8de5I8kdWYRMCCPMhCEiMhQBUURBaxGrVatSlaq1WqtWxV3rpO2vWmndLVpX25/jV9sKVmUpGwRRwg4JhOy9c2/u+f1xyYVACCGMm/F+Ph73kdxzvuecz0ku+njnO45hmqaJiIiIiIiIiASdJdgFiIiIiIiIiIifQrqIiIiIiIhIO6GQLiIiIiIiItJOKKSLiIiIiIiItBMK6SIiIiIiIiLthEK6iIiIiIiISDuhkC4iIiIiIiLSTiiki4iIiIiIiLQTCukiIiIiIiIi7YRCuoiItGuGYbTqtXjx4uO6zmOPPYZhGG06dvHixSekhvZu1qxZ9OnT54j758+f36rfVUvnOBbLly/nscceo7S0tFXtG3/HhYWFJ+T6IiIiJ4Mt2AWIiIi0ZMWKFU3e//rXv2bRokV88cUXTbanpaUd13VuuukmLrzwwjYdO3LkSFasWHHcNXR006dPP+z3NW7cOC6//HLuvvvuwLbQ0NATcr3ly5fzq1/9ilmzZhEVFXVCzikiIhJsCukiItKujR07tsn72NhYLBbLYdsPVV1djdPpbPV1evbsSc+ePdtUY0RExFHr6QpiY2OJjY09bHt8fLx+PiIiIq2k4e4iItLhTZo0iaFDh7J06VLOOussnE4nN9xwAwB/+9vfuOCCC0hMTMThcJCamsr9999PVVVVk3M0N9y9T58+XHzxxSxYsICRI0ficDgYPHgwf/7zn5u0a264+6xZs3C73ezYsYNp06bhdrtJTk7m7rvvpq6ursnxe/fu5fLLLyc8PJyoqCiuueYa1qxZg2EYzJ8/v8V7Lygo4Kc//SlpaWm43W7i4uKYPHkyy5Yta9Ju9+7dGIbBb3/7W/7nf/6Hvn374na7GTduHCtXrjzsvPPnz2fQoEGEhoaSmprKm2++2WIdx2L79u1cffXVxMXFBc7/xz/+sUkbn8/HE088waBBg3A4HERFRTF8+HCef/55wP/7+uUvfwlA3759T9i0B4B//vOfjBs3DqfTSXh4OFOmTDlshEBBQQE/+clPSE5OJjQ0lNjYWM4++2w+++yzQJuvv/6aiy++OHCfSUlJTJ8+nb179wbamKbJn/70J0aMGIHD4SA6OprLL7+cXbt2Nblea84lIiKdg3rSRUSkU8jJyeFHP/oR9957L0899RQWi//v0Nu3b2fatGnceeeduFwutmzZwrPPPsvq1asPGzLfnI0bN3L33Xdz//33Ex8fz2uvvcaNN95I//79mTBhQovHejwevve973HjjTdy9913s3TpUn79618TGRnJI488AkBVVRXnnnsuxcXFPPvss/Tv358FCxZw5ZVXtuq+i4uLAXj00UdJSEigsrKSDz/8kEmTJvH5558zadKkJu3/+Mc/MnjwYJ577jkAHn74YaZNm0ZGRgaRkZGAP6D/+Mc/5tJLL+V3v/sdZWVlPPbYY9TV1QV+rm21efNmzjrrLHr16sXvfvc7EhISWLhwIXfccQeFhYU8+uijAMydO5fHHnuMhx56iAkTJuDxeNiyZUtg/vlNN91EcXExL7zwAh988AGJiYnA8U97eOedd7jmmmu44IILePfdd6mrq2Pu3LmBn+f48eMBuPbaa1m/fj1PPvkkAwcOpLS0lPXr11NUVAT4f69Tpkyhb9++/PGPfyQ+Pp7c3FwWLVpERUVF4Ho333wz8+fP54477uDZZ5+luLiYxx9/nLPOOouNGzcSHx/f6nOJiEgnYYqIiHQg119/velyuZpsmzhxogmYn3/+eYvH+nw+0+PxmEuWLDEBc+PGjYF9jz76qHno/xZ79+5thoWFmZmZmYFtNTU1Zrdu3cybb745sG3RokUmYC5atKhJnYD597//vck5p02bZg4aNCjw/o9//KMJmJ988kmTdjfffLMJmH/5y19avKdDeb1e0+PxmOedd545Y8aMwPaMjAwTMIcNG2Z6vd7A9tWrV5uA+e6775qmaZoNDQ1mUlKSOXLkSNPn8wXa7d6927Tb7Wbv3r2PqR7AvO222wLvp06davbs2dMsKytr0u5nP/uZGRYWZhYXF5umaZoXX3yxOWLEiBbP/Zvf/MYEzIyMjFbV0vg7LigoaHZ/470PGzbMbGhoCGyvqKgw4+LizLPOOiuwze12m3feeecRr7V27VoTMD/66KMjtlmxYoUJmL/73e+abN+zZ4/pcDjMe++9t9XnEhGRzkPD3UVEpFOIjo5m8uTJh23ftWsXV199NQkJCVitVux2OxMnTgQgPT39qOcdMWIEvXr1CrwPCwtj4MCBZGZmHvVYwzC45JJLmmwbPnx4k2OXLFlCeHj4YYvWXXXVVUc9f6OXXnqJkSNHEhYWhs1mw2638/nnnzd7f9OnT8dqtTapBwjUtHXrVvbt28fVV1/dZPh/7969Oeuss1pdU3Nqa2v5/PPPmTFjBk6nE6/XG3hNmzaN2trawND70aNHs3HjRn7605+ycOFCysvLj+vardF479dee22TEQNut5uZM2eycuVKqqurA/XNnz+fJ554gpUrV+LxeJqcq3///kRHR3Pffffx0ksvsXnz5sOu969//QvDMPjRj37U5GeRkJDAaaedFhi635pziYhI56GQLiIinULjcOeDVVZWcs4557Bq1SqeeOIJFi9ezJo1a/jggw8AqKmpOep5Y2JiDtsWGhraqmOdTidhYWGHHVtbWxt4X1RURHx8/GHHNretOf/zP//DrbfeypgxY3j//fdZuXIla9as4cILL2y2xkPvp3Gl9ca2jcO1ExISDju2uW3HoqioCK/XywsvvIDdbm/ymjZtGkDg8Whz5szht7/9LStXruSiiy4iJiaG8847j7Vr1x5XDUerD5r/LCUlJeHz+SgpKQH8ax1cf/31vPbaa4wbN45u3bpx3XXXkZubC0BkZCRLlixhxIgRPPDAAwwZMoSkpCQeffTRQKDPy8vDNE3i4+MP+3msXLky8LNozblERKTz0Jx0ERHpFJp7xvkXX3zBvn37WLx4caD3HGj1c7VPhZiYGFavXn3Y9sawdzRvvfUWkyZN4sUXX2yyva1zlRtDfHPXb21NRxIdHY3VauXaa6/ltttua7ZN3759AbDZbNx1113cddddlJaW8tlnn/HAAw8wdepU9uzZc0wr97dW473n5OQctm/fvn1YLBaio6MB6N69O8899xzPPfccWVlZ/POf/+T+++8nPz+fBQsWADBs2DDee+89TNPkm2++Yf78+Tz++OM4HA7uv/9+unfvjmEYLFu2rNnH0h287WjnEhGRzkM96SIi0mk1BvdDA9DLL78cjHKaNXHiRCoqKvjkk0+abH/vvfdadbxhGIfd3zfffHPYauStNWjQIBITE3n33XcxTTOwPTMzk+XLl7fpnI2cTifnnnsuX3/9NcOHD2fUqFGHvZobuRAVFcXll1/ObbfdRnFxMbt37wYOHwVwvAYNGkSPHj145513mtx7VVUV77//fmDF90P16tWLn/3sZ0yZMoX169cftt8wDE477TR+//vfExUVFWhz8cUXY5om2dnZzf4shg0b1upziYhI56GedBER6bTOOussoqOjueWWW3j00Uex2+28/fbbbNy4MdilBVx//fX8/ve/50c/+hFPPPEE/fv355NPPmHhwoUAR11N/eKLL+bXv/41jz76KBMnTmTr1q08/vjj9O3bF6/Xe8z1WCwWfv3rX3PTTTcxY8YMZs+eTWlpKY899thxD3cHeP755xk/fjznnHMOt956K3369KGiooIdO3bw8ccfB1bcv+SSSxg6dCijRo0iNjaWzMxMnnvuOXr37s2AAQMAAiH2+eef5/rrr8dutzNo0CDCw8NbrOHjjz9uts3ll1/O3Llzueaaa7j44ou5+eabqaur4ze/+Q2lpaU888wzAJSVlXHuuedy9dVXM3jwYMLDw1mzZg0LFizgsssuA/zzzf/0pz/x/e9/n5SUFEzT5IMPPqC0tJQpU6YAcPbZZ/OTn/yEH//4x6xdu5YJEybgcrnIycnhyy+/ZNiwYdx6662tOpeIiHQeCukiItJpxcTE8O9//5u7776bH/3oR7hcLi699FL+9re/MXLkyGCXB4DL5eKLL77gzjvv5N5778UwDC644AL+9Kc/MW3aNKKiolo8/sEHH6S6uprXX3+duXPnkpaWxksvvcSHH37Y5meG33jjjQA8++yzXHbZZfTp04cHHniAJUuWHPdzyNPS0li/fj2//vWveeihh8jPzycqKooBAwYE5qUDnHvuubz//vu89tprlJeXk5CQwJQpU3j44Yex2+0ATJo0iTlz5vDGG2/w6quv4vP5WLRo0WGPnTvUDTfc0Ox20zS5+uqrcblcPP3001x55ZVYrVbGjh3LokWLAgvnhYWFMWbMGP7617+ye/duPB4PvXr14r777uPee+8FYMCAAURFRTF37lz27dtHSEgIgwYNYv78+Vx//fWBa7788suMHTuWl19+mT/96U/4fD6SkpI4++yzGT169DGdS0REOgfDPHg8l4iIiLQLTz31FA899BBZWVn07Nkz2OWIiIjIKaKedBERkSCbN28eAIMHD8bj8fDFF1/whz/8gR/96EcK6CIiIl2MQrqIiEiQOZ1Ofv/737N7927q6uoCQ6cfeuihYJcmIiIip5iGu4uIiIiIiIi0E3oEm4iIiIiIiEg7oZAuIiIiIiIi0k4opIuIiIiIiIi0E11u4Tifz8e+ffsIDw/HMIxglyMiIiIiIiKdnGmaVFRUkJSUhMXScl95lwvp+/btIzk5OdhliIiIiIiISBezZ8+eoz5etcuF9PDwcMD/w4mIiAhyNSIiIiIiItLZlZeXk5ycHMijLelyIb1xiHtERIRCuoiIiIiIiJwyrZlyrYXjRERERERERNoJhXQRERERERGRdkIhXURERERERKSd6HJz0kVERERERI6koaEBj8cT7DKkA7Lb7Vit1uM+j0K6iIiIiIgIUFlZyd69ezFNM9ilSAdkGAY9e/bE7XYf13kU0kVEREREpMtraGhg7969OJ1OYmNjW7UKt0gj0zQpKChg7969DBgw4Lh61BXSRURERESky/N4PJimSWxsLA6HI9jlSAcUGxvL7t278Xg8xxXStXCciIiIiIjIfupBl7Y6UZ8dhXQRERERERGRdkIhXURERERERKSdUEgXERERERHp4l555RWSk5OxWCw899xzwS4nKB577DFGjBgR7DIU0kVERERERDqq/Px8br75Znr16kVoaCgJCQlMnTqVFStWtPoc5eXl/OxnP+O+++4jOzubn/zkJ0yaNIk777zz5BV+HCZNmoRhGEd89enTp03nveeee/j8889PbLFtoNXdRUREREREOqiZM2fi8Xh44403SElJIS8vj88//5zi4uJWnyMrKwuPx8P06dNJTEw8idWeGB988AH19fUA7Nmzh9GjR/PZZ58xZMgQgMNWVq+vryckJOSo53W73cf9jPMTQT3pIiIiIiIihzBNk+p6b1Bepmm2qsbS0lK+/PJLnn32Wc4991x69+7N6NGjmTNnDtOnTw+0y8rK4tJLL8XtdhMREcEVV1xBXl4eAPPnz2fYsGEApKSkYBgGs2bNYsmSJTz//POB3undu3ezePFiDMNg4cKFnH766TgcDiZPnkx+fj6ffPIJqampREREcNVVV1FdXR24/oIFCxg/fjxRUVHExMRw8cUXs3PnzsD+N998E7fbzfbt2wPbbr/9dgYOHEhVVdVh992tWzcSEhJISEggNjYWgJiYmMC2M888kyeeeIJZs2YRGRnJ7NmzAbjvvvsYOHAgTqeTlJQUHn74YTweT+C8hw53nzVrFt///vf57W9/S2JiIjExMdx2221NjjkZ1JMuIiIiIiJyiBpPA2mPLAzKtTc/PhVnyNGjWmPP70cffcTYsWMJDQ09rI1pmnz/+9/H5XKxZMkSvF4vP/3pT7nyyitZvHgxV155JcnJyZx//vmsXr2a5ORkHA4H27ZtY+jQoTz++OPAgWeAgz/Mzps3D6fTyRVXXMEVV1xBaGgo77zzDpWVlcyYMYMXXniB++67D4Cqqiruuusuhg0bRlVVFY888ggzZsxgw4YNWCwWrrvuOv71r39xzTXXsHz5cj777DNefvllvvrqK1wuV5t+hr/5zW94+OGHeeihhwLbwsPDmT9/PklJSWzatInZs2cTHh7Ovffee8TzLFq0iMTERBYtWsSOHTu48sorGTFiRCD4nwwK6SIiIiIiIh2QzWZj/vz5zJ49m5deeomRI0cyceJEfvjDHzJ8+HAAPvvsM7755hsyMjJITk4G4K9//StDhgxhzZo1nHnmmcTExAD+IJ6QkABASEgITqcz8P5gTzzxBGeffTYAN954I3PmzGHnzp2kpKQAcPnll7No0aJASJ85c2aT419//XXi4uLYvHkzQ4cOBeDll19m+PDh3HHHHXzwwQc8+uijnHnmmW3+2UyePJl77rmnybaDA3ufPn24++67+dvf/tZiSI+OjmbevHlYrVYGDx7M9OnT+fzzzxXSu6SKXMheB6ER0PecYFcjIiIiItKlOOxWNj8+NWjXbq2ZM2cyffp0li1bxooVK1iwYAFz587ltddeY9asWaSnp5OcnBwI6ABpaWlERUWRnp7epiDc+AcAgPj4+MDw8YO3rV69OvB+586dPPzww6xcuZLCwkJ8Ph/gH4bfGNKjo6N5/fXXmTp1KmeddRb333//Mdd1sFGjRh227X//93957rnn2LFjB5WVlXi9XiIiIlo8z5AhQ5rMcU9MTGTTpk3HVdvRaE56e5X+Mbx3Naz4Y7ArERERERHpcgzDwBliC8rLMIxjqjUsLIwpU6bwyCOPsHz5cmbNmsWjjz4K+Ie7N3e+I21vDbvd3uTndPD7xm2NQRzgkksuoaioiFdffZVVq1axatUqgMDib42WLl2K1Wpl3759zc5FPxaHDpNfuXIlP/zhD7nooov417/+xddff82DDz54WA2HOtq9nQwK6e3UbqMnABXZm4NciYiIiIiIdCRpaWmBkJuWlkZWVhZ79uwJ7N+8eTNlZWWkpqYe8RwhISE0NDQcdy1FRUWkp6fz0EMPcd5555GamkpJSclh7ZYvX87cuXP5+OOPiYiI4Pbbbz/uax/sq6++onfv3jz44IOMGjWKAQMGkJmZeUKvcaJouHs7tbE2lj6Aq2oPeOvAdvgiECIiIiIi0nUVFRXxgx/8gBtuuIHhw4cTHh7O2rVrmTt3LpdeeikA559/PsOHD+eaa67hueeeCywcN3HixGaHhDfq06cPq1atYvfu3bjdbrp169amGqOjo4mJieGVV14hMTGRrKysw4ayV1RUcO2113L77bdz0UUX0atXL0aNGsXFF1/MD37wgzZd91D9+/cnKyuL9957jzPPPJN///vffPjhhyfk3CeaetLbqaSefakwHVjwQfGuYJcjIiIiIiLtjNvtZsyYMfz+979nwoQJDB06lIcffpjZs2czb948wD88+6OPPiI6OpoJEyZw/vnnk5KSwt/+9rcWz33PPfdgtVpJS0sjNjaWrKysNtVosVh47733WLduHUOHDuUXv/gFv/nNb5q0+fnPf47L5eKpp54C/PPAn332WW655Rays7PbdN1DXXrppfziF7/gZz/7GSNGjGD58uU8/PDDJ+TcJ5phtvYhfJ1EeXk5kZGRlJWVHXWRgGAqrqon69mxDLPsxHvZfEKHzwh2SSIiIiIinVZtbS0ZGRn07duXsLCwYJcjHVBLn6FjyaHqSW+nvspdyG29fTzSvRtlWd8GuxwRERERERE5BRTS26kwWxjltgZ2hdipz9sa7HJERERERETkFNDCce1Un69zeXK+l4wkK7bTtwe7HBERERERETkFFNLbqRhbFANyoMECdbVZ4POBRQMfREREREREOjOlvnbKOWAQAD2LINvmg4p9Qa5IRERERERETjaF9HYqpG8ffAa4ayHLG4JZsC3YJYmIiIiIiMhJppDeTllCQqiJjwSgtNxO+d7vglyRiIiIiIiInGwK6e2Y2acnAHUVNqr3pQe5GhERERERETnZFNLbMdf+eekhJVaMAj2GTUREREREpLNTSG/HuqeeDkBsMdRXZQS5GhERERER6axeeeUVkpOTsVgsPPfcc8Eup0tTSG/HwgelAtCzEPItlVBTGtyCRERERESkXcnPz+fmm2+mV69ehIaGkpCQwNSpU1mxYkWrz1FeXs7PfvYz7rvvPrKzs/nJT37CpEmTuPPOO09e4cdh0qRJGIZxxFefPn3afO5Zs2bx/e9//4TV2hZ6Tno7Ftq3LwCR1bC5IYRRhdsh+cwgVyUiIiIiIu3FzJkz8Xg8vPHGG6SkpJCXl8fnn39OcXFxq8+RlZWFx+Nh+vTpJCYmnsRqT4wPPviA+vp6APbs2cPo0aP57LPPGDJkCABWqzWY5R039aS3Yxank6rubgCKK+zU520JckUiIiIiIl2EaUJ9VXBeptmqEktLS/nyyy959tlnOffcc+nduzejR49mzpw5TJ8+PdAuKyuLSy+9FLfbTUREBFdccQV5eXkAzJ8/n2HDhgGQkpKCYRjMmjWLJUuW8Pzzzwd6p3fv3s3ixYsxDIOFCxdy+umn43A4mDx5Mvn5+XzyySekpqYSERHBVVddRXV1deD6CxYsYPz48URFRRETE8PFF1/Mzp07A/vffPNN3G4327dvD2y7/fbbGThwIFVVVYfdd7du3UhISCAhIYHY2FgAYmJiAtsKCgqYNm0abreb+Ph4rr32WgoLCwPH/+///i/Dhg3D4XAQExPD+eefT1VVFY899hhvvPEG//d//xe478WLF7fqd3EiqSe9nfP1SYLCbdSV2ynb8x2xo4JdkYiIiIhIF+CphqeSgnPtB/ZBiOuozdxuN263m48++oixY8cSGhp6WBvTNPn+97+Py+ViyZIleL1efvrTn3LllVeyePFirrzySpKTkzn//PNZvXo1ycnJOBwOtm3bxtChQ3n88ccBiI2NZffu3QA89thjzJs3D6fTyRVXXMEVV1xBaGgo77zzDpWVlcyYMYMXXniB++67D4Cqqiruuusuhg0bRlVVFY888ggzZsxgw4YNWCwWrrvuOv71r39xzTXXsHz5cj777DNefvllvvrqK1yuo/8cDpaTk8PEiROZPXs2//M//0NNTQ333XcfV1xxBV988QU5OTlcddVVzJ07lxkzZlBRUcGyZcswTZN77rmH9PR0ysvL+ctf/gL4/yBwqimkt3OO/gNg7TaspVYa8rTCu4iIiIiI+NlsNubPn8/s2bN56aWXGDlyJBMnTuSHP/whw4cPB+Czzz7jm2++ISMjg+TkZAD++te/MmTIENasWcOZZ55JTEwM4A/iCQkJAISEhOB0OgPvD/bEE09w9tlnA3DjjTcyZ84cdu7cSUpKCgCXX345ixYtCoT0mTNnNjn+9ddfJy4ujs2bNzN06FAAXn75ZYYPH84dd9zBBx98wKOPPsqZZx77VN8XX3yRkSNH8tRTTwW2/fnPfyY5OZlt27ZRWVmJ1+vlsssuo3fv3gCBkQQADoeDurq6Zu/7VFFIb+e6Dx5BKf+mW7GBt3xHsMsREREREeka7E5/j3awrt1KM2fOZPr06SxbtowVK1awYMEC5s6dy2uvvcasWbNIT08nOTk5ENAB0tLSiIqKIj09vU1BuPEPAADx8fE4nc5AQG/ctnr16sD7nTt38vDDD7Ny5UoKCwvx+XyAfxh+Y0iPjo7m9ddfZ+rUqZx11lncf//9x1wXwLp161i0aBFut/uwfTt37uSCCy7gvPPOY9iwYUydOpULLriAyy+/nOjo6DZd72RQSG/nogYPpRToWWhS5M2jp7cebCHBLktEREREpHMzjFYNOW8PwsLCmDJlClOmTOGRRx7hpptu4tFHH2XWrFmYpolhGIcdc6TtrWG32wPfG4bR5H3jtsYgDnDJJZeQnJzMq6++SlJSEj6fj6FDhwYWf2u0dOlSrFYr+/bto6qqioiIiGOuzefzcckll/Dss88eti8xMRGr1cp///tfli9fzqeffsoLL7zAgw8+yKpVq+i7f+HuYNPCce1cSL9+AHSrhD3YoHhXkCsSEREREZH2LC0tLbDgWlpaGllZWezZsyewf/PmzZSVlZGamnrEc4SEhNDQ0HDctRQVFZGens5DDz3EeeedR2pqKiUlJYe1W758OXPnzuXjjz8mIiKC22+/vU3XGzlyJN999x19+vShf//+TV6N89sNw+Dss8/mV7/6FV9//TUhISF8+OGHwIm77+OhkN7OWcPDqYpyAJBfEYJZqHnpIiIiIiLiD8CTJ0/mrbfeCsw7/8c//sHcuXO59NJLATj//PMZPnw411xzDevXr2f16tVcd911TJw4kVGjjrwqdZ8+fVi1ahW7d+9uMkT9WEVHRxMTE8Mrr7zCjh07+OKLL7jrrruatKmoqODaa6/l9ttv56KLLuKdd97h73//O//4xz+O+Xq33XYbxcXFXHXVVaxevZpdu3bx6aefcsMNN9DQ0MCqVat46qmnWLt2LVlZWXzwwQcUFBQE/mDRp08fvvnmG7Zu3UphYSEej6dN9308FNI7AG9v/6IFNeV2KvduDnI1IiIiIiLSHrjdbsaMGcPvf/97JkyYwNChQ3n44YeZPXs28+bNA/y9xh999BHR0dFMmDCB888/n5SUFP72t7+1eO577rkHq9VKWloasbGxZGVltalGi8XCe++9x7p16xg6dCi/+MUv+M1vftOkzc9//nNcLldgsbchQ4bw7LPPcsstt5CdnX1M10tKSuKrr76ioaGBqVOnMnToUH7+858TGRmJxWIhIiKCpUuXMm3aNAYOHMhDDz3E7373Oy666CIAZs+ezaBBgxg1ahSxsbF89dVXbbrv42GYZisfwtdJlJeXExkZSVlZWZvmOATDhgduJ/SDz1gyyuSy88cQN+uNYJckIiIiItKp1NbWkpGRQd++fQkLCwt2OdIBtfQZOpYcqp70DqDb4NMACC8x8BZvC3I1IiIiIiIicrIopHcAMan+kN6zEErqsqBrDX4QERERERHpMhTSO4DQ/v0BiCuDvaYPyoP0vEYRERERERE5qRTSOwBbdDTVbv+z0fOrQqBQQ95FREREREQ6I4X0DqIuOQ6AqnI79bnpQa5GRERERERETgaF9A4irH8/AMwyG5XZCukiIiIiIiKdkUJ6BxE5eCgA7hKD+vwtQa5GRERERERETgaF9A4iLnUkAEmFUFKVEeRqRERERERE5GRQSO8gwvav8J5QCrkNlVBbFtyCRERERERE5IRTSO8gbLGx1DhsWEzIrQ6Bwu3BLklERERERDoAwzD46KOPgl0GixcvxjAMSktLj9hm/vz5REVFnbKaAHbv3o1hGGzYsOGUXvdIFNI7CMMwqO7RHYCKCju+gq1BrkhERERERIItPz+fm2++mV69ehEaGkpCQgJTp05lxYoVgTY5OTlcdNFFQazS76yzziInJ4fIyMg2HT9//nwMw2jxtXjx4mM+b3JyMjk5OQwdOrRNdZ1otmAXIK0X0q8P7MjFV2ajKjud8NODXZGIiIiIiATTzJkz8Xg8vPHGG6SkpJCXl8fnn39OcXFxoE1CQkIQKzwgJCTkuGq58sorufDCCwPvL7vsMoYOHcrjjz8e2NatW7fA9x6PB7vdftTzWq3WdvMzAvWkdyhR+1d4d5ZYqMvdHORqREREREQ6L9M0qfZUB+VlmmaraiwtLeXLL7/k2Wef5dxzz6V3796MHj2aOXPmMH369EC7Q4e7L1++nBEjRhAWFsaoUaP46KOPmgz3bhyWvnDhQk4//XQcDgeTJ08mPz+fTz75hNTUVCIiIrjqqquorq4OnLeuro477riDuLg4wsLCGD9+PGvWrAnsb264+/z58+nVqxdOp5MZM2ZQVFR0xPt1OBwkJCQEXiEhITidzsD7l156idGjR/PnP/+ZlJQUQkNDMU2TBQsWMH78eKKiooiJieHiiy9m586dgfMeOty9sc7PP/+cUaNG4XQ6Oeuss9i69dSMZlZPegcSlzqSHCChCEpKd9A92AWJiIiIiHRSNd4axrwzJijXXnX1Kpx251Hbud1u3G43H330EWPHjiU0NPSox1RUVHDJJZcwbdo03nnnHTIzM7nzzjubbfvYY48xb948nE4nV1xxBVdccQWhoaG88847VFZWMmPGDF544QXuu+8+AO69917ef/993njjDXr37s3cuXOZOnUqO3bsaNLDHbjPVau44YYbeOqpp7jssstYsGABjz766FHvoSU7duzg73//O++//z5WqxWAqqoq7rrrLoYNG0ZVVRWPPPIIM2bMYMOGDVgsR+63fvDBB/nd735HbGwst9xyCzfccANfffXVcdXXGgrpHYhr4CAAEosh11PAgAYPWI8+fENERERERDofm83G/PnzmT17Ni+99BIjR45k4sSJ/PCHP2T48OHNHvP2229jGAavvvoqYWFhpKWlkZ2dzezZsw9r+8QTT3D22WcDcOONNzJnzhx27txJSkoKAJdffjmLFi3ivvvuo6qqihdffJH58+cH5r+/+uqr/Pe//+X111/nl7/85WHnf/7555k6dSr3338/AAMHDmT58uUsWLCgzT+T+vp6/vrXvxIbGxvYNnPmzCZtXn/9deLi4ti8eXOL89CffPJJJk6cCMD999/P9OnTqa2tJSwsrM31tYZCegdiS0ykLsRKaH0DuTV2KN4FsYOCXZaIiIiISKfjsDlYdfWqoF27tWbOnMn06dNZtmwZK1asYMGCBcydO5fXXnuNWbNmHdZ+69atDB8+vEnQHD16dLPnPjjox8fH43Q6AwG9cdvq1asB2LlzJx6PJxDqAex2O6NHjyY9Pb3Z86enpzNjxowm28aNG3dcIb13795NAnpjbQ8//DArV66ksLAQn88HQFZWVosh/eD7T0xMBPwL9fXq1avN9bWGQnoHYhgGFYnRhGYWUl5ug8JtCukiIiIiIieBYRitGnLeHoSFhTFlyhSmTJnCI488wk033cSjjz7abEg3TRPDMA7b1pyDF10zDOOwRdgMwwgE3sZzNHfuQ7cd7brHw+VyHbbtkksuITk5mVdffZWkpCR8Ph9Dhw6lvr6+xXMdev9A4H5PJi0c18FYUvx/tfGW2ajPbf4vUiIiIiIi0nWlpaVRVVXV7L7BgwfzzTffUFdXF9i2du3a475m//79CQkJ4csvvwxs83g8rF27ltTU1CPWuXLlyibbDn1/vIqKikhPT+ehhx7ivPPOIzU1lZKSkhN6jRNNIb2DiRrs/4CHlVqpylZIFxERERHpqoqKipg8eTJvvfUW33zzDRkZGfzjH/9g7ty5XHrppc0ec/XVV+Pz+fjJT35Ceno6Cxcu5Le//S1weC/4sXC5XNx666388pe/ZMGCBWzevJnZs2dTXV3NjTfe2Owxd9xxR2B4/rZt25g3b95xDXVvTnR0NDExMbzyyivs2LGDL774grvuuuuEXuNEU0jvYOLTRgEQWwylxafmEQAiIiIiItL+uN1uxowZw+9//3smTJjA0KFDefjhh5k9ezbz5s1r9piIiAg+/vhjNmzYwIgRI3jwwQd55JFHAI57QbRnnnmGmTNncu211zJy5Eh27NjBwoULiY6Obrb92LFjee2113jhhRcYMWIEn376KQ899NBx1XAoi8XCe++9x7p16xg6dCi/+MUv+M1vfnNCr3GiGebJmAjQjpWXlxMZGUlZWRkRERHBLueY1WdlsfOCqdTboOZHVYy9LxOO4y9eIiIiIiICtbW1ZGRk0Ldv35O+end78/bbb/PjH/+YsrIyHI7WL1onTbX0GTqWHKqF4zoYe48eeGwGIV6T3TU+qMiBiKRglyUiIiIiIh3Em2++SUpKCj169GDjxo3cd999XHHFFQro7YRCegdjWK2UxUXSfV8pJRUh/hXeFdJFRERERKSVcnNzeeSRR8jNzSUxMZEf/OAHPPnkk8EuS/ZTSO+AGvr0gH2leMps+PK3YkmZFOySRERERESkg7j33nu59957g12GHIEWjuuAGld4t5daqcnRCu8iIiIiIiKdhUJ6B5Qw5AwAYoqhLHdzkKsRERERERGRE0UhvQOKTh0OQM9C2FeZEeRqRERERERE5ERRSO+AQpKTabBAmAfyq6qhtjzYJYmIiIiIiMgJ0G5C+tNPP41hGNx5550ttluyZAlnnHEGYWFhpKSk8NJLL52aAtsRw26nJDYcgOKKECjcHuSKRERERERE5ERoFyF9zZo1vPLKKwwfPrzFdhkZGUybNo1zzjmHr7/+mgceeIA77riD999//xRV2n7U9UwAoKbc5n8Mm4iIiIiIiHR4QQ/plZWVXHPNNbz66qtER0e32Pall16iV69ePPfcc6SmpnLTTTdxww038Nvf/vYUVdt+hA8aBPhXePfkbwlyNSIiIiIi0l4ZhsFHH30U7DJYvHgxhmFQWlp6xDbz588nKirqlNXUHgU9pN92221Mnz6d888//6htV6xYwQUXXNBk29SpU1m7di0ej6fZY+rq6igvL2/y6gyShvlXeI8qNijL/i7I1YiIiIiISDDk5+dz880306tXL0JDQ0lISGDq1KmsWLEi0CYnJ4eLLrooiFX6nXXWWeTk5BAZGdmm4+fPn49hGC2+Fi9e3KZzt+YPCKeKLZgXf++991i/fj1r1qxpVfvc3Fzi4+ObbIuPj8fr9VJYWEhiYuJhxzz99NP86le/OiH1tifdU0dQCfQogr0l2+ke7IJEREREROSUmzlzJh6PhzfeeIOUlBTy8vL4/PPPKS4uDrRJSEgIYoUHhISEHFctV155JRdeeGHg/WWXXcbQoUN5/PHHA9u6det2XDW2B0HrSd+zZw8///nPeeuttwgLC2v1cYZhNHlvmmaz2xvNmTOHsrKywGvPnj1tL7odCe3bF58B7lrIqyiChuZHEoiIiIiIyLEzTRNfdXVQXo0Z52hKS0v58ssvefbZZzn33HPp3bs3o0ePZs6cOUyfPj3Q7tDh7suXL2fEiBGEhYUxatQoPvroIwzDYMOGDcCBXuWFCxdy+umn43A4mDx5Mvn5+XzyySekpqYSERHBVVddRXV1deC8dXV13HHHHcTFxREWFsb48eObdMg211s9f/58evXqhdPpZMaMGRQVFR3xfh0OBwkJCYFXSEgITqcz8L5bt2489NBD9OjRA5fLxZgxY5r0rGdmZnLJJZcQHR2Ny+ViyJAh/Oc//2H37t2ce+65AERHR2MYBrNmzWrV7+BkCFpP+rp168jPz+eMM84IbGtoaGDp0qXMmzePuro6rFZrk2MSEhLIzc1tsi0/Px+bzUZMTEyz1wkNDSU0NPTE30CQWUJDKe7mpHtRNcWVNijOgNiBwS5LRERERKRTMGtq2DryjKM3PAkGrV+H4XQetZ3b7cbtdvPRRx8xduzYVuWeiooKLrnkEqZNm8Y777xDZmbmEZ+w9dhjjzFv3jycTidXXHEFV1xxBaGhobzzzjtUVlYyY8YMXnjhBe677z4A7r33Xt5//33eeOMNevfuzdy5c5k6dSo7duxotod71apV3HDDDTz11FNcdtllLFiwgEcfffSo93AkP/7xj9m9ezfvvfceSUlJfPjhh1x44YVs2rSJAQMGcNttt1FfX8/SpUtxuVxs3rwZt9tNcnIy77//PjNnzmTr1q1ERETgcDjaXMfxClpIP++889i0aVOTbT/+8Y8ZPHgw991332EBHWDcuHF8/PHHTbZ9+umnjBo1CrvdflLrbY9qesRB0W5qyvav8K6QLiIiIiLSZdhsNubPn8/s2bN56aWXGDlyJBMnTuSHP/zhEZ+c9fbbb2MYBq+++iphYWGkpaWRnZ3N7NmzD2v7xBNPcPbZZwNw4403MmfOHHbu3ElKSgoAl19+OYsWLeK+++6jqqqKF198kfnz5wfmv7/66qv897//5fXXX+eXv/zlYed//vnnmTp1Kvfffz8AAwcOZPny5SxYsOCYfxY7d+7k3XffZe/evSQlJQFwzz33sGDBAv7yl7/w1FNPkZWVxcyZMxk2bBhA4D7gwDD5uLi4oC9cF7SQHh4eztChQ5tsc7lcxMTEBLbPmTOH7Oxs3nzzTQBuueUW5s2bx1133cXs2bNZsWIFr7/+Ou++++4pr789CO3XD77ZjaXMhq9gK5bUi4NdkoiIiIhIp2A4HAxavy5o126tmTNnMn36dJYtW8aKFStYsGABc+fO5bXXXmt2yPbWrVsZPnx4kynHo0ePbvbcBwf9+Ph4nE5nk2AbHx/P6tWrAX9I9ng8gVAPYLfbGT16NOnp6c2ePz09nRkzZjTZNm7cuDaF9PXr12OaJgMHNu24rKurC4y6vuOOO7j11lv59NNPOf/885k5c+ZRHwMeDEFdOO5ocnJyyMrKCrzv27cv//nPf/jFL37BH//4R5KSkvjDH/7AzJkzg1hl8PQYPhLPh58TUWxQtW8z4cEuSERERESkkzAMo1VDztuDsLAwpkyZwpQpU3jkkUe46aabePTRR5sN6aZpHnGdr0MdPFrZMIzDRi8bhoHP52tyjubOfaT1w1o79741fD4fVquVdevWHTYq2+12A3DTTTcxdepU/v3vf/Ppp5/y9NNP87vf/Y7bb7/9hNVxIgT9EWwHW7x4Mc8991zg/fz58w9bQn/ixImsX7+euro6MjIyuOWWW05tke1I4tBR/q9FsKdQz0oXERERERFIS0ujqqqq2X2DBw/mm2++oa6uLrBt7dq1x33N/v37ExISwpdffhnY5vF4WLt2LampqUesc+XKlU22Hfq+tU4//XQaGhrIz8+nf//+TV4HryifnJzMLbfcwgcffMDdd9/Nq6++CvhXngf/OmnB1q5CuhwbR7/+AERVQ27pXjiBf4kSEREREZH2raioiMmTJ/PWW2/xzTffkJGRwT/+8Q/mzp3LpZde2uwxV199NT6fj5/85Cekp6ezcOFCfvvb3wJHfmJWa7hcLm699VZ++ctfsmDBAjZv3szs2bOprq7mxhtvbPaYO+64IzA8f9u2bcybN69NQ93BP5/9mmuu4brrruODDz4gIyODNWvW8Oyzz/Kf//wHgDvvvJOFCxeSkZHB+vXr+eKLLwJ/QOjduzeGYfCvf/2LgoICKisr2/aDOAEU0jswi9NJcZR/BcfCcqAit+UDRERERESk03C73YwZM4bf//73TJgwgaFDh/Lwww8ze/Zs5s2b1+wxERERfPzxx2zYsIERI0bw4IMP8sgjjwAc06Oxm/PMM88wc+ZMrr32WkaOHMmOHTtYuHAh0dHRzbYfO3Ysr732Gi+88AIjRozg008/5aGHHmrz9f/yl79w3XXXcffddzNo0CC+973vsWrVKpKTkwF/L/ltt91GamoqF154IYMGDeJPf/oTAD169OBXv/oV999/P/Hx8fzsZz9rcx3HyzBP5ESADqC8vJzIyEjKysqIiIgIdjnHbcH3z6f3lmxWTfIw697XIWVisEsSEREREelwamtrycjIoG/fvscdVjuat99+mx//+MeUlZUF9dFjHV1Ln6FjyaHteuE4OTpL376wJRtKrf7HsCmki4iIiIhIC958801SUlLo0aMHGzdu5L777uOKK65QQG8nFNI7uIShp8EnX+IqtVCfm05IsAsSEREREZF2LTc3l0ceeYTc3FwSExP5wQ9+wJNPPhnssmQ/hfQOru/pY9nLH0ksgr2535Jy9ENERERERKQLu/fee7n33nuDXYYcgRaO6+CcAwYBEFMBe4ozg1yNiIiIiIiIHA+F9A7OGh5OqdsOQHFJDdRVBLkiEREREZGOq4utqy0n0In67CikdwKlCd0AqKiw+xePExERERGRY2K1WgGor68PciXSUTV+dho/S22lOemdQEOvXrAjD1+ZDQq3Q48zgl2SiIiIiEiHYrPZcDqdFBQUYLfbsVjUnymt5/P5KCgowOl0YrMdX8xWSO8EuqUNhS/W4CyxYBZswwh2QSIiIiIiHYxhGCQmJpKRkUFmptZ6kmNnsVjo1asXhnF8iUwhvRMYOHIsBfyF+CLI3beJxGAXJCIiIiLSAYWEhDBgwAANeZc2CQkJOSEjMBTSO4Go1GEUAHFlsKtgm0K6iIiIiEgbWSwWwsLCgl2GdGGaaNEJ2KKjqXD6FycoKiyFBk9wCxIREREREZE2UUjvJArjogCoKLdCye6g1iIiIiIiIiJto5DeSdT37AmAt8ymx7CJiIiIiIh0UArpnYR7wGAAwkqtULA1yNWIiIiIiIhIWyikdxL9Ro4FoHsRlOVtDnI1IiIiIiIi0hYK6Z1Ez9NGApBQCjty04NbjIiIiIiIiLSJQnonYYuNpSrUgsWEvNwcMM1glyQiIiIiIiLHSCG9kzAMg4LYcADKy0yozAtyRSIiIiIiInKsFNI7keqkRAA8ZXat8C4iIiIiItIBKaR3IiEpAwGwl1q0wruIiIiIiEgHpJDeifQcfiYA3YoN6gq2BLkaEREREREROVYK6Z3IoDP9j2FLLIbt2ZuCXI2IiIiIiIgcK4X0TsTRswe1dgObD/btywp2OSIiIiIiInKMFNI7EcMwyI9xAlBWXAt1FUGuSERERERERI6FQnonU5aYAEBduR0Ktwe5GhERERERETkWCumdjNE7BQBbqVUhXUREREREpINRSO9kYoeMBCCy2KBBK7yLiIiIiIh0KArpnUzamWcBkFQMu3O/CXI1IiIiIiIiciwU0juZ2AH98FghxAu7s3YEuxwRERERERE5BgrpnYxhtZLXzQFAWX45NHiDXJGIiIiIiIi0lkJ6J1QcHwtATbkVSnYHtxgRERERERFpNYX0Tsib3AcAa6kVCrcFtxgRERERERFpNYX0Tihy4DAAwkssmFrhXUREREREpMNQSO+EBp4xHoDEIsjJ2xTkakRERERERKS1FNI7ob7DU/FawFEPOzPVky4iIiIiItJRKKR3QvawUPKjQwAo2lcAphnkikRERERERKQ1FNI7qYLu3QCoLTWhMj/I1YiIiIiIiEhrKKR3UrU9evu/KbNphXcREREREZEOQiG9k3L0HwKAq8QChVuDXI2IiIiIiIi0hkJ6J9VnxFgAEoqgJO/bIFcjIiIiIiIiraGQ3kkNOv10fAa4a2FnpkK6iIiIiIhIR6CQ3klFRbspiLQBkJuVHeRqREREREREpDUU0juxvJgoAKqK66GuMrjFiIiIiIiIyFEppHdiFYk9ATDLrFC0PcjViIiIiIiIyNEopHditr6DAXCWWKFQIV1ERERERKS9U0jvxJKGjAIgtgiq878LcjUiIiIiIiJyNArpndiAkaMBiKqGXZkbgluMiIiIiIiIHJVCeieW3COGwnArAHszdge3GBERERERETkqhfROzGa1kBvjBqCioAoavEGuSERERERERFqikN7JFcf1AMBXZoHSzCBXIyIiIiIiIi1RSO/kzOT+AISVWKBwW5CrERERERERkZYopHdy3VLPACCm2KA+f3OQqxEREREREZGWKKR3cv1PHwNATAXszvo6yNWIiIiIiIhISxTSO7l+KUmUuPy/5j27tge5GhEREREREWmJQnonF+mwk9PNCUBZTjGYZpArEhERERERkSNRSO8CCrrHA+AtAaoKgluMiIiIiIiIHJFCehdQ19O/wntIqVUrvIuIiIiIiLRjCuldQHj/0wCILjZoyE8PcjUiIiIiIiJyJArpXUCfEWMB6F4G2Xu0wruIiIiIiEh7pZDeBfTr34sKh4EFyNypZ6WLiIiIiIi0VwrpXUDPaCf7osMAKN6bH+RqRERERERE5EgU0rsAq8UgL6Y7APVFHqivCnJFIiIiIiIi0hyF9C6iPCEFAFupFQq3B7kaERERERERaY5CehcRljIUgMgSA7NAj2ETERERERFpjxTSu4ikIf4V3uNLIG/v+iBXIyIiIiIiIs0Jakh/8cUXGT58OBEREURERDBu3Dg++eSTI7ZfvHgxhmEc9tqyZcsprLpjShmUQnUoWEzI3L4x2OWIiIiIiIhIM2zBvHjPnj155pln6N+/PwBvvPEGl156KV9//TVDhgw54nFbt24lIiIi8D42Nvak19rRpcSFsyw6lH65dRTs2RfsckRERERERKQZQQ3pl1xySZP3Tz75JC+++CIrV65sMaTHxcURFRV1kqvrXCIddvZFR9EvN4/a/BrwNYDFGuyyRERERERE5CDtZk56Q0MD7733HlVVVYwbN67FtqeffjqJiYmcd955LFq0qMW2dXV1lJeXN3l1VUVxfQCwlhpQsjuotYiIiIiIiMjhgh7SN23ahNvtJjQ0lFtuuYUPP/yQtLS0ZtsmJibyyiuv8P777/PBBx8waNAgzjvvPJYuXXrE8z/99NNERkYGXsnJySfrVto9o5f/5xpeYtFj2ERERERERNohwzRNM5gF1NfXk5WVRWlpKe+//z6vvfYaS5YsOWJQP9Qll1yCYRj885//bHZ/XV0ddXV1gffl5eUkJydTVlbWZF57V/D6B4s564Fb8Vqg5x+vI/rcOcEuSUREREREpNMrLy8nMjKyVTk06D3pISEh9O/fn1GjRvH0009z2mmn8fzzz7f6+LFjx7J9+5F7hUNDQwOrxze+uqqUwYOptYPNB7u3rgt2OSIiIiIiInKIoIf0Q5mm2aTn+2i+/vprEhMTT2JFnUdKfDi50XYA8jIyg1yNiIiIiIiIHCqoq7s/8MADXHTRRSQnJ1NRUcF7773H4sWLWbBgAQBz5swhOzubN998E4DnnnuOPn36MGTIEOrr63nrrbd4//33ef/994N5Gx1Gz2gnn0WF0ye/mOq8cjBNMIxglyUiIiIiIiL7BTWk5+Xlce2115KTk0NkZCTDhw9nwYIFTJkyBYCcnByysrIC7evr67nnnnvIzs7G4XAwZMgQ/v3vfzNt2rRg3UKHYrUY5MUkA8UYJUBVIbj1jHkREREREZH2IugLx51qxzJhvzN67OHHuPIff2NvHEx58zXoc3awSxIREREREenUOtTCcXJqRQwYBUBcMVTlbAxyNSIiIiIiInIwhfQupnfqMOqtEOKF3VtXB7scEREREREROYhCehfTLyGS/GgrALk7dwS5GhERERERETmYQnoXkxLrYl+UC4CKfcVBrkZEREREREQOppDexUSE2cmO9j9X3iz2Qn11kCsSERERERGRRgrpXVB5wiAAnCUWKNoe5GpERERERESkkUJ6F+RIOR2A2GKD+rz0IFcjIiIiIiIijRTSu6Aeg07HawFHPWRuWR7sckRERERERGQ/hfQuqF9SNAVR/l/9vu3fBbkaERERERERaaSQ3gX1i3WTE+UAoDy7IMjViIiIiIiISCOF9C6oR7SDPZHdAWgoqANfQ5ArEhEREREREVBI75KsFoPC7gMACCsxoDQzyBWJiIiIiIgIKKR3WUbv0wCIKTbw5m8JcjUiIiIiIiICCuldVvzAM/AZ4K6F7K1fBbscERERERERQSG9y0rpGUthpAFA9taNQa5GREREREREQCG9y0qJdZMbFQZASda+IFcjIiIiIiIioJDeZaXEusiOiAbAm18NphnkikREREREREQhvYuKCLOzJ7ovACHFQHVRcAsSERERERERhfSurL7nMACiiw3Mgq1BrkZEREREREQU0ruwqP5nAhBZDbnbtcK7iIiIiIhIsCmkd2EpyYkUhftXeN/z3dogVyMiIiIiIiIK6V1YSqyLvOgQAIozs4JcjYiIiIiIiCikd2H9urvZFxkBQF1ueZCrEREREREREYX0LqxHtIOMiGQA7EUNUF8d5IpERERERES6NoX0LsxqMSiNHwpAZIkFinYEuSIREREREZGuTSG9i3OnjAKgWwUU71oV5GpERERERES6NoX0Lq5Xci9KXf7vMzctD24xIiIiIiIiXZxCeheXEusir5sdgKJdO4NcjYiIiIiISNemkN7F9Yt1kxMZDkBNTkmQqxEREREREenaFNK7uJRYF5nuBAAshR7wNQS5IhERERERka5LIb2LCw+zszc6FYCIYgNKs4JckYiIiIiISNelkC5Yeo8EoFsZVGauD3I1IiIiIiIiXZdCupDcewAVDv+HIXPjkmCXIyIiIiIi0mUppAspsW7yulkByN+RHuRqREREREREui6FdPE/hi3S/7D06uyCIFcjIiIiIiLSdSmkC/1j3ewJjwXAyK8LcjUiIiIiIiJdl0K6kBTlYHv4AADcxSZUFQW5IhERERERka5JIV2wWgzqks4AIKbUoH7vxiBXJCIiIiIi0jUppAsAccmDqAoFiwmZGz4LdjkiIiIiIiJdkkK6ANA/PoL8bv6PQ+7Wb4NcjYiIiIiISNekkC6Af4X3/EgHAJV79gW5GhERERERka5JIV0ASOnuJjsiBgAzrzrI1YiIiIiIiHRNCukC+HvSt7v6AOAs8oGnJrgFiYiIiIiIdEEK6QJAeJidnJjTAIgpMfDmpQe5IhERERERka5HIV0CopOHU2sHmw+yv/5vsMsRERERERHpchTSJaBfXGRghfd9360LcjUiIiIiIiJdj0K6BPSLdZMfFQJAeeaeIFcjIiIiIiLS9SikS0BKrIuciCgAGnIrgluMiIiIiIhIF6SQLgH9Yt1kuJIBCCvygs8X5IpERERERES6FoV0CUiKcrDVPRyAmGIDX9GuIFckIiIiIiLStSikS4DVYhCaNJx6K4R4Ie+bL4JdkoiIiIiISJeikC5NpMR3J7+bAcDeb1YEuRoREREREZGuRSFdmkiJdVEQZQegNEPD3UVERERERE4lhXRpIiXWRX5EBADefSVBrkZERERERKRrUUiXJvrFusl0JwEQUugJcjUiIiIiIiJdS5tC+p49e9i7d2/g/erVq7nzzjt55ZVXTlhhEhx9u7v41pkKQHQxmJWFQa5IRERERESk62hTSL/66qtZtGgRALm5uUyZMoXVq1fzwAMP8Pjjj5/QAuXUCg+zUxlzOl4LOOqh5LvFwS5JRERERESky2hTSP/2228ZPXo0AH//+98ZOnQoy5cv55133mH+/Pknsj4Jgj5x8eRH+b/fs35xMEsRERERERHpUtoU0j0eD6GhoQB89tlnfO973wNg8ODB5OTknLjqJCj6xbkoirYBULRja5CrERERERER6TraFNKHDBnCSy+9xLJly/jvf//LhRdeCMC+ffuIiYk5oQXKqZfS3U1+pBuA+mzNSRcRERERETlV2hTSn332WV5++WUmTZrEVVddxWmnnQbAP//5z8AweOm4UmJd7HXHA2AvqA1yNSIiIiIiIl2HrS0HTZo0icLCQsrLy4mOjg5s/8lPfoLT6TxhxUlw9It1s8U5CNhKZBGY9TUYIY5glyUiIiIiItLptaknvaamhrq6ukBAz8zM5LnnnmPr1q3ExcWd0ALl1OsR5WCn8zR8gKsWKrcuD3ZJIiIiIiIiXUKbQvqll17Km2++CUBpaSljxozhd7/7Hd///vd58cUXT2iBcupZLAY94npSEOV/n7Xuv0GtR0REREREpKtoU0hfv34955xzDgD/+7//S3x8PJmZmbz55pv84Q9/OKEFSnD0i3VTGG0FoGjrd0GuRkREREREpGtoU0ivrq4mPDwcgE8//ZTLLrsMi8XC2LFjyczMPKEFSnD0i3VRGOVfX6BmT26QqxEREREREeka2hTS+/fvz0cffcSePXtYuHAhF1xwAQD5+flERESc0AIlOFJi3exz+x+nZ82vDnI1IiIiIiIiXUObQvojjzzCPffcQ58+fRg9ejTjxo0D/L3qp59++gktUIIjJdbFDlcKABGFPvD5glyRiIiIiIhI59emR7BdfvnljB8/npycnMAz0gHOO+88ZsyYccKKk+BJiXWzKfR04AvCq6F29wbCUkYGuywREREREZFOrU096QAJCQmcfvrp7Nu3j+zsbABGjx7N4MGDW32OF198keHDhxMREUFERATjxo3jk08+afGYJUuWcMYZZxAWFkZKSgovvfRSW29BWuAOteGI7E3h/tkLWWsWBLcgERERERGRLqBNId3n8/H4448TGRlJ79696dWrF1FRUfz617/GdwzDonv27MkzzzzD2rVrWbt2LZMnT+bSSy/lu++aX008IyODadOmcc455/D111/zwAMPcMcdd/D++++35TbkKPrHhlPQzf8RKUj/OsjViIiIiIiIdH5tGu7+4IMP8vrrr/PMM89w9tlnY5omX331FY899hi1tbU8+eSTrTrPJZdc0uT9k08+yYsvvsjKlSsZMmTIYe1feuklevXqxXPPPQdAamoqa9eu5be//S0zZ85sy61IC1JiXRRFhgI1VGfuDXY5IiIiIiIinV6bQvobb7zBa6+9xve+973AttNOO40ePXrw05/+tNUh/WANDQ384x//oKqqKrAQ3aFWrFgRWEm+0dSpU3n99dfxeDzY7fbDjqmrq6Ouri7wvry8/Jhr66pSurvZFx4N1GDkVgS7HBERERERkU6vTcPdi4uLm517PnjwYIqLi4/pXJs2bcLtdhMaGsott9zChx9+SFpaWrNtc3NziY+Pb7ItPj4er9dLYWFhs8c8/fTTREZGBl7JycnHVF9X1i/OTYarNwDuwoYgVyMiIiIiItL5tSmkn3baacybN++w7fPmzWP48OHHdK5BgwaxYcMGVq5cya233sr111/P5s2bj9jeMIwm703TbHZ7ozlz5lBWVhZ47dmz55jq68pSurv4JmwEAJEVUJ+3O6j1iIiIiIiIdHZtGu4+d+5cpk+fzmeffca4ceMwDIPly5ezZ88e/vOf/xzTuUJCQujfvz8Ao0aNYs2aNTz//PO8/PLLh7VNSEggNze3ybb8/HxsNhsxMTHNnj80NJTQ0NBjqkn8ekQ5KLf3o8QN0ZWwb/V/6HPJT4NdloiIiIiISKfVpp70iRMnsm3bNmbMmEFpaSnFxcVcdtllfPfdd/zlL385roJM02wyh/xg48aN47///W+TbZ9++imjRo1qdj66HB+LxaBv93AKov2jFPI2rQxyRSIiIiIiIp1bm3rSAZKSkg5bIG7jxo288cYb/PnPf27VOR544AEuuugikpOTqaio4L333mPx4sUsWOB/JvecOXPIzs7mzTffBOCWW25h3rx53HXXXcyePZsVK1bw+uuv8+6777b1NuQo+sW5KY4KgT11VGbsDnY5IiIiIiIinVqbQ/qJkJeXx7XXXktOTg6RkZEMHz6cBQsWMGXKFABycnLIysoKtO/bty//+c9/+MUvfsEf//hHkpKS+MMf/qDHr51E/bq7yI+IBPIxc0qDXY6IiIiIiEinFtSQ/vrrr7e4f/78+YdtmzhxIuvXrz9JFcmhUmLdfObqAeTjLPAEuxwREREREZFOrU1z0qXr6Bfr5lvHMAAiy6Ch7NgesSciIiIiIiKtd0w96ZdddlmL+0tLS4+nFmmH+sa62G1JpdwBETWQt3YBSeddHeyyREREREREOqVjCumRkZFH3X/dddcdV0HSvrhDbSSEh5PfDSKyIWfjUoV0ERERERGRk+SYQvrxPl5NOqaUWBfF0XbI9lC+Y0ewyxEREREREem0NCddjqpfrJuiCDcADdlFQa5GRERERESk81JIl6NKiXWR7U4AIKygLsjViIiIiIiIdF4K6XJUKbFutjpSAYgqMfHV1ga5IhERERERkc5JIV2Oql+si3TjNKpDwWpC8YZFwS5JRERERESkU1JIl6NKinRgt4eT283/Pnv9Z8EtSEREREREpJNSSJejslgM+nZ3UxxtBaB025YgVyQiIiIiItI5KaRLq6TEuiiJcAHg2ZMX5GpEREREREQ6J4V0aZV+sW5y3d0BCM2vCXI1IiIiIiIinZNCurRKv1gXO5z9AYgq8mF6vUGuSEREREREpPNRSJdWSenu5jvbSGrtYPNBefraYJckIiIiIiLS6SikS6ukxLqo8nUPrPC+d83C4BYkIiIiIiLSCSmkS6u4Qm0kRIRREu3/yJSkbwxyRSIiIiIiIp2PQrq0Wr84FyWRYQDUZ+0LcjUiIiIiIiKdj0K6tFpKdzf54TEA2HIrg1yNiIiIiIhI56OQLq2WEusiy9kbgMiiBkyfL8gViYiIiIiIdC4K6dJq/WLdfGc7DY8VQrxQk7E92CWJiIiIiIh0Kgrp0mopsS7yfb0DK7xnr/4kuAWJiIiIiIh0Mgrp0mpJkQ7C7DaKow0ACr9bHeSKREREREREOheFdGk1i8Wgb3c3pVEhANRmZAW5IhERERERkc5FIV2OSb9YF0XhkQBYcsuDXI2IiIiIiEjnopAuxyQl1k22MxmAiAIPpmkGuSIREREREZHOQyFdjkm/WBfpIcNoMCCsHuqy9wa7JBERERERkU5DIV2OSb9YN3saBpIb7X+fs+7T4BYkIiIiIiLSiSikyzHp292FlxCK9z+GrXDj8uAWJCIiIiIi0okopMsxcYXaSIwMoyzKDkDVrl1BrkhERERERKTzUEiXY5YS66I4IhwAY19xkKsRERERERHpPBTS5Zj1i3WT60oEIDy/Xiu8i4iIiIiInCAK6XLMUrq72B6Sig9w1IK3sDDYJYmIiIiIiHQKCulyzFJi3WT4hpAf5X+fv2FpUOsRERERERHpLBTS5Zj1i3NTZYZTtH+F97yvFdJFREREREROBIV0OWaJEWGE2S2URVkBqNy+JcgViYiIiIiIdA4K6XLMLBaDlO5uyiKcAJh7C4JckYiIiIiISOegkC5tkhLrIs8dD4ArvzbI1YiIiIiIiHQOCunSJv1i3exyDADAVWXSUFoa3IJEREREREQ6AYV0aZOUWBe7G4ZQGOF/X/TduuAWJCIiIiIi0gkopEub9It1U+CLJ3//Cu+56z8PbkEiIiIiIiKdgEK6tEnf7i7AoDza/xEq37IpuAWJiIiIiIh0Agrp0iauUBuJkWGUR4QB0JCVG+SKREREREREOj6FdGmzfrFuitwxADjyqoNcjYiIiIiISMenkC5tlhLrIsvRF4Dwch8NlZVBrkhERERERKRjU0iXNkvp7mI3Qyhx+d+Xb/0uuAWJiIiIiIh0cArp0mb94tzs9fYm1z/inZz1XwS3IBERERERkQ5OIV3aLCXWjRcbFdEGAKWb1we5IhERERERkY5NIV3aLDEiDIfdSkVkCADe3XuDXJGIiIiIiEjHppAubWaxGPTt7qLEHQVASG5FcAsSERERERHp4BTS5bj0i3Ozz5kMQHhJA76amiBXJCIiIiIi0nEppMtxSenuIpPBlDv8H6bqnduDXZKIiIiIiEiHpZAuxyUl1kWWdwA5+1d43/f1kuAWJCIiIiIi0oEppMtx6RfrphIX5dH+9yXfrQluQSIiIiIiIh2YQrocl5RYFwCVkTYA6nZlBLMcERERERGRDk0hXY6LM8RGUmQYZeHhANj3lQa3IBERERERkQ5MIV2OW0qsmzxXEgDhRV7M+vogVyQiIiIiItIxKaTLcesX62KPZRDVoWA1IeuWW6hYvBjT5wt2aSIiIiIiIh2KQroct5RYN3vqB/LZ6QY+oHr5Cvbecis7L7yIovnzaSgvD3aJIiIiIiIiHYJCuhy3lFgX+cSwc4yXn99i5ePRBlVh4MnKIv+ZZ9k+cRI5jz5G7bZtwS5VRERERESkXVNIl+PWL9YNGJy5bwQ/9xaxbryXW26z8vJFFjJjwaypofRvfyPje5eSed31lC/8FNPrDXbZIiIiIiIi7Y5hmqYZ7CJOpfLyciIjIykrKyMiIiLY5XQKPp/JkEcXUuNpYMlP+tMr9zM2pv+Dt2uz+K/TwcC9Bheu8zF6q4l1/6fNlphI9A9/SNQPLsfWrVtwb0BEREREROQkOpYcqpAuJ8T0Pyzju33lvHbdKM5Pi/dvLMsm95t3+PuOD/iHWYa1ysL5X/uYssEkotrfxAixEzFtOtHXXINj2NDg3YCIiIiIiMhJciw5VMPd5YRIiXUDsLOg8sDGyB4knPNL7vjxCv4749/8fOB5bJgQxq23WZl3sYUdCWDWeyj76CN2/+AH7L78Mso+/liPcBMRERERkS5LIV1OiH6xLgB2FVQ1uz8sui8zpj7PP65fxyvn/o7QcQN4aJaNB66zsnSIgddiUvNtOvt+eS/bx4+l4DdP4snLP5W3ICIiIiIiEnS2YBcgnUOzPenNMAyDUSlTGZUylX2V+3jvm9d5M/n/+Ot5tZy3wWTqeh/R5TUUvv4WhX95i4hR/Ym+4VYcEy/CMIxTcSsiIiIiIiJBoznpckJ8m13GxS98STdXCOsfnnJMx9Z4a/jXrn/xzndvklGSwehtJlPX+Ujbc6BNaKyNbtPHE3H9L7AkDjzB1YuIiIiIiJw8WjiuBQrpJ0d1vZe0RxYC8I9bxnFGr2gslmPr+TZNk9W5q3k7/W0W71lMrzwfF67zcc53PkK8/nNZQ3xEDQ8n6geXEjLhOojufaJvRURERERE5IRSSG+BQvrJM2HuIrKK/cu2x4WHMnlwHJMHxzF+QHecIcc2s2JPxR7e2/IeH27/EF95OZM3mly43kds2f4Ghok7qZZuYxNxXngFxtAZEN3nxN6QiIiIiIjICaCQ3gKF9JNndUYxf/4yg2XbC6iqbwhsD7FZGJcSw3mp/tDeM9rZ6nNWe6r5585/8nb622SWZjByp8mFa01O233gYxsS4SF6QBWRY/tjPf0ySPs+dOt7Im9NRERERESkzTpMSH/66af54IMP2LJlCw6Hg7POOotnn32WQYMGHfGYxYsXc+655x62PT09ncGDBx/1mgrpJ1+dt4HVGcV8np7P51vy2FNc02T/oPhwJqfGcd7gOE7vFY21FcPifaaPlftW8lb6WyzLXkZSkcmFa32c+y2E1vs/wha7j8i+1UT3ryJ00FB/WB/yfeiWchLuUkREREREpHU6TEi/8MIL+eEPf8iZZ56J1+vlwQcfZNOmTWzevBmXy9XsMY0hfevWrU1uLjY2FqvVetRrKqSfWqZpsrOgcn9gz2ft7mJ8B33iop12Jg3y97BPGBhLpMN+1HNmlmfy7pZ3+WjHR/gqK5m4yWTaekgo8gXauBJqiR5QhTupDiPpNAV2EREREREJmg4T0g9VUFBAXFwcS5YsYcKECc22aQzpJSUlREVFHfWcdXV11NXVBd6Xl5eTnJyskB4kpdX1LNlWwOfp+Szemk95rTewz2oxOLNPNOcNjmdyahwp3V0tPnatsr6S/9v5f7yT/g57yjMZlmFy0TqTkTtNjP2farvbS3T/KqJSqrGGmJAw3B/W074PMf1O7s2KiIiIiIjQgUP6jh07GDBgAJs2bWLo0KHNtmkM6X369KG2tpa0tDQeeuihZofAAzz22GP86le/Omy7QnrweRt8rMss4Yst/l72HflNn7HeJ8bJ5MHxnJcax5l9uhFiszR7Hp/p48vsL3k7/W2W71tOXIn/eevnbzJw1Ph71w27QWSvKqIHVBIWtf8PAwnD/GF90EUQlwZ6DruIiIiIiJwEHTKkm6bJpZdeSklJCcuWLTtiu61bt7J06VLOOOMM6urq+Otf/8pLL73E4sWLm+19V096x5FZVMUXW/L5Yks+K3cV4Wk48NF0h9qYMLA7kwfHM2lQLN3doc2eY1fpLt7Z8g7/3PlPGmqqOec7k+nrDHrmH1jIztnLQXRyDuE9qjEac787HlLOhX7nQsokCE84iXcqIiIiIiJdSYcM6bfddhv//ve/+fLLL+nZs+cxHXvJJZdgGAb//Oc/j9pWc9I7hso6L19u9w+LX7Q1n8LK+sA+w4ARyVGcNziOyYPjSU0MP2xYfHl9OR9u/5B3t7xLdsVeUvfARetMRm8zseyfFG/rFk70cAeR0duwh1Y3LSBuyP7Afi70PgtCWr8ivYiIiIiIyME6XEi//fbb+eijj1i6dCl9+x77o7OefPJJ3nrrLdLT04/aViG94/H5TL7JLuOL9Dw+35LPd/vKm+xPigzbv1p8POP6xRBmP7CAYIOvgSV7l/BO+jusyl1Ft3KTKV/7mLrRgrvqQO96aO8k3AMjcUfn4CAdw3LQPwtrCPQaC/0m+0N7wnCwND/0XkRERERE5FAdJqSbpsntt9/Ohx9+yOLFixkwYECbznP55ZdTXFzMF198cdS2CukdX05ZDYu2FPDFljy+3FFIrefAqu5hdgvj+/uHxU8eHEdCZFhg3/aS7byd/jb/3vVvvHU1jEs3ufAbK/33eDEO+mdgiQjHPbwv7p4NuEI2Y6vPblqAM8Y/JL5xeHzksY38EBERERGRrqXDhPSf/vSnvPPOO/zf//1fk2ejR0ZG4nA4AJgzZw7Z2dm8+eabADz33HP06dOHIUOGUF9fz1tvvcUzzzzD+++/z2WXXXbUayqkdy61ngZW7Czi8y15fJGez76y2ib7hyRF+IfFp8YzvEckFotBaW0pH+z4gHe3vEtuVS7uapPTMkzOzghl+K4GQqoODK3HYsExZBDuQd1wxxQQWr0Ww9N0gTu6DzwQ2PuMh9DwU3DnIiIiIiLSUXSYkH6kx2v95S9/YdasWQDMmjWL3bt3s3jxYgDmzp3LK6+8QnZ2Ng6HgyFDhjBnzhymTZvWqmsqpHdepmmyJbfCv1p8eh5f7ynl4E93d3cI5w6K47zUOMYPiCXMDl9mf8mnuz9l0Z5FVHoqsfhMBmTD+MwwxmbYidxb2uQatoQE3CMH4e5t4ArZiiX/azAP9ORjsUHP0f7A3m8yJJ0OFisiIiIiItJ1dZiQHgwK6V1HUWUdi7cW8MWWfJZuK6Ci7sAz2e1Wg7EpMUweHMekQXEkRdlYlbuKT3d/yhd7vqCivgKAmDKTczIdTNobQWJ6AUa9J3AOIzQU56iRuNPicMeWEFKyEkp2Ny0iLBL6Tjgwn73bsa+5ICIiIiIiHZtCegsU0rumeq+PtbuL+Xx/L/vuoqarufeIcjBhYHfOGRDL6L6RbClbHwjsZXVlANg9JmP3ubgwJ46U74qx5hc3OUfogP64R4/A3deGw7INI2sZ1JY1LSS6z4HA3ncCOKJO4l2LiIiIiEh7oJDeAoV0AdhVULl/WHw+azOLmzyT3WLA8J5RTBjQnXH9o/HYt/HFns/4IusLSupK/I1Mk9QyNzPyezFkWy32zbug4cBq8ZaICNxnn4V7eG9cceXY8lbA3tXgO9Cbj2GBHmfsn88+GXqOAqv9VP0IRERERETkFFFIb4FCuhyqut7Lql3FLNteyLLtBWzPb7ownDvUxrh+MYzvH01Utz1sLFnK51mfU1x7oCc9vsHFFaUDOXOXBefarfhKSw+cwDBwnHYa7vFjcfdzEerbipGxGAq3NS0kJNy/8Fy/yf457TH9/Q+FFxERERGRDk0hvQUK6XI0OWU1+wN7IV9uL6Ck2tNkf3I3B2f3i6FnYg75vtUs27eIwprCwP4Iq5vLPcMZn+Wg2/oM6rc2DeO2hATcEybgPnMIrpgKLNlfwq7FUNN0+DwRPfcvQHcu9J0ErpiTc8MiIiIiInJSKaS3QCFdjoXPZ/LdvnKWbi9g2fYC1mWWHD40PjmCgb2K8IR9zcbiZRTUFAT2u+wuLnKN5oJ9MSRt3EfNqjWYNTWB/UZICM7Ro3FPnIA7LZ6Q6m9h1yLIWgkNBz0KDgMST/MH9gFToddY9bKLiIiIiHQQCuktUEiX41FV52VVRlGgp33HIUPjw0MtDEkpJSz6OzJrVlBYmx/Y57A5mBw/nmmlvUnZXELt0q/w7N3b5PiQfv1wT5yI++wxOLvVYGQthZ2LIP+7poXEDYGxt8CwH4DdcdLuV0REREREjp9CegsU0uVE2ldaw5fbC1m6vYAvdxRS2mRovI/E+ALiErZSzFpK65sG9nOSxnOR9TSGbK2hfulyqtevb7r4XHg4rvFn+0P7yMHYSjbAzi8g/WPw7F+d3tENRt0AZ94EEYmn5qZFREREROSYKKS3QCFdTpYGn8l3+8pYtr2Qpdv8Q+O9vsZ/XiY2x16SkrbhcWyksuFAYA+zhjG+x3imxoxnZKYV75crqVy6lIaSkgMnNwzChg/zB/YxpxNWvQpj7etQluXfb7HBkBkw5lboecapu2kRERERETkqhfQWKKTLqVJZ52XVLv/Q+KXbC9hVULV/j4klLBtX1GbCor+llgOBPcQSwtk9zuaC5PMZW9od3/I1VC5ZQt3m9CbntoSH4zzjDJx9XDh96wmrXYdh2b+z52j/UPjU7+mRbiIiIiIi7YBCegsU0iVYsktr+HJ7AUu3F/JVYGi8iSU0B1vEJhxR3+GzHQjsdouds5LO4oI+F3BO6BBYsZ7KJUuoXrkKX2XTufAWlxNnLwfOsEycsTWERXkwonr4h8GfMQuc3U7tzYqIiIiISIBCegsU0qU9aPCZfJtdxrL9oX19Zglenw9LaB628G+wR2zCEnpglXibxca4xHFM6T2FCYln48zIp3r1av9r3brDQ3uIibN7Hc64OpyJBmGTZmKMuxXiUk/1rYqIiIiIdHkK6S1QSJf2qLLOy8qdRSzbXsCy7YXsKqzCEpKHLWITtvBNWMPymrQPDwmnp7snPcN70svZg34FVnpsK8H9XSbmhu8OD+12H87YepypyTgvvo6w86/HsGsovIiIiIjIqaCQ3gKFdOkI9hRX8+WOQpZtL+DL7YVU+vZhC9+ELWIT1rDcFo+1mxbOKIvhjH2hDMioJ25bPrZab5M2lhBwpqXgPO97OM8aT9jgwRhW68m8JRERERGRLkshvQUK6dLRNPhMNmWXsWxbAUu3F7BxbwFeSxFGSBEWezGWkGIs9mJCHCWYtmJMPE2Ot/hM+uTBkCyTtEyT1D0mzvqm1zBdYdhGjiB63HjcY8YqtIuIiIiInEAK6S1QSJeOrs7bwJacCjbuLWXDHv/rwMrxPgxbBRZ7MdbQEmKjKwl3l2PYC6lsyKe0vvjw0L7XxFnX9Bq1DivFgxKoP20AjjPPJH74aHpG9SY8JPyU36+IiIiISEenkN4ChXTpjMpqPGzaW9YkuBdU1B3WzhnawIAe9fSMrSYivAx77TcUFWzEt7ecmH0WBmfRbGivCoX0ZINdKQ5K03piHzSAHpHJgXnxyeHJxDvjsVrU+y4iIiIiciiF9BYopEtXYJomOWW1bNxTyoa9pWzcU8qmvWVU1Tcc1jY2PJSL4kq5vOFjuhd/yj4aKCgLpb7IhSPfRWxWLWG1vibHNIb273obbO5lsDsOrFY7Se4keob3pKfbH9x7hvtDfL/Iftj1zHYRERER6aIU0lugkC5dVYPPZEd+ZZPgviW3ggbfgf8ERFHBVdZF/Djkv8SZRf7jsFMbfQHl3gGUfrcL34ZvsVTXNjl3c6HdtBiB/Q6bgxGxIzgz4UzOTDiTITFDFNpFREREpMtQSG+BQrrIATX1DWzOKePrrFI27i1j455SsoqrseFlqmUtN9g+4QzL9kD7TNdwcgZeT0zUEKK3b6ZmzRqq167FV1XV5LweZwg5/aPZ2sfGyvgKskOqaLCAzwI+A+z2MIbGD2dk0pmMShrN0NjhCu0iIiIi0mkppLdAIV2kZcVV9Wzc39O+YU8p3qw1XO79F9Mtq7Ab/uHye83uvGdcyLakGQzsmcgobyF9927FsnF9s6H9aHyAaTXAasVis2O12TCsNrBZMaw2/0rzViuG1Yphs8Kh26xWsNkwLJYDxzS2O2ybFcNibbrN0ngO/zZrRDihgwYTNmggFpfrJPyURURERKQrUUhvgUK6yLExTZM9xTWkb9+Gc+NfGJ77AZFmOQDVZij/2zCB+Q1T2WUmkRQZxoge4ZxlFpOWt53o7d9S/81GfNXV4PVCR/vPjWEQ0rs3YWmphKamEpaaRljqYGwxMcGuTEREREQ6EIX0FiikixwnTw3ejX/Hu/xPhBVvCWxe1HAaf2m4kKW+4YB/PrrFgAFx4fxgVE+uG9cHuwVoaMBsaMD0NkCDF19DA7uLd7Ixdz3f5H7Nt/mbqKwtw+oDqw8sPnAYIQyK7E9a5CAGRw2kr7s3VhNMbwNmgzdwTvaf98jbfAf2Ndnmr8X0NmD6GvAWFFCXvgVvfn6zPwJbXBxhqamEpqUSNjiVsLRU7D17YhhGs+1FREREpGtTSG+BQrrICWKakLEUVr0EWz8B/P8pKXb2ZYHrUl4rG82u8gPN+8W6eOSSIUwcGHuU05rsKtvFmtw1rMldw9q8tRTXFjdpE2YN47S40zgz3r8Q3dDuQwmxhpzoO8RbVETt5nRq09Op25JO7eZ06jMzmx0RYAkPJ2zwYEJTB/t73NNSCU1JwbBrrr2IiIhIV6eQ3gKFdJGToHgXrHoFvn4L6iv828KiqBr6Iz51X8ITyyooqqoH4PzUeB6+OJXeMa2b692eQjuAr6qK2q1bqU3fH943p1O3fTumx3NYW8NuJ3TAAH+Pe+Nwec1zFxEREelyFNJboJAuchLVlsOGt2HVy1CS4d9mWPEmj2WZdygvZPZkQ0NfbFYbsyf05aeT+uMKtR3TJQ4O7Wvz1rImd81hoT3UGsqI2BGMShjFmQlnMqz7sJMW2gFMj4e6Xbv297pvpm5zOrVbtuCrrDy88aHz3PcPl9c8dxEREZHOSyG9BQrpIqeArwG2LYRVL/qHxB+kynCz1JvKV76hpDvP4Lpp5/K9ET3aPJ/bNE0yyjL8Pe15a9pFaG+sy7N374Hgnr6F2vT0o89zP2i4vOa5i4iIiHQOCuktUEgXOcWKdsKuRbBzEWQsg7qyJrv3mt3Z4hzFwHEX0+uMaeA6vh7l9hraGwXmuW9Jpy79KPPc3W7/PPe0VM1zFxEREenAFNJboJAuEkQNXsjZALsW4duxCHPPKqymt0kTb9wwbAMmQ8ok6DUO7I7juqRpmmSUZ7A2d21gXntRbVGTNqHWUE6LPc0f2uPPZEj3IThsx3fdY+Gf576N2vTNbZjnnkrYoEEYDgem14tZ7wGvB9Pj8b8PfPV/3+y++oPfe6Dxe4/3kLae/ec4aH+Ta9T799Ufcv5Dj/d4sbhcOEeNwjl2LK6xY7AnJp6yn7eIiIjIqaaQ3gKFdJF2pL6K4s2L2LDk/0gsWkGqZU/T/dZQ6DXWH9j7nQsJp4HFclyXbE1oB7Bb7ISHhOOyu3Db3bhD3P6vB3/fzDaX3dXkOKvF2rY6D53nnr7FP8+9ouK47r+9svfuhWv0GJxjx+AaMwZb9+7BLklERETkhFFIb4FCukj7tDqjmOc++pLYghWMt3zLJPt3xJqHhGdHNPSdeCC0R/c57uu2NrS3lcPmINwejivE5f9qdzUb8gPBvpl9DpsDwzCOeZ67YbeD3Y5ht2PYbM1+xd74vpk2dpv/eJsNwx5yyP79x9sOOn7/MU3Ob7M1u8+Tm0f1qlVUrV5F7aZvwedrUnvogP44G0P7mWdijYo6Yb8TERERkVNNIb0FCuki7VeDz+S9NVn8duFWSqrr6WfsY3aPTC4N344je/mBx7s1iu4DKef6Q3vfCeDsdtw1mKZJhaeCqvoqKj2V/ld95eHfH/S1ylPlP8ZTRUW9/2tdQ91x19LIalgP79EP2d9rbw8nymOnt7Mn/WMHkdJ9EKGhTrBaO8yicw0VFVSvXUv1ylVUrV5NXXp60waGQWjqYFxj/EPjHWeMwurWY+xERESk41BIb4FCukj7V1pdz+//u42/rszEZ0KozcKt5/Tm1gGlhGYthV2LYe8a8B08n92ApBEHQnuvsWALDc4NAPUN9VR5qpoP+M2F/YO+PzjsN5gNx3Rdm8VGv8h+DO42OPAa1G0Q4SHhJ+lOTzxvSQnVq9f4e9pXraJ+586mDaxWHEOHBuazO04/HUtYWHCKFREREWkFhfQWKKSLdBxbcsv51T83s2KXf/h5UmQYD05PY9qwBIz6Stj9lT+w71oEBVuaHmxzQO9xB0J7/NDjns9+qpmmSY235kBPff2BHvuDQ35ZfRk7SnaQXpxOeX15s+fq6e5Jakxqk/Ae64jtEL3tnvz8/aF9JVUrV+HZ03TtAsNuxzFiRGA+u2P4cIyQU7Nav4iIiEhrKKS3QCFdpGMxTZNPvs3lyX+nk11aA8CYvt147HtDSE086N9wec7+wL7/VZnb9ETO7pAy8UBoj0o+RXdw6pimSU5VDluKt7CleAvpxelsLd5KTlVOs+27hXUjtVsqg7oNIrWbP8D3iuiFxWjff8zwZGdTtWp1ILR78/Ka7DccDpwjR+IcMwbX2DGEpaVh2GxBqlZEREREIb1FCukiHVNNfQMvL93Ji4t3Uuf1YTHgmjG9uWvKQKJdh/Samqa/Z33nIn9g3/0leKqatonp7w/rKedCn/HgiDpFd3LqldaWsqVkC1uKtgS+ZpRn4DN9h7V12BwMih7E4G6DSY3xB/gBUQNO2XPkj5VpmngyM6lauYqqVSupXrWahuLiJm0sbjfOM8/EOWY0rrFjCR04EKODjaoQERGRjk0hvQUK6SId296Sap76Tzr/2eTvKY9y2rn7gkFcPboXVssRhm576yF7rT+w71wE2evg4LnehgWSRvpXjE+ZBD1Hg619htITpcZbExgi39jzvq1kW7ML3tkMGylRKU2Gyg/uNrhdznM3TZO67dv9i9CtWkX1mjX4yptOAbBGReEcPdo/PH7sWEL69u0Qw/5FRESk41JIb4FCukjnsHxnIb/652a25vlXfE9NjOCxS9IYkxJz9INry/y9642hvWh70/3WEIgd5J/HHpcG8UP837vjoBOHOa/PS2Z5pj+4N/a6F2+hrK6s2fY93D0Cw+RTY1IZFD2IOGdcuwq8ZkMDtelb/EPjV62ieu06zOrqJm1ssbGBofHOsWMJ6dkzSNWKiIhIZ6WQ3gKFdJHOw9vg4+1VWfzPf7dRVuMB4OLhiTwwLZWkKEfrT1S2t+l89qqC5ts5Yw4E9sbwHjsYQpzHeyvtlmma5FblNpnnvqV4S4vz3BtXlA/Mcw/vhdViPcWVN8/0eKjZ9C3Vq1dRtXIVNevXY9bXN2ljT0oKrBzvHDMGe3x8kKoVERGRzkIhvQUK6SKdT3FVPb/7dCvvrM7CNCHMbuGnk/rzkwkphNmPMRz6fFCWBXnfNX0V74Rm5nBjWKBbyuHhPap3h1tN/liU1ZUdtkDdrrJdR5znPjB6oL/HfX9w7xPZB4fNEfRF6nx1ddR8veFAaP/mG/B6m7QJ6dMH55gxOIYNJXTgQEL79cPi0nPaRUREpPUU0lugkC7SeX23r4xf/XMzq3f7Fw7rGe3goelpTB0Sf/xDsD01/sXomoT3b6G6qPn2IW6ISz0kvKeBI/r46mjHar21bC/ZfmCRuv3z3Gsbao94jMPmaPHltDuP2ubQtk6bkzBbWJv+AOCrqqJ6/deBleNrN2/2/+HmEPbkZH9gH9CfsIEDCR0wgJA+fTDs9mO+poiIiHR+CuktUEgX6dxM0+Tjb3J46t/p5Jb7w+H4/t159JI0BsSf4IXOTBMq8yH/kF73gi3QUN/8MRE9/WH94PDefQBYO2e4a/A1HJjnftBw+SPNcz+Rwqxhh4d5uz/EHzHs2w8K/TYnjhofYd/uwrZxK7aMbOq376ChsLDZ6xl2OyEpKYQOGNAkwNuSktrVPH0RERE59RTSW6CQLtI1VNd7eXHxTl5euot6rw+rxeC6cb258/yBRDpOciBu8EDRzkPC+2b/MPrmWOz+ue2B8D4E4oZAeEKnXKjONE1qvDWBV7W3+sB7T02TfYe18zSz7ZB2J4vNsNEnsg9Drb0YWh5B7wKIyanClrGP+u3b8R2yIF0ji8t1UHDf/3XgAGzRnXdUhYiIiDSlkN4ChXSRriWrqJon/r2ZTzfnAdDNFcIvpw7iilHJR35k28lSUwr56YeH9/qK5ts7uh0I7Y2v2NROvVDd8TJNk9qGWqo9h4f3ZoP/Qe2aC/yNfxSo8lQdcdi+w+agX3gKp5lJpJY4SS40idpbDjszqcvIOGyOeyNrbHfCBhwc3AcS2r8fFscxLHooIiIiHYJCegsU0kW6pmXbC/jVx5vZkV8JwNAeEfzqe0M4o3e34BZmmlC6f6G6g8N70Y7mF6rDOGihuoNeUX069UJ1wda4yv320u1sL9nOjtId7Cjdwa7SXdT7mp/aEBkaySB3P0bUxjGwJIzEPA/he4pp2JmBZ+/e5i9kGNh7JRM6YIB/rnvjfPfevTFstpN4hyIiInIyKaS3QCFdpOvyNPh4c0Umz/13GxV1/t7NGaf34P6LBhMfERbk6g7hqYGCrfvD+2b/InV53x358XB2V9O57vFD/e/DIk9t3V2M1+clqyKLHSU7AsF9e8l2siqyml3pHiDeGU+qow8jKmPoV2wnLqcGR1Yhnu07aCgubvYYw24npF8/QgcOCCxUFzpwILaEBM13FxER6QAU0lugkC4ihZV1/HbhVv62dg+mCc4QKz+b3J8bx/cl1NY+nud9RJX5B3rbG8N7/hZoqGu+fWSvQ3rdh0JMP2gnzy3vrGq9tWSUZQRC+/ZSf+97blVus+0thoXk8GSGW/zz3fsUGsTsq8a+O4f6HTswa5qfa28JD98f2Acc6H0fMABrVNRJvDsRERE5VgrpLVBIF5FG3+wt5bF/fsf6rFIAesc4eXh6GuelxnWs3skGr/857o297XnfQe63UH6EIdW2sKaPh2v86gzy0P8uoKK+IhDcD+55L60rbba93WKnX3hfTvMlkVbqole+SVR2OcauPdTv3g0NDc0eZ4uLI6RXLywu1/6XE4vT6f/e6cRwOrG6XBhO//YD3+9v63Jh2O0d69+BiIhIO6aQ3gKFdBE5mM9n8tGGbJ75ZAv5Ff7e6NN6RhIfEYYr1IYzxIor1IYrxIYr1Irz4K+N+wLvbThDrdit7WRueE2Jf2G6xme6N/a+e5pfhZzwpKY97vFDOvXj4doL0zQpqi06LLjvKN1xxNXqXXYXA919Ob0mnkElYSTlewnfU4K5czeefftOTGE2W5Ng36rvXS2303PkRUSkq1JIb4FCuog0p7LOy7wvdvD6l7vwNBzffxZDrBacodajBPv9fwDYH+xdIf737lAbzlB/u8DXEBshthMU/H0NULK7aa973rf+bc2xhkDsoIN63PcHeHfcialHjshn+thXue+wIfMZZRl4fc2vGN8trBtDwvpyWmU3kqvDcHgshNWbhNab2GsbsNd5sdV5sdTUY6mpg5pafNXV+Kqq/F+rqzFrm1/F/kQwQkKaDfaNPfsWdziWcDfW8AgsEeFYw8OxhIdjjYjA4nb7v4aHYwkJOWk1ioiInAwK6S1QSBeRluwprmZdZglV9V6q6rxU1TVQXe+lqr6B6rr9X+u9VNb531fXN1BV76W6roH6huYXCjsRDg7+zoMCvGv/1yhnCNHOELq5Q+jmDCHaZSfGFUq0y060M+Tovft1Fft73Q8O798d+fFwrrjDe91jB4Et9MTfvDTh8XnILMv0h/fS7ewo8X/dW7EXk2P7X7rNYsNpc+Kyu3DZXf7vLQ4ifSFENIQS4bXharDh9lhxea046iHMA6F1JiF1PkLqG7DVerHW1mOt8UBNLWZ1Nb7qKnxV1YE/AJgezwn9GRihof4Q7w7fH+YjAuHeGhHuD/sHbz8o5FvDwzGcTg3lFxGRU0ohvQUK6SJystR7fdTUN1BZ7z0Q6A8K9lV1Df7gX78/3Nc1/VpZ5w20a/zDQL33xAT/iDAb3VwhRLv8Ib6bK+TAe1djsA8hZv+2iDAbBux/PNy3TYfMF+2E5sKgxQbdBx4e3sMTQYHopKv2VJNRlhF4TFxOVQ7V3mqqPdVUeaqo9lRT7fV/X3ekhQaPk8Ww4LQ5/S+7P/w77U7CcRDhCyG8wU6E147L6w/9znoDh8cgrB5C63zENIRhr/HiqyinobwCX0UFDRUVNFSU46uoxFdxhD8aHSurFavbjSXi8HBvjQjHEh6BNdzt/9pk+/7efbdbj8QTEZFjopDeAoV0EelIPA0+qgMhf3/Q399z3xj2K2u9lNbUU1zlobiqjpIqD8XV9RRX1VNSXU9b/itvsxiBQB/tsgdCfTdnCLFhDfRqyCKxdgcxldtxl20lpDAdo7a0+ZM5uh2ySN0QiOoNmP7h96YPzP1fA+99R9hn+t+3tC/w/nj3+Q5cAyAuDfqcA66Ytv462w2vzxsI8IEQvz/AV3mqqPHWBL5vEvS9hwf+xu9PlP5R/Tkj/gxGxo3kjPgziHfFB/aZDQ3+ofmN4b28HF9lpf9rRaU/zJdX0FBZ4f9accj2igrwNj9V4FhZnE7/0HuH48Cce6fDv/ie03lgu8v/feMifRaHs8n2A8c6tVifiEgnppDeAoV0EelKGnwm5TUeivYH9uKqA6+SqvoDYb7x+8p6quqbXzG8ZSZ9Q8o4I2wfw217GEgmfby7iavfg4W2nK8dix8KfSf4A3vvs8ARFeyKgs5n+qjx1gTCfJW3KvAHgIPD/xEDv6ea0rpS9lYe/kSCnu6ejIwfyaj4UYyMH0mv8F5tDrKmaWLW1vp76SsPDfn+EN9suG8M/ZWVmNUn7g8Sh2lcrO+g8G5xODBchwR8pxOL86CAH2jvarK98TyG5vCLiASdQnoLFNJFRFpW62mgtNpD0cG98pV1FFd7/GG+6kAvfdH+gO/1Nf+/klDq6W/sY7CRxWBLFqlGJqmWLGIM/7BlEwPDYgXDAsb+rxarf3i8Yd3//cH7LM203f/+iPsszbRtYV/gOgedp6Ee9q6D/O+a3qBhgcQR0Pccf3DvNQ5CXCf5N9R5FdUU8XX+16zLW8e6vHVsLdmKz2w65aO7o3uTnvYB0QOwGKfuiQqmx0NDZWUg1Js1NfsX3asJLL7nq9m/CF/j+yb7avxz9qurMfdvN+vrT27Rdrs/sLtchPTsSeiAAYQOHEBo//6EDhiANTLy5F5fREQU0luikC4icmKZpklFnZeSqgOhPdBjX11/SLD3UFhZR1VtPT4MwODcQbHMPieFcf1i2v9Q38oC2L3M/8pYCkU7mu632KDHKH9g73sO9BwN9rDg1NoJVNZXsqFgA+vy1rE+bz2bCjfh8TVdhC48JDwQ2EfGjyQtJg27pWM96s30eveH92r/gns1BwX8mpoDi/AdHPAP2159YIX+xq+tXLDPFhfnD+79+/vD+4ABhPbrh8WlPziJiJwoCuktUEgXEQku0zRZm1nCa8t28enmvMCc+bTECG46py8XD086cY+cO9nK90HG/sCesRTKsprut4ZC8mjoO9Ef3HuM1HPnj0Ott5ZNhZtYn7eedXnr2FCw4bBnyTtsDoZ3H84Z8WdwRvwZDIsdhsPmCFLFwWV6PE0DfEUFdRkZ1G3fTt2OHdRt3453X84Rj7f36OEP7Af1vIekpGAJ1VMcRESOlUJ6CxTSRUTaj92FVfz5qwz+sXYvNR7/3PX4iFCuP6sP14zuTaSzgwXakt37A/v+4F6Z23S/3QW9x/nns/edAImn+YfTS5t4fV62FG8JDI9fn7+esrqyJm1sFhtDYoYE5rWPiBtBRIj+/9+ooaLCH9j3h/bGAN9QUNj8ARYLIb17H9bzHtKrF4a9g/17FRE5hRTSW6CQLiLS/pRW1/P2qizeWL6b/Ar/48EcditXjOrJDeP70jumAw67NU3/cPiMJf7AvvtLqC5q2iY0EvqcfWAhurg0/3x4aROf6WNX6S5/aM/3B/f86vwmbQwMBkYPDAyPPyP+DLo7ugep4vbLW1LSJLT7v9+Br6ys2faG3U5ISkpgnntjeLf36IGhz7SIiEJ6SxTSRUTar3qvj4837uPVZbvYkutfXM4w4IK0eGafk8IZvaPb/7z1I/H5IH/zgfnsu7+CQ3p9ccbs72U/xz9EPqa/njF/HEzTZG/l3sDw+PX568kszzysXZ+IPoHAPjJuJD3cPTru5+wkMk0Tb34BdTu2HxLgd/x/e3ce3UZ57w38q323vG+x8ZrETpyNkJ2lLS+QUJbbcoFQXprQlgOUsr8shXKhtL2l93JL7y2FFgqBFmh7WhJKSws3tCQEsgIOJI6zeclqx7tsy9rnef8YSdbYkhwH25Ls7+ccnRmNnhmNMmdiffV75pmYo96rTCYYKioGu81Pl0O8Ni+P/8ZENKUwpMfBkE5ElPyEENja0InntzRi04H28PL5xen41nllWDk7H1pNilfnpADQ8ung9exHtwG+IUHHmh8chC44EF1GaUJ2dTJpH2jHx20fh4P7oe5DEFB+Fcoz54W7xy/MW4hyezkDZRxCkuA72QLPoYOKqru3oSHmyPVqm22wy3zEde/azMwJ3nsioonBkB4HQzoRUWo5dKoPL3zQhPW1J+D1y7fjmpZuwo0rSnHtomLYjJPkOli/Fzj5SfB69s3AsZ1AwKNsk35WsGt8MLSnFSZmXycRh8eB3W27w13k93Xsg1/4FW3SDek4O/fscHCfmTkTWrU2QXssE0IgIAKQhAS/5EdABBCQAvALPyQhhecDUkB+LdjWqrMi05gJk9Y07j88CL8f3mPHhnWb9zY1A4FA1HU0mZmDob2yEvqzioP3gg/eG94kP1QmE7vRE1FKYUiPgyGdiCg1tfd58Mr2I/jt9iPocsrVOZtBi9WLi7F2RRmmpU+yEbx9buD4zsFK+4mPAUkZHpE1ffAe7aXnARZeW/15DfgGsKdjT3gwus/aP4M74Fa0MWvNmJ87H5XplRAQ4SDsl4IBOTgfCs6hkBwZnCUhDQvRkdsJiAAkKdgmcjsR7T8Pg8aAdEM6Mo2ZyDBmyA9DRng+0yAvTzemI9OQiTRD2pjdj17yeuFtao7oNi+Hd9+xY8AovpaqjEaojUaozCaoTebBAD/kudosh3p1RNCP+twcsY0pMgheKAawpwjR+GNIj4MhnYgotbl9AWyoPYFfb2lEQ7sTAKBRq/DlOQX41nllmFuUntgdHC+efuDodqA5GNpbPgWEpGyTO3swtBcvka9x55fvz8UX8KGusw6ftMnd42tP1aLP15fo3YpLo9LID7UGWpUWarUaGpUGKqjQ5+2DV4reBX2kbdoNdmWQN2Yi3ZAenlcEfUMGdKO83aDkcsHT0Bisuh+C5+Ah+Ftb5dvIBR+xrn0fczrdYMiPFfyNUX4EMJkACIhAAAgEIPwBQJKnIhA57wcCkjz1ByAkCQj45dekgLwstI1AjPZ+v/xa3PYBuZ2i/eA8AgGorVaY5s6Baf58+TFvHjR2+8T8OxNNIQzpcTCkExFNDpIksOlgG369pQlbGwZHTV9clolvnVuG/1OdB7V6EgdUVw9wZGtwELotwKm9w9toTYC9CLBPA9KKIuanAfZieV6fgiPnJ1BACuBwz2F8dOojtDpboVbJAVir1oaDcbSQrFVpB1+LaBNaT61SK7ahVWnlbQfnNepgm4jtaNXaYe+vVqnjVkWFEHD5Xehyd6Hb3Y1uT7c8dXejy9OFHndPeL7b3Y0ed88Z/yhh09kU1fioFftg0D/dLvhCCAi3O3j/dxeEayA8L7kGIEKBfiAU7AcgBlwRQT/4PLgNMTCg+BEAfn/c958q9OXlwdA+D6b582GorOTlBUSfE0N6HAzpRESTz94TDrz4QRPe/PQk/JL8Z60s24JvrCjFvy4shkk/Be5F7uwIjhwfHD2+89DprWdMHwzsadOCQb5ocD6tEBhlRZQmF1/AFw7zXe4u9Hh6BkP+kKDf7elGj6cH0tBeHqfBoDEMC/G55lyU28tRmV6Jcns5zDrzOHzCQcLrHQztUYN/8HnkjwCRz90uqFRqQKOBSqOBSqsB1PI8tBqo1MGpRguVRg1otPJrGrW8LNReq5G3Ea29VgOoY7QPPhCtvTb0XhrF1N/eDtfu3XDV7oZr9254jwy/A4JcbZ8rB/cFwWo7v0cTjQpDehwM6UREk1erw42XtzXj1e1H0OuWK2LpZh3+75ISfH1ZCXLTjAnewwnkcwO9J+SH4zjgOAH0Hh+cdxwHvKdTIVUB1rzBKry9OBjgQyG+CLDk8P7uFCYJCb2eXkU1PjQfGfRD893u7tPugl9oKURFesXgwy5Pxzu8TyX+7m45tO/+VJ7u2RP1MgN9RQVMC+bDHOwmry8vZ7WdKA6G9DgY0omIJj+nx48/fnQML37YjKNd8pdLvUaNK+YX4pvnlqG6gP//AwDcjmB4PwE4jkXMB8N87wkgcBrhSaOXK+5pRRHhfUhV3mjn9fEUVbwu+Cf6T6DR0YjDPYfR5e6KuY0CSwEq0ivCFffK9EqUp5fDouPlHJ+X8PvhOXQoGNx3Y2D3bviOHB3WTp2WNlhtnz8fpnlzobHZErDHRMmJIT0OhnQioqkjIAls3NeKX29pwkdHusPLz5uejW+dV47zp2dzVON4JAkY6FCGdsX8CaCvBcBpfJXQWyPC+9CKfLEc8nWTbIR+GlPd7m409DSEQ3tjjzztdHfGXKfAUoDy9HJU2isVFXiG98/H39kJ16efhrvIu/buhXC5lI1UKhgqKwZD+/z50JeVsdpOU1bKhPQf//jHWL9+Pfbv3w+TyYTly5fjJz/5CWbOnBl3vc2bN+Oee+5BXV0dCgsLcf/99+OWW245rfdkSCcimppqj3bj11ua8Pe9LQheto4ZeVZ869xyXDG/EEbdFLhufTwEfHJQD3Wh7z0+ZP444OoeeTsAoDMDhjS56m4MTg1pQ+btg4+hr+mt7HY/BfW4e9DgaEBDT8TD0YAOV0fMdfIt+eHu8qGqe4W9Ala9dQL3fPIQfj/cBw4ousn7jh0b1k5ttwer7fMGR5K38t88kYQQCPT0wN/ejkBHB/wdHQh0d0NXXAzj7Bro8nITvYuTRsqE9JUrV2L16tVYtGgR/H4/Hn74YezZswf79u2DxRL9F86mpibU1NTgpptuws0334wPP/wQ3/72t/G73/0OV1111YjvyZBORDS1HesawLoPm/GHXUfh9Mr3ms626vH1ZaX4v0tLkGnRJ3gPJyGvE+g9OaRL/ZDu9b4xuLWWSg0YbMHQPjToxwr+duVrWsPn3w9KCj3unnDVPRTcG3pOI7zbKxRV93J7OWx6dtseLX9Hh1xtDw1Kt3cvhNutbKRSwTB9+pBqeyl7OI0ByeOBv70DgY52+Ds64G9vh789OA2GcX97O/ydnYDPF3M72pwcGGtqYJxTA1NNDYyzZ0OblTWBn2TySJmQPlR7eztyc3OxefNmnH/++VHbPPDAA3jzzTdRX18fXnbLLbfg008/xbZt20Z8D4Z0IiICAIfLhz/sOop1HzajxSF/cTRo1bhqYRG+eW4ZKnJY3ZkwQgDuHvm2cm4H4OmVp+5e5bzbAXgc0V+TYn/JHBWNYYRwH1HFt+YAOdVyV32GipTh8DgUoT30aHe1x1wnz5wXrriHrnuvSK9geB8F4fPBfeBg+Np2V20tfCdODGunsdthnD8vPCCdcc5caKy8PAGIUvUOBe7I8B2cSr29o9q2xm6HNjcHmuxsaNLs8DY1wXP4sHzZ0xDawgKYZtfI4b1mNkyzZ0OTnj5Gn3LyStmQfvjwYUyfPh179uxBTU1N1Dbnn38+FixYgP/+7/8OL9uwYQOuueYaDAwMQKdT3ibG4/HA4/GEn/f29qK4uJghnYiIAAC+gIS/7WnBr7c0Yc8JR3j5hVW5+NZ55VhanjnuVR0hBPySgNsXgNsnweMfPvX4JLh9AXj8ymms9nqtGtNzbZiRZ8WMPBumpZsm733jhQB8riHh3jEk3McL/sFlZ8qQBuRWAzlVQO4sIDc4teQwvKcQh8cx7Hr3xp5GtLnaYq6Ta84dNlhdljELZp0ZVp0VBo2BVeE4/O3tGAiF9t2fwr13L0TE93YAgFqtrLbPmwdNRjpUOp18WzmtVr6dXAr/O0dWvX3hAD48eI9U9R5KpdNBm5MDTU42tDk50GZnQ5udI8/nZMvPc3KgycqCWj+8F5nkcsFdvx/uvXvhrtsL1946eBsb5f9zh9AVF8M0pwbGUHifPYuXMgyRkiFdCIErr7wS3d3d2LJlS8x2M2bMwNq1a/HQQw+Fl23duhUrVqzAyZMnUVBQoGj/2GOP4fvf//6w7TCkExFRJCEEdjZ14fktTfjH/lPh7yA109LwzXPLUJpliR6cIwK0e+gyvwSPLwB3RLD2KAL24Lw0zn+NzXoNKnPlwD4jz4rpeTbMyLOh0G5M6S+3Y0YKAJ6+6AE+WhXf7ZC76nc2ACIQfZumzGBorx4M7jlVgDlzYj8bfS4OjwNNjqbBbvPBR7zwHqJRaWDRWWDVWcPB3aK3wKK1wKq3wqKzDH89YlnkvE6jG/H9Up3weuVr20MD0u3eDd/Jk6e3sk4XvDd9MLjrtFBpB4P86JdFLNdpAW3kMvl9MKSNYplucHuS1xu+3tvfNqS7+ZlUvdPT5ZCdI1e+w8E7Ozu8XJudDXVa2pj//x7o74d73z6499bBvXcvXHV7o472DwD6sjIYa2pgqpkN45w5MFZVQW2eurdLTMmQftttt+Gtt97CBx98gKKiopjtZsyYgRtvvBHf/e53w8s+/PBDnHvuuWhpaUF+fr6iPSvpREQ0Wo3t/Xjxwyb86ePjcPuGd/UbbwatGgatGkadBkadJjw/uEwNg1YDQ3BqjJiG2jk9fhw81Y+Dp/rQ2O6ENxD9c1gNWlTmWjEzz4bpeaEQb0NeGiuAp8XvAToPA231g4/2eqCrCTFHvbfmK0N77iwgZ6bcrZ6UhEja3gi93l409jSioadBrro7GtHkaEKvtxdOn3PM30+v1odDezi8662waC2w6JVBP1boN+vMsOgs0Kq1Y75/48XX1qYYkM5dVze82j4JqPT6wcp2RJV7WOU7KwuqKFXvRAo4HHDv2wfXnr1y1X3v3ug/rqjVMFRUDHaTr6mBoaoKasPUGAsk5UL67bffjjfeeAPvv/8+ysrK4rYdbXf3oXhNOhERna4upxev7TiC9bUn4PVLcQNzZHA2xpkaddGWRYZu9ZiHY39AQnPnAA6d6pODe1sfDgXDuz9GCT/NqA1X20Nd5qfnWZFjZXg/Ld4BoOMA0LYfaNsHtO+XA7xj+IjXYfbi4d3ms2cC+klWeZICgLMD6G8F+tuAvlag/5Q8r1jWJg8oaLANPvRW5fNhy9OC04h2+uBUa5iwwC8JCS6/C/3efjh9Tjh9TvT7os/HWtbv7ceAfwAuv2vkNxwlk9YEs9YMq96KDEMGytPLUW4PPtLLUWApgFqVvHdKEIEAhN8P4fMDfp88H3r4fIDiuR/CP2SZ3z/43Bda5ou9zBe5bsSy0H74fYBPuf3IZSqtdsTKt9pmm1T/t/q7uuCuC1bbg1V3/6lTwxtqtTDMmK64xt04fXrS/RAxFlImpAshcPvtt2PDhg3YtGkTpk+fPuI6DzzwAP7yl79g37594WW33nordu/ezYHjiIiIRsHrl9Dc6cTBYHiXQ3wfmjsHEIgR3tPNOszItWFGfjC4B697z7JOjUrI5+buBdoPyNX2yOp7f2uMFVRARmlEt/ngI6sy+Uai9zrlsN13Khi6Tw15HgzeznZATHwPFah1EeE9LXbgHxb6h7TT2wDNxFWi/ZIfTp8TA76BYUE/tCxy3ul1wukfDPmRPxR4Je9pvadJa0JpWmn41nSh8F5sK06pKjwlF9+ptsHgXrcX7j17EejqGtZOpdPBUFUVrrYba+bAUFEuX06QwlImpH/729/Ga6+9hj//+c+Ke6Pb7XaYTCYAwHe/+12cOHECv/nNbwAM3oLt5ptvxk033YRt27bhlltu4S3YiIiIxojHH0BThxMHWvtwKNhl/lBbP450OmNeO59l0Q+73n1GnhXp5slXDRkXA13Bavu+YPW9Xp53Df8CCwBQaeSgPrTbfGb52AZISQIGOpUhO1TlHvrc2zeKDavkgfWseYAtT56GHpHP9RbA0y+PDeDtD44bEOURfq032D5iubd/7P49QrSmIRX7iNCfVghklADpJfIPLPZiQJsc54E34B1WwT81cAqNjkY09jSi0dGI5t5m+CV/1PW1ai1K00pRZi8Lj3Bfbi9Hqb0UBk2S/WhESU8IAX9rK1x75cAuh/c6SA7HsLYqoxHG6urBa9xraqAvLYVKo0nAnp+ZlAnpsbp0rFu3DmvXrgUArF27Fs3Nzdi0aVP49c2bN+Puu+9GXV0dCgsL8cADD+CWW245rfdkSCciIjozbl8ADe39Qyrv/TjWPRBtsF8AQI7NMOR6dznEpxkn/yBYn5sQctU58lr3tno5xHuGf4kFAGj0QPaM4d3m00sBdUT3ZZ8rRtV7SADvb4s9MF40WlMwZOcD1lzAFpxa85UB3Jw9cdVoKRAR4iPCvCL0B38IGCn0B87kWmgVkDYtIrgPmdoKlMcmwfySH8f6joWvsW/oaQjPx+p6r1apMc06DRX2CpSllymq7xYdb59Gp08IAd/x48Fu8nvlAerq6iD1D/+xTW02wzhrljwoXc1sWJYvhzYjIwF7fXpSJqQnAkM6ERHR2Brw+nG4rV/RZf7gqX6c6Il9LW1+mhEz8m2YkTt4vXuB3YRMix56bfIElqQkBNB7ckhwr5cr8b6B6OvozEBWBeBzy8E7VsiPxZw9QtU7GMYNtqQd6G1M+L3BAN8bPfS7HYDjONDdDHQfAXqOAiNdU67Ry9X2yOp75LwpIyn+TSUhodXZqgjtofvN98XpRZFnzgtX3cvsZeHqe4YxecMUJRchSfA2H4G7bu/gNe779kG4lOdWye9eg3nBggTt5cgY0uNgSCciIpoY/R4/Dp0a7DJ/sE0O8S0Od9z10oxaZFsNyLTokWXVI8tqQLZFnmZZ9ci06JFtNSDLoke6WQ/NZL3/+2hJEuA4Onyk+faD0SvAGkOU0B1Z+Q5Wwi05wBS4/de4EEL+UaTnSDC0NwenweeO4yP3VNDbYlfhM0rkSwISSAiBTnenPNK9owGNPcEA72hAh6sj5nqRA9aFA7y9Arnm3Ek1gBqNDxEIwNvYGB6Uzl1Xh7NeWge10ZjoXYuJIT0OhnQiIqLE6nX7BoN7MMQfbutHe78n5oB1sahVkMO8ZTDUhwJ8KNRHztsM2qkXAAJ+ubLbeUgOdKFAbrQnRYV2Sgv4gd4TESH+SEQV/oh8CcJILDmxA7y9OKE/sITuMd/oaFRU4E/0n4i5jlVnRbm9XL7uPWLgukJrITTq1Ln+mGgohvQ4GNKJiIiSkyQJ9Lp96Oj3orPfgy6nFx1Oeb6z34tOpwcd/V50BZd1D/hG/R56jTpcjR+s0AdDfGjeYgiHfaOOoYASyOeSu8xHBvjIQO8e4bIFlVq+Hn5oeM8oleeteQm5Hn7AN4Dm3mY09DQorns/1ncMgRg9Cwwag2LE+cr0SlSkV6DYVszwTimBIT0OhnQiIqLJwR+Q0DXglQN8MMRHTjsi5rucXvR7oo9YHY9Fr0FWsOt9dkSAz7IakGHWwWbUwWrQwmYMPeTnvK6eJoSrJ3YVvuco4I9/aQk0BrkSr1IBUAEqBKcq5RQYvizqFPG3o2gz/DUfgCOqABrhRwN8aFL50AAfmlU+xLp5nF6lQZkxGxXWIlTay1GRVY3K3HmYll7B8E5JhSE9DoZ0IiKiqcntC6AzojLfEazWdzrleUXQ7/fCGzjze3kbtOpwaLcZteEgbzXoIgL94HOrUYu0Ic+tei3UvN6ezpQQcnf5yGvgI6+Jd5wY3cj9CRQAcFKrRYNOhwa9Fo06HQ7rdWjU6eCO0RPAIATKA0CFyoAKrQ2VhixUWAoxzVoEtSULMA95mDKT5lZ5NDkxpMfBkE5EREQjEUKg3+NXdLOXK/LBeacXPQNe9Ln96HP70Of2o9/jx4B3bEPPYLgPToOh32ZQVu6HVvIHg78OBq166l2HTyML+OTr4Qe6AAhAIDgVyikwfFnU6dD1x3J7EetIAbmb/0AnJGcHTjhb0OBuw2GfA4clFxrUEpq0GnhihHeTJKHM50Ol14eKiGmBPwC1IQ0wZw4P8NGWmTLlkfcn6laCI5EkIOCVe0/4PfJgkf7Ih3v4soAHAZ8LXZ4etHt60OHtRbuvD90BF8rtFTin7GKkFS0GDNZEf7pJgSE9DoZ0IiIiGi/+gASnJ4Betw/9Hn8wvPuCYV75vN/tR2/kc48/HPp9gbH7eqZVq8IBvrrAhsVlWVhSlonqgjSOjE+TjxAIuHpwvGMfDnfWoaH7MA73HUHDQCuavD3wIXoPGZMkocLnQ4VXGeDzAwHEPUuM6SOHep0pGKDjBGa/OyJke4e0iVwWEcQjwjYCygsCfAA6NBq0azVo12jQodGgTaNBR8Tzdo0GXRo1pBg/4qmFQLXXh8UwYnFaBc4uXApz4dlA/jzAmnNGh2cqY0iPgyGdiIiIkp3bF4gI7r6IQD/4vC8i1Id/EAhV9j1y23jf8mwGLRaWZmBxWSaWlGVizrR0XktPk5pf8uNY3zE09DTgcM/h8LS5txl+KfqYFRaVFhVqEyokDSr8AVS6B1DR34O8ge744X2cDKhU4YDdrg2FbfWwQN6jOf3r8dUAMlV65KgNyNaaYFXpUO9uR/OQkQC0QqDG48UitxtLVBbMy5wNY+F8IH8uUDBXHoyQvXZiYkiPgyGdiIiIpgJJEhjwBcKhvtPpRe3RHuxs6sRHzd3oGzKQnkGrxoKz0sOV9gVnpcOsT5KuvETjyCf5cKz3mCK4N/Q04EjvEfhF9PBu01lRbjsLleYCVBgyUaG1ohJ65HjdULm6gYHOwYfPBWj0gNYIaA2DD40B0BohNHr0aTToUEloRwDtwo8O4UO75Ea75EaHfwDtfic6fP3oD4wwGGAErVqLbFM2ckw5g1OzPM0x5SDHLE8zjBnQqoef66ecp7DzyD/kR9c+nPT3K17XSwLzPB45tLs8mKMyQpc/F8ifMxjcs2cmzyUBCcaQHgdDOhEREU11AUmgvqUXO5u6sKu5CzubutDpHFI1U6swp8gerrQvLMmE3ZS4e24TTTRfwIcjvUdw2CGH9lCAP9p7NOat4mx6W/j2cKFphiEDHa4OtLva5elAO9pd7eFph6sDnoDntPfLpDUpw7d5MIRHBnG7wQ61aux6xxzvO45drbuw48SH2NWyA23eHuV+SRIWuD1Y5PZgicuNaq8XWo0ByJs1GNrz5wF5swG9ecz2K1UwpMfBkE5ERESkJIRAQ7sTO5u6sLOpEzuautDiUFbsVCqgKj8NS8oysbgsE4tKM5FjMyRoj4kSxxvwhu/zHqq6N/Q04GjfUUjizO8KYdPZwgE7HLrNyvkcUw4sOkvCB4MUQqC5t1kO7S07sKt1J7o9PYo2VklgoduNxS43FrvdmOH1QQ0AKjWQNT0Y2kNV93nytfyTGEN6HAzpRERERPEJIXC826WotDd2OIe1K8+xKEJ7UcbUq44RhXgCHjQ7mod1m+/39ceufEfMG7XGRH+EMyYJCYd7DmNny07sbN2Jj1o/Qp+vT9EmHWqc4/Zhcb8DS9xulPn8yuv604qCwX3u4NReNGmuc2dIj4MhnYiIiGj02vrc2NXUHa60HzjVN2xgumnpJiwOhvbFZZkoz058xY+IJl5ACmB/937sbNmJHa078MmpT+DyuxRtsjUmLIIRi/t6sKTzBIr8/uGD8ZkylKE9fy6QPR1Qn/7AeMmCIT0OhnQiIiKiz69nwIuPmruxM1hp33PCgYCk/FqZbdXLgb00E4vLsjAz38bbvhFNQT7Jh7qOOuxs3YmdLTuxu333sOvw8w0ZWGzIxWKvhCXdLchvOwhEG3Vfa5Kva4+suufOBnTJ3ROBIT0OhnQiIiKisef0+MOjx+9o6kLtsR54/crrc21GLRaVDlba50yzQ6fhbd+IphpPwIPP2j8Lh/bPOj4bdhu8YmsRFtsrsVhtxWJnP7LbDgCtewHf8EtvoNIAa/8KlCyfoE8wegzpcTCkExEREY0/jz+Az447sLOpCzuauvBxcxecXuWI2CadBmeXpGNxaRYWB2/7ZtSlXjdWIvp8BnwD2N22Ww7trTtR11k3bBC+CnsFFuWfgyWWEpzjF0jvOAy0fAa0fibf6u7/HQKsuQn6BCNjSI+DIZ2IiIho4vkDEupb+rCjqTM8IF33gE/RRqdRYW5RerjSXpVvgyp4laqA/JVVCCD05TX0NTby22xoXkBEzEe0VbQTQ9ZRrh9+vyjLIt8vtFytUqE4wwy7mbeqI/o8+rx9+OTUJ9jRugO7Wndhf9d+xesqqDAzcyYW5S/CkvzFWGieBmvW9ATt7elhSI+DIZ2IiIgo8SRJ4HB7P3Y0dYVv/Xaq9/TvFZ3MpqWbUJVvQ3VBWvBhQ0mWhdfjE52hHncPPjr1UfB2b7vQ4GhQvK5WqfGri36FpQVLE7SHI2NIj4MhnYiIiCj5CCFwrMsVrrTvbO7C8W4XVBi8A1Ooqg4VwqNAD31NpXhNFWqOiFWVr8VoPzgo/WC7oa8NfU+fJNDeF/2HBpNOgxn5NswqGAzvVfk22IysuhONVoerI+Ie7btwrO8YtqzeArvBnuhdi4khPQ6GdCIiIiIaLw6XDwda+1Df0ht+HDjVB7dPitq+KMMUDu2hAF+cYYaaVXei09bh6kC2KTvRuxEXQ3ocDOlERERENJECkkBThxP7W0PBXQ7xLQ531PYWvQZVwUp7ZNXdYtBO8J4T0VhhSI+DIZ2IiIiIkkG304v61l7sD4b2+tZeHDzVP+zWdYDcpb4k04yq/MHr3KsL0lCUYQp30yf6vCRJ4ESPC3lpRui1vD3iWGJIj4MhnYiIiIiSlT8gobHDqai417f0oi3Gte42gxZVBZGD1KVhZp4NJj1vZUcjE0Lu5fFhQye2NXRgW0Mnugd8sJt0uHhWHi6dW4AVFdkM7GOAIT0OhnQiIiIiSjWd/R7Ut/Rhf2sv9gUD/OG2PvgCw7/Kq1VAabZFDu0RXeYL7EZW3Qkne1z48LAcyLc2dKK1V3nZhUqlvM2g3aTDJbPzcOmcAqyozIZOw8B+JhjS42BIJyIiIqLJwOuX0NDej/qWXuyPGKyuo98btb3dpEN1gQ1V+WmYFQzu0/OsMOpYdZ/MOvo94UC+raEDzZ0Ditf1GjUWlmRgeUUWlldmYXahHbVHe/C3PS34+95WdPQP9uJINwcr7Azso8aQHgdDOhERERFNZm19brnq3jI4UF1Dez/8UvSv/ZkWPfLTjChMNyLfbkSB3YQCuzxfaDch325kkE8hvW4fdjR2YWuw+/r+1j7F6xq1CnOL7HIor8jGwpKMmMc3IAnsbOoKBvYWxQ9AocD+5bmFWF6RxcA+Aob0OBjSiYiIiGiq8fgDOHRqeNW9e8B3WuvHCvKRgZ5BPjFc3gA+OtKFrcFq+Z7jPRj6e0xVvg0rKrOxvCILi8oykWbUjfp9QoH9rT0n8fbe1mGB/ZJZ+bh0bgEDewwM6XEwpBMRERERyYOG9br8OOlwodXhHpz2uNHa60JLj7ws1j3eh8q06IPBnUF+PHn9Ej493oOthzuxtaEDtUd74A0oj1F5tgXLgpXypeWZyLIaxnQfApLAjqZO/G1PS8zA/uW5BVjGwB7GkB4HQzoRERER0ekRQsDh8qHF4R7zIF8Q7EpfmG5EfhqDfCwBSWDfyV5sbejA1oZO7GruwoA3oGhTYDdieYVcKV9WkYXCdNOE7t+Opk689Zkc2DudysC+cnY+Lp3DwM6QHgdDOhERERHR2IkM8i0OlzztccvBfhyCfLbNAIteM2lHqhdC4HBbP7Y2dOLDwx3Y3tiJXrdf0SbTog9WyuVqeWmWOSn+PfwBKdglfnhgzzDrcMnsYIW9PAvaKRbYGdLjYEgnIiIiIppY8YJ8S0R3+9MN8jqNChlmPTIt+sGpRYcMc+RzPTLNeqSbdci06GFO4mB/rGsAWxs68OFh+bryyBHVAcBm0GJJeSaWBavlM/NsUKuT87OEhAL7X/e04J0ogX1lTbDCPkUCO0N6HAzpRERERETJ53SCfIvDDZcvMPLGotBr1cg0y+E9w6wLh3h5qgsuVwZ8k358ut639bqxrVGulG9t6MTxbpfidYNWjUWlmeFq+Zxp9pQOsv6AhB0RFfauiMCeadHjktl5+PKcQiwtz0zpzxkPQ3ocDOlERERERKnL5Q2ge8CLLqd3cOr0omvAhx7Fcl9wuRde/+lV6Icy6tRRqvODgX4w6OvCVf1o19T3DHixvbEzPAL74bZ+xetatQrzi9OD15Rn4+ySdBi0k/Pa/FBg/+tnLXinLlpgz8eX5xRMusDOkB4HQzoRERER0dQhhIDLFwiGeR+6BuRQ3z3gDYf4bqcvHO67g8+Hjph+ukw6jaL7ffeAF3UnexGZulQqYHZhGpZXZGNZRRYWl2bCYtCO0SdOHf6AhO2NoQp7i+KWgKHAftncAiwpS/3AzpAeB0M6ERERERHFI4SA0xsIh/lo1flw1T4i+PuH3qA8QmWuNTjQWxaWlmch3ayfwE+U/AYDu3wf9sjAnmXR45IaucKeqoGdIT0OhnQiIiIiIhprQgj0e/yK0N7l9EKnVWNpWSZy04yJ3sWU4QtI2N44eB/2aIH9sjkFWJxCgZ0hPQ6GdCIiIiIiotQQCuxvBa9hHxrYVwYr7Mke2BnS42BIJyIiIiIiSj2+gIRtDcEKe10reiIC+wtrzsGF1XkJ3Lv4RpNDp97oBERERERERJRydBo1zp+Rg/Nn5OAH/1KDbQ1yhX1rYwfOnZ6d6N0bMwzpRERERERElFIiA7sQAiqVKtG7NGaSt9M+ERERERER0QgmU0AHGNKJiIiIiIiIkgZDOhEREREREVGSYEgnIiIiIiIiShIM6URERERERERJgiGdiIiIiIiIKEkwpBMRERERERElCYZ0IiIiIiIioiTBkE5ERERERESUJBjSiYiIiIiIiJIEQzoRERERERFRkmBIJyIiIiIiIkoSDOlERERERERESYIhnYiIiIiIiChJMKQTERERERERJQmGdCIiIiIiIqIkwZBORERERERElCQY0omIiIiIiIiShDbROzDRhBAAgN7e3gTvCREREREREU0FofwZyqPxTLmQ3tfXBwAoLi5O8J4QERERERHRVNLX1we73R63jUqcTpSfRCRJwsmTJ2Gz2aBSqRK9O3H19vaiuLgYx44dQ1paWqJ3h0aBxy518dilLh671Mbjl7p47FIXj13q4rFLPUII9PX1obCwEGp1/KvOp1wlXa1Wo6ioKNG7MSppaWk8+VIUj13q4rFLXTx2qY3HL3Xx2KUuHrvUxWOXWkaqoIdw4DgiIiIiIiKiJMGQTkRERERERJQkGNKTmMFgwKOPPgqDwZDoXaFR4rFLXTx2qYvHLrXx+KUuHrvUxWOXunjsJrcpN3AcERERERERUbJiJZ2IiIiIiIgoSTCkExERERERESUJhnQiIiIiIiKiJMGQTkRERERERJQkGNIT6JlnnkFZWRmMRiMWLlyILVu2xG2/efNmLFy4EEajEeXl5fjlL385QXtKkX784x9j0aJFsNlsyM3Nxb/8y7/gwIEDcdfZtGkTVCrVsMf+/fsnaK8JAB577LFhxyA/Pz/uOjzvkkNpaWnUc+i2226L2p7nXGK9//77uPzyy1FYWAiVSoU33nhD8boQAo899hgKCwthMpnwhS98AXV1dSNu9/XXX8esWbNgMBgwa9YsbNiwYZw+wdQV79j5fD488MADmDNnDiwWCwoLC/H1r38dJ0+ejLvNl156Ker56Ha7x/nTTC0jnXdr164ddgyWLl064nZ53o2/kY5dtPNHpVLhP//zP2Nuk+ddamNIT5A//OEPuOuuu/Dwww+jtrYW5513HlatWoWjR49Gbd/U1IRLL70U5513Hmpra/HQQw/hjjvuwOuvvz7Be06bN2/Gbbfdhu3bt2Pjxo3w+/24+OKL4XQ6R1z3wIEDaGlpCT+mT58+AXtMkWbPnq04Bnv27InZludd8ti1a5fiuG3cuBEAcPXVV8ddj+dcYjidTsybNw9PP/101Nf/4z/+Az/96U/x9NNPY9euXcjPz8dFF12Evr6+mNvctm0brr32Wtxwww349NNPccMNN+Caa67Bjh07xutjTEnxjt3AwAA++eQTPPLII/jkk0+wfv16HDx4EFdcccWI201LS1Ociy0tLTAajePxEaaskc47AFi5cqXiGPztb3+Lu02edxNjpGM39Nx58cUXoVKpcNVVV8XdLs+7FCYoIRYvXixuueUWxbKqqirx4IMPRm1///33i6qqKsWym2++WSxdunTc9pFOT1tbmwAgNm/eHLPNe++9JwCI7u7uidsxGubRRx8V8+bNO+32PO+S15133ikqKiqEJElRX+c5lzwAiA0bNoSfS5Ik8vPzxRNPPBFe5na7hd1uF7/85S9jbueaa64RK1euVCy75JJLxOrVq8d8n0k29NhFs3PnTgFAHDlyJGabdevWCbvdPrY7R3FFO3Zr1qwRV1555ai2w/Nu4p3OeXfllVeKL33pS3Hb8LxLbaykJ4DX68XHH3+Miy++WLH84osvxtatW6Ous23btmHtL7nkEnz00Ufw+Xzjtq80MofDAQDIzMwcse2CBQtQUFCACy+8EO+999547xpFcejQIRQWFqKsrAyrV69GY2NjzLY875KT1+vFK6+8gm984xtQqVRx2/KcSz5NTU1obW1VnFsGgwEXXHBBzL+BQOzzMd46NP4cDgdUKhXS09Pjtuvv70dJSQmKiopw2WWXoba2dmJ2kBQ2bdqE3NxczJgxAzfddBPa2tritud5l3xOnTqFt956C9/85jdHbMvzLnUxpCdAR0cHAoEA8vLyFMvz8vLQ2toadZ3W1tao7f1+Pzo6OsZtXyk+IQTuuecenHvuuaipqYnZrqCgAM899xxef/11rF+/HjNnzsSFF16I999/fwL3lpYsWYLf/OY3eOedd/D888+jtbUVy5cvR2dnZ9T2PO+S0xtvvIGenh6sXbs2Zhuec8kr9HduNH8DQ+uNdh0aX263Gw8++CC+9rWvIS0tLWa7qqoqvPTSS3jzzTfxu9/9DkajEStWrMChQ4cmcG9p1apVePXVV/HPf/4T//Vf/4Vdu3bhS1/6EjweT8x1eN4ln5dffhk2mw1f/epX47bjeZfatInegalsaAVICBG3KhStfbTlNHG+853v4LPPPsMHH3wQt93MmTMxc+bM8PNly5bh2LFjePLJJ3H++eeP925S0KpVq8Lzc+bMwbJly1BRUYGXX34Z99xzT9R1eN4lnxdeeAGrVq1CYWFhzDY855LfaP8Gnuk6ND58Ph9Wr14NSZLwzDPPxG27dOlSxQBlK1aswNlnn42f//zn+J//+Z/x3lUKuvbaa8PzNTU1OOecc1BSUoK33norbuDjeZdcXnzxRVx//fUjXlvO8y61sZKeANnZ2dBoNMN+hWxraxv2a2VIfn5+1PZarRZZWVnjtq8U2+23344333wT7733HoqKika9/tKlS/lrZoJZLBbMmTMn5nHgeZd8jhw5gnfffRff+ta3Rr0uz7nkELqjwmj+BobWG+06ND58Ph+uueYaNDU1YePGjXGr6NGo1WosWrSI52OCFRQUoKSkJO5x4HmXXLZs2YIDBw6c0d9AnnephSE9AfR6PRYuXBgenThk48aNWL58edR1li1bNqz9//7v/+Kcc86BTqcbt32l4YQQ+M53voP169fjn//8J8rKys5oO7W1tSgoKBjjvaPR8Hg8qK+vj3kceN4ln3Xr1iE3Nxdf/vKXR70uz7nkUFZWhvz8fMW55fV6sXnz5ph/A4HY52O8dWjshQL6oUOH8O67757RD5ZCCOzevZvnY4J1dnbi2LFjcY8Dz7vk8sILL2DhwoWYN2/eqNfleZdiEjVi3VT3+9//Xuh0OvHCCy+Iffv2ibvuuktYLBbR3NwshBDiwQcfFDfccEO4fWNjozCbzeLuu+8W+/btEy+88ILQ6XTiT3/6U6I+wpR16623CrvdLjZt2iRaWlrCj4GBgXCbocfvqaeeEhs2bBAHDx4Ue/fuFQ8++KAAIF5//fVEfIQp69577xWbNm0SjY2NYvv27eKyyy4TNpuN512KCAQC4qyzzhIPPPDAsNd4ziWXvr4+UVtbK2prawUA8dOf/lTU1taGRwB/4oknhN1uF+vXrxd79uwR1113nSgoKBC9vb3hbdxwww2KO558+OGHQqPRiCeeeELU19eLJ554Qmi1WrF9+/YJ/3yTWbxj5/P5xBVXXCGKiorE7t27FX8DPR5PeBtDj91jjz0m3n77bdHQ0CBqa2vFjTfeKLRardixY0ciPuKkFe/Y9fX1iXvvvVds3bpVNDU1iffee08sW7ZMTJs2jeddEhjp/0whhHA4HMJsNotnn3026jZ43k0uDOkJ9Itf/EKUlJQIvV4vzj77bMUtvNasWSMuuOACRftNmzaJBQsWCL1eL0pLS2OepDS+AER9rFu3Ltxm6PH7yU9+IioqKoTRaBQZGRni3HPPFW+99dbE7/wUd+2114qCggKh0+lEYWGh+OpXvyrq6urCr/O8S27vvPOOACAOHDgw7DWec8kldAu8oY81a9YIIeTbsD366KMiPz9fGAwGcf7554s9e/YotnHBBReE24f88Y9/FDNnzhQ6nU5UVVXxR5dxEO/YNTU1xfwb+N5774W3MfTY3XXXXeKss84Ser1e5OTkiIsvvlhs3bp14j/cJBfv2A0MDIiLL75Y5OTkCJ1OJ8466yyxZs0acfToUcU2eN4lxkj/ZwohxK9+9SthMplET09P1G3wvJtcVEIER0EiIiIiIiIiooTiNelERERERERESYIhnYiIiIiIiChJMKQTERERERERJQmGdCIiIiIiIqIkwZBORERERERElCQY0omIiIiIiIiSBEM6ERERERERUZJgSCciIiIiIiJKEgzpREREKaS5uRkqlQq7d+8e9/d66aWXkJ6ePu7vQ0RERIMY0omIiMbI2rVroVKphj1WrlyZ6F0bUWlpKX72s58pll177bU4ePDguL93Y2MjrrvuOhQWFsJoNKKoqAhXXnll+L0n8ocJIiKiRNMmegeIiIgmk5UrV2LdunWKZQaDIUF78/mYTCaYTKZxfQ+v14uLLroIVVVVWL9+PQoKCnD8+HH87W9/g8PhGNf3JiIiSkaspBMREY0hg8GA/Px8xSMjIwMAcN1112H16tWK9j6fD9nZ2eFg//bbb+Pcc89Feno6srKycNlll6GhoSHm+0Xrkv7GG29ApVKFnzc0NODKK69EXl4erFYrFi1ahHfffTf8+he+8AUcOXIEd999d7j6H2vbzz77LCoqKqDX6zFz5kz89re/VbyuUqnw61//Gl/5yldgNpsxffp0vPnmmzH3f9++fWhsbMQzzzyDpUuXoqSkBCtWrMCPfvQjLFq0CABQVlYGAFiwYAFUKhW+8IUvhNdft24dqqurYTQaUVVVhWeeeSb8WqgC//vf/x7Lly+H0WjE7NmzsWnTpnCb7u5uXH/99cjJyYHJZML06dOH/chCREQ0kRjSiYiIJsj111+PN998E/39/eFl77zzDpxOJ6666ioAgNPpxD333INdu3bhH//4B9RqNb7yla9AkqQzft/+/n5ceumlePfdd1FbW4tLLrkEl19+OY4ePQoAWL9+PYqKivD444+jpaUFLS0tUbezYcMG3Hnnnbj33nuxd+9e3Hzzzbjxxhvx3nvvKdp9//vfxzXXXIPPPvsMl156Ka6//np0dXVF3WZOTg7UajX+9Kc/IRAIRG2zc+dOAMC7776LlpYWrF+/HgDw/PPP4+GHH8aPfvQj1NfX49///d/xyCOP4OWXX1asf9999+Hee+9FbW0tli9fjiuuuAKdnZ0AgEceeQT79u3D3//+d9TX1+PZZ59Fdnb2af7LEhERjQNBREREY2LNmjVCo9EIi8WieDz++ONCCCG8Xq/Izs4Wv/nNb8LrXHfddeLqq6+Ouc22tjYBQOzZs0cIIURTU5MAIGpra4UQQqxbt07Y7XbFOhs2bBAj/YmfNWuW+PnPfx5+XlJSIp566ilFm6HbXr58ubjpppsUba6++mpx6aWXhp8DEN/73vfCz/v7+4VKpRJ///vfY+7L008/Lcxms7DZbOKLX/yiePzxx0VDQ0P49aGfOaS4uFi89tprimU/+MEPxLJlyxTrPfHEE+HXfT6fKCoqEj/5yU+EEEJcfvnl4sYbb4y5b0RERBONlXQiIqIx9MUvfhG7d+9WPG677TYAgE6nw9VXX41XX30VgFw1//Of/4zrr78+vH5DQwO+9rWvoby8HGlpaeGu3qGq95lwOp24//77MWvWLKSnp8NqtWL//v2j3mZ9fT1WrFihWLZixQrU19crls2dOzc8b7FYYLPZ0NbWFnO7t912G1pbW/HKK69g2bJl+OMf/4jZs2dj48aNMddpb2/HsWPH8M1vfhNWqzX8+OEPfzjs8oBly5aF57VaLc4555zwPt966634/e9/j/nz5+P+++/H1q1bR/6HICIiGkccOI6IiGgMWSwWVFZWxnz9+uuvxwUXXIC2tjZs3LgRRqMRq1atCr9++eWXo7i4GM8//zwKCwshSRJqamrg9Xqjbk+tVkMIoVjm8/kUz++77z688847ePLJJ1FZWQmTyYR//dd/jbnNeCKvdQcAIcSwZTqdbtg6I3XXt9lsuOKKK3DFFVfghz/8IS655BL88Ic/xEUXXRS1fWh7zz//PJYsWaJ4TaPRnPbnWLVqFY4cOYK33noL7777Li688ELcdtttePLJJ0fcBhER0XhgJZ2IiGgCLV++HMXFxfjDH/6AV199FVdffTX0ej0AoLOzE/X19fje976HCy+8ENXV1eju7o67vZycHPT19cHpdIaXDb1V2ZYtW7B27Vp85StfwZw5c5Cfn4/m5mZFG71eH/Oa8JDq6mp88MEHimVbt25FdXX1CJ96dFQqFaqqqsKfKfTvE7l/eXl5mDZtGhobG1FZWal4hHofhGzfvj087/f78fHHH6Oqqiq8LCcnB2vXrsUrr7yCn/3sZ3juuefG9PMQERGNBivpREREY8jj8aC1tVWxTKvVhgcjU6lU+NrXvoZf/vKXOHjwoGLQtYyMDGRlZeG5555DQUEBjh49igcffDDu+y1ZsgRmsxkPPfQQbr/9duzcuRMvvfSSok1lZSXWr1+Pyy+/HCqVCo888siwynZpaSnef/99rF69GgaDIergaffddx+uueYanH322bjwwgvxl7/8BevXr1eMFD9au3fvxqOPPoobbrgBs2bNgl6vx+bNm/Hiiy/igQceAADk5ubCZDLh7bffRlFREYxGI+x2Ox577DHccccdSEtLw6pVq+DxePDRRx+hu7sb99xzT/g9fvGLX2D69Omorq7GU089he7ubnzjG98AAPzbv/0bFi5ciNmzZ8Pj8eCvf/3rmP/oQERENCqJviieiIhoslizZo0AMOwxc+ZMRbu6ujoBQJSUlAhJkhSvbdy4UVRXVwuDwSDmzp0rNm3aJACIDRs2CCGiD6K2YcMGUVlZKYxGo7jsssvEc889pxg4rqmpSXzxi18UJpNJFBcXi6efflpccMEF4s477wy32bZtm5g7d64wGAzhdaMNSvfMM8+I8vJyodPpxIwZMxSD4AkhFPsaYrfbxbp166L+m7W3t4s77rhD1NTUCKvVKmw2m5gzZ4548sknRSAQCLd7/vnnRXFxsVCr1eKCCy4IL3/11VfF/PnzhV6vFxkZGeL8888X69evV/xbvfbaa2LJkiVCr9eL6upq8Y9//CO8/g9+8ANRXV0tTCaTyMzMFFdeeaVobGyMuq9EREQTQSXEkAvZiIiIiCaB5uZmlJWVoba2FvPnz0/07hAREZ0WXpNORERERERElCQY0omIiIiIiIiSBLu7ExERERERESUJVtKJiIiIiIiIkgRDOhEREREREVGSYEgnIiIiIiIiShIM6URERERERERJgiGdiIiIiIiIKEkwpBMRERERERElCYZ0IiIiIiIioiTBkE5ERERERESUJP4/79EkOXXvWIYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Softmax Attention Generated Text:\n",
      "First Citizen:\n",
      "Mehery Mose for wail, to am thirse: comanincess.\n",
      "So the peast thy wirlace, you: ip him.\n",
      "\n",
      "WARY Sigha\n",
      "\n",
      "Sigmoid Attention Generated Text:\n",
      "First Citizen:\n",
      "Forie with yat sorsent his forcaty:\n",
      "Sewere thop enoto detltsemang.\n",
      "\n",
      "NENIUSS:\n",
      "I dim he seenlanceor f\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import math\n",
    "\n",
    "# Load the saved weights\n",
    "model_softmax.load_state_dict(torch.load('checkpoints/out-softmax.pt', weights_only=False))\n",
    "model_sigmoid.load_state_dict(torch.load('checkpoints/out-sigmoid.pt', weights_only=False))\n",
    "\n",
    "# Plot training and test losses\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.plot(train_losses_softmax, label='Softmax Train')\n",
    "plt.plot(test_losses_softmax, label='Softmax Test')\n",
    "plt.plot(train_losses_sigmoid, label='Sigmoid Train')\n",
    "plt.plot(test_losses_sigmoid, label='Sigmoid Test')\n",
    "plt.xlabel('Evaluation Steps')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Training and Test Losses')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "# Generate text samples\n",
    "def generate_text(model, prompt, max_new_tokens=100):\n",
    "    model.eval()\n",
    "    context = torch.tensor([stoi[c] for c in prompt], dtype=torch.long, device=device).unsqueeze(0)\n",
    "    generated = model.generate(context, max_new_tokens=max_new_tokens)[0]\n",
    "    return ''.join([itos[int(i)] for i in generated])\n",
    "\n",
    "prompt = \"First Citizen:\"\n",
    "print(\"Softmax Attention Generated Text:\")\n",
    "print(generate_text(model_softmax, prompt))\n",
    "print(\"\\nSigmoid Attention Generated Text:\")\n",
    "print(generate_text(model_sigmoid, prompt))\n",
    "\n",
    "# Compare perplexity\n",
    "@torch.no_grad()\n",
    "def calculate_perplexity(model, data):\n",
    "    model.eval()\n",
    "    total_loss = 0\n",
    "    total_tokens = 0\n",
    "    for i in range(0, len(data) - block_size, block_size):\n",
    "        x = data[i:i+block_size].unsqueeze(0).to(device)\n",
    "        y = data[i+1:i+block_size+1].unsqueeze(0).to(device)\n",
    "        logits, loss = model(x, y)\n",
    "        total_loss += loss.item() * block_size\n",
    "        total_tokens += block_size\n",
    "    return math.exp(total_loss / total_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Softmax Attention Perplexity: 6.94\n",
      "Sigmoid Attention Perplexity: 7.77\n"
     ]
    }
   ],
   "source": [
    "softmax_perplexity = calculate_perplexity(model_softmax, test_data)\n",
    "sigmoid_perplexity = calculate_perplexity(model_sigmoid, test_data)\n",
    "\n",
    "print(f\"Softmax Attention Perplexity: {softmax_perplexity:.2f}\")\n",
    "print(f\"Sigmoid Attention Perplexity: {sigmoid_perplexity:.2f}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}