Copy_of_unit6.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "a34fabc6986e48f2bb6e12c108409ae6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "VBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "VBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "VBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_24a103c1373b41068a12faed02dab379",
              "IPY_MODEL_c962e045f1b94cabb62563753d455e31",
              "IPY_MODEL_658d54a1adb846dca2d910275f84231e",
              "IPY_MODEL_41231ee052944c9086767d9d7b2b7c1e",
              "IPY_MODEL_ea3cf8733cd6422e800957a69d6a6f3f"
            ],
            "layout": "IPY_MODEL_47d307b9ec0749579914e691576787c4"
          }
        },
        "24a103c1373b41068a12faed02dab379": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a2b053047b5641f88453e4acd56a312b",
            "placeholder": "​",
            "style": "IPY_MODEL_0ff1b530f7134711b62ac3572efad623",
            "value": "<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.svg\nalt='Hugging Face'> <br> Copy a token from <a\nhref=\"https://huggingface.co/settings/tokens\" target=\"_blank\">your Hugging Face\ntokens page</a> and paste it below. <br> Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file. </center>"
          }
        },
        "c962e045f1b94cabb62563753d455e31": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "PasswordModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "PasswordModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "PasswordView",
            "continuous_update": true,
            "description": "Token:",
            "description_tooltip": null,
            "disabled": false,
            "layout": "IPY_MODEL_93c9254e31074aeab27d17151f4c10be",
            "placeholder": "​",
            "style": "IPY_MODEL_eeb08bf5205c4eaab84cd70e699d2f10",
            "value": ""
          }
        },
        "658d54a1adb846dca2d910275f84231e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "CheckboxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "CheckboxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "CheckboxView",
            "description": "Add token as git credential?",
            "description_tooltip": null,
            "disabled": false,
            "indent": true,
            "layout": "IPY_MODEL_2623aab1ad264746aefbe5db1120b58d",
            "style": "IPY_MODEL_bb335f5acdf549c3a809eeeb1d12cc6f",
            "value": true
          }
        },
        "41231ee052944c9086767d9d7b2b7c1e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ButtonModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ButtonView",
            "button_style": "",
            "description": "Login",
            "disabled": false,
            "icon": "",
            "layout": "IPY_MODEL_99cc1849b8cf494c826350c6062d00ac",
            "style": "IPY_MODEL_4489b095f33943b88613df10dfc393a0",
            "tooltip": ""
          }
        },
        "ea3cf8733cd6422e800957a69d6a6f3f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4e8dcfbf15d84dbbbfb7fc1605352aff",
            "placeholder": "​",
            "style": "IPY_MODEL_b515fe462dd547d7983bb823b09b1623",
            "value": "\n<b>Pro Tip:</b> If you don't already have one, you can create a dedicated\n'notebooks' token with 'write' access, that you can then easily reuse for all\nnotebooks. </center>"
          }
        },
        "47d307b9ec0749579914e691576787c4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": "center",
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "flex",
            "flex": null,
            "flex_flow": "column",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "50%"
          }
        },
        "a2b053047b5641f88453e4acd56a312b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0ff1b530f7134711b62ac3572efad623": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "93c9254e31074aeab27d17151f4c10be": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eeb08bf5205c4eaab84cd70e699d2f10": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2623aab1ad264746aefbe5db1120b58d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bb335f5acdf549c3a809eeeb1d12cc6f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "99cc1849b8cf494c826350c6062d00ac": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4489b095f33943b88613df10dfc393a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ButtonStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "button_color": null,
            "font_weight": ""
          }
        },
        "4e8dcfbf15d84dbbbfb7fc1605352aff": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b515fe462dd547d7983bb823b09b1623": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "68eb672001184bc2b671390c312e9314": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_37bfe909ddcd4577a90cded7027c21ec",
              "IPY_MODEL_c009131f0b4b4cfb99fea816934f0a48",
              "IPY_MODEL_a1d9861fe7d343608cdae8ee9a47519b"
            ],
            "layout": "IPY_MODEL_d498d465aa094011ac5d229aa896ecdd"
          }
        },
        "37bfe909ddcd4577a90cded7027c21ec": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cc64ee8a978a4e858c3a6318ddae8928",
            "placeholder": "​",
            "style": "IPY_MODEL_e51e992c74084ddbb55b017bfffcd1ff",
            "value": "a2c-AntBulletEnv-v0.zip: 100%"
          }
        },
        "c009131f0b4b4cfb99fea816934f0a48": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_263aa7aab49648369436845307d6a973",
            "max": 129260,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c424805af3d0456da20ba0f24c3b3b93",
            "value": 129260
          }
        },
        "a1d9861fe7d343608cdae8ee9a47519b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bc03289624c54d848b2616206aa1a094",
            "placeholder": "​",
            "style": "IPY_MODEL_724340b6f03d4334ad06dac943251c8e",
            "value": " 129k/129k [00:02&lt;00:00, 146kB/s]"
          }
        },
        "d498d465aa094011ac5d229aa896ecdd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cc64ee8a978a4e858c3a6318ddae8928": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e51e992c74084ddbb55b017bfffcd1ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "263aa7aab49648369436845307d6a973": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c424805af3d0456da20ba0f24c3b3b93": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "bc03289624c54d848b2616206aa1a094": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "724340b6f03d4334ad06dac943251c8e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "503ffbb5b1e74b1fbe03c012e5f8a738": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d6c5f606aa6d4e7caf6e77073bf155d3",
              "IPY_MODEL_efcbebc94bde4515aa470a8fc2c9fa80",
              "IPY_MODEL_bd32ba77fe1c4044a654db5d08027801"
            ],
            "layout": "IPY_MODEL_97bda6a665a248d3a2e5b87cec5fb95f"
          }
        },
        "d6c5f606aa6d4e7caf6e77073bf155d3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1b601f2dca70414eb88a4586be3d84b5",
            "placeholder": "​",
            "style": "IPY_MODEL_d26bc3f87753444a8c3de5eeccd4baa8",
            "value": "vec_normalize.pkl: 100%"
          }
        },
        "efcbebc94bde4515aa470a8fc2c9fa80": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_db6bef2236c2450fb4d8572a10228eb1",
            "max": 2136,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c68c4db80fba4e07823bbfe005fa4ac9",
            "value": 2136
          }
        },
        "bd32ba77fe1c4044a654db5d08027801": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cdf0ab2d494440f58004d3283b328340",
            "placeholder": "​",
            "style": "IPY_MODEL_e681656f605944dfbddc127ba571865e",
            "value": " 2.14k/2.14k [00:02&lt;00:00, 2.67kB/s]"
          }
        },
        "97bda6a665a248d3a2e5b87cec5fb95f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1b601f2dca70414eb88a4586be3d84b5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d26bc3f87753444a8c3de5eeccd4baa8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "db6bef2236c2450fb4d8572a10228eb1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c68c4db80fba4e07823bbfe005fa4ac9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "cdf0ab2d494440f58004d3283b328340": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e681656f605944dfbddc127ba571865e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6548ec07c4224995a6f968e80eb61eb1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_2e142a5d69c64382a974cdda9bd4bb6d",
              "IPY_MODEL_75305559778e44629c881bc48d36af4d",
              "IPY_MODEL_835fa566e8dd4211acf13880d020b8dc"
            ],
            "layout": "IPY_MODEL_fe2a3d5ad75a46249011b58154f62478"
          }
        },
        "2e142a5d69c64382a974cdda9bd4bb6d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_edb23deac5bf45dcb7dde3d53a33b165",
            "placeholder": "​",
            "style": "IPY_MODEL_df51699b3979418786205186c0f17ec6",
            "value": "Upload 6 LFS files: 100%"
          }
        },
        "75305559778e44629c881bc48d36af4d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f96a4005a4c9489aba897e6f9cf1178e",
            "max": 6,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a606c822615f4c8baec8748fc7aa255c",
            "value": 6
          }
        },
        "835fa566e8dd4211acf13880d020b8dc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7249db09a122433caf03f6fc0c99cb5b",
            "placeholder": "​",
            "style": "IPY_MODEL_9c8e25fb846b489a8f213ee108b9a1b0",
            "value": " 6/6 [00:04&lt;00:00,  1.72it/s]"
          }
        },
        "fe2a3d5ad75a46249011b58154f62478": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "edb23deac5bf45dcb7dde3d53a33b165": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "df51699b3979418786205186c0f17ec6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f96a4005a4c9489aba897e6f9cf1178e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a606c822615f4c8baec8748fc7aa255c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7249db09a122433caf03f6fc0c99cb5b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9c8e25fb846b489a8f213ee108b9a1b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7678399be04742219bd6a8cc27886057": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0fead3213f6a4f3f8dcc61ad514af322",
              "IPY_MODEL_5446389ac76c43b98ba6497b359386f9",
              "IPY_MODEL_0a98f9e64c624a2497fb6cae4c92c971"
            ],
            "layout": "IPY_MODEL_36b460e71dd64b3584f85af9d3f52d87"
          }
        },
        "0fead3213f6a4f3f8dcc61ad514af322": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_02ca26a2ab084fea94043bf42d1c797b",
            "placeholder": "​",
            "style": "IPY_MODEL_e11636a343d9476da28d6665770bfb19",
            "value": "replay.mp4: 100%"
          }
        },
        "5446389ac76c43b98ba6497b359386f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8dc0467ca3454f4282d96194c76f0543",
            "max": 1076406,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_04f278e10f5a48b08a3bebbb9cbd2599",
            "value": 1076406
          }
        },
        "0a98f9e64c624a2497fb6cae4c92c971": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6b04ce624a194392b24d3b96c8a52e0d",
            "placeholder": "​",
            "style": "IPY_MODEL_bdd624c82ee24df3a6f8cca44831339e",
            "value": " 1.08M/1.08M [00:03&lt;00:00, 547kB/s]"
          }
        },
        "36b460e71dd64b3584f85af9d3f52d87": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "02ca26a2ab084fea94043bf42d1c797b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e11636a343d9476da28d6665770bfb19": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8dc0467ca3454f4282d96194c76f0543": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "04f278e10f5a48b08a3bebbb9cbd2599": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6b04ce624a194392b24d3b96c8a52e0d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bdd624c82ee24df3a6f8cca44831339e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ae35b56363a142dd83dec4b4f32f4d9c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_79e1484ef15440fcafb7e477ec58a1ba",
              "IPY_MODEL_f114800172b540e8bd4c7b0d8f0a7a26",
              "IPY_MODEL_17e1863f5fe04b7eb83ad5a3bea15668"
            ],
            "layout": "IPY_MODEL_c1a2ea9a3c9a491b84e02311ac57a59c"
          }
        },
        "79e1484ef15440fcafb7e477ec58a1ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_974ffb94b0ee435986e33851d4ced978",
            "placeholder": "​",
            "style": "IPY_MODEL_8fc015bbe24b4d6099fa82c2e93e7878",
            "value": "policy.pth: 100%"
          }
        },
        "f114800172b540e8bd4c7b0d8f0a7a26": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_657da5c546914b2c9d68dc6fab932622",
            "max": 56958,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_647d510b5580498c882bd82b56d0405d",
            "value": 56958
          }
        },
        "17e1863f5fe04b7eb83ad5a3bea15668": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7ad122fb00d2478da69cd75e7ffe8748",
            "placeholder": "​",
            "style": "IPY_MODEL_0b2b157b55fe423e8fc7fc19d886788c",
            "value": " 57.0k/57.0k [00:02&lt;00:00, 10.1kB/s]"
          }
        },
        "c1a2ea9a3c9a491b84e02311ac57a59c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "974ffb94b0ee435986e33851d4ced978": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8fc015bbe24b4d6099fa82c2e93e7878": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "657da5c546914b2c9d68dc6fab932622": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "647d510b5580498c882bd82b56d0405d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7ad122fb00d2478da69cd75e7ffe8748": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0b2b157b55fe423e8fc7fc19d886788c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a74d10d7a11649c49c335861243aaf7c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4f45935a6b104ab0bb612df7c6109051",
              "IPY_MODEL_247c19da4be146948526caeeaf6d55ac",
              "IPY_MODEL_1a125311748b488c80427c8aa5498525"
            ],
            "layout": "IPY_MODEL_1f146f7ce3894a0aac22db792ec6c30c"
          }
        },
        "4f45935a6b104ab0bb612df7c6109051": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_728dfcc6f2084950911bb447febf9f4a",
            "placeholder": "​",
            "style": "IPY_MODEL_e1a08c16eb28403c9015753b46b1de36",
            "value": "pytorch_variables.pth: 100%"
          }
        },
        "247c19da4be146948526caeeaf6d55ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bdec093d21d7466a80cbbd006aeaa41a",
            "max": 431,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_18d7eb175e6f41a99016fbb522049131",
            "value": 431
          }
        },
        "1a125311748b488c80427c8aa5498525": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0a112f2cb90147b6aaa4ee1e9763f0cb",
            "placeholder": "​",
            "style": "IPY_MODEL_390b64f7f8244ab6867a34daba51919f",
            "value": " 431/431 [00:02&lt;00:00, 548B/s]"
          }
        },
        "1f146f7ce3894a0aac22db792ec6c30c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "728dfcc6f2084950911bb447febf9f4a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e1a08c16eb28403c9015753b46b1de36": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bdec093d21d7466a80cbbd006aeaa41a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "18d7eb175e6f41a99016fbb522049131": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0a112f2cb90147b6aaa4ee1e9763f0cb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "390b64f7f8244ab6867a34daba51919f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3b1edea109d54474b3aff38345aacf10": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_17c94d0b4247461f99266a1621bbcc91",
              "IPY_MODEL_6f720c1611aa4dc3ba690eee1e1f7594",
              "IPY_MODEL_796c4dbf273048968c27502917bf52e7"
            ],
            "layout": "IPY_MODEL_3ea60dd3cc23472faee54b0f9b990e22"
          }
        },
        "17c94d0b4247461f99266a1621bbcc91": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2c9f69117f794b689d5f083dfb7a5905",
            "placeholder": "​",
            "style": "IPY_MODEL_3665b8ceb29e4cd38f9039ed8b4abd72",
            "value": "policy.optimizer.pth: 100%"
          }
        },
        "6f720c1611aa4dc3ba690eee1e1f7594": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e79bec7d2eb04e3f94580cb6b74152b3",
            "max": 56190,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_7a486346d7574a86afd2aa758a85f175",
            "value": 56190
          }
        },
        "796c4dbf273048968c27502917bf52e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1ea24607e6ff45298cfa62aebae76124",
            "placeholder": "​",
            "style": "IPY_MODEL_38223374a3a542d89e3415bd44470767",
            "value": " 56.2k/56.2k [00:02&lt;00:00, 11.4kB/s]"
          }
        },
        "3ea60dd3cc23472faee54b0f9b990e22": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2c9f69117f794b689d5f083dfb7a5905": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3665b8ceb29e4cd38f9039ed8b4abd72": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e79bec7d2eb04e3f94580cb6b74152b3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7a486346d7574a86afd2aa758a85f175": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1ea24607e6ff45298cfa62aebae76124": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "38223374a3a542d89e3415bd44470767": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e26e2fa08e2448b599ae19b47423d050": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6de3ea06cb6d4d50be950012e4b98a55",
              "IPY_MODEL_fae640cc5a8c462884a9cd615e3c2682",
              "IPY_MODEL_e924ae7a5954423980fbc8751c1bd900"
            ],
            "layout": "IPY_MODEL_e5bd6c7f5f7a4c30ac45cfb605ad4303"
          }
        },
        "6de3ea06cb6d4d50be950012e4b98a55": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_20538c22adec46a68cf61e4b3a86950f",
            "placeholder": "​",
            "style": "IPY_MODEL_5c81f0c24c554b29ae0becefa688d355",
            "value": "a2c-PandaReachDense-v2.zip: 100%"
          }
        },
        "fae640cc5a8c462884a9cd615e3c2682": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_16a9e0378aa54d6c8f6b6bd2eb35fd94",
            "max": 108024,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_51ba58f98d904d868246cbe1dff5a565",
            "value": 108024
          }
        },
        "e924ae7a5954423980fbc8751c1bd900": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_96f7d176addb4e63aaad69f86454c743",
            "placeholder": "​",
            "style": "IPY_MODEL_1a2dfcd55d1f44cdbe98af3b6ba52b87",
            "value": " 108k/108k [00:02&lt;00:00, 125kB/s]"
          }
        },
        "e5bd6c7f5f7a4c30ac45cfb605ad4303": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "20538c22adec46a68cf61e4b3a86950f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5c81f0c24c554b29ae0becefa688d355": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "16a9e0378aa54d6c8f6b6bd2eb35fd94": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "51ba58f98d904d868246cbe1dff5a565": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "96f7d176addb4e63aaad69f86454c743": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1a2dfcd55d1f44cdbe98af3b6ba52b87": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "41ca82d652664ea18f7573ce4f55ab1f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_40dfffa0bd634c10832c62ea606df622",
              "IPY_MODEL_e0c9825ac17540a6b8b0320631923f6e",
              "IPY_MODEL_fa020e55ffd94116afe3ce6f303a7518"
            ],
            "layout": "IPY_MODEL_7399aa7a4e8f442696db78a76f61a830"
          }
        },
        "40dfffa0bd634c10832c62ea606df622": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4ef9bcc993794a0cbcc6ba57b055ea2d",
            "placeholder": "​",
            "style": "IPY_MODEL_dfe61677995e41d4a132a0fccce05e1a",
            "value": "vec_normalize.pkl: 100%"
          }
        },
        "e0c9825ac17540a6b8b0320631923f6e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_968b4e38f8d84ab29fc1bb802a762ce9",
            "max": 3056,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8b4520ad34fe4922a19aae8e64bd47aa",
            "value": 3056
          }
        },
        "fa020e55ffd94116afe3ce6f303a7518": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_79b06a88db2d45bb9adafb479f1328af",
            "placeholder": "​",
            "style": "IPY_MODEL_ee426146660649d6b0f7b89c141f7a64",
            "value": " 3.06k/3.06k [00:02&lt;00:00, 3.54kB/s]"
          }
        },
        "7399aa7a4e8f442696db78a76f61a830": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4ef9bcc993794a0cbcc6ba57b055ea2d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dfe61677995e41d4a132a0fccce05e1a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "968b4e38f8d84ab29fc1bb802a762ce9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8b4520ad34fe4922a19aae8e64bd47aa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "79b06a88db2d45bb9adafb479f1328af": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ee426146660649d6b0f7b89c141f7a64": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cc2dce85d39940c89dffdaffbf2c91fb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6e2f0398430a4bffa7f048f83884d711",
              "IPY_MODEL_6d64726e416c44b4bd9d195f426f573d",
              "IPY_MODEL_9697c4a45c92434086342c0e750267cc"
            ],
            "layout": "IPY_MODEL_5ae0765a12dc4c5686fc84947c293257"
          }
        },
        "6e2f0398430a4bffa7f048f83884d711": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2a5b7d631c8747c28b2f88e062b6c23a",
            "placeholder": "​",
            "style": "IPY_MODEL_a27612ae7b044a2ca392af8a14b2b491",
            "value": "policy.pth: 100%"
          }
        },
        "6d64726e416c44b4bd9d195f426f573d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ceff407b85254fb4898dfe2e74f10c95",
            "max": 46014,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_67b5ba536b254a81bf3f9501c477b0c0",
            "value": 46014
          }
        },
        "9697c4a45c92434086342c0e750267cc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_afae0e81ffb04c7e9d497b731207e681",
            "placeholder": "​",
            "style": "IPY_MODEL_2eccf98cc0544b9c8cbfe938356ed745",
            "value": " 46.0k/46.0k [00:02&lt;00:00, 9.54kB/s]"
          }
        },
        "5ae0765a12dc4c5686fc84947c293257": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a5b7d631c8747c28b2f88e062b6c23a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a27612ae7b044a2ca392af8a14b2b491": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ceff407b85254fb4898dfe2e74f10c95": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "67b5ba536b254a81bf3f9501c477b0c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "afae0e81ffb04c7e9d497b731207e681": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2eccf98cc0544b9c8cbfe938356ed745": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "82dc7446550048c1a962119ff1d0097e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_451fd2eed21c475894e218c6e8b74bd3",
              "IPY_MODEL_0122668d4f07427aac6f0f2c7d543686",
              "IPY_MODEL_8dc3f46be2f94cdbb1baffc60211edfc"
            ],
            "layout": "IPY_MODEL_a36a7df9f04d48339845a8c4c266c1fc"
          }
        },
        "451fd2eed21c475894e218c6e8b74bd3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0ca9d512574b4d329969b7b6fcf13b36",
            "placeholder": "​",
            "style": "IPY_MODEL_612ec11319814c279e55438997d129b1",
            "value": "pytorch_variables.pth: 100%"
          }
        },
        "0122668d4f07427aac6f0f2c7d543686": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_873488239dc445e1ae139e06040d7eeb",
            "max": 431,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0e1c0a01b61043549528bdbd62b153b4",
            "value": 431
          }
        },
        "8dc3f46be2f94cdbb1baffc60211edfc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ead13d07941c497884baa0a617ccee9c",
            "placeholder": "​",
            "style": "IPY_MODEL_5f8ba321ce0e4426af1ef0fd0314f4c8",
            "value": " 431/431 [00:02&lt;00:00, 509B/s]"
          }
        },
        "a36a7df9f04d48339845a8c4c266c1fc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0ca9d512574b4d329969b7b6fcf13b36": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "612ec11319814c279e55438997d129b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "873488239dc445e1ae139e06040d7eeb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0e1c0a01b61043549528bdbd62b153b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ead13d07941c497884baa0a617ccee9c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5f8ba321ce0e4426af1ef0fd0314f4c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6216c51a7f074b088bc9fcc3c08e0df6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_49e10565811f41919c484d1d1fda3e67",
              "IPY_MODEL_24c76f9d966c48788fd9da971c353aa6",
              "IPY_MODEL_bcc4ed12941c4540a699967177865252"
            ],
            "layout": "IPY_MODEL_00f030e66abb4b7bbcf773408802ab6b"
          }
        },
        "49e10565811f41919c484d1d1fda3e67": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_95dac9ad006f4aefa9ce69ab170dea24",
            "placeholder": "​",
            "style": "IPY_MODEL_dc4852a69c9c4bc688d084b117129add",
            "value": "Upload 5 LFS files: 100%"
          }
        },
        "24c76f9d966c48788fd9da971c353aa6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d9b2b0d38aef42ef81ba6bd70e6dccf7",
            "max": 5,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_bf1d8aa8635e4d0c97a32ad2037f0250",
            "value": 5
          }
        },
        "bcc4ed12941c4540a699967177865252": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ea326412469948518cbc4f3afc2c6311",
            "placeholder": "​",
            "style": "IPY_MODEL_e87b3b26439349b88b1c37a28553dc98",
            "value": " 5/5 [00:02&lt;00:00,  2.68s/it]"
          }
        },
        "00f030e66abb4b7bbcf773408802ab6b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "95dac9ad006f4aefa9ce69ab170dea24": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dc4852a69c9c4bc688d084b117129add": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d9b2b0d38aef42ef81ba6bd70e6dccf7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bf1d8aa8635e4d0c97a32ad2037f0250": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ea326412469948518cbc4f3afc2c6311": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e87b3b26439349b88b1c37a28553dc98": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "43e09e204dd146a88b6717837ebf7f72": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_2296f0e560d74ad3b09558b27a74963b",
              "IPY_MODEL_6fffcdd8ace14a17bfe65453980652bc",
              "IPY_MODEL_a17e56d7eaf44e35ad79e71a0cb3e773"
            ],
            "layout": "IPY_MODEL_1332380fd3374eacb9a6059d6aa3b0c0"
          }
        },
        "2296f0e560d74ad3b09558b27a74963b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d65e7f78d8b94028822a66cb71b98efc",
            "placeholder": "​",
            "style": "IPY_MODEL_37562c17b5f54e4aa417092348543490",
            "value": "policy.optimizer.pth: 100%"
          }
        },
        "6fffcdd8ace14a17bfe65453980652bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d887c679a8864a008d7e09fd51e5abc9",
            "max": 44734,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d32667a445014cc787536fd36e78c7f6",
            "value": 44734
          }
        },
        "a17e56d7eaf44e35ad79e71a0cb3e773": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e9afd0ead8824c76817ae5c9ec6df20e",
            "placeholder": "​",
            "style": "IPY_MODEL_2960f3f6f09b4f49a52ce45ad3131deb",
            "value": " 44.7k/44.7k [00:02&lt;00:00, 10.2kB/s]"
          }
        },
        "1332380fd3374eacb9a6059d6aa3b0c0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d65e7f78d8b94028822a66cb71b98efc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "37562c17b5f54e4aa417092348543490": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d887c679a8864a008d7e09fd51e5abc9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d32667a445014cc787536fd36e78c7f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e9afd0ead8824c76817ae5c9ec6df20e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2960f3f6f09b4f49a52ce45ad3131deb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Ryukijano/Deep-Reinforcement-Learning-and-Hugging-Face/blob/master/Copy_of_unit6.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Unit 6: Advantage Actor Critic (A2C) using Robotics Simulations with PyBullet and Panda-Gym 🤖\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit8/thumbnail.png\"  alt=\"Thumbnail\"/>\n",
        "\n",
        "In this notebook, you'll learn to use A2C with PyBullet and Panda-Gym, two set of robotics environments. \n",
        "\n",
        "With [PyBullet](https://github.com/bulletphysics/bullet3), you're going to **train a robot to move**:\n",
        "- `AntBulletEnv-v0` 🕸️ More precisely, a spider (they say Ant but come on... it's a spider 😆) 🕸️\n",
        "\n",
        "Then, with [Panda-Gym](https://github.com/qgallouedec/panda-gym), you're going **to train a robotic arm** (Franka Emika Panda robot) to perform a task:\n",
        "- `Reach`: the robot must place its end-effector at a target position.\n",
        "\n",
        "After that, you'll be able **to train in other robotics environments**.\n"
      ],
      "metadata": {
        "id": "-PTReiOw-RAN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit8/environments.gif\" alt=\"Robotics environments\"/>"
      ],
      "metadata": {
        "id": "2VGL_0ncoAJI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 🎮 Environments: \n",
        "\n",
        "- [PyBullet](https://github.com/bulletphysics/bullet3)\n",
        "- [Panda-Gym](https://github.com/qgallouedec/panda-gym)\n",
        "\n",
        "###📚 RL-Library: \n",
        "\n",
        "- [Stable-Baselines3](https://stable-baselines3.readthedocs.io/)"
      ],
      "metadata": {
        "id": "QInFitfWno1Q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)."
      ],
      "metadata": {
        "id": "2CcdX4g3oFlp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Objectives of this notebook 🏆\n",
        "\n",
        "At the end of the notebook, you will:\n",
        "\n",
        "- Be able to use **PyBullet** and **Panda-Gym**, the environment libraries.\n",
        "- Be able to **train robots using A2C**.\n",
        "- Understand why **we need to normalize the input**.\n",
        "- Be able to **push your trained agent and the code to the Hub** with a nice video replay and an evaluation score 🔥.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "MoubJX20oKaQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## This notebook is from the Deep Reinforcement Learning Course\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\" alt=\"Deep RL Course illustration\"/>\n",
        "\n",
        "In this free course, you will:\n",
        "\n",
        "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
        "- 🧑‍💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
        "- 🤖 Train **agents in unique environments** \n",
        "\n",
        "And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
        "\n",
        "Don’t forget to **<a href=\"http://eepurl.com/ic5ZUD\">sign up to the course</a>** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
        "\n",
        "\n",
        "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
      ],
      "metadata": {
        "id": "DoUNkTExoUED"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prerequisites 🏗️\n",
        "Before diving into the notebook, you need to:\n",
        "\n",
        "🔲 📚 Study [Actor-Critic methods by reading Unit 6](https://huggingface.co/deep-rl-course/unit6/introduction) 🤗  "
      ],
      "metadata": {
        "id": "BTuQAUAPoa5E"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Let's train our first robots 🤖"
      ],
      "metadata": {
        "id": "iajHvVDWoo01"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To validate this hands-on for the [certification process](https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process),  you need to push your two trained models to the Hub and get the following results:\n",
        "\n",
        "- `AntBulletEnv-v0` get a result of >= 650.\n",
        "- `PandaReachDense-v2` get a result of >= -3.5.\n",
        "\n",
        "To find your result, go to the [leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) and find your model, **the result = mean_reward - std of reward**\n",
        "\n",
        "If you don't find your model, **go to the bottom of the page and click on the refresh button**\n",
        "\n",
        "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process"
      ],
      "metadata": {
        "id": "zbOENTE2os_D"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Set the GPU 💪\n",
        "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\" alt=\"GPU Step 1\">"
      ],
      "metadata": {
        "id": "PU4FVzaoM6fC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "- `Hardware Accelerator > GPU`\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\" alt=\"GPU Step 2\">"
      ],
      "metadata": {
        "id": "KV0NyFdQM9ZG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create a virtual display 🔽\n",
        "\n",
        "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
        "\n",
        "Hence the following cell will install the librairies and create and run a virtual screen 🖥"
      ],
      "metadata": {
        "id": "bTpYcVZVMzUI"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "jV6wjQ7Be7p5"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!apt install python-opengl\n",
        "!apt install ffmpeg\n",
        "!apt install xvfb\n",
        "!pip3 install pyvirtualdisplay"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Virtual display\n",
        "from pyvirtualdisplay import Display\n",
        "\n",
        "virtual_display = Display(visible=0, size=(1400, 900))\n",
        "virtual_display.start()"
      ],
      "metadata": {
        "id": "ww5PQH1gNLI4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b8a6e766-7518-4fa0-fe64-679856bbabbf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyvirtualdisplay.display.Display at 0x7fd9c0160a00>"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Install dependencies 🔽\n",
        "The first step is to install the dependencies, we’ll install multiple ones:\n",
        "\n",
        "- `pybullet`: Contains the walking robots environments.\n",
        "- `panda-gym`: Contains the robotics arm environments.\n",
        "- `stable-baselines3[extra]`: The SB3 deep reinforcement learning library.\n",
        "- `huggingface_sb3`: Additional code for Stable-baselines3 to load and upload models from the Hugging Face 🤗 Hub.\n",
        "- `huggingface_hub`: Library allowing anyone to work with the Hub repositories."
      ],
      "metadata": {
        "id": "e1obkbdJ_KnG"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2yZRi_0bQGPM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "631b3474-178f-4c7c-f253-e2755defccde"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting stable-baselines3[extra]\n",
            "  Downloading stable_baselines3-1.7.0-py3-none-any.whl (171 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m171.8/171.8 KB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting huggingface_sb3\n",
            "  Downloading huggingface_sb3-2.2.4-py3-none-any.whl (9.4 kB)\n",
            "Collecting panda_gym==2.0.0\n",
            "  Downloading panda_gym-2.0.0-py3-none-any.whl (26 kB)\n",
            "Collecting pyglet==1.5.1\n",
            "  Downloading pyglet-1.5.1-py2.py3-none-any.whl (1.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m68.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from panda_gym==2.0.0->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 3)) (1.22.4)\n",
            "Collecting pybullet\n",
            "  Downloading pybullet-3.2.5-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (91.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m91.7/91.7 MB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: gym in /usr/local/lib/python3.8/dist-packages (from panda_gym==2.0.0->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 3)) (0.25.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from panda_gym==2.0.0->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 3)) (1.7.3)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2.2.1)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (3.5.3)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.3.5)\n",
            "Collecting importlib-metadata~=4.13\n",
            "  Downloading importlib_metadata-4.13.0-py3-none-any.whl (23 kB)\n",
            "Requirement already satisfied: torch>=1.11 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.13.1+cu116)\n",
            "Collecting gym\n",
            "  Downloading gym-0.21.0.tar.gz (1.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m78.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (4.64.1)\n",
            "Collecting ale-py==0.7.4\n",
            "  Downloading ale_py-0.7.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m76.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pillow in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (8.4.0)\n",
            "Requirement already satisfied: tensorboard>=2.9.1 in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2.11.2)\n",
            "Collecting autorom[accept-rom-license]~=0.4.2\n",
            "  Downloading AutoROM-0.4.2-py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (5.4.8)\n",
            "Collecting rich\n",
            "  Downloading rich-13.3.1-py3-none-any.whl (239 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m239.0/239.0 KB\u001b[0m \u001b[31m29.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: opencv-python in /usr/local/lib/python3.8/dist-packages (from stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (4.6.0.66)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from ale-py==0.7.4->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (5.12.0)\n",
            "Requirement already satisfied: pyyaml~=6.0 in /usr/local/lib/python3.8/dist-packages (from huggingface_sb3->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 2)) (6.0)\n",
            "Requirement already satisfied: wasabi in /usr/local/lib/python3.8/dist-packages (from huggingface_sb3->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 2)) (0.10.1)\n",
            "Collecting huggingface-hub~=0.8\n",
            "  Downloading huggingface_hub-0.12.1-py3-none-any.whl (190 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m190.3/190.3 KB\u001b[0m \u001b[31m23.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from autorom[accept-rom-license]~=0.4.2->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (8.1.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from autorom[accept-rom-license]~=0.4.2->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2.25.1)\n",
            "Collecting AutoROM.accept-rom-license\n",
            "  Downloading AutoROM.accept-rom-license-0.5.5.tar.gz (22 kB)\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 2)) (3.9.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 2)) (4.5.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub~=0.8->huggingface_sb3->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 2)) (23.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata~=4.13->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (3.15.0)\n",
            "Requirement already satisfied: protobuf<4,>=3.9.2 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (3.19.6)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (0.4.6)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (0.38.4)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2.16.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (3.4.1)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2.2.3)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.8.1)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.51.3)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (57.4.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (0.6.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (4.38.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.4.4)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2.8.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (0.11.0)\n",
            "Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (3.0.9)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2022.7.1)\n",
            "Collecting pygments<3.0.0,>=2.14.0\n",
            "  Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m74.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting markdown-it-py<3.0.0,>=2.1.0\n",
            "  Downloading markdown_it_py-2.2.0-py3-none-any.whl (84 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.5/84.5 KB\u001b[0m \u001b[31m606.3 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.15.0)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (5.3.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.3.1)\n",
            "Collecting mdurl~=0.1\n",
            "  Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (4.0.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (1.26.14)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.8/dist-packages (from werkzeug>=1.0.1->tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (2.1.2)\n",
            "Collecting libtorrent\n",
            "  Using cached libtorrent-2.0.7-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (8.6 MB)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.8/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.9.1->stable-baselines3[extra]->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt (line 1)) (3.2.2)\n",
            "Building wheels for collected packages: gym, AutoROM.accept-rom-license\n",
            "  Building wheel for gym (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gym: filename=gym-0.21.0-py3-none-any.whl size=1616821 sha256=24d0c98dbeb037dd4898ebbfee9148f678e7f796597eb5fcf3c17d1217f68b9a\n",
            "  Stored in directory: /root/.cache/pip/wheels/27/6d/b3/a3a6e10704795c9b9000f1ab2dc480dfe7bed42f5972806e73\n",
            "  Building wheel for AutoROM.accept-rom-license (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for AutoROM.accept-rom-license: filename=AutoROM.accept_rom_license-0.5.5-py3-none-any.whl size=441098 sha256=9273c519778119ad85725807e4be80e02757b816fa0d1055f8eb275389c53b04\n",
            "  Stored in directory: /root/.cache/pip/wheels/c3/86/6f/e96885ff274388b9f0636418a2926f46f076cd7e891670321d\n",
            "Successfully built gym AutoROM.accept-rom-license\n",
            "Installing collected packages: pyglet, pybullet, libtorrent, pygments, mdurl, importlib-metadata, gym, panda_gym, markdown-it-py, huggingface-hub, AutoROM.accept-rom-license, autorom, ale-py, stable-baselines3, rich, huggingface_sb3\n",
            "  Attempting uninstall: pygments\n",
            "    Found existing installation: Pygments 2.6.1\n",
            "    Uninstalling Pygments-2.6.1:\n",
            "      Successfully uninstalled Pygments-2.6.1\n",
            "  Attempting uninstall: importlib-metadata\n",
            "    Found existing installation: importlib-metadata 6.0.0\n",
            "    Uninstalling importlib-metadata-6.0.0:\n",
            "      Successfully uninstalled importlib-metadata-6.0.0\n",
            "  Attempting uninstall: gym\n",
            "    Found existing installation: gym 0.25.2\n",
            "    Uninstalling gym-0.25.2:\n",
            "      Successfully uninstalled gym-0.25.2\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "ipython 7.9.0 requires jedi>=0.10, which is not installed.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed AutoROM.accept-rom-license-0.5.5 ale-py-0.7.4 autorom-0.4.2 gym-0.21.0 huggingface-hub-0.12.1 huggingface_sb3-2.2.4 importlib-metadata-4.13.0 libtorrent-2.0.7 markdown-it-py-2.2.0 mdurl-0.1.2 panda_gym-2.0.0 pybullet-3.2.5 pyglet-1.5.1 pygments-2.14.0 rich-13.3.1 stable-baselines3-1.7.0\n"
          ]
        }
      ],
      "source": [
        "!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit6/requirements-unit6.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Import the packages 📦"
      ],
      "metadata": {
        "id": "QTep3PQQABLr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pybullet_envs\n",
        "import panda_gym\n",
        "import gym\n",
        "\n",
        "import os\n",
        "\n",
        "from huggingface_sb3 import load_from_hub, package_to_hub\n",
        "\n",
        "from stable_baselines3 import A2C\n",
        "from stable_baselines3.common.evaluation import evaluate_policy\n",
        "from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize\n",
        "from stable_baselines3.common.env_util import make_vec_env\n",
        "\n",
        "from huggingface_hub import notebook_login"
      ],
      "metadata": {
        "id": "HpiB8VdnQ7Bk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Environment 1: AntBulletEnv-v0 🕸\n",
        "\n"
      ],
      "metadata": {
        "id": "lfBwIS_oAVXI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create the AntBulletEnv-v0\n",
        "#### The environment 🎮\n",
        "In this environment, the agent needs to use correctly its different joints to walk correctly.\n",
        "You can find a detailled explanation of this environment here: https://hackmd.io/@jeffreymo/SJJrSJh5_#PyBullet"
      ],
      "metadata": {
        "id": "frVXOrnlBerQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env_id = \"AntBulletEnv-v0\"\n",
        "# Create the env\n",
        "env = gym.make(env_id)\n",
        "\n",
        "# Get the state space and action space\n",
        "s_size = env.observation_space.shape[0]\n",
        "a_size = env.action_space"
      ],
      "metadata": {
        "id": "JpU-JCDQYYax"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
        "print(\"The State Space is: \", s_size)\n",
        "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
      ],
      "metadata": {
        "id": "2ZfvcCqEYgrg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb55a26e-dc38-48d5-ece6-00ce5e4ac21e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "_____OBSERVATION SPACE_____ \n",
            "\n",
            "The State Space is:  28\n",
            "Sample observation [ 0.9079969   0.44507018  0.18524839 -0.29020578  0.9130252  -0.5412519\n",
            " -0.00579384 -0.8938713  -0.85982805  0.38960713  0.17623445  0.7413489\n",
            " -1.3120481   0.03487071  0.9744689  -0.12380921 -0.54346013  1.9468037\n",
            " -1.2590312   1.4217961  -0.3373394   1.7405407  -1.4088     -0.7180425\n",
            " -2.1430063   0.07882362 -0.8116915   0.08001423]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The observation Space (from [Jeffrey Y Mo](https://hackmd.io/@jeffreymo/SJJrSJh5_#PyBullet)):\n",
        "\n",
        "The difference is that our observation space is 28 not 29.\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit8/obs_space.png\" alt=\"PyBullet Ant Obs space\"/>\n"
      ],
      "metadata": {
        "id": "QzMmsdMJS7jh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
        "print(\"The Action Space is: \", a_size)\n",
        "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
      ],
      "metadata": {
        "id": "Tc89eLTYYkK2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cf6de60f-9038-4def-c934-2e67d793da45"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            " _____ACTION SPACE_____ \n",
            "\n",
            "The Action Space is:  Box([-1. -1. -1. -1. -1. -1. -1. -1.], [1. 1. 1. 1. 1. 1. 1. 1.], (8,), float32)\n",
            "Action Space Sample [ 0.7409361   0.6281361   0.7241572   0.6204377  -0.7129181   0.86028594\n",
            "  0.25823456  0.71500236]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The action Space (from [Jeffrey Y Mo](https://hackmd.io/@jeffreymo/SJJrSJh5_#PyBullet)):\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit8/action_space.png\" alt=\"PyBullet Ant Obs space\"/>\n"
      ],
      "metadata": {
        "id": "3RfsHhzZS9Pw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Normalize observation and rewards"
      ],
      "metadata": {
        "id": "S5sXcg469ysB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "A good practice in reinforcement learning is to [normalize input features](https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html). \n",
        "\n",
        "For that purpose, there is a wrapper that will compute a running average and standard deviation of input features.\n",
        "\n",
        "We also normalize rewards with this same wrapper by adding `norm_reward = True`\n",
        "\n",
        "[You should check the documentation to fill this cell](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#vecnormalize)"
      ],
      "metadata": {
        "id": "1ZyX6qf3Zva9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env = make_vec_env(env_id, n_envs=4)\n",
        "\n",
        "# Adding this wrapper to normalize the observation and the reward\n",
        "env = VecNormalize(env, norm_obs = True, norm_reward = True, clip_obs=10.)"
      ],
      "metadata": {
        "id": "1RsDtHHAQ9Ie"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Solution"
      ],
      "metadata": {
        "id": "tF42HvI7-gs5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "env = make_vec_env(env_id, n_envs=4)\n",
        "\n",
        "env = VecNormalize(env, norm_obs=True, norm_reward=True, clip_obs=10.)"
      ],
      "metadata": {
        "id": "2O67mqgC-hol"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create the A2C Model 🤖\n",
        "\n",
        "In this case, because we have a vector of 28 values as input, we'll use an MLP (multi-layer perceptron) as policy.\n",
        "\n",
        "For more information about A2C implementation with StableBaselines3 check: https://stable-baselines3.readthedocs.io/en/master/modules/a2c.html#notes\n",
        "\n",
        "To find the best parameters I checked the [official trained agents by Stable-Baselines3 team](https://huggingface.co/sb3)."
      ],
      "metadata": {
        "id": "4JmEVU6z1ZA-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = A2C(policy = \"MlpPolicy\",\n",
        "            env = env,\n",
        "            gae_lambda = 0.9,\n",
        "            gamma = 0.99,\n",
        "            learning_rate = 0.00096,\n",
        "            max_grad_norm = 0.5,\n",
        "            n_steps = 8,\n",
        "            vf_coef = 0.4,\n",
        "            ent_coef = 0.0,\n",
        "            policy_kwargs=dict(\n",
        "            log_std_init=-2, ortho_init=False),\n",
        "            normalize_advantage=False,\n",
        "            use_rms_prop= True,\n",
        "            use_sde= True,\n",
        "            verbose=1)"
      ],
      "metadata": {
        "id": "vR3T4qFt164I",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc5d2baf-01f1-4223-9845-4467c1d6751c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using cuda device\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Solution"
      ],
      "metadata": {
        "id": "nWAuOOLh-oQf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = A2C(policy = \"MlpPolicy\",\n",
        "            env = env,\n",
        "            gae_lambda = 0.9,\n",
        "            gamma = 0.99,\n",
        "            learning_rate = 0.00096,\n",
        "            max_grad_norm = 0.5,\n",
        "            n_steps = 8,\n",
        "            vf_coef = 0.4,\n",
        "            ent_coef = 0.0,\n",
        "            policy_kwargs=dict(\n",
        "            log_std_init=-2, ortho_init=False),\n",
        "            normalize_advantage=False,\n",
        "            use_rms_prop= True,\n",
        "            use_sde= True,\n",
        "            verbose=1)"
      ],
      "metadata": {
        "id": "FKFLY54T-pU1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Train the A2C agent 🏃\n",
        "- Let's train our agent for 2,000,000 timesteps, don't forget to use GPU on Colab. It will take approximately ~25-40min"
      ],
      "metadata": {
        "id": "opyK3mpJ1-m9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.learn(2_000_000)"
      ],
      "metadata": {
        "id": "4TuGHZD7RF1G",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6c67aefa-59a9-4540-9500-f17ca38540c4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1;30;43mStreaming output truncated to the last 5000 lines.\u001b[0m\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 34800    |\n",
            "|    time_elapsed       | 2308     |\n",
            "|    total_timesteps    | 1113600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.05    |\n",
            "|    explained_variance | 0.983    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 34799    |\n",
            "|    policy_loss        | 0.131    |\n",
            "|    std                | 0.0688   |\n",
            "|    value_loss         | 0.0074   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.44e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 34900    |\n",
            "|    time_elapsed       | 2314     |\n",
            "|    total_timesteps    | 1116800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.16    |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 34899    |\n",
            "|    policy_loss        | -0.261   |\n",
            "|    std                | 0.0686   |\n",
            "|    value_loss         | 0.0114   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.45e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35000    |\n",
            "|    time_elapsed       | 2321     |\n",
            "|    total_timesteps    | 1120000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.48    |\n",
            "|    explained_variance | 0.97     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 34999    |\n",
            "|    policy_loss        | -0.0735  |\n",
            "|    std                | 0.0685   |\n",
            "|    value_loss         | 0.00525  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.45e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35100    |\n",
            "|    time_elapsed       | 2327     |\n",
            "|    total_timesteps    | 1123200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.24    |\n",
            "|    explained_variance | 0.98     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35099    |\n",
            "|    policy_loss        | 0.14     |\n",
            "|    std                | 0.0681   |\n",
            "|    value_loss         | 0.00713  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.48e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35200    |\n",
            "|    time_elapsed       | 2334     |\n",
            "|    total_timesteps    | 1126400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.97    |\n",
            "|    explained_variance | 0.9      |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35199    |\n",
            "|    policy_loss        | -0.104   |\n",
            "|    std                | 0.0681   |\n",
            "|    value_loss         | 0.00142  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.5e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35300    |\n",
            "|    time_elapsed       | 2340     |\n",
            "|    total_timesteps    | 1129600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.1     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35299    |\n",
            "|    policy_loss        | 0.124    |\n",
            "|    std                | 0.0683   |\n",
            "|    value_loss         | 0.0012   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.48e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35400    |\n",
            "|    time_elapsed       | 2346     |\n",
            "|    total_timesteps    | 1132800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.8     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35399    |\n",
            "|    policy_loss        | -0.0441  |\n",
            "|    std                | 0.0679   |\n",
            "|    value_loss         | 0.0071   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.49e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35500    |\n",
            "|    time_elapsed       | 2353     |\n",
            "|    total_timesteps    | 1136000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.03    |\n",
            "|    explained_variance | 0.959    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35499    |\n",
            "|    policy_loss        | 0.046    |\n",
            "|    std                | 0.0678   |\n",
            "|    value_loss         | 0.00106  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.49e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35600    |\n",
            "|    time_elapsed       | 2359     |\n",
            "|    total_timesteps    | 1139200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.06    |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35599    |\n",
            "|    policy_loss        | 0.254    |\n",
            "|    std                | 0.068    |\n",
            "|    value_loss         | 0.00628  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.5e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35700    |\n",
            "|    time_elapsed       | 2366     |\n",
            "|    total_timesteps    | 1142400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.53    |\n",
            "|    explained_variance | 0.99     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35699    |\n",
            "|    policy_loss        | 0.0458   |\n",
            "|    std                | 0.068    |\n",
            "|    value_loss         | 0.0087   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.48e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35800    |\n",
            "|    time_elapsed       | 2374     |\n",
            "|    total_timesteps    | 1145600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.52    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35799    |\n",
            "|    policy_loss        | 0.134    |\n",
            "|    std                | 0.0679   |\n",
            "|    value_loss         | 0.00468  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.46e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 35900    |\n",
            "|    time_elapsed       | 2381     |\n",
            "|    total_timesteps    | 1148800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.84    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35899    |\n",
            "|    policy_loss        | 0.0844   |\n",
            "|    std                | 0.0677   |\n",
            "|    value_loss         | 0.00173  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.46e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36000    |\n",
            "|    time_elapsed       | 2387     |\n",
            "|    total_timesteps    | 1152000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.41    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 35999    |\n",
            "|    policy_loss        | 0.0902   |\n",
            "|    std                | 0.0673   |\n",
            "|    value_loss         | 0.00175  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.45e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36100    |\n",
            "|    time_elapsed       | 2393     |\n",
            "|    total_timesteps    | 1155200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.85    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36099    |\n",
            "|    policy_loss        | -0.144   |\n",
            "|    std                | 0.0674   |\n",
            "|    value_loss         | 0.00622  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.41e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36200    |\n",
            "|    time_elapsed       | 2400     |\n",
            "|    total_timesteps    | 1158400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.9     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36199    |\n",
            "|    policy_loss        | 0.131    |\n",
            "|    std                | 0.0672   |\n",
            "|    value_loss         | 0.00524  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.39e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36300    |\n",
            "|    time_elapsed       | 2406     |\n",
            "|    total_timesteps    | 1161600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.01    |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36299    |\n",
            "|    policy_loss        | 0.0627   |\n",
            "|    std                | 0.0673   |\n",
            "|    value_loss         | 0.0303   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36400    |\n",
            "|    time_elapsed       | 2413     |\n",
            "|    total_timesteps    | 1164800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.71    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36399    |\n",
            "|    policy_loss        | 0.0324   |\n",
            "|    std                | 0.0669   |\n",
            "|    value_loss         | 0.00335  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36500    |\n",
            "|    time_elapsed       | 2419     |\n",
            "|    total_timesteps    | 1168000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.84    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36499    |\n",
            "|    policy_loss        | -0.103   |\n",
            "|    std                | 0.0671   |\n",
            "|    value_loss         | 0.00603  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.34e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36600    |\n",
            "|    time_elapsed       | 2426     |\n",
            "|    total_timesteps    | 1171200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.99    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36599    |\n",
            "|    policy_loss        | 0.124    |\n",
            "|    std                | 0.0673   |\n",
            "|    value_loss         | 0.00702  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.34e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36700    |\n",
            "|    time_elapsed       | 2432     |\n",
            "|    total_timesteps    | 1174400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.95    |\n",
            "|    explained_variance | 0.92     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36699    |\n",
            "|    policy_loss        | -0.453   |\n",
            "|    std                | 0.0672   |\n",
            "|    value_loss         | 0.0263   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36800    |\n",
            "|    time_elapsed       | 2438     |\n",
            "|    total_timesteps    | 1177600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.83    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36799    |\n",
            "|    policy_loss        | -0.0742  |\n",
            "|    std                | 0.0669   |\n",
            "|    value_loss         | 0.00237  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 36900    |\n",
            "|    time_elapsed       | 2445     |\n",
            "|    total_timesteps    | 1180800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.88    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36899    |\n",
            "|    policy_loss        | -0.0238  |\n",
            "|    std                | 0.0668   |\n",
            "|    value_loss         | 0.000575 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.34e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37000    |\n",
            "|    time_elapsed       | 2451     |\n",
            "|    total_timesteps    | 1184000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.88    |\n",
            "|    explained_variance | 0.614    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 36999    |\n",
            "|    policy_loss        | 0.0788   |\n",
            "|    std                | 0.0669   |\n",
            "|    value_loss         | 0.00189  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.34e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 482      |\n",
            "|    iterations         | 37100    |\n",
            "|    time_elapsed       | 2458     |\n",
            "|    total_timesteps    | 1187200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5       |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37099    |\n",
            "|    policy_loss        | 0.0101   |\n",
            "|    std                | 0.067    |\n",
            "|    value_loss         | 0.00279  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.33e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37200    |\n",
            "|    time_elapsed       | 2464     |\n",
            "|    total_timesteps    | 1190400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.19    |\n",
            "|    explained_variance | 0.575    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37199    |\n",
            "|    policy_loss        | 0.0808   |\n",
            "|    std                | 0.0674   |\n",
            "|    value_loss         | 0.000333 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37300    |\n",
            "|    time_elapsed       | 2470     |\n",
            "|    total_timesteps    | 1193600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.14    |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37299    |\n",
            "|    policy_loss        | 0.00256  |\n",
            "|    std                | 0.0671   |\n",
            "|    value_loss         | 0.00186  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37400    |\n",
            "|    time_elapsed       | 2477     |\n",
            "|    total_timesteps    | 1196800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.77    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37399    |\n",
            "|    policy_loss        | -0.00222 |\n",
            "|    std                | 0.0671   |\n",
            "|    value_loss         | 0.000298 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37500    |\n",
            "|    time_elapsed       | 2482     |\n",
            "|    total_timesteps    | 1200000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.93    |\n",
            "|    explained_variance | 0.545    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37499    |\n",
            "|    policy_loss        | 0.0815   |\n",
            "|    std                | 0.0669   |\n",
            "|    value_loss         | 0.00447  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.36e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37600    |\n",
            "|    time_elapsed       | 2490     |\n",
            "|    total_timesteps    | 1203200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.81    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37599    |\n",
            "|    policy_loss        | 0.0529   |\n",
            "|    std                | 0.0667   |\n",
            "|    value_loss         | 0.00206  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37700    |\n",
            "|    time_elapsed       | 2496     |\n",
            "|    total_timesteps    | 1206400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.85    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37699    |\n",
            "|    policy_loss        | 0.00415  |\n",
            "|    std                | 0.0663   |\n",
            "|    value_loss         | 0.00143  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37800    |\n",
            "|    time_elapsed       | 2503     |\n",
            "|    total_timesteps    | 1209600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.75    |\n",
            "|    explained_variance | 0.876    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37799    |\n",
            "|    policy_loss        | 0.205    |\n",
            "|    std                | 0.0665   |\n",
            "|    value_loss         | 0.00603  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 37900    |\n",
            "|    time_elapsed       | 2509     |\n",
            "|    total_timesteps    | 1212800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.75    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37899    |\n",
            "|    policy_loss        | 0.131    |\n",
            "|    std                | 0.0662   |\n",
            "|    value_loss         | 0.00141  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.36e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38000    |\n",
            "|    time_elapsed       | 2515     |\n",
            "|    total_timesteps    | 1216000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.06    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 37999    |\n",
            "|    policy_loss        | 0.0422   |\n",
            "|    std                | 0.066    |\n",
            "|    value_loss         | 0.00111  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.36e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38100    |\n",
            "|    time_elapsed       | 2522     |\n",
            "|    total_timesteps    | 1219200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.43    |\n",
            "|    explained_variance | 0.98     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38099    |\n",
            "|    policy_loss        | 0.00407  |\n",
            "|    std                | 0.0661   |\n",
            "|    value_loss         | 0.000983 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38200    |\n",
            "|    time_elapsed       | 2527     |\n",
            "|    total_timesteps    | 1222400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.93    |\n",
            "|    explained_variance | 0.945    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38199    |\n",
            "|    policy_loss        | -0.0625  |\n",
            "|    std                | 0.0664   |\n",
            "|    value_loss         | 0.000898 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38300    |\n",
            "|    time_elapsed       | 2535     |\n",
            "|    total_timesteps    | 1225600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.15    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38299    |\n",
            "|    policy_loss        | -0.0745  |\n",
            "|    std                | 0.0663   |\n",
            "|    value_loss         | 0.00303  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 992      |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38400    |\n",
            "|    time_elapsed       | 2540     |\n",
            "|    total_timesteps    | 1228800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.99    |\n",
            "|    explained_variance | 0.544    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38399    |\n",
            "|    policy_loss        | -0.57    |\n",
            "|    std                | 0.0667   |\n",
            "|    value_loss         | 0.105    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.4e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38500    |\n",
            "|    time_elapsed       | 2547     |\n",
            "|    total_timesteps    | 1232000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.7     |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38499    |\n",
            "|    policy_loss        | -0.0126  |\n",
            "|    std                | 0.0667   |\n",
            "|    value_loss         | 0.000525 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.41e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38600    |\n",
            "|    time_elapsed       | 2553     |\n",
            "|    total_timesteps    | 1235200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.83    |\n",
            "|    explained_variance | 0.986    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38599    |\n",
            "|    policy_loss        | 0.126    |\n",
            "|    std                | 0.0667   |\n",
            "|    value_loss         | 0.00253  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.42e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38700    |\n",
            "|    time_elapsed       | 2559     |\n",
            "|    total_timesteps    | 1238400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.99    |\n",
            "|    explained_variance | 0.98     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38699    |\n",
            "|    policy_loss        | 0.169    |\n",
            "|    std                | 0.0665   |\n",
            "|    value_loss         | 0.00546  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.42e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38800    |\n",
            "|    time_elapsed       | 2566     |\n",
            "|    total_timesteps    | 1241600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.11    |\n",
            "|    explained_variance | 0.813    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38799    |\n",
            "|    policy_loss        | -0.0191  |\n",
            "|    std                | 0.0664   |\n",
            "|    value_loss         | 0.000921 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.45e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 38900    |\n",
            "|    time_elapsed       | 2572     |\n",
            "|    total_timesteps    | 1244800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.71    |\n",
            "|    explained_variance | 0.99     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38899    |\n",
            "|    policy_loss        | -0.0997  |\n",
            "|    std                | 0.066    |\n",
            "|    value_loss         | 0.00285  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.46e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39000    |\n",
            "|    time_elapsed       | 2580     |\n",
            "|    total_timesteps    | 1248000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.87    |\n",
            "|    explained_variance | 0.982    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 38999    |\n",
            "|    policy_loss        | -0.0736  |\n",
            "|    std                | 0.0663   |\n",
            "|    value_loss         | 0.000517 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.48e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39100    |\n",
            "|    time_elapsed       | 2587     |\n",
            "|    total_timesteps    | 1251200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.08    |\n",
            "|    explained_variance | 0.908    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39099    |\n",
            "|    policy_loss        | 0.0162   |\n",
            "|    std                | 0.0661   |\n",
            "|    value_loss         | 0.00205  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.5e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39200    |\n",
            "|    time_elapsed       | 2594     |\n",
            "|    total_timesteps    | 1254400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -5.02    |\n",
            "|    explained_variance | 0.984    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39199    |\n",
            "|    policy_loss        | 0.00773  |\n",
            "|    std                | 0.0659   |\n",
            "|    value_loss         | 0.00112  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.53e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39300    |\n",
            "|    time_elapsed       | 2600     |\n",
            "|    total_timesteps    | 1257600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.55    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39299    |\n",
            "|    policy_loss        | 0.0219   |\n",
            "|    std                | 0.066    |\n",
            "|    value_loss         | 0.00102  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.57e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39400    |\n",
            "|    time_elapsed       | 2606     |\n",
            "|    total_timesteps    | 1260800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.81    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39399    |\n",
            "|    policy_loss        | 0.139    |\n",
            "|    std                | 0.0659   |\n",
            "|    value_loss         | 0.00226  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.59e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39500    |\n",
            "|    time_elapsed       | 2613     |\n",
            "|    total_timesteps    | 1264000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.99    |\n",
            "|    explained_variance | 0.952    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39499    |\n",
            "|    policy_loss        | 0.129    |\n",
            "|    std                | 0.0659   |\n",
            "|    value_loss         | 0.00223  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.6e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39600    |\n",
            "|    time_elapsed       | 2619     |\n",
            "|    total_timesteps    | 1267200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.91    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39599    |\n",
            "|    policy_loss        | 0.131    |\n",
            "|    std                | 0.0659   |\n",
            "|    value_loss         | 0.00107  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.64e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39700    |\n",
            "|    time_elapsed       | 2626     |\n",
            "|    total_timesteps    | 1270400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.75    |\n",
            "|    explained_variance | 0.531    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39699    |\n",
            "|    policy_loss        | -0.0197  |\n",
            "|    std                | 0.0658   |\n",
            "|    value_loss         | 0.000686 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39800    |\n",
            "|    time_elapsed       | 2632     |\n",
            "|    total_timesteps    | 1273600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.88    |\n",
            "|    explained_variance | 0.782    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39799    |\n",
            "|    policy_loss        | 0.00047  |\n",
            "|    std                | 0.0655   |\n",
            "|    value_loss         | 0.00058  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 39900    |\n",
            "|    time_elapsed       | 2638     |\n",
            "|    total_timesteps    | 1276800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.97    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39899    |\n",
            "|    policy_loss        | 0.0629   |\n",
            "|    std                | 0.0652   |\n",
            "|    value_loss         | 0.00122  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.69e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 40000    |\n",
            "|    time_elapsed       | 2645     |\n",
            "|    total_timesteps    | 1280000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.84    |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 39999    |\n",
            "|    policy_loss        | -0.16    |\n",
            "|    std                | 0.065    |\n",
            "|    value_loss         | 0.00138  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.69e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 40100    |\n",
            "|    time_elapsed       | 2650     |\n",
            "|    total_timesteps    | 1283200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.76    |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40099    |\n",
            "|    policy_loss        | 0.0481   |\n",
            "|    std                | 0.0651   |\n",
            "|    value_loss         | 0.00202  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.71e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 483      |\n",
            "|    iterations         | 40200    |\n",
            "|    time_elapsed       | 2658     |\n",
            "|    total_timesteps    | 1286400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.92    |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40199    |\n",
            "|    policy_loss        | 0.0499   |\n",
            "|    std                | 0.0651   |\n",
            "|    value_loss         | 0.000306 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.72e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 40300    |\n",
            "|    time_elapsed       | 2663     |\n",
            "|    total_timesteps    | 1289600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.77    |\n",
            "|    explained_variance | 0.964    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40299    |\n",
            "|    policy_loss        | 0.102    |\n",
            "|    std                | 0.0652   |\n",
            "|    value_loss         | 0.00277  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 40400    |\n",
            "|    time_elapsed       | 2671     |\n",
            "|    total_timesteps    | 1292800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.66    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40399    |\n",
            "|    policy_loss        | 0.0246   |\n",
            "|    std                | 0.065    |\n",
            "|    value_loss         | 0.00106  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 40500    |\n",
            "|    time_elapsed       | 2676     |\n",
            "|    total_timesteps    | 1296000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.77    |\n",
            "|    explained_variance | 0.969    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40499    |\n",
            "|    policy_loss        | 0.21     |\n",
            "|    std                | 0.0655   |\n",
            "|    value_loss         | 0.00425  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.74e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 40600    |\n",
            "|    time_elapsed       | 2683     |\n",
            "|    total_timesteps    | 1299200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.63    |\n",
            "|    explained_variance | 0.953    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40599    |\n",
            "|    policy_loss        | -0.148   |\n",
            "|    std                | 0.0654   |\n",
            "|    value_loss         | 0.00593  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 40700    |\n",
            "|    time_elapsed       | 2689     |\n",
            "|    total_timesteps    | 1302400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.85    |\n",
            "|    explained_variance | 0.601    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40699    |\n",
            "|    policy_loss        | -0.287   |\n",
            "|    std                | 0.0653   |\n",
            "|    value_loss         | 0.0177   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.75e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 40800    |\n",
            "|    time_elapsed       | 2695     |\n",
            "|    total_timesteps    | 1305600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.85    |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40799    |\n",
            "|    policy_loss        | 0.097    |\n",
            "|    std                | 0.0652   |\n",
            "|    value_loss         | 0.0023   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.75e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 40900    |\n",
            "|    time_elapsed       | 2702     |\n",
            "|    total_timesteps    | 1308800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.9     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40899    |\n",
            "|    policy_loss        | 0.0589   |\n",
            "|    std                | 0.0652   |\n",
            "|    value_loss         | 0.00115  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41000    |\n",
            "|    time_elapsed       | 2708     |\n",
            "|    total_timesteps    | 1312000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.98    |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 40999    |\n",
            "|    policy_loss        | -0.0674  |\n",
            "|    std                | 0.0651   |\n",
            "|    value_loss         | 0.000824 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41100    |\n",
            "|    time_elapsed       | 2715     |\n",
            "|    total_timesteps    | 1315200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.67    |\n",
            "|    explained_variance | 0.908    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41099    |\n",
            "|    policy_loss        | 0.0907   |\n",
            "|    std                | 0.0656   |\n",
            "|    value_loss         | 0.0132   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.77e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41200    |\n",
            "|    time_elapsed       | 2721     |\n",
            "|    total_timesteps    | 1318400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.75    |\n",
            "|    explained_variance | 0.771    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41199    |\n",
            "|    policy_loss        | -0.0864  |\n",
            "|    std                | 0.0654   |\n",
            "|    value_loss         | 0.00135  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.77e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41300    |\n",
            "|    time_elapsed       | 2727     |\n",
            "|    total_timesteps    | 1321600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.78    |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41299    |\n",
            "|    policy_loss        | -0.219   |\n",
            "|    std                | 0.0655   |\n",
            "|    value_loss         | 0.0108   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41400    |\n",
            "|    time_elapsed       | 2734     |\n",
            "|    total_timesteps    | 1324800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.95    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41399    |\n",
            "|    policy_loss        | -0.0449  |\n",
            "|    std                | 0.0652   |\n",
            "|    value_loss         | 0.000885 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41500    |\n",
            "|    time_elapsed       | 2740     |\n",
            "|    total_timesteps    | 1328000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.82    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41499    |\n",
            "|    policy_loss        | 0.0644   |\n",
            "|    std                | 0.0652   |\n",
            "|    value_loss         | 0.00489  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41600    |\n",
            "|    time_elapsed       | 2747     |\n",
            "|    total_timesteps    | 1331200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.7     |\n",
            "|    explained_variance | 0.951    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41599    |\n",
            "|    policy_loss        | 0.0733   |\n",
            "|    std                | 0.0649   |\n",
            "|    value_loss         | 0.00663  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41700    |\n",
            "|    time_elapsed       | 2753     |\n",
            "|    total_timesteps    | 1334400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.76    |\n",
            "|    explained_variance | 0.729    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41699    |\n",
            "|    policy_loss        | -0.0646  |\n",
            "|    std                | 0.0646   |\n",
            "|    value_loss         | 0.000802 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41800    |\n",
            "|    time_elapsed       | 2759     |\n",
            "|    total_timesteps    | 1337600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.48    |\n",
            "|    explained_variance | 0.975    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41799    |\n",
            "|    policy_loss        | -0.0116  |\n",
            "|    std                | 0.0643   |\n",
            "|    value_loss         | 0.00221  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 41900    |\n",
            "|    time_elapsed       | 2766     |\n",
            "|    total_timesteps    | 1340800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.85    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41899    |\n",
            "|    policy_loss        | 0.0999   |\n",
            "|    std                | 0.0642   |\n",
            "|    value_loss         | 0.000937 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42000    |\n",
            "|    time_elapsed       | 2771     |\n",
            "|    total_timesteps    | 1344000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.64    |\n",
            "|    explained_variance | 0.961    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 41999    |\n",
            "|    policy_loss        | 0.0205   |\n",
            "|    std                | 0.064    |\n",
            "|    value_loss         | 0.00107  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42100    |\n",
            "|    time_elapsed       | 2778     |\n",
            "|    total_timesteps    | 1347200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.76    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42099    |\n",
            "|    policy_loss        | -0.00438 |\n",
            "|    std                | 0.0639   |\n",
            "|    value_loss         | 0.00104  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42200    |\n",
            "|    time_elapsed       | 2784     |\n",
            "|    total_timesteps    | 1350400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.89    |\n",
            "|    explained_variance | 0.976    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42199    |\n",
            "|    policy_loss        | 0.0972   |\n",
            "|    std                | 0.0641   |\n",
            "|    value_loss         | 0.00213  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42300    |\n",
            "|    time_elapsed       | 2791     |\n",
            "|    total_timesteps    | 1353600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.77    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42299    |\n",
            "|    policy_loss        | -0.165   |\n",
            "|    std                | 0.0639   |\n",
            "|    value_loss         | 0.00508  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.74e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42400    |\n",
            "|    time_elapsed       | 2798     |\n",
            "|    total_timesteps    | 1356800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.64    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42399    |\n",
            "|    policy_loss        | -0.0723  |\n",
            "|    std                | 0.0639   |\n",
            "|    value_loss         | 0.00274  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42500    |\n",
            "|    time_elapsed       | 2805     |\n",
            "|    total_timesteps    | 1360000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.21    |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42499    |\n",
            "|    policy_loss        | -0.125   |\n",
            "|    std                | 0.0637   |\n",
            "|    value_loss         | 0.00926  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.72e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42600    |\n",
            "|    time_elapsed       | 2812     |\n",
            "|    total_timesteps    | 1363200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.78    |\n",
            "|    explained_variance | 0.699    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42599    |\n",
            "|    policy_loss        | -0.0652  |\n",
            "|    std                | 0.0638   |\n",
            "|    value_loss         | 0.00982  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.7e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42700    |\n",
            "|    time_elapsed       | 2818     |\n",
            "|    total_timesteps    | 1366400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.61    |\n",
            "|    explained_variance | 0.848    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42699    |\n",
            "|    policy_loss        | -0.00936 |\n",
            "|    std                | 0.0637   |\n",
            "|    value_loss         | 0.00331  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.7e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 42800    |\n",
            "|    time_elapsed       | 2825     |\n",
            "|    total_timesteps    | 1369600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.3     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42799    |\n",
            "|    policy_loss        | 0.0245   |\n",
            "|    std                | 0.0636   |\n",
            "|    value_loss         | 0.00105  |\n",
            "------------------------------------\n",
            "-------------------------------------\n",
            "| rollout/              |           |\n",
            "|    ep_len_mean        | 1e+03     |\n",
            "|    ep_rew_mean        | 1.68e+03  |\n",
            "| time/                 |           |\n",
            "|    fps                | 484       |\n",
            "|    iterations         | 42900     |\n",
            "|    time_elapsed       | 2831      |\n",
            "|    total_timesteps    | 1372800   |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -4.42     |\n",
            "|    explained_variance | 0.985     |\n",
            "|    learning_rate      | 0.00096   |\n",
            "|    n_updates          | 42899     |\n",
            "|    policy_loss        | -0.000183 |\n",
            "|    std                | 0.0634    |\n",
            "|    value_loss         | 0.00256   |\n",
            "-------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 43000    |\n",
            "|    time_elapsed       | 2838     |\n",
            "|    total_timesteps    | 1376000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.55    |\n",
            "|    explained_variance | 0.934    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 42999    |\n",
            "|    policy_loss        | -0.0156  |\n",
            "|    std                | 0.0631   |\n",
            "|    value_loss         | 0.00139  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 43100    |\n",
            "|    time_elapsed       | 2844     |\n",
            "|    total_timesteps    | 1379200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.74    |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43099    |\n",
            "|    policy_loss        | -0.045   |\n",
            "|    std                | 0.0629   |\n",
            "|    value_loss         | 0.00216  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 43200    |\n",
            "|    time_elapsed       | 2850     |\n",
            "|    total_timesteps    | 1382400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.53    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43199    |\n",
            "|    policy_loss        | -0.0431  |\n",
            "|    std                | 0.0623   |\n",
            "|    value_loss         | 0.00179  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 484      |\n",
            "|    iterations         | 43300    |\n",
            "|    time_elapsed       | 2857     |\n",
            "|    total_timesteps    | 1385600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.62    |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43299    |\n",
            "|    policy_loss        | -0.0217  |\n",
            "|    std                | 0.0623   |\n",
            "|    value_loss         | 0.000583 |\n",
            "------------------------------------\n",
            "-------------------------------------\n",
            "| rollout/              |           |\n",
            "|    ep_len_mean        | 1e+03     |\n",
            "|    ep_rew_mean        | 1.67e+03  |\n",
            "| time/                 |           |\n",
            "|    fps                | 485       |\n",
            "|    iterations         | 43400     |\n",
            "|    time_elapsed       | 2862      |\n",
            "|    total_timesteps    | 1388800   |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -4.57     |\n",
            "|    explained_variance | 0.993     |\n",
            "|    learning_rate      | 0.00096   |\n",
            "|    n_updates          | 43399     |\n",
            "|    policy_loss        | -4.42e-05 |\n",
            "|    std                | 0.0625    |\n",
            "|    value_loss         | 0.00175   |\n",
            "-------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 43500    |\n",
            "|    time_elapsed       | 2869     |\n",
            "|    total_timesteps    | 1392000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.53    |\n",
            "|    explained_variance | 0.981    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43499    |\n",
            "|    policy_loss        | -0.0388  |\n",
            "|    std                | 0.0626   |\n",
            "|    value_loss         | 0.00328  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.68e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 43600    |\n",
            "|    time_elapsed       | 2875     |\n",
            "|    total_timesteps    | 1395200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.59    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43599    |\n",
            "|    policy_loss        | -0.0147  |\n",
            "|    std                | 0.0624   |\n",
            "|    value_loss         | 0.00335  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 43700    |\n",
            "|    time_elapsed       | 2882     |\n",
            "|    total_timesteps    | 1398400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.39    |\n",
            "|    explained_variance | 0.925    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43699    |\n",
            "|    policy_loss        | 0.19     |\n",
            "|    std                | 0.0623   |\n",
            "|    value_loss         | 0.00367  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 43800    |\n",
            "|    time_elapsed       | 2888     |\n",
            "|    total_timesteps    | 1401600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.41    |\n",
            "|    explained_variance | 0.922    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43799    |\n",
            "|    policy_loss        | -0.062   |\n",
            "|    std                | 0.0625   |\n",
            "|    value_loss         | 0.00158  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 43900    |\n",
            "|    time_elapsed       | 2894     |\n",
            "|    total_timesteps    | 1404800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.71    |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43899    |\n",
            "|    policy_loss        | 0.0669   |\n",
            "|    std                | 0.0624   |\n",
            "|    value_loss         | 0.00141  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44000    |\n",
            "|    time_elapsed       | 2901     |\n",
            "|    total_timesteps    | 1408000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.24    |\n",
            "|    explained_variance | 0.968    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 43999    |\n",
            "|    policy_loss        | -0.0544  |\n",
            "|    std                | 0.0625   |\n",
            "|    value_loss         | 0.0139   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44100    |\n",
            "|    time_elapsed       | 2906     |\n",
            "|    total_timesteps    | 1411200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.61    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44099    |\n",
            "|    policy_loss        | 0.305    |\n",
            "|    std                | 0.0626   |\n",
            "|    value_loss         | 0.00711  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44200    |\n",
            "|    time_elapsed       | 2914     |\n",
            "|    total_timesteps    | 1414400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.51    |\n",
            "|    explained_variance | 0.899    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44199    |\n",
            "|    policy_loss        | -0.0558  |\n",
            "|    std                | 0.0625   |\n",
            "|    value_loss         | 0.000444 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44300    |\n",
            "|    time_elapsed       | 2919     |\n",
            "|    total_timesteps    | 1417600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.52    |\n",
            "|    explained_variance | 0.94     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44299    |\n",
            "|    policy_loss        | -0.14    |\n",
            "|    std                | 0.0626   |\n",
            "|    value_loss         | 0.0063   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44400    |\n",
            "|    time_elapsed       | 2925     |\n",
            "|    total_timesteps    | 1420800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.45    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44399    |\n",
            "|    policy_loss        | -0.0191  |\n",
            "|    std                | 0.0626   |\n",
            "|    value_loss         | 0.00233  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.63e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44500    |\n",
            "|    time_elapsed       | 2932     |\n",
            "|    total_timesteps    | 1424000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.48    |\n",
            "|    explained_variance | 0.976    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44499    |\n",
            "|    policy_loss        | 0.00958  |\n",
            "|    std                | 0.0625   |\n",
            "|    value_loss         | 0.00357  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.64e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44600    |\n",
            "|    time_elapsed       | 2938     |\n",
            "|    total_timesteps    | 1427200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.76    |\n",
            "|    explained_variance | 0.972    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44599    |\n",
            "|    policy_loss        | -0.175   |\n",
            "|    std                | 0.0628   |\n",
            "|    value_loss         | 0.00174  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.64e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44700    |\n",
            "|    time_elapsed       | 2945     |\n",
            "|    total_timesteps    | 1430400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.51    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44699    |\n",
            "|    policy_loss        | 0.0215   |\n",
            "|    std                | 0.063    |\n",
            "|    value_loss         | 0.00381  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.64e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44800    |\n",
            "|    time_elapsed       | 2951     |\n",
            "|    total_timesteps    | 1433600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.24    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44799    |\n",
            "|    policy_loss        | 0.313    |\n",
            "|    std                | 0.0628   |\n",
            "|    value_loss         | 0.0186   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.64e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 44900    |\n",
            "|    time_elapsed       | 2957     |\n",
            "|    total_timesteps    | 1436800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.52    |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44899    |\n",
            "|    policy_loss        | 0.212    |\n",
            "|    std                | 0.0627   |\n",
            "|    value_loss         | 0.0026   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.64e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 45000    |\n",
            "|    time_elapsed       | 2964     |\n",
            "|    total_timesteps    | 1440000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.41    |\n",
            "|    explained_variance | 0.974    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 44999    |\n",
            "|    policy_loss        | 0.00726  |\n",
            "|    std                | 0.0622   |\n",
            "|    value_loss         | 0.00229  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 45100    |\n",
            "|    time_elapsed       | 2969     |\n",
            "|    total_timesteps    | 1443200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.42    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45099    |\n",
            "|    policy_loss        | -0.0575  |\n",
            "|    std                | 0.0622   |\n",
            "|    value_loss         | 0.00113  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 45200    |\n",
            "|    time_elapsed       | 2977     |\n",
            "|    total_timesteps    | 1446400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.51    |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45199    |\n",
            "|    policy_loss        | 0.0571   |\n",
            "|    std                | 0.0622   |\n",
            "|    value_loss         | 0.00296  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 45300    |\n",
            "|    time_elapsed       | 2982     |\n",
            "|    total_timesteps    | 1449600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.31    |\n",
            "|    explained_variance | 0.98     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45299    |\n",
            "|    policy_loss        | 0.0624   |\n",
            "|    std                | 0.0623   |\n",
            "|    value_loss         | 0.000579 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 45400    |\n",
            "|    time_elapsed       | 2989     |\n",
            "|    total_timesteps    | 1452800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.51    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45399    |\n",
            "|    policy_loss        | 0.00855  |\n",
            "|    std                | 0.062    |\n",
            "|    value_loss         | 0.00117  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 45500    |\n",
            "|    time_elapsed       | 2995     |\n",
            "|    total_timesteps    | 1456000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.31    |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45499    |\n",
            "|    policy_loss        | -0.0101  |\n",
            "|    std                | 0.0618   |\n",
            "|    value_loss         | 0.000646 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 45600    |\n",
            "|    time_elapsed       | 3001     |\n",
            "|    total_timesteps    | 1459200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.42    |\n",
            "|    explained_variance | 0.963    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45599    |\n",
            "|    policy_loss        | 0.0526   |\n",
            "|    std                | 0.0618   |\n",
            "|    value_loss         | 0.000758 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.7e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 45700    |\n",
            "|    time_elapsed       | 3008     |\n",
            "|    total_timesteps    | 1462400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.08    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45699    |\n",
            "|    policy_loss        | 0.0127   |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.0027   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.71e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 45800    |\n",
            "|    time_elapsed       | 3016     |\n",
            "|    total_timesteps    | 1465600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.14    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45799    |\n",
            "|    policy_loss        | 0.0366   |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.000914 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.71e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 45900    |\n",
            "|    time_elapsed       | 3023     |\n",
            "|    total_timesteps    | 1468800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.87    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45899    |\n",
            "|    policy_loss        | 0.085    |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.00115  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.7e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 46000    |\n",
            "|    time_elapsed       | 3029     |\n",
            "|    total_timesteps    | 1472000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.44    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 45999    |\n",
            "|    policy_loss        | -0.0962  |\n",
            "|    std                | 0.0614   |\n",
            "|    value_loss         | 0.00208  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.69e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 46100    |\n",
            "|    time_elapsed       | 3036     |\n",
            "|    total_timesteps    | 1475200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.31    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46099    |\n",
            "|    policy_loss        | 0.0298   |\n",
            "|    std                | 0.0614   |\n",
            "|    value_loss         | 0.00358  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 46200    |\n",
            "|    time_elapsed       | 3042     |\n",
            "|    total_timesteps    | 1478400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.42    |\n",
            "|    explained_variance | 0.972    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46199    |\n",
            "|    policy_loss        | 0.0335   |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.000751 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 46300    |\n",
            "|    time_elapsed       | 3048     |\n",
            "|    total_timesteps    | 1481600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.51    |\n",
            "|    explained_variance | 0.975    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46299    |\n",
            "|    policy_loss        | 0.106    |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.0229   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.63e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 485      |\n",
            "|    iterations         | 46400    |\n",
            "|    time_elapsed       | 3055     |\n",
            "|    total_timesteps    | 1484800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.42    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46399    |\n",
            "|    policy_loss        | -0.0459  |\n",
            "|    std                | 0.0611   |\n",
            "|    value_loss         | 0.000933 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.62e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 46500    |\n",
            "|    time_elapsed       | 3060     |\n",
            "|    total_timesteps    | 1488000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.49    |\n",
            "|    explained_variance | 0.979    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46499    |\n",
            "|    policy_loss        | 0.0538   |\n",
            "|    std                | 0.0613   |\n",
            "|    value_loss         | 0.00241  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.61e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 46600    |\n",
            "|    time_elapsed       | 3068     |\n",
            "|    total_timesteps    | 1491200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.89    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46599    |\n",
            "|    policy_loss        | -0.0898  |\n",
            "|    std                | 0.0613   |\n",
            "|    value_loss         | 0.0033   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.58e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 46700    |\n",
            "|    time_elapsed       | 3073     |\n",
            "|    total_timesteps    | 1494400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.4     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46699    |\n",
            "|    policy_loss        | 0.15     |\n",
            "|    std                | 0.0614   |\n",
            "|    value_loss         | 0.0022   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.58e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 46800    |\n",
            "|    time_elapsed       | 3080     |\n",
            "|    total_timesteps    | 1497600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.15    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46799    |\n",
            "|    policy_loss        | 0.137    |\n",
            "|    std                | 0.0614   |\n",
            "|    value_loss         | 0.00226  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.57e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 46900    |\n",
            "|    time_elapsed       | 3086     |\n",
            "|    total_timesteps    | 1500800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.41    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46899    |\n",
            "|    policy_loss        | 0.0677   |\n",
            "|    std                | 0.0614   |\n",
            "|    value_loss         | 0.00479  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.57e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47000    |\n",
            "|    time_elapsed       | 3092     |\n",
            "|    total_timesteps    | 1504000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.33    |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 46999    |\n",
            "|    policy_loss        | -0.279   |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.0135   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.56e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47100    |\n",
            "|    time_elapsed       | 3099     |\n",
            "|    total_timesteps    | 1507200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.18    |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47099    |\n",
            "|    policy_loss        | -0.148   |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.00428  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.56e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47200    |\n",
            "|    time_elapsed       | 3105     |\n",
            "|    total_timesteps    | 1510400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.23    |\n",
            "|    explained_variance | 0.964    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47199    |\n",
            "|    policy_loss        | -0.0931  |\n",
            "|    std                | 0.0611   |\n",
            "|    value_loss         | 0.00152  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.57e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47300    |\n",
            "|    time_elapsed       | 3113     |\n",
            "|    total_timesteps    | 1513600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.41    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47299    |\n",
            "|    policy_loss        | -0.0731  |\n",
            "|    std                | 0.0614   |\n",
            "|    value_loss         | 0.00105  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.56e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47400    |\n",
            "|    time_elapsed       | 3118     |\n",
            "|    total_timesteps    | 1516800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.36    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47399    |\n",
            "|    policy_loss        | 0.0833   |\n",
            "|    std                | 0.0613   |\n",
            "|    value_loss         | 0.00119  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.56e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47500    |\n",
            "|    time_elapsed       | 3125     |\n",
            "|    total_timesteps    | 1520000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.5     |\n",
            "|    explained_variance | 0.944    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47499    |\n",
            "|    policy_loss        | -0.0165  |\n",
            "|    std                | 0.0613   |\n",
            "|    value_loss         | 0.000232 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.57e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47600    |\n",
            "|    time_elapsed       | 3131     |\n",
            "|    total_timesteps    | 1523200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.09    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47599    |\n",
            "|    policy_loss        | -0.0225  |\n",
            "|    std                | 0.0615   |\n",
            "|    value_loss         | 0.00139  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.57e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47700    |\n",
            "|    time_elapsed       | 3137     |\n",
            "|    total_timesteps    | 1526400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.1     |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47699    |\n",
            "|    policy_loss        | -0.219   |\n",
            "|    std                | 0.0615   |\n",
            "|    value_loss         | 0.00652  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.56e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47800    |\n",
            "|    time_elapsed       | 3144     |\n",
            "|    total_timesteps    | 1529600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.98    |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47799    |\n",
            "|    policy_loss        | -0.0082  |\n",
            "|    std                | 0.0614   |\n",
            "|    value_loss         | 0.00057  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.54e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 47900    |\n",
            "|    time_elapsed       | 3150     |\n",
            "|    total_timesteps    | 1532800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.13    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47899    |\n",
            "|    policy_loss        | 0.0206   |\n",
            "|    std                | 0.0614   |\n",
            "|    value_loss         | 0.000822 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.53e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48000    |\n",
            "|    time_elapsed       | 3157     |\n",
            "|    total_timesteps    | 1536000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.03    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 47999    |\n",
            "|    policy_loss        | 0.0407   |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.000547 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.52e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48100    |\n",
            "|    time_elapsed       | 3163     |\n",
            "|    total_timesteps    | 1539200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.81    |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48099    |\n",
            "|    policy_loss        | 0.0409   |\n",
            "|    std                | 0.061    |\n",
            "|    value_loss         | 0.000695 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.51e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48200    |\n",
            "|    time_elapsed       | 3169     |\n",
            "|    total_timesteps    | 1542400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.22    |\n",
            "|    explained_variance | 0.973    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48199    |\n",
            "|    policy_loss        | -0.0974  |\n",
            "|    std                | 0.0611   |\n",
            "|    value_loss         | 0.00226  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.5e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48300    |\n",
            "|    time_elapsed       | 3176     |\n",
            "|    total_timesteps    | 1545600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.08    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48299    |\n",
            "|    policy_loss        | 0.0546   |\n",
            "|    std                | 0.0609   |\n",
            "|    value_loss         | 0.00188  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.5e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48400    |\n",
            "|    time_elapsed       | 3181     |\n",
            "|    total_timesteps    | 1548800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.01    |\n",
            "|    explained_variance | 0.986    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48399    |\n",
            "|    policy_loss        | 0.0833   |\n",
            "|    std                | 0.0611   |\n",
            "|    value_loss         | 0.00206  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.49e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48500    |\n",
            "|    time_elapsed       | 3189     |\n",
            "|    total_timesteps    | 1552000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.65    |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48499    |\n",
            "|    policy_loss        | -0.0133  |\n",
            "|    std                | 0.0613   |\n",
            "|    value_loss         | 0.00373  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.48e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48600    |\n",
            "|    time_elapsed       | 3194     |\n",
            "|    total_timesteps    | 1555200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.62    |\n",
            "|    explained_variance | 0.969    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48599    |\n",
            "|    policy_loss        | 0.0732   |\n",
            "|    std                | 0.0615   |\n",
            "|    value_loss         | 0.00258  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.45e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48700    |\n",
            "|    time_elapsed       | 3201     |\n",
            "|    total_timesteps    | 1558400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -2.94    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48699    |\n",
            "|    policy_loss        | 0.00716  |\n",
            "|    std                | 0.0616   |\n",
            "|    value_loss         | 0.00032  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.42e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48800    |\n",
            "|    time_elapsed       | 3207     |\n",
            "|    total_timesteps    | 1561600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.52    |\n",
            "|    explained_variance | 0.981    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48799    |\n",
            "|    policy_loss        | -0.0282  |\n",
            "|    std                | 0.0617   |\n",
            "|    value_loss         | 0.000663 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.38e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 48900    |\n",
            "|    time_elapsed       | 3213     |\n",
            "|    total_timesteps    | 1564800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -2.23    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48899    |\n",
            "|    policy_loss        | -0.0546  |\n",
            "|    std                | 0.0618   |\n",
            "|    value_loss         | 0.000957 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49000    |\n",
            "|    time_elapsed       | 3221     |\n",
            "|    total_timesteps    | 1568000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.17    |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 48999    |\n",
            "|    policy_loss        | 0.0696   |\n",
            "|    std                | 0.0617   |\n",
            "|    value_loss         | 0.00161  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49100    |\n",
            "|    time_elapsed       | 3227     |\n",
            "|    total_timesteps    | 1571200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.78    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49099    |\n",
            "|    policy_loss        | 0.0261   |\n",
            "|    std                | 0.0617   |\n",
            "|    value_loss         | 0.000428 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49200    |\n",
            "|    time_elapsed       | 3236     |\n",
            "|    total_timesteps    | 1574400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.45    |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49199    |\n",
            "|    policy_loss        | 0.106    |\n",
            "|    std                | 0.0615   |\n",
            "|    value_loss         | 0.00278  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49300    |\n",
            "|    time_elapsed       | 3241     |\n",
            "|    total_timesteps    | 1577600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.33    |\n",
            "|    explained_variance | 0.986    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49299    |\n",
            "|    policy_loss        | -0.18    |\n",
            "|    std                | 0.0618   |\n",
            "|    value_loss         | 0.00525  |\n",
            "------------------------------------\n",
            "-------------------------------------\n",
            "| rollout/              |           |\n",
            "|    ep_len_mean        | 1e+03     |\n",
            "|    ep_rew_mean        | 1.35e+03  |\n",
            "| time/                 |           |\n",
            "|    fps                | 486       |\n",
            "|    iterations         | 49400     |\n",
            "|    time_elapsed       | 3248      |\n",
            "|    total_timesteps    | 1580800   |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -4.08     |\n",
            "|    explained_variance | 0.972     |\n",
            "|    learning_rate      | 0.00096   |\n",
            "|    n_updates          | 49399     |\n",
            "|    policy_loss        | -0.000211 |\n",
            "|    std                | 0.0615    |\n",
            "|    value_loss         | 0.000898  |\n",
            "-------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49500    |\n",
            "|    time_elapsed       | 3254     |\n",
            "|    total_timesteps    | 1584000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.93    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49499    |\n",
            "|    policy_loss        | 0.0192   |\n",
            "|    std                | 0.0619   |\n",
            "|    value_loss         | 0.00105  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.35e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49600    |\n",
            "|    time_elapsed       | 3260     |\n",
            "|    total_timesteps    | 1587200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.45    |\n",
            "|    explained_variance | 0.77     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49599    |\n",
            "|    policy_loss        | 0.0158   |\n",
            "|    std                | 0.0619   |\n",
            "|    value_loss         | 0.000483 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.36e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49700    |\n",
            "|    time_elapsed       | 3267     |\n",
            "|    total_timesteps    | 1590400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.25    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49699    |\n",
            "|    policy_loss        | 0.0114   |\n",
            "|    std                | 0.062    |\n",
            "|    value_loss         | 0.00164  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.37e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49800    |\n",
            "|    time_elapsed       | 3273     |\n",
            "|    total_timesteps    | 1593600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.44    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49799    |\n",
            "|    policy_loss        | -0.05    |\n",
            "|    std                | 0.062    |\n",
            "|    value_loss         | 0.00109  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.38e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 49900    |\n",
            "|    time_elapsed       | 3280     |\n",
            "|    total_timesteps    | 1596800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.13    |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49899    |\n",
            "|    policy_loss        | -0.0414  |\n",
            "|    std                | 0.0625   |\n",
            "|    value_loss         | 0.00133  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.38e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 50000    |\n",
            "|    time_elapsed       | 3286     |\n",
            "|    total_timesteps    | 1600000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.35    |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 49999    |\n",
            "|    policy_loss        | -0.116   |\n",
            "|    std                | 0.0625   |\n",
            "|    value_loss         | 0.00433  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.38e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 50100    |\n",
            "|    time_elapsed       | 3292     |\n",
            "|    total_timesteps    | 1603200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.43    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50099    |\n",
            "|    policy_loss        | 0.0358   |\n",
            "|    std                | 0.0624   |\n",
            "|    value_loss         | 0.000287 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.39e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 50200    |\n",
            "|    time_elapsed       | 3299     |\n",
            "|    total_timesteps    | 1606400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.42    |\n",
            "|    explained_variance | 0.972    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50199    |\n",
            "|    policy_loss        | -0.117   |\n",
            "|    std                | 0.0624   |\n",
            "|    value_loss         | 0.00255  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.39e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 50300    |\n",
            "|    time_elapsed       | 3304     |\n",
            "|    total_timesteps    | 1609600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.51    |\n",
            "|    explained_variance | 0.972    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50299    |\n",
            "|    policy_loss        | 0.0695   |\n",
            "|    std                | 0.0629   |\n",
            "|    value_loss         | 0.00209  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.39e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 50400    |\n",
            "|    time_elapsed       | 3312     |\n",
            "|    total_timesteps    | 1612800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.61    |\n",
            "|    explained_variance | 0.986    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50399    |\n",
            "|    policy_loss        | -0.0695  |\n",
            "|    std                | 0.0632   |\n",
            "|    value_loss         | 0.00134  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.4e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 50500    |\n",
            "|    time_elapsed       | 3317     |\n",
            "|    total_timesteps    | 1616000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.59    |\n",
            "|    explained_variance | 0.54     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50499    |\n",
            "|    policy_loss        | 0.023    |\n",
            "|    std                | 0.0629   |\n",
            "|    value_loss         | 0.000147 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.41e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 486      |\n",
            "|    iterations         | 50600    |\n",
            "|    time_elapsed       | 3325     |\n",
            "|    total_timesteps    | 1619200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.51    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50599    |\n",
            "|    policy_loss        | 0.0161   |\n",
            "|    std                | 0.0626   |\n",
            "|    value_loss         | 0.000275 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.41e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 50700    |\n",
            "|    time_elapsed       | 3330     |\n",
            "|    total_timesteps    | 1622400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.44    |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50699    |\n",
            "|    policy_loss        | -0.105   |\n",
            "|    std                | 0.0627   |\n",
            "|    value_loss         | 0.00428  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.42e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 50800    |\n",
            "|    time_elapsed       | 3336     |\n",
            "|    total_timesteps    | 1625600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.69    |\n",
            "|    explained_variance | 0.829    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50799    |\n",
            "|    policy_loss        | 0.0071   |\n",
            "|    std                | 0.063    |\n",
            "|    value_loss         | 0.000565 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.44e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 50900    |\n",
            "|    time_elapsed       | 3343     |\n",
            "|    total_timesteps    | 1628800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.37    |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50899    |\n",
            "|    policy_loss        | 0.0219   |\n",
            "|    std                | 0.0627   |\n",
            "|    value_loss         | 0.00195  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.46e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51000    |\n",
            "|    time_elapsed       | 3349     |\n",
            "|    total_timesteps    | 1632000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.61    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 50999    |\n",
            "|    policy_loss        | -0.0428  |\n",
            "|    std                | 0.0631   |\n",
            "|    value_loss         | 0.00114  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.48e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51100    |\n",
            "|    time_elapsed       | 3356     |\n",
            "|    total_timesteps    | 1635200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.66    |\n",
            "|    explained_variance | 0.784    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51099    |\n",
            "|    policy_loss        | -0.0136  |\n",
            "|    std                | 0.0631   |\n",
            "|    value_loss         | 0.000234 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.5e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51200    |\n",
            "|    time_elapsed       | 3362     |\n",
            "|    total_timesteps    | 1638400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.18    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51199    |\n",
            "|    policy_loss        | -0.00625 |\n",
            "|    std                | 0.0631   |\n",
            "|    value_loss         | 0.00133  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.51e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51300    |\n",
            "|    time_elapsed       | 3368     |\n",
            "|    total_timesteps    | 1641600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.55    |\n",
            "|    explained_variance | 0.978    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51299    |\n",
            "|    policy_loss        | 0.079    |\n",
            "|    std                | 0.0634   |\n",
            "|    value_loss         | 0.00122  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.51e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51400    |\n",
            "|    time_elapsed       | 3374     |\n",
            "|    total_timesteps    | 1644800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.52    |\n",
            "|    explained_variance | 0.976    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51399    |\n",
            "|    policy_loss        | 0.243    |\n",
            "|    std                | 0.0635   |\n",
            "|    value_loss         | 0.00353  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.52e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51500    |\n",
            "|    time_elapsed       | 3380     |\n",
            "|    total_timesteps    | 1648000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.52    |\n",
            "|    explained_variance | 0.829    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51499    |\n",
            "|    policy_loss        | 0.00712  |\n",
            "|    std                | 0.0633   |\n",
            "|    value_loss         | 0.000435 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.53e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51600    |\n",
            "|    time_elapsed       | 3387     |\n",
            "|    total_timesteps    | 1651200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.48    |\n",
            "|    explained_variance | 0.981    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51599    |\n",
            "|    policy_loss        | 0.0539   |\n",
            "|    std                | 0.063    |\n",
            "|    value_loss         | 0.00101  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.56e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51700    |\n",
            "|    time_elapsed       | 3393     |\n",
            "|    total_timesteps    | 1654400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.57    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51699    |\n",
            "|    policy_loss        | 0.00173  |\n",
            "|    std                | 0.0628   |\n",
            "|    value_loss         | 0.00198  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.58e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51800    |\n",
            "|    time_elapsed       | 3400     |\n",
            "|    total_timesteps    | 1657600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.6     |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51799    |\n",
            "|    policy_loss        | 0.118    |\n",
            "|    std                | 0.0629   |\n",
            "|    value_loss         | 0.0018   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.61e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 51900    |\n",
            "|    time_elapsed       | 3406     |\n",
            "|    total_timesteps    | 1660800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.31    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51899    |\n",
            "|    policy_loss        | -0.0761  |\n",
            "|    std                | 0.0628   |\n",
            "|    value_loss         | 0.000682 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.64e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52000    |\n",
            "|    time_elapsed       | 3411     |\n",
            "|    total_timesteps    | 1664000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.26    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 51999    |\n",
            "|    policy_loss        | 0.103    |\n",
            "|    std                | 0.0627   |\n",
            "|    value_loss         | 0.00176  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52100    |\n",
            "|    time_elapsed       | 3419     |\n",
            "|    total_timesteps    | 1667200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.49    |\n",
            "|    explained_variance | 0.945    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52099    |\n",
            "|    policy_loss        | -0.0392  |\n",
            "|    std                | 0.0627   |\n",
            "|    value_loss         | 0.000474 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.7e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52200    |\n",
            "|    time_elapsed       | 3424     |\n",
            "|    total_timesteps    | 1670400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.42    |\n",
            "|    explained_variance | 0.92     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52199    |\n",
            "|    policy_loss        | 0.0109   |\n",
            "|    std                | 0.0627   |\n",
            "|    value_loss         | 0.000394 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.72e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52300    |\n",
            "|    time_elapsed       | 3431     |\n",
            "|    total_timesteps    | 1673600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.25    |\n",
            "|    explained_variance | 0.942    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52299    |\n",
            "|    policy_loss        | -0.0385  |\n",
            "|    std                | 0.0629   |\n",
            "|    value_loss         | 0.000462 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.75e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52400    |\n",
            "|    time_elapsed       | 3437     |\n",
            "|    total_timesteps    | 1676800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.34    |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52399    |\n",
            "|    policy_loss        | -0.0175  |\n",
            "|    std                | 0.0631   |\n",
            "|    value_loss         | 0.00375  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52500    |\n",
            "|    time_elapsed       | 3444     |\n",
            "|    total_timesteps    | 1680000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.33    |\n",
            "|    explained_variance | 0.953    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52499    |\n",
            "|    policy_loss        | -0.0203  |\n",
            "|    std                | 0.0626   |\n",
            "|    value_loss         | 0.00119  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.8e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52600    |\n",
            "|    time_elapsed       | 3452     |\n",
            "|    total_timesteps    | 1683200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.34    |\n",
            "|    explained_variance | 0.835    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52599    |\n",
            "|    policy_loss        | 0.0354   |\n",
            "|    std                | 0.0624   |\n",
            "|    value_loss         | 0.000314 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.84e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52700    |\n",
            "|    time_elapsed       | 3457     |\n",
            "|    total_timesteps    | 1686400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.27    |\n",
            "|    explained_variance | 0.959    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52699    |\n",
            "|    policy_loss        | -0.033   |\n",
            "|    std                | 0.0618   |\n",
            "|    value_loss         | 0.000857 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.84e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52800    |\n",
            "|    time_elapsed       | 3465     |\n",
            "|    total_timesteps    | 1689600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.05    |\n",
            "|    explained_variance | 0.976    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52799    |\n",
            "|    policy_loss        | -0.0534  |\n",
            "|    std                | 0.0617   |\n",
            "|    value_loss         | 0.00728  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.85e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 52900    |\n",
            "|    time_elapsed       | 3470     |\n",
            "|    total_timesteps    | 1692800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.3     |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52899    |\n",
            "|    policy_loss        | -0.112   |\n",
            "|    std                | 0.0618   |\n",
            "|    value_loss         | 0.00155  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.86e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 53000    |\n",
            "|    time_elapsed       | 3477     |\n",
            "|    total_timesteps    | 1696000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.2     |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 52999    |\n",
            "|    policy_loss        | 0.23     |\n",
            "|    std                | 0.0616   |\n",
            "|    value_loss         | 0.0133   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.86e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 53100    |\n",
            "|    time_elapsed       | 3483     |\n",
            "|    total_timesteps    | 1699200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.05    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53099    |\n",
            "|    policy_loss        | -0.336   |\n",
            "|    std                | 0.0613   |\n",
            "|    value_loss         | 0.0139   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.84e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 53200    |\n",
            "|    time_elapsed       | 3489     |\n",
            "|    total_timesteps    | 1702400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.41    |\n",
            "|    explained_variance | 0.971    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53199    |\n",
            "|    policy_loss        | 0.0162   |\n",
            "|    std                | 0.0615   |\n",
            "|    value_loss         | 0.00245  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.83e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 53300    |\n",
            "|    time_elapsed       | 3496     |\n",
            "|    total_timesteps    | 1705600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.18    |\n",
            "|    explained_variance | 0.981    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53299    |\n",
            "|    policy_loss        | 0.0435   |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.00264  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.82e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 53400    |\n",
            "|    time_elapsed       | 3502     |\n",
            "|    total_timesteps    | 1708800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.21    |\n",
            "|    explained_variance | 0.961    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53399    |\n",
            "|    policy_loss        | -0.121   |\n",
            "|    std                | 0.0611   |\n",
            "|    value_loss         | 0.00489  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.82e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 53500    |\n",
            "|    time_elapsed       | 3509     |\n",
            "|    total_timesteps    | 1712000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.2     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53499    |\n",
            "|    policy_loss        | -0.0497  |\n",
            "|    std                | 0.0612   |\n",
            "|    value_loss         | 0.0027   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.82e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 53600    |\n",
            "|    time_elapsed       | 3515     |\n",
            "|    total_timesteps    | 1715200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.34    |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53599    |\n",
            "|    policy_loss        | 0.0155   |\n",
            "|    std                | 0.0609   |\n",
            "|    value_loss         | 0.0021   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.82e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 53700    |\n",
            "|    time_elapsed       | 3520     |\n",
            "|    total_timesteps    | 1718400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.1     |\n",
            "|    explained_variance | 0.96     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53699    |\n",
            "|    policy_loss        | 0.459    |\n",
            "|    std                | 0.0611   |\n",
            "|    value_loss         | 0.012    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.82e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 53800    |\n",
            "|    time_elapsed       | 3528     |\n",
            "|    total_timesteps    | 1721600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.37    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53799    |\n",
            "|    policy_loss        | 0.00287  |\n",
            "|    std                | 0.0609   |\n",
            "|    value_loss         | 0.000546 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.82e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 53900    |\n",
            "|    time_elapsed       | 3533     |\n",
            "|    total_timesteps    | 1724800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.06    |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53899    |\n",
            "|    policy_loss        | 0.0619   |\n",
            "|    std                | 0.0608   |\n",
            "|    value_loss         | 0.000698 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.82e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 487      |\n",
            "|    iterations         | 54000    |\n",
            "|    time_elapsed       | 3541     |\n",
            "|    total_timesteps    | 1728000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.09    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 53999    |\n",
            "|    policy_loss        | 0.00511  |\n",
            "|    std                | 0.0606   |\n",
            "|    value_loss         | 0.00241  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.82e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54100    |\n",
            "|    time_elapsed       | 3546     |\n",
            "|    total_timesteps    | 1731200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.26    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54099    |\n",
            "|    policy_loss        | -0.15    |\n",
            "|    std                | 0.0605   |\n",
            "|    value_loss         | 0.00435  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.8e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54200    |\n",
            "|    time_elapsed       | 3552     |\n",
            "|    total_timesteps    | 1734400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.27    |\n",
            "|    explained_variance | 0.987    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54199    |\n",
            "|    policy_loss        | -0.223   |\n",
            "|    std                | 0.0604   |\n",
            "|    value_loss         | 0.00903  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54300    |\n",
            "|    time_elapsed       | 3559     |\n",
            "|    total_timesteps    | 1737600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.23    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54299    |\n",
            "|    policy_loss        | -0.129   |\n",
            "|    std                | 0.0605   |\n",
            "|    value_loss         | 0.00184  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54400    |\n",
            "|    time_elapsed       | 3565     |\n",
            "|    total_timesteps    | 1740800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.81    |\n",
            "|    explained_variance | 0.979    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54399    |\n",
            "|    policy_loss        | -0.0364  |\n",
            "|    std                | 0.0604   |\n",
            "|    value_loss         | 0.00129  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54500    |\n",
            "|    time_elapsed       | 3572     |\n",
            "|    total_timesteps    | 1744000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.14    |\n",
            "|    explained_variance | 0.938    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54499    |\n",
            "|    policy_loss        | -0.018   |\n",
            "|    std                | 0.0603   |\n",
            "|    value_loss         | 0.000525 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54600    |\n",
            "|    time_elapsed       | 3578     |\n",
            "|    total_timesteps    | 1747200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.2     |\n",
            "|    explained_variance | 0.907    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54599    |\n",
            "|    policy_loss        | -0.0788  |\n",
            "|    std                | 0.06     |\n",
            "|    value_loss         | 0.00103  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54700    |\n",
            "|    time_elapsed       | 3584     |\n",
            "|    total_timesteps    | 1750400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.1     |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54699    |\n",
            "|    policy_loss        | -0.127   |\n",
            "|    std                | 0.0598   |\n",
            "|    value_loss         | 0.00504  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.77e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54800    |\n",
            "|    time_elapsed       | 3591     |\n",
            "|    total_timesteps    | 1753600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.16    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54799    |\n",
            "|    policy_loss        | 0.0571   |\n",
            "|    std                | 0.0599   |\n",
            "|    value_loss         | 0.00829  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.75e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 54900    |\n",
            "|    time_elapsed       | 3596     |\n",
            "|    total_timesteps    | 1756800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.91    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54899    |\n",
            "|    policy_loss        | 0.0363   |\n",
            "|    std                | 0.0599   |\n",
            "|    value_loss         | 0.00193  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55000    |\n",
            "|    time_elapsed       | 3604     |\n",
            "|    total_timesteps    | 1760000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.29    |\n",
            "|    explained_variance | 0.963    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 54999    |\n",
            "|    policy_loss        | 0.0504   |\n",
            "|    std                | 0.0597   |\n",
            "|    value_loss         | 0.00785  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55100    |\n",
            "|    time_elapsed       | 3610     |\n",
            "|    total_timesteps    | 1763200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.16    |\n",
            "|    explained_variance | 0.957    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55099    |\n",
            "|    policy_loss        | 0.366    |\n",
            "|    std                | 0.0595   |\n",
            "|    value_loss         | 0.0118   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.72e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55200    |\n",
            "|    time_elapsed       | 3617     |\n",
            "|    total_timesteps    | 1766400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.16    |\n",
            "|    explained_variance | 0.982    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55199    |\n",
            "|    policy_loss        | -0.12    |\n",
            "|    std                | 0.0596   |\n",
            "|    value_loss         | 0.012    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.7e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55300    |\n",
            "|    time_elapsed       | 3623     |\n",
            "|    total_timesteps    | 1769600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4       |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55299    |\n",
            "|    policy_loss        | 0.0369   |\n",
            "|    std                | 0.0596   |\n",
            "|    value_loss         | 0.00521  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55400    |\n",
            "|    time_elapsed       | 3630     |\n",
            "|    total_timesteps    | 1772800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.16    |\n",
            "|    explained_variance | 0.99     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55399    |\n",
            "|    policy_loss        | -0.0758  |\n",
            "|    std                | 0.0599   |\n",
            "|    value_loss         | 0.00395  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55500    |\n",
            "|    time_elapsed       | 3636     |\n",
            "|    total_timesteps    | 1776000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.97    |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55499    |\n",
            "|    policy_loss        | -0.112   |\n",
            "|    std                | 0.0594   |\n",
            "|    value_loss         | 0.00777  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55600    |\n",
            "|    time_elapsed       | 3642     |\n",
            "|    total_timesteps    | 1779200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.05    |\n",
            "|    explained_variance | 0.876    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55599    |\n",
            "|    policy_loss        | 0.196    |\n",
            "|    std                | 0.0592   |\n",
            "|    value_loss         | 0.00459  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55700    |\n",
            "|    time_elapsed       | 3649     |\n",
            "|    total_timesteps    | 1782400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.22    |\n",
            "|    explained_variance | 0.893    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55699    |\n",
            "|    policy_loss        | -0.117   |\n",
            "|    std                | 0.0593   |\n",
            "|    value_loss         | 0.00177  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.65e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55800    |\n",
            "|    time_elapsed       | 3655     |\n",
            "|    total_timesteps    | 1785600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.04    |\n",
            "|    explained_variance | 0.961    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55799    |\n",
            "|    policy_loss        | 0.0365   |\n",
            "|    std                | 0.0594   |\n",
            "|    value_loss         | 0.00128  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 55900    |\n",
            "|    time_elapsed       | 3664     |\n",
            "|    total_timesteps    | 1788800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.08    |\n",
            "|    explained_variance | 0.987    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55899    |\n",
            "|    policy_loss        | -0.0243  |\n",
            "|    std                | 0.0592   |\n",
            "|    value_loss         | 0.00163  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.66e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56000    |\n",
            "|    time_elapsed       | 3670     |\n",
            "|    total_timesteps    | 1792000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.97    |\n",
            "|    explained_variance | 0.915    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 55999    |\n",
            "|    policy_loss        | 0.0391   |\n",
            "|    std                | 0.0591   |\n",
            "|    value_loss         | 0.000638 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.67e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56100    |\n",
            "|    time_elapsed       | 3676     |\n",
            "|    total_timesteps    | 1795200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.19    |\n",
            "|    explained_variance | 0.898    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56099    |\n",
            "|    policy_loss        | -0.0108  |\n",
            "|    std                | 0.0592   |\n",
            "|    value_loss         | 0.00119  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.68e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56200    |\n",
            "|    time_elapsed       | 3683     |\n",
            "|    total_timesteps    | 1798400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.2     |\n",
            "|    explained_variance | 0.973    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56199    |\n",
            "|    policy_loss        | 0.0117   |\n",
            "|    std                | 0.0594   |\n",
            "|    value_loss         | 0.000215 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.72e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56300    |\n",
            "|    time_elapsed       | 3689     |\n",
            "|    total_timesteps    | 1801600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.07    |\n",
            "|    explained_variance | 0.426    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56299    |\n",
            "|    policy_loss        | -0.0239  |\n",
            "|    std                | 0.059    |\n",
            "|    value_loss         | 0.000439 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.74e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56400    |\n",
            "|    time_elapsed       | 3696     |\n",
            "|    total_timesteps    | 1804800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.88    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56399    |\n",
            "|    policy_loss        | -0.00768 |\n",
            "|    std                | 0.0588   |\n",
            "|    value_loss         | 0.000234 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56500    |\n",
            "|    time_elapsed       | 3702     |\n",
            "|    total_timesteps    | 1808000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.91    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56499    |\n",
            "|    policy_loss        | -0.018   |\n",
            "|    std                | 0.0588   |\n",
            "|    value_loss         | 0.000545 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.77e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56600    |\n",
            "|    time_elapsed       | 3709     |\n",
            "|    total_timesteps    | 1811200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.88    |\n",
            "|    explained_variance | 0.941    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56599    |\n",
            "|    policy_loss        | -0.0784  |\n",
            "|    std                | 0.0586   |\n",
            "|    value_loss         | 0.00131  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.77e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56700    |\n",
            "|    time_elapsed       | 3715     |\n",
            "|    total_timesteps    | 1814400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.89    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56699    |\n",
            "|    policy_loss        | -0.0941  |\n",
            "|    std                | 0.0585   |\n",
            "|    value_loss         | 0.00177  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.75e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56800    |\n",
            "|    time_elapsed       | 3721     |\n",
            "|    total_timesteps    | 1817600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.77    |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56799    |\n",
            "|    policy_loss        | 0.283    |\n",
            "|    std                | 0.0588   |\n",
            "|    value_loss         | 0.00776  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.72e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 56900    |\n",
            "|    time_elapsed       | 3728     |\n",
            "|    total_timesteps    | 1820800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.88    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56899    |\n",
            "|    policy_loss        | 0.129    |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.00264  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.7e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57000    |\n",
            "|    time_elapsed       | 3734     |\n",
            "|    total_timesteps    | 1824000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.88    |\n",
            "|    explained_variance | 0.749    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 56999    |\n",
            "|    policy_loss        | 0.0399   |\n",
            "|    std                | 0.0587   |\n",
            "|    value_loss         | 0.00347  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.7e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57100    |\n",
            "|    time_elapsed       | 3741     |\n",
            "|    total_timesteps    | 1827200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.93    |\n",
            "|    explained_variance | 0.956    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57099    |\n",
            "|    policy_loss        | 0.0071   |\n",
            "|    std                | 0.0586   |\n",
            "|    value_loss         | 0.000327 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.71e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57200    |\n",
            "|    time_elapsed       | 3747     |\n",
            "|    total_timesteps    | 1830400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -4.14    |\n",
            "|    explained_variance | 0.907    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57199    |\n",
            "|    policy_loss        | 0.000709 |\n",
            "|    std                | 0.0587   |\n",
            "|    value_loss         | 0.000849 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.72e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57300    |\n",
            "|    time_elapsed       | 3753     |\n",
            "|    total_timesteps    | 1833600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.69    |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57299    |\n",
            "|    policy_loss        | 0.0063   |\n",
            "|    std                | 0.0588   |\n",
            "|    value_loss         | 0.000274 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57400    |\n",
            "|    time_elapsed       | 3760     |\n",
            "|    total_timesteps    | 1836800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.96    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57399    |\n",
            "|    policy_loss        | -0.0262  |\n",
            "|    std                | 0.0587   |\n",
            "|    value_loss         | 0.000497 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.75e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57500    |\n",
            "|    time_elapsed       | 3766     |\n",
            "|    total_timesteps    | 1840000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.77    |\n",
            "|    explained_variance | 0.97     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57499    |\n",
            "|    policy_loss        | 0.0639   |\n",
            "|    std                | 0.0587   |\n",
            "|    value_loss         | 0.00805  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57600    |\n",
            "|    time_elapsed       | 3773     |\n",
            "|    total_timesteps    | 1843200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.81    |\n",
            "|    explained_variance | 0.893    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57599    |\n",
            "|    policy_loss        | 0.00472  |\n",
            "|    std                | 0.0588   |\n",
            "|    value_loss         | 0.00112  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57700    |\n",
            "|    time_elapsed       | 3779     |\n",
            "|    total_timesteps    | 1846400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.85    |\n",
            "|    explained_variance | 0.86     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57699    |\n",
            "|    policy_loss        | 0.00447  |\n",
            "|    std                | 0.0586   |\n",
            "|    value_loss         | 0.000515 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57800    |\n",
            "|    time_elapsed       | 3786     |\n",
            "|    total_timesteps    | 1849600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.76    |\n",
            "|    explained_variance | 0.922    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57799    |\n",
            "|    policy_loss        | 0.0881   |\n",
            "|    std                | 0.0585   |\n",
            "|    value_loss         | 0.00115  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.77e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 57900    |\n",
            "|    time_elapsed       | 3792     |\n",
            "|    total_timesteps    | 1852800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.98    |\n",
            "|    explained_variance | 0.984    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57899    |\n",
            "|    policy_loss        | -0.0534  |\n",
            "|    std                | 0.0585   |\n",
            "|    value_loss         | 0.00193  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58000    |\n",
            "|    time_elapsed       | 3798     |\n",
            "|    total_timesteps    | 1856000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.89    |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 57999    |\n",
            "|    policy_loss        | 0.00482  |\n",
            "|    std                | 0.0586   |\n",
            "|    value_loss         | 0.00176  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.8e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58100    |\n",
            "|    time_elapsed       | 3805     |\n",
            "|    total_timesteps    | 1859200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.86    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58099    |\n",
            "|    policy_loss        | 0.554    |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.0388   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.81e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58200    |\n",
            "|    time_elapsed       | 3811     |\n",
            "|    total_timesteps    | 1862400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.29    |\n",
            "|    explained_variance | 0.968    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58199    |\n",
            "|    policy_loss        | 0.0511   |\n",
            "|    std                | 0.0588   |\n",
            "|    value_loss         | 0.0068   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58300    |\n",
            "|    time_elapsed       | 3818     |\n",
            "|    total_timesteps    | 1865600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.7     |\n",
            "|    explained_variance | 0.974    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58299    |\n",
            "|    policy_loss        | -0.329   |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.0136   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58400    |\n",
            "|    time_elapsed       | 3824     |\n",
            "|    total_timesteps    | 1868800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.91    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58399    |\n",
            "|    policy_loss        | 0.092    |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.00106  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58500    |\n",
            "|    time_elapsed       | 3831     |\n",
            "|    total_timesteps    | 1872000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.83    |\n",
            "|    explained_variance | 0.945    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58499    |\n",
            "|    policy_loss        | -0.0131  |\n",
            "|    std                | 0.0588   |\n",
            "|    value_loss         | 0.00077  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58600    |\n",
            "|    time_elapsed       | 3837     |\n",
            "|    total_timesteps    | 1875200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.86    |\n",
            "|    explained_variance | 0.976    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58599    |\n",
            "|    policy_loss        | -0.0393  |\n",
            "|    std                | 0.0591   |\n",
            "|    value_loss         | 0.00222  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.8e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58700    |\n",
            "|    time_elapsed       | 3843     |\n",
            "|    total_timesteps    | 1878400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.88    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58699    |\n",
            "|    policy_loss        | -0.0474  |\n",
            "|    std                | 0.059    |\n",
            "|    value_loss         | 0.00247  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.8e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58800    |\n",
            "|    time_elapsed       | 3850     |\n",
            "|    total_timesteps    | 1881600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.81    |\n",
            "|    explained_variance | 0.986    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58799    |\n",
            "|    policy_loss        | 0.00446  |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.000656 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 58900    |\n",
            "|    time_elapsed       | 3856     |\n",
            "|    total_timesteps    | 1884800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.93    |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58899    |\n",
            "|    policy_loss        | -0.36    |\n",
            "|    std                | 0.0585   |\n",
            "|    value_loss         | 0.00831  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59000    |\n",
            "|    time_elapsed       | 3863     |\n",
            "|    total_timesteps    | 1888000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.9     |\n",
            "|    explained_variance | 0.632    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 58999    |\n",
            "|    policy_loss        | 0.077    |\n",
            "|    std                | 0.058    |\n",
            "|    value_loss         | 0.00205  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59100    |\n",
            "|    time_elapsed       | 3869     |\n",
            "|    total_timesteps    | 1891200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.84    |\n",
            "|    explained_variance | 0.955    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59099    |\n",
            "|    policy_loss        | 0.0209   |\n",
            "|    std                | 0.0577   |\n",
            "|    value_loss         | 0.000342 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59200    |\n",
            "|    time_elapsed       | 3875     |\n",
            "|    total_timesteps    | 1894400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.84    |\n",
            "|    explained_variance | 0.691    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59199    |\n",
            "|    policy_loss        | -0.0447  |\n",
            "|    std                | 0.0578   |\n",
            "|    value_loss         | 0.00152  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59300    |\n",
            "|    time_elapsed       | 3884     |\n",
            "|    total_timesteps    | 1897600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.42    |\n",
            "|    explained_variance | 0.981    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59299    |\n",
            "|    policy_loss        | -0.152   |\n",
            "|    std                | 0.0577   |\n",
            "|    value_loss         | 0.0186   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.75e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59400    |\n",
            "|    time_elapsed       | 3890     |\n",
            "|    total_timesteps    | 1900800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.87    |\n",
            "|    explained_variance | 0.934    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59399    |\n",
            "|    policy_loss        | 0.0947   |\n",
            "|    std                | 0.0576   |\n",
            "|    value_loss         | 0.0029   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.74e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59500    |\n",
            "|    time_elapsed       | 3897     |\n",
            "|    total_timesteps    | 1904000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.91    |\n",
            "|    explained_variance | 0.952    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59499    |\n",
            "|    policy_loss        | 0.0809   |\n",
            "|    std                | 0.0575   |\n",
            "|    value_loss         | 0.00085  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.74e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59600    |\n",
            "|    time_elapsed       | 3903     |\n",
            "|    total_timesteps    | 1907200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.72    |\n",
            "|    explained_variance | 0.958    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59599    |\n",
            "|    policy_loss        | -0.0604  |\n",
            "|    std                | 0.0577   |\n",
            "|    value_loss         | 0.0059   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59700    |\n",
            "|    time_elapsed       | 3910     |\n",
            "|    total_timesteps    | 1910400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.66    |\n",
            "|    explained_variance | 0.863    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59699    |\n",
            "|    policy_loss        | 0.109    |\n",
            "|    std                | 0.0576   |\n",
            "|    value_loss         | 0.00264  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.72e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59800    |\n",
            "|    time_elapsed       | 3916     |\n",
            "|    total_timesteps    | 1913600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.83    |\n",
            "|    explained_variance | 0.633    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59799    |\n",
            "|    policy_loss        | 0.015    |\n",
            "|    std                | 0.058    |\n",
            "|    value_loss         | 0.00131  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.73e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 59900    |\n",
            "|    time_elapsed       | 3922     |\n",
            "|    total_timesteps    | 1916800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.85    |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59899    |\n",
            "|    policy_loss        | 0.058    |\n",
            "|    std                | 0.0583   |\n",
            "|    value_loss         | 0.00191  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.76e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60000    |\n",
            "|    time_elapsed       | 3929     |\n",
            "|    total_timesteps    | 1920000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.84    |\n",
            "|    explained_variance | 0.87     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 59999    |\n",
            "|    policy_loss        | -0.0622  |\n",
            "|    std                | 0.0582   |\n",
            "|    value_loss         | 0.000708 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60100    |\n",
            "|    time_elapsed       | 3934     |\n",
            "|    total_timesteps    | 1923200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.56    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60099    |\n",
            "|    policy_loss        | 0.116    |\n",
            "|    std                | 0.0578   |\n",
            "|    value_loss         | 0.0025   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60200    |\n",
            "|    time_elapsed       | 3942     |\n",
            "|    total_timesteps    | 1926400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.79    |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60199    |\n",
            "|    policy_loss        | 0.155    |\n",
            "|    std                | 0.0579   |\n",
            "|    value_loss         | 0.00518  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60300    |\n",
            "|    time_elapsed       | 3947     |\n",
            "|    total_timesteps    | 1929600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.84    |\n",
            "|    explained_variance | 0.902    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60299    |\n",
            "|    policy_loss        | -0.0972  |\n",
            "|    std                | 0.058    |\n",
            "|    value_loss         | 0.00443  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60400    |\n",
            "|    time_elapsed       | 3954     |\n",
            "|    total_timesteps    | 1932800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.73    |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60399    |\n",
            "|    policy_loss        | -0.00538 |\n",
            "|    std                | 0.0581   |\n",
            "|    value_loss         | 0.00144  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.8e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60500    |\n",
            "|    time_elapsed       | 3961     |\n",
            "|    total_timesteps    | 1936000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.69    |\n",
            "|    explained_variance | 0.86     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60499    |\n",
            "|    policy_loss        | -0.07    |\n",
            "|    std                | 0.0578   |\n",
            "|    value_loss         | 0.00152  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.8e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60600    |\n",
            "|    time_elapsed       | 3966     |\n",
            "|    total_timesteps    | 1939200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.73    |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60599    |\n",
            "|    policy_loss        | -0.0323  |\n",
            "|    std                | 0.058    |\n",
            "|    value_loss         | 0.000435 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.79e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60700    |\n",
            "|    time_elapsed       | 3974     |\n",
            "|    total_timesteps    | 1942400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.78    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60699    |\n",
            "|    policy_loss        | 0.0484   |\n",
            "|    std                | 0.0583   |\n",
            "|    value_loss         | 0.00102  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60800    |\n",
            "|    time_elapsed       | 3979     |\n",
            "|    total_timesteps    | 1945600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.72    |\n",
            "|    explained_variance | 0.982    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60799    |\n",
            "|    policy_loss        | -0.0212  |\n",
            "|    std                | 0.0587   |\n",
            "|    value_loss         | 0.0015   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 60900    |\n",
            "|    time_elapsed       | 3987     |\n",
            "|    total_timesteps    | 1948800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.76    |\n",
            "|    explained_variance | 0.929    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60899    |\n",
            "|    policy_loss        | 0.0104   |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.00784  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.77e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 61000    |\n",
            "|    time_elapsed       | 3992     |\n",
            "|    total_timesteps    | 1952000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.61    |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 60999    |\n",
            "|    policy_loss        | 0.0203   |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.00401  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.77e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 61100    |\n",
            "|    time_elapsed       | 3999     |\n",
            "|    total_timesteps    | 1955200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.76    |\n",
            "|    explained_variance | 0.778    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61099    |\n",
            "|    policy_loss        | -0.165   |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.00828  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.78e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 61200    |\n",
            "|    time_elapsed       | 4005     |\n",
            "|    total_timesteps    | 1958400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.65    |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61199    |\n",
            "|    policy_loss        | -0.0135  |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.00116  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.8e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 61300    |\n",
            "|    time_elapsed       | 4011     |\n",
            "|    total_timesteps    | 1961600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.6     |\n",
            "|    explained_variance | 0.983    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61299    |\n",
            "|    policy_loss        | -0.00821 |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.0014   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.84e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 61400    |\n",
            "|    time_elapsed       | 4018     |\n",
            "|    total_timesteps    | 1964800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.84    |\n",
            "|    explained_variance | 0.99     |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61399    |\n",
            "|    policy_loss        | -0.096   |\n",
            "|    std                | 0.0589   |\n",
            "|    value_loss         | 0.00181  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.87e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 61500    |\n",
            "|    time_elapsed       | 4024     |\n",
            "|    total_timesteps    | 1968000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.62    |\n",
            "|    explained_variance | 0.951    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61499    |\n",
            "|    policy_loss        | -0.0396  |\n",
            "|    std                | 0.0582   |\n",
            "|    value_loss         | 0.00525  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.88e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 61600    |\n",
            "|    time_elapsed       | 4031     |\n",
            "|    total_timesteps    | 1971200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.76    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61599    |\n",
            "|    policy_loss        | -0.0851  |\n",
            "|    std                | 0.0582   |\n",
            "|    value_loss         | 0.0028   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.88e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 488      |\n",
            "|    iterations         | 61700    |\n",
            "|    time_elapsed       | 4037     |\n",
            "|    total_timesteps    | 1974400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.57    |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61699    |\n",
            "|    policy_loss        | 0.0247   |\n",
            "|    std                | 0.0585   |\n",
            "|    value_loss         | 0.00171  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.88e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 61800    |\n",
            "|    time_elapsed       | 4043     |\n",
            "|    total_timesteps    | 1977600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.66    |\n",
            "|    explained_variance | 0.982    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61799    |\n",
            "|    policy_loss        | 0.0757   |\n",
            "|    std                | 0.0586   |\n",
            "|    value_loss         | 0.00163  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.89e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 61900    |\n",
            "|    time_elapsed       | 4050     |\n",
            "|    total_timesteps    | 1980800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.78    |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61899    |\n",
            "|    policy_loss        | -0.0383  |\n",
            "|    std                | 0.0587   |\n",
            "|    value_loss         | 0.000743 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.89e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 62000    |\n",
            "|    time_elapsed       | 4056     |\n",
            "|    total_timesteps    | 1984000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.83    |\n",
            "|    explained_variance | 0.987    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 61999    |\n",
            "|    policy_loss        | -0.0475  |\n",
            "|    std                | 0.059    |\n",
            "|    value_loss         | 0.000935 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.9e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 62100    |\n",
            "|    time_elapsed       | 4063     |\n",
            "|    total_timesteps    | 1987200  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.71    |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 62099    |\n",
            "|    policy_loss        | -0.0379  |\n",
            "|    std                | 0.0592   |\n",
            "|    value_loss         | 0.00667  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.9e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 62200    |\n",
            "|    time_elapsed       | 4069     |\n",
            "|    total_timesteps    | 1990400  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.88    |\n",
            "|    explained_variance | 0.956    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 62199    |\n",
            "|    policy_loss        | 0.0148   |\n",
            "|    std                | 0.0592   |\n",
            "|    value_loss         | 0.000666 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.9e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 62300    |\n",
            "|    time_elapsed       | 4075     |\n",
            "|    total_timesteps    | 1993600  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.88    |\n",
            "|    explained_variance | 0.937    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 62299    |\n",
            "|    policy_loss        | 0.00297  |\n",
            "|    std                | 0.0593   |\n",
            "|    value_loss         | 0.00126  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.9e+03  |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 62400    |\n",
            "|    time_elapsed       | 4082     |\n",
            "|    total_timesteps    | 1996800  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.7     |\n",
            "|    explained_variance | 0.972    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 62399    |\n",
            "|    policy_loss        | -0.129   |\n",
            "|    std                | 0.0599   |\n",
            "|    value_loss         | 0.00441  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 1e+03    |\n",
            "|    ep_rew_mean        | 1.93e+03 |\n",
            "| time/                 |          |\n",
            "|    fps                | 489      |\n",
            "|    iterations         | 62500    |\n",
            "|    time_elapsed       | 4087     |\n",
            "|    total_timesteps    | 2000000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -3.97    |\n",
            "|    explained_variance | 0.729    |\n",
            "|    learning_rate      | 0.00096  |\n",
            "|    n_updates          | 62499    |\n",
            "|    policy_loss        | 0.0802   |\n",
            "|    std                | 0.0596   |\n",
            "|    value_loss         | 0.00409  |\n",
            "------------------------------------\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<stable_baselines3.a2c.a2c.A2C at 0x7fa8bdf61130>"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Save the model and  VecNormalize statistics when saving the agent\n",
        "model.save(\"a2c-AntBulletEnv-v0\")\n",
        "env.save(\"vec_normalize.pkl\")"
      ],
      "metadata": {
        "id": "MfYtjj19cKFr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Evaluate the agent 📈\n",
        "- Now that's our  agent is trained, we need to **check its performance**.\n",
        "- Stable-Baselines3 provides a method to do that: `evaluate_policy`\n",
        "- In my case, I got a mean reward of `2371.90 +/- 16.50`"
      ],
      "metadata": {
        "id": "01M9GCd32Ig-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize\n",
        "\n",
        "# Load the saved statistics\n",
        "eval_env = DummyVecEnv([lambda: gym.make(\"AntBulletEnv-v0\")])\n",
        "eval_env = VecNormalize.load(\"vec_normalize.pkl\", eval_env)\n",
        "\n",
        "#  do not update them at test time\n",
        "eval_env.training = False\n",
        "# reward normalization is not needed at test time\n",
        "eval_env.norm_reward = False\n",
        "\n",
        "# Load the agent\n",
        "model = A2C.load(\"a2c-AntBulletEnv-v0\")\n",
        "\n",
        "mean_reward, std_reward = evaluate_policy(model, eval_env)\n",
        "\n",
        "print(f\"Mean reward = {mean_reward:.2f} +/- {std_reward:.2f}\")"
      ],
      "metadata": {
        "id": "liirTVoDkHq3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08d0032e-b46a-4df6-82a7-95dc78f27d2a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/evaluation.py:67: UserWarning: Evaluation environment is not wrapped with a ``Monitor`` wrapper. This may result in reporting modified episode lengths and rewards, if other wrappers happen to modify these. Consider wrapping environment first with ``Monitor`` wrapper.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mean reward = 1888.63 +/- 282.14\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Publish your trained model on the Hub 🔥\n",
        "Now that we saw we got good results after the training, we can publish our trained model on the Hub with one line of code.\n",
        "\n",
        "📚 The libraries documentation 👉 https://github.com/huggingface/huggingface_sb3/tree/main#hugging-face--x-stable-baselines3-v20\n",
        "\n",
        "Here's an example of a Model Card (with a PyBullet environment):\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit8/modelcardpybullet.png\" alt=\"Model Card Pybullet\"/>"
      ],
      "metadata": {
        "id": "44L9LVQaavR8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "By using `package_to_hub`, as we already mentionned in the former units, **you evaluate, record a replay, generate a model card of your agent and push it to the hub**.\n",
        "\n",
        "This way:\n",
        "- You can **showcase our work** 🔥\n",
        "- You can **visualize your agent playing** 👀\n",
        "- You can **share with the community an agent that others can use** 💾\n",
        "- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n"
      ],
      "metadata": {
        "id": "MkMk99m8bgaQ"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JquRrWytA6eo"
      },
      "source": [
        "To be able to share your model with the community there are three more steps to follow:\n",
        "\n",
        "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
        "\n",
        "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
        "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/create-token.jpg\" alt=\"Create HF Token\">\n",
        "\n",
        "- Copy the token \n",
        "- Run the cell below and paste the token"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GZiFBBlzxzxY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331,
          "referenced_widgets": [
            "a34fabc6986e48f2bb6e12c108409ae6",
            "24a103c1373b41068a12faed02dab379",
            "c962e045f1b94cabb62563753d455e31",
            "658d54a1adb846dca2d910275f84231e",
            "41231ee052944c9086767d9d7b2b7c1e",
            "ea3cf8733cd6422e800957a69d6a6f3f",
            "47d307b9ec0749579914e691576787c4",
            "a2b053047b5641f88453e4acd56a312b",
            "0ff1b530f7134711b62ac3572efad623",
            "93c9254e31074aeab27d17151f4c10be",
            "eeb08bf5205c4eaab84cd70e699d2f10",
            "2623aab1ad264746aefbe5db1120b58d",
            "bb335f5acdf549c3a809eeeb1d12cc6f",
            "99cc1849b8cf494c826350c6062d00ac",
            "4489b095f33943b88613df10dfc393a0",
            "4e8dcfbf15d84dbbbfb7fc1605352aff",
            "b515fe462dd547d7983bb823b09b1623"
          ]
        },
        "outputId": "d84619fc-a690-4952-f9d8-0bed3c00a3c6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Token is valid.\n",
            "Your token has been saved in your configured git credential helpers (store).\n",
            "Your token has been saved to /root/.cache/huggingface/token\n",
            "Login successful\n"
          ]
        }
      ],
      "source": [
        "notebook_login()\n",
        "!git config --global credential.helper store"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_tsf2uv0g_4p"
      },
      "source": [
        "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FGNh9VsZok0i"
      },
      "source": [
        "3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using `package_to_hub()` function"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "package_to_hub(\n",
        "    model=model,\n",
        "    model_name=f\"a2c-{env_id}\",\n",
        "    model_architecture=\"A2C\",\n",
        "    env_id=env_id,\n",
        "    eval_env=eval_env,\n",
        "    repo_id=f\"Ryukijano/a2c-{env_id}\", # Change the username\n",
        "    commit_message=\"Initial commit\",\n",
        ")"
      ],
      "metadata": {
        "id": "ueuzWVCUTkfS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 453,
          "referenced_widgets": [
            "68eb672001184bc2b671390c312e9314",
            "37bfe909ddcd4577a90cded7027c21ec",
            "c009131f0b4b4cfb99fea816934f0a48",
            "a1d9861fe7d343608cdae8ee9a47519b",
            "d498d465aa094011ac5d229aa896ecdd",
            "cc64ee8a978a4e858c3a6318ddae8928",
            "e51e992c74084ddbb55b017bfffcd1ff",
            "263aa7aab49648369436845307d6a973",
            "c424805af3d0456da20ba0f24c3b3b93",
            "bc03289624c54d848b2616206aa1a094",
            "724340b6f03d4334ad06dac943251c8e",
            "503ffbb5b1e74b1fbe03c012e5f8a738",
            "d6c5f606aa6d4e7caf6e77073bf155d3",
            "efcbebc94bde4515aa470a8fc2c9fa80",
            "bd32ba77fe1c4044a654db5d08027801",
            "97bda6a665a248d3a2e5b87cec5fb95f",
            "1b601f2dca70414eb88a4586be3d84b5",
            "d26bc3f87753444a8c3de5eeccd4baa8",
            "db6bef2236c2450fb4d8572a10228eb1",
            "c68c4db80fba4e07823bbfe005fa4ac9",
            "cdf0ab2d494440f58004d3283b328340",
            "e681656f605944dfbddc127ba571865e",
            "6548ec07c4224995a6f968e80eb61eb1",
            "2e142a5d69c64382a974cdda9bd4bb6d",
            "75305559778e44629c881bc48d36af4d",
            "835fa566e8dd4211acf13880d020b8dc",
            "fe2a3d5ad75a46249011b58154f62478",
            "edb23deac5bf45dcb7dde3d53a33b165",
            "df51699b3979418786205186c0f17ec6",
            "f96a4005a4c9489aba897e6f9cf1178e",
            "a606c822615f4c8baec8748fc7aa255c",
            "7249db09a122433caf03f6fc0c99cb5b",
            "9c8e25fb846b489a8f213ee108b9a1b0",
            "7678399be04742219bd6a8cc27886057",
            "0fead3213f6a4f3f8dcc61ad514af322",
            "5446389ac76c43b98ba6497b359386f9",
            "0a98f9e64c624a2497fb6cae4c92c971",
            "36b460e71dd64b3584f85af9d3f52d87",
            "02ca26a2ab084fea94043bf42d1c797b",
            "e11636a343d9476da28d6665770bfb19",
            "8dc0467ca3454f4282d96194c76f0543",
            "04f278e10f5a48b08a3bebbb9cbd2599",
            "6b04ce624a194392b24d3b96c8a52e0d",
            "bdd624c82ee24df3a6f8cca44831339e",
            "ae35b56363a142dd83dec4b4f32f4d9c",
            "79e1484ef15440fcafb7e477ec58a1ba",
            "f114800172b540e8bd4c7b0d8f0a7a26",
            "17e1863f5fe04b7eb83ad5a3bea15668",
            "c1a2ea9a3c9a491b84e02311ac57a59c",
            "974ffb94b0ee435986e33851d4ced978",
            "8fc015bbe24b4d6099fa82c2e93e7878",
            "657da5c546914b2c9d68dc6fab932622",
            "647d510b5580498c882bd82b56d0405d",
            "7ad122fb00d2478da69cd75e7ffe8748",
            "0b2b157b55fe423e8fc7fc19d886788c",
            "a74d10d7a11649c49c335861243aaf7c",
            "4f45935a6b104ab0bb612df7c6109051",
            "247c19da4be146948526caeeaf6d55ac",
            "1a125311748b488c80427c8aa5498525",
            "1f146f7ce3894a0aac22db792ec6c30c",
            "728dfcc6f2084950911bb447febf9f4a",
            "e1a08c16eb28403c9015753b46b1de36",
            "bdec093d21d7466a80cbbd006aeaa41a",
            "18d7eb175e6f41a99016fbb522049131",
            "0a112f2cb90147b6aaa4ee1e9763f0cb",
            "390b64f7f8244ab6867a34daba51919f",
            "3b1edea109d54474b3aff38345aacf10",
            "17c94d0b4247461f99266a1621bbcc91",
            "6f720c1611aa4dc3ba690eee1e1f7594",
            "796c4dbf273048968c27502917bf52e7",
            "3ea60dd3cc23472faee54b0f9b990e22",
            "2c9f69117f794b689d5f083dfb7a5905",
            "3665b8ceb29e4cd38f9039ed8b4abd72",
            "e79bec7d2eb04e3f94580cb6b74152b3",
            "7a486346d7574a86afd2aa758a85f175",
            "1ea24607e6ff45298cfa62aebae76124",
            "38223374a3a542d89e3415bd44470767"
          ]
        },
        "outputId": "0ca00d29-54e4-4c23-8866-6101cf49c790"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[38;5;4mℹ This function will save, evaluate, generate a video of your agent,\n",
            "create a model card and push everything to the hub. It might take up to 1min.\n",
            "This is a work in progress: if you encounter a bug, please open an issue.\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/evaluation.py:67: UserWarning: Evaluation environment is not wrapped with a ``Monitor`` wrapper. This may result in reporting modified episode lengths and rewards, if other wrappers happen to modify these. Consider wrapping environment first with ``Monitor`` wrapper.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving video to /tmp/tmp6fw1i8ij/-step-0-to-step-1000.mp4\n",
            "\u001b[38;5;4mℹ Pushing repo Ryukijano/a2c-AntBulletEnv-v0 to the Hugging Face\n",
            "Hub\u001b[0m\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "a2c-AntBulletEnv-v0.zip:   0%|          | 0.00/129k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "68eb672001184bc2b671390c312e9314"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vec_normalize.pkl:   0%|          | 0.00/2.14k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "503ffbb5b1e74b1fbe03c012e5f8a738"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Upload 6 LFS files:   0%|          | 0/6 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "6548ec07c4224995a6f968e80eb61eb1"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "replay.mp4:   0%|          | 0.00/1.08M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7678399be04742219bd6a8cc27886057"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "policy.pth:   0%|          | 0.00/57.0k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ae35b56363a142dd83dec4b4f32f4d9c"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "pytorch_variables.pth:   0%|          | 0.00/431 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a74d10d7a11649c49c335861243aaf7c"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "policy.optimizer.pth:   0%|          | 0.00/56.2k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3b1edea109d54474b3aff38345aacf10"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[38;5;4mℹ Your model is pushed to the Hub. You can view your model here:\n",
            "https://huggingface.co/Ryukijano/a2c-AntBulletEnv-v0/tree/main/\u001b[0m\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'https://huggingface.co/Ryukijano/a2c-AntBulletEnv-v0/tree/main/'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Take a coffee break ☕\n",
        "- You already trained your first robot that learned to move congratutlations 🥳!\n",
        "- It's **time to take a break**. Don't hesitate to **save this notebook** `File > Save a copy to Drive` to work on this second part later.\n"
      ],
      "metadata": {
        "id": "Qk9ykOk9D6Qh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Environment 2: PandaReachDense-v2 🦾\n",
        "\n",
        "The agent we're going to train is a robotic arm that needs to do controls (moving the arm and using the end-effector).\n",
        "\n",
        "In robotics, the *end-effector* is the device at the end of a robotic arm designed to interact with the environment.\n",
        "\n",
        "In `PandaReach`, the robot must place its end-effector at a target position (green ball).\n",
        "\n",
        "We're going to use the dense version of this environment. It means we'll get a *dense reward function* that **will provide a reward at each timestep** (the closer the agent is to completing the task, the higher the reward). Contrary to a *sparse reward function* where the environment **return a reward if and only if the task is completed**.\n",
        "\n",
        "Also, we're going to use the *End-effector displacement control*, it means the **action corresponds to the displacement of the end-effector**. We don't control the individual motion of each joint (joint control).\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit8/robotics.jpg\"  alt=\"Robotics\"/>\n",
        "\n",
        "\n",
        "This way **the training will be easier**.\n",
        "\n"
      ],
      "metadata": {
        "id": "5VWfwAA7EJg7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "In `PandaReachDense-v2` the robotic arm must place its end-effector at a target position (green ball).\n",
        "\n"
      ],
      "metadata": {
        "id": "oZ7FyDEi7G3T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import gym\n",
        "\n",
        "env_id = \"PandaReachDense-v2\"\n",
        "\n",
        "# Create the env\n",
        "env = gym.make(env_id)\n",
        "\n",
        "# Get the state space and action space\n",
        "s_size = env.observation_space.shape\n",
        "a_size = env.action_space"
      ],
      "metadata": {
        "id": "zXzAu3HYF1WD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
        "print(\"The State Space is: \", s_size)\n",
        "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
      ],
      "metadata": {
        "id": "E-U9dexcF-FB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0599de84-cb54-48f7-8648-eec968292a36"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "_____OBSERVATION SPACE_____ \n",
            "\n",
            "The State Space is:  None\n",
            "Sample observation OrderedDict([('achieved_goal', array([-0.41542676,  0.68229276, -8.004612  ], dtype=float32)), ('desired_goal', array([6.2410693, 6.319219 , 9.930942 ], dtype=float32)), ('observation', array([-6.6189137, -8.045314 , -7.458237 , -8.594829 ,  5.6395845,\n",
            "        7.5126705], dtype=float32))])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The observation space **is a dictionary with 3 different elements**:\n",
        "- `achieved_goal`: (x,y,z) position of the goal.\n",
        "- `desired_goal`: (x,y,z) distance between the goal position and the current object position.\n",
        "- `observation`: position (x,y,z) and velocity of the end-effector (vx, vy, vz).\n",
        "\n",
        "Given it's a dictionary as observation, **we will need to use a MultiInputPolicy policy instead of MlpPolicy**."
      ],
      "metadata": {
        "id": "g_JClfElGFnF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
        "print(\"The Action Space is: \", a_size)\n",
        "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
      ],
      "metadata": {
        "id": "ib1Kxy4AF-FC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8d99ad78-97cd-49cb-eed7-1ec83c3e3100"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            " _____ACTION SPACE_____ \n",
            "\n",
            "The Action Space is:  Box([-1. -1. -1.], [1. 1. 1.], (3,), float32)\n",
            "Action Space Sample [-0.8225759   0.8757768  -0.34699088]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The action space is a vector with 3 values:\n",
        "- Control x, y, z movement"
      ],
      "metadata": {
        "id": "5MHTHEHZS4yp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now it's your turn:\n",
        "\n",
        "1. Define the environment called \"PandaReachDense-v2\"\n",
        "2. Make a vectorized environment\n",
        "3. Add a wrapper to normalize the observations and rewards. [Check the documentation](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#vecnormalize)\n",
        "4. Create the A2C Model (don't forget verbose=1 to print the training logs).\n",
        "5. Train it for 1M Timesteps\n",
        "6. Save the model and  VecNormalize statistics when saving the agent\n",
        "7. Evaluate your agent\n",
        "8. Publish your trained model on the Hub 🔥 with `package_to_hub`"
      ],
      "metadata": {
        "id": "nIhPoc5t9HjG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Solution (fill the todo)"
      ],
      "metadata": {
        "id": "sKGbFXZq9ikN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 1 - 2\n",
        "env_id = \"PandaReachDense-v2\"\n",
        "env = make_vec_env(env_id, n_envs=4)\n",
        "\n",
        "# 3\n",
        "env = VecNormalize(env, norm_obs=True, norm_reward=False, clip_obs=10.)\n",
        "\n",
        "# 4\n",
        "model = A2C(policy = \"MultiInputPolicy\",\n",
        "            env = env,\n",
        "            verbose=1)\n",
        "# 5\n",
        "model.learn(2_000_000)"
      ],
      "metadata": {
        "id": "J-cC-Feg9iMm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb8e20b2-d75a-422b-d79e-9bc514d159d1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1;30;43mStreaming output truncated to the last 5000 lines.\u001b[0m\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 72300    |\n",
            "|    time_elapsed       | 5109     |\n",
            "|    total_timesteps    | 1446000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.91     |\n",
            "|    explained_variance | 0.876    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 72299    |\n",
            "|    policy_loss        | 8.09     |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 2.32     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -30.8    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 72400    |\n",
            "|    time_elapsed       | 5116     |\n",
            "|    total_timesteps    | 1448000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.9      |\n",
            "|    explained_variance | 0.941    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 72399    |\n",
            "|    policy_loss        | 8.15     |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 1.83     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -31.8    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 72500    |\n",
            "|    time_elapsed       | 5126     |\n",
            "|    total_timesteps    | 1450000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.86     |\n",
            "|    explained_variance | 0.973    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 72499    |\n",
            "|    policy_loss        | 3.81     |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.518    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -32.6    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 72600    |\n",
            "|    time_elapsed       | 5132     |\n",
            "|    total_timesteps    | 1452000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.971    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 72599    |\n",
            "|    policy_loss        | 7.16     |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 1.29     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -32.8    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 72700    |\n",
            "|    time_elapsed       | 5139     |\n",
            "|    total_timesteps    | 1454000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.973    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 72699    |\n",
            "|    policy_loss        | 6.62     |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 1.12     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -32.8    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 72800    |\n",
            "|    time_elapsed       | 5146     |\n",
            "|    total_timesteps    | 1456000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 72799    |\n",
            "|    policy_loss        | 3.74     |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.447    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -32.5    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 72900    |\n",
            "|    time_elapsed       | 5153     |\n",
            "|    total_timesteps    | 1458000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.77     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 72899    |\n",
            "|    policy_loss        | 2.62     |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.226    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -32      |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73000    |\n",
            "|    time_elapsed       | 5161     |\n",
            "|    total_timesteps    | 1460000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.0955   |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 72999    |\n",
            "|    policy_loss        | 2        |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.113    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -31.2    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73100    |\n",
            "|    time_elapsed       | 5166     |\n",
            "|    total_timesteps    | 1462000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73099    |\n",
            "|    policy_loss        | 0.481    |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.0167   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -30      |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73200    |\n",
            "|    time_elapsed       | 5175     |\n",
            "|    total_timesteps    | 1464000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.74     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73199    |\n",
            "|    policy_loss        | 1.3      |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.0721   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -29.4    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73300    |\n",
            "|    time_elapsed       | 5181     |\n",
            "|    total_timesteps    | 1466000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.74     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73299    |\n",
            "|    policy_loss        | 1.41     |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.0748   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -28.8    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73400    |\n",
            "|    time_elapsed       | 5189     |\n",
            "|    total_timesteps    | 1468000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73399    |\n",
            "|    policy_loss        | 0.895    |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.0631   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -26.9    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73500    |\n",
            "|    time_elapsed       | 5195     |\n",
            "|    total_timesteps    | 1470000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73499    |\n",
            "|    policy_loss        | 1.05     |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.079    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -26.6    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73600    |\n",
            "|    time_elapsed       | 5204     |\n",
            "|    total_timesteps    | 1472000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.881    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73599    |\n",
            "|    policy_loss        | -0.0946  |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.0283   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -22.9    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73700    |\n",
            "|    time_elapsed       | 5210     |\n",
            "|    total_timesteps    | 1474000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73699    |\n",
            "|    policy_loss        | -7.62    |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 1.58     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -20.2    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73800    |\n",
            "|    time_elapsed       | 5218     |\n",
            "|    total_timesteps    | 1476000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.75     |\n",
            "|    explained_variance | 0.976    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73799    |\n",
            "|    policy_loss        | -1.13    |\n",
            "|    std                | 0.0258   |\n",
            "|    value_loss         | 0.158    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -13.8    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 73900    |\n",
            "|    time_elapsed       | 5225     |\n",
            "|    total_timesteps    | 1478000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | -2.6     |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73899    |\n",
            "|    policy_loss        | -10.4    |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 3.38     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -10.6    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74000    |\n",
            "|    time_elapsed       | 5233     |\n",
            "|    total_timesteps    | 1480000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.77     |\n",
            "|    explained_variance | 0.699    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 73999    |\n",
            "|    policy_loss        | 0.618    |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.518    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -6.45    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74100    |\n",
            "|    time_elapsed       | 5240     |\n",
            "|    total_timesteps    | 1482000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74099    |\n",
            "|    policy_loss        | -1.11    |\n",
            "|    std                | 0.0256   |\n",
            "|    value_loss         | 0.0441   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -6.26    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74200    |\n",
            "|    time_elapsed       | 5248     |\n",
            "|    total_timesteps    | 1484000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74199    |\n",
            "|    policy_loss        | -0.419   |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.00593  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.16    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74300    |\n",
            "|    time_elapsed       | 5255     |\n",
            "|    total_timesteps    | 1486000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74299    |\n",
            "|    policy_loss        | -0.0307  |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.00254  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.39    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74400    |\n",
            "|    time_elapsed       | 5261     |\n",
            "|    total_timesteps    | 1488000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74399    |\n",
            "|    policy_loss        | 0.16     |\n",
            "|    std                | 0.0258   |\n",
            "|    value_loss         | 0.00851  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.92    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74500    |\n",
            "|    time_elapsed       | 5270     |\n",
            "|    total_timesteps    | 1490000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74499    |\n",
            "|    policy_loss        | 0.00865  |\n",
            "|    std                | 0.0258   |\n",
            "|    value_loss         | 0.000563 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.37    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74600    |\n",
            "|    time_elapsed       | 5275     |\n",
            "|    total_timesteps    | 1492000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74599    |\n",
            "|    policy_loss        | 0.00612  |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.00144  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.34    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74700    |\n",
            "|    time_elapsed       | 5284     |\n",
            "|    total_timesteps    | 1494000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.75     |\n",
            "|    explained_variance | 0.978    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74699    |\n",
            "|    policy_loss        | -0.354   |\n",
            "|    std                | 0.0258   |\n",
            "|    value_loss         | 0.0225   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.57    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74800    |\n",
            "|    time_elapsed       | 5290     |\n",
            "|    total_timesteps    | 1496000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.74     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74799    |\n",
            "|    policy_loss        | 0.0127   |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.00257  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.55    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 74900    |\n",
            "|    time_elapsed       | 5298     |\n",
            "|    total_timesteps    | 1498000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.71     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74899    |\n",
            "|    policy_loss        | 0.0161   |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.00358  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.58    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75000    |\n",
            "|    time_elapsed       | 5304     |\n",
            "|    total_timesteps    | 1500000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.71     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 74999    |\n",
            "|    policy_loss        | -0.193   |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.00724  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.84    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75100    |\n",
            "|    time_elapsed       | 5311     |\n",
            "|    total_timesteps    | 1502000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.72     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75099    |\n",
            "|    policy_loss        | -0.2     |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.0101   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.22    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75200    |\n",
            "|    time_elapsed       | 5318     |\n",
            "|    total_timesteps    | 1504000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75199    |\n",
            "|    policy_loss        | -0.186   |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.00821  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.14    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75300    |\n",
            "|    time_elapsed       | 5325     |\n",
            "|    total_timesteps    | 1506000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.73     |\n",
            "|    explained_variance | 0.429    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75299    |\n",
            "|    policy_loss        | 1.19     |\n",
            "|    std                | 0.026    |\n",
            "|    value_loss         | 0.17     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.03    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75400    |\n",
            "|    time_elapsed       | 5332     |\n",
            "|    total_timesteps    | 1508000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.75     |\n",
            "|    explained_variance | 0.961    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75399    |\n",
            "|    policy_loss        | 0.817    |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.0568   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.97    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75500    |\n",
            "|    time_elapsed       | 5338     |\n",
            "|    total_timesteps    | 1510000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.75     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75499    |\n",
            "|    policy_loss        | 0.255    |\n",
            "|    std                | 0.0258   |\n",
            "|    value_loss         | 0.0145   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.53    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75600    |\n",
            "|    time_elapsed       | 5346     |\n",
            "|    total_timesteps    | 1512000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.74     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75599    |\n",
            "|    policy_loss        | -0.19    |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.00433  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.1     |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75700    |\n",
            "|    time_elapsed       | 5352     |\n",
            "|    total_timesteps    | 1514000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.72     |\n",
            "|    explained_variance | 0.962    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75699    |\n",
            "|    policy_loss        | 0.588    |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.0236   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.84    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75800    |\n",
            "|    time_elapsed       | 5360     |\n",
            "|    total_timesteps    | 1516000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.73     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75799    |\n",
            "|    policy_loss        | -0.257   |\n",
            "|    std                | 0.026    |\n",
            "|    value_loss         | 0.0073   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.61    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 75900    |\n",
            "|    time_elapsed       | 5366     |\n",
            "|    total_timesteps    | 1518000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.72     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75899    |\n",
            "|    policy_loss        | -0.00479 |\n",
            "|    std                | 0.026    |\n",
            "|    value_loss         | 0.0049   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.23    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 76000    |\n",
            "|    time_elapsed       | 5373     |\n",
            "|    total_timesteps    | 1520000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.74     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 75999    |\n",
            "|    policy_loss        | -0.101   |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.00189  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.09    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 76100    |\n",
            "|    time_elapsed       | 5381     |\n",
            "|    total_timesteps    | 1522000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.71     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76099    |\n",
            "|    policy_loss        | -0.165   |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.00484  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.08    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 76200    |\n",
            "|    time_elapsed       | 5387     |\n",
            "|    total_timesteps    | 1524000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.72     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76199    |\n",
            "|    policy_loss        | 0.105    |\n",
            "|    std                | 0.026    |\n",
            "|    value_loss         | 0.00743  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.86    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 76300    |\n",
            "|    time_elapsed       | 5395     |\n",
            "|    total_timesteps    | 1526000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76299    |\n",
            "|    policy_loss        | -0.0117  |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.000836 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.68    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 76400    |\n",
            "|    time_elapsed       | 5400     |\n",
            "|    total_timesteps    | 1528000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.71     |\n",
            "|    explained_variance | 0.929    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76399    |\n",
            "|    policy_loss        | 0.0925   |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.00324  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.55    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 76500    |\n",
            "|    time_elapsed       | 5408     |\n",
            "|    total_timesteps    | 1530000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.71     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76499    |\n",
            "|    policy_loss        | -0.0665  |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.00117  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.54    |\n",
            "| time/                 |          |\n",
            "|    fps                | 282      |\n",
            "|    iterations         | 76600    |\n",
            "|    time_elapsed       | 5413     |\n",
            "|    total_timesteps    | 1532000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.72     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76599    |\n",
            "|    policy_loss        | -0.0446  |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.000927 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.67    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 76700    |\n",
            "|    time_elapsed       | 5419     |\n",
            "|    total_timesteps    | 1534000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76699    |\n",
            "|    policy_loss        | -0.122   |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.00255  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.78    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 76800    |\n",
            "|    time_elapsed       | 5426     |\n",
            "|    total_timesteps    | 1536000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.69     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76799    |\n",
            "|    policy_loss        | 0.116    |\n",
            "|    std                | 0.0263   |\n",
            "|    value_loss         | 0.00159  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.04    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 76900    |\n",
            "|    time_elapsed       | 5432     |\n",
            "|    total_timesteps    | 1538000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.68     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76899    |\n",
            "|    policy_loss        | 0.315    |\n",
            "|    std                | 0.0264   |\n",
            "|    value_loss         | 0.00404  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.46    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77000    |\n",
            "|    time_elapsed       | 5439     |\n",
            "|    total_timesteps    | 1540000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.67     |\n",
            "|    explained_variance | 0.984    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 76999    |\n",
            "|    policy_loss        | 0.0994   |\n",
            "|    std                | 0.0264   |\n",
            "|    value_loss         | 0.00875  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.77    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77100    |\n",
            "|    time_elapsed       | 5444     |\n",
            "|    total_timesteps    | 1542000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.68     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77099    |\n",
            "|    policy_loss        | 0.392    |\n",
            "|    std                | 0.0263   |\n",
            "|    value_loss         | 0.0148   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.94    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77200    |\n",
            "|    time_elapsed       | 5449     |\n",
            "|    total_timesteps    | 1544000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77199    |\n",
            "|    policy_loss        | 1.04     |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.0447   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.89    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77300    |\n",
            "|    time_elapsed       | 5456     |\n",
            "|    total_timesteps    | 1546000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.69     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77299    |\n",
            "|    policy_loss        | -0.0812  |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.00951  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.58    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77400    |\n",
            "|    time_elapsed       | 5461     |\n",
            "|    total_timesteps    | 1548000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.69     |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77399    |\n",
            "|    policy_loss        | 0.0178   |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.0202   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.28    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77500    |\n",
            "|    time_elapsed       | 5467     |\n",
            "|    total_timesteps    | 1550000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.69     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77499    |\n",
            "|    policy_loss        | -0.171   |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.0054   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.11    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77600    |\n",
            "|    time_elapsed       | 5474     |\n",
            "|    total_timesteps    | 1552000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.67     |\n",
            "|    explained_variance | 0.828    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77599    |\n",
            "|    policy_loss        | -0.409   |\n",
            "|    std                | 0.0264   |\n",
            "|    value_loss         | 0.0405   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.23    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77700    |\n",
            "|    time_elapsed       | 5479     |\n",
            "|    total_timesteps    | 1554000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.67     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77699    |\n",
            "|    policy_loss        | 0.303    |\n",
            "|    std                | 0.0264   |\n",
            "|    value_loss         | 0.00753  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.22    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77800    |\n",
            "|    time_elapsed       | 5485     |\n",
            "|    total_timesteps    | 1556000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.67     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77799    |\n",
            "|    policy_loss        | 0.221    |\n",
            "|    std                | 0.0264   |\n",
            "|    value_loss         | 0.00652  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.16    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 77900    |\n",
            "|    time_elapsed       | 5491     |\n",
            "|    total_timesteps    | 1558000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.67     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77899    |\n",
            "|    policy_loss        | 0.139    |\n",
            "|    std                | 0.0264   |\n",
            "|    value_loss         | 0.00219  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.98    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78000    |\n",
            "|    time_elapsed       | 5496     |\n",
            "|    total_timesteps    | 1560000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.68     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 77999    |\n",
            "|    policy_loss        | 0.156    |\n",
            "|    std                | 0.0263   |\n",
            "|    value_loss         | 0.00357  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.86    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78100    |\n",
            "|    time_elapsed       | 5504     |\n",
            "|    total_timesteps    | 1562000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.915    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78099    |\n",
            "|    policy_loss        | -0.0704  |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.00208  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.74    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78200    |\n",
            "|    time_elapsed       | 5509     |\n",
            "|    total_timesteps    | 1564000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.69     |\n",
            "|    explained_variance | 0.981    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78199    |\n",
            "|    policy_loss        | -0.0218  |\n",
            "|    std                | 0.0262   |\n",
            "|    value_loss         | 0.00379  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.67    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78300    |\n",
            "|    time_elapsed       | 5516     |\n",
            "|    total_timesteps    | 1566000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.68     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78299    |\n",
            "|    policy_loss        | -0.102   |\n",
            "|    std                | 0.0263   |\n",
            "|    value_loss         | 0.00551  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.61    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78400    |\n",
            "|    time_elapsed       | 5522     |\n",
            "|    total_timesteps    | 1568000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78399    |\n",
            "|    policy_loss        | -0.0335  |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.00389  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.54    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78500    |\n",
            "|    time_elapsed       | 5529     |\n",
            "|    total_timesteps    | 1570000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.983    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78499    |\n",
            "|    policy_loss        | 0.15     |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.0032   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.67    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78600    |\n",
            "|    time_elapsed       | 5537     |\n",
            "|    total_timesteps    | 1572000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78599    |\n",
            "|    policy_loss        | -0.0712  |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.00043  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.65    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78700    |\n",
            "|    time_elapsed       | 5543     |\n",
            "|    total_timesteps    | 1574000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78699    |\n",
            "|    policy_loss        | -0.0886  |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.00182  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.52    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78800    |\n",
            "|    time_elapsed       | 5551     |\n",
            "|    total_timesteps    | 1576000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78799    |\n",
            "|    policy_loss        | 0.0115   |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.000317 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.26    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 78900    |\n",
            "|    time_elapsed       | 5557     |\n",
            "|    total_timesteps    | 1578000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.71     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78899    |\n",
            "|    policy_loss        | 0.225    |\n",
            "|    std                | 0.026    |\n",
            "|    value_loss         | 0.00544  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.6     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79000    |\n",
            "|    time_elapsed       | 5566     |\n",
            "|    total_timesteps    | 1580000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 78999    |\n",
            "|    policy_loss        | -0.275   |\n",
            "|    std                | 0.026    |\n",
            "|    value_loss         | 0.00699  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.94    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79100    |\n",
            "|    time_elapsed       | 5572     |\n",
            "|    total_timesteps    | 1582000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.72     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79099    |\n",
            "|    policy_loss        | 0.0851   |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.000618 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.8     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79200    |\n",
            "|    time_elapsed       | 5580     |\n",
            "|    total_timesteps    | 1584000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.72     |\n",
            "|    explained_variance | 0.963    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79199    |\n",
            "|    policy_loss        | 0.585    |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.04     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.64    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79300    |\n",
            "|    time_elapsed       | 5586     |\n",
            "|    total_timesteps    | 1586000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.7      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79299    |\n",
            "|    policy_loss        | 0.355    |\n",
            "|    std                | 0.0261   |\n",
            "|    value_loss         | 0.013    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.14    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79400    |\n",
            "|    time_elapsed       | 5593     |\n",
            "|    total_timesteps    | 1588000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.71     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79399    |\n",
            "|    policy_loss        | 0.74     |\n",
            "|    std                | 0.026    |\n",
            "|    value_loss         | 0.0348   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.68    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79500    |\n",
            "|    time_elapsed       | 5600     |\n",
            "|    total_timesteps    | 1590000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.73     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79499    |\n",
            "|    policy_loss        | -0.0673  |\n",
            "|    std                | 0.0259   |\n",
            "|    value_loss         | 0.0271   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.06    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79600    |\n",
            "|    time_elapsed       | 5606     |\n",
            "|    total_timesteps    | 1592000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.75     |\n",
            "|    explained_variance | 0.973    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79599    |\n",
            "|    policy_loss        | 0.197    |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.0145   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.37    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79700    |\n",
            "|    time_elapsed       | 5614     |\n",
            "|    total_timesteps    | 1594000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79699    |\n",
            "|    policy_loss        | -0.705   |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.0237   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.07    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 79800    |\n",
            "|    time_elapsed       | 5619     |\n",
            "|    total_timesteps    | 1596000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79799    |\n",
            "|    policy_loss        | 0.128    |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.0268   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.1     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 79900    |\n",
            "|    time_elapsed       | 5626     |\n",
            "|    total_timesteps    | 1598000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79899    |\n",
            "|    policy_loss        | -0.0996  |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.00436  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.78    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 80000    |\n",
            "|    time_elapsed       | 5633     |\n",
            "|    total_timesteps    | 1600000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 79999    |\n",
            "|    policy_loss        | 0.128    |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.00333  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.29    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 80100    |\n",
            "|    time_elapsed       | 5639     |\n",
            "|    total_timesteps    | 1602000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80099    |\n",
            "|    policy_loss        | 0.0273   |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.000451 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.76    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 80200    |\n",
            "|    time_elapsed       | 5647     |\n",
            "|    total_timesteps    | 1604000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80199    |\n",
            "|    policy_loss        | 0.041    |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.000793 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.44    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 80300    |\n",
            "|    time_elapsed       | 5653     |\n",
            "|    total_timesteps    | 1606000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80299    |\n",
            "|    policy_loss        | -0.00179 |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.000596 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.46    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 80400    |\n",
            "|    time_elapsed       | 5662     |\n",
            "|    total_timesteps    | 1608000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80399    |\n",
            "|    policy_loss        | 0.031    |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.000362 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.65    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 80500    |\n",
            "|    time_elapsed       | 5668     |\n",
            "|    total_timesteps    | 1610000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80499    |\n",
            "|    policy_loss        | -0.0913  |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.0036   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.95    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 80600    |\n",
            "|    time_elapsed       | 5676     |\n",
            "|    total_timesteps    | 1612000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80599    |\n",
            "|    policy_loss        | -0.0986  |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00143  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.83    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 80700    |\n",
            "|    time_elapsed       | 5682     |\n",
            "|    total_timesteps    | 1614000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80699    |\n",
            "|    policy_loss        | -0.161   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.005    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.79    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 80800    |\n",
            "|    time_elapsed       | 5689     |\n",
            "|    total_timesteps    | 1616000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80799    |\n",
            "|    policy_loss        | -0.284   |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0215   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.87    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 80900    |\n",
            "|    time_elapsed       | 5696     |\n",
            "|    total_timesteps    | 1618000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80899    |\n",
            "|    policy_loss        | -0.0164  |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 2.64e-05 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.76    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 81000    |\n",
            "|    time_elapsed       | 5703     |\n",
            "|    total_timesteps    | 1620000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.87     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 80999    |\n",
            "|    policy_loss        | -0.00623 |\n",
            "|    std                | 0.0247   |\n",
            "|    value_loss         | 2.29e-05 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.44    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 81100    |\n",
            "|    time_elapsed       | 5711     |\n",
            "|    total_timesteps    | 1622000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81099    |\n",
            "|    policy_loss        | -0.0341  |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.000384 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.24    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 81200    |\n",
            "|    time_elapsed       | 5717     |\n",
            "|    total_timesteps    | 1624000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81199    |\n",
            "|    policy_loss        | -0.066   |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 0.00697  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.35    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 81300    |\n",
            "|    time_elapsed       | 5725     |\n",
            "|    total_timesteps    | 1626000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.87     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81299    |\n",
            "|    policy_loss        | 0.0297   |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.000601 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.35    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 81400    |\n",
            "|    time_elapsed       | 5731     |\n",
            "|    total_timesteps    | 1628000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81399    |\n",
            "|    policy_loss        | -0.0378  |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 0.000147 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.16    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 81500    |\n",
            "|    time_elapsed       | 5740     |\n",
            "|    total_timesteps    | 1630000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81499    |\n",
            "|    policy_loss        | -0.0483  |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.000221 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -1.76    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 81600    |\n",
            "|    time_elapsed       | 5746     |\n",
            "|    total_timesteps    | 1632000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.87     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81599    |\n",
            "|    policy_loss        | 0.0181   |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.000109 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -1.79    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 81700    |\n",
            "|    time_elapsed       | 5755     |\n",
            "|    total_timesteps    | 1634000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.86     |\n",
            "|    explained_variance | 0.974    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81699    |\n",
            "|    policy_loss        | 0.029    |\n",
            "|    std                | 0.0247   |\n",
            "|    value_loss         | 0.00215  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.11    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 81800    |\n",
            "|    time_elapsed       | 5762     |\n",
            "|    total_timesteps    | 1636000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.937    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81799    |\n",
            "|    policy_loss        | 0.433    |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0414   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.6     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 81900    |\n",
            "|    time_elapsed       | 5771     |\n",
            "|    total_timesteps    | 1638000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.968    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81899    |\n",
            "|    policy_loss        | -0.756   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0266   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.68    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82000    |\n",
            "|    time_elapsed       | 5777     |\n",
            "|    total_timesteps    | 1640000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 81999    |\n",
            "|    policy_loss        | -0.0704  |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00102  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.39    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82100    |\n",
            "|    time_elapsed       | 5785     |\n",
            "|    total_timesteps    | 1642000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.839    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82099    |\n",
            "|    policy_loss        | -0.635   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0256   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.28    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82200    |\n",
            "|    time_elapsed       | 5792     |\n",
            "|    total_timesteps    | 1644000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82199    |\n",
            "|    policy_loss        | 0.15     |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00277  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.2     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82300    |\n",
            "|    time_elapsed       | 5799     |\n",
            "|    total_timesteps    | 1646000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82299    |\n",
            "|    policy_loss        | 0.0471   |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00377  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.13    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82400    |\n",
            "|    time_elapsed       | 5806     |\n",
            "|    total_timesteps    | 1648000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82399    |\n",
            "|    policy_loss        | 0.272    |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.0105   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.25    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82500    |\n",
            "|    time_elapsed       | 5812     |\n",
            "|    total_timesteps    | 1650000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82499    |\n",
            "|    policy_loss        | 0.137    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00206  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.85    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82600    |\n",
            "|    time_elapsed       | 5821     |\n",
            "|    total_timesteps    | 1652000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.987    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82599    |\n",
            "|    policy_loss        | 0.068    |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00078  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.73    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82700    |\n",
            "|    time_elapsed       | 5827     |\n",
            "|    total_timesteps    | 1654000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82699    |\n",
            "|    policy_loss        | 0.0326   |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00114  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.95    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82800    |\n",
            "|    time_elapsed       | 5835     |\n",
            "|    total_timesteps    | 1656000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82799    |\n",
            "|    policy_loss        | -0.00733 |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.000898 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3       |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 82900    |\n",
            "|    time_elapsed       | 5841     |\n",
            "|    total_timesteps    | 1658000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82899    |\n",
            "|    policy_loss        | 0.106    |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00145  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.54    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83000    |\n",
            "|    time_elapsed       | 5850     |\n",
            "|    total_timesteps    | 1660000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 82999    |\n",
            "|    policy_loss        | 0.366    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0152   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.57    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83100    |\n",
            "|    time_elapsed       | 5856     |\n",
            "|    total_timesteps    | 1662000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83099    |\n",
            "|    policy_loss        | 0.24     |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0135   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.47    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83200    |\n",
            "|    time_elapsed       | 5863     |\n",
            "|    total_timesteps    | 1664000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83199    |\n",
            "|    policy_loss        | 0.13     |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00367  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.05    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83300    |\n",
            "|    time_elapsed       | 5870     |\n",
            "|    total_timesteps    | 1666000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83299    |\n",
            "|    policy_loss        | 0.127    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00459  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.06    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83400    |\n",
            "|    time_elapsed       | 5876     |\n",
            "|    total_timesteps    | 1668000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83399    |\n",
            "|    policy_loss        | -0.188   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00792  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.89    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83500    |\n",
            "|    time_elapsed       | 5883     |\n",
            "|    total_timesteps    | 1670000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.969    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83499    |\n",
            "|    policy_loss        | -0.177   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.015    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.88    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83600    |\n",
            "|    time_elapsed       | 5889     |\n",
            "|    total_timesteps    | 1672000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83599    |\n",
            "|    policy_loss        | 0.0502   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00464  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -2.97    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83700    |\n",
            "|    time_elapsed       | 5897     |\n",
            "|    total_timesteps    | 1674000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83699    |\n",
            "|    policy_loss        | 0.0152   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00302  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.06    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83800    |\n",
            "|    time_elapsed       | 5903     |\n",
            "|    total_timesteps    | 1676000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83799    |\n",
            "|    policy_loss        | -0.564   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0162   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.1     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 83900    |\n",
            "|    time_elapsed       | 5911     |\n",
            "|    total_timesteps    | 1678000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83899    |\n",
            "|    policy_loss        | 0.258    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00637  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.11    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84000    |\n",
            "|    time_elapsed       | 5918     |\n",
            "|    total_timesteps    | 1680000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 83999    |\n",
            "|    policy_loss        | 0.198    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00234  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.22    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84100    |\n",
            "|    time_elapsed       | 5925     |\n",
            "|    total_timesteps    | 1682000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84099    |\n",
            "|    policy_loss        | -0.00381 |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0038   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.4     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84200    |\n",
            "|    time_elapsed       | 5932     |\n",
            "|    total_timesteps    | 1684000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84199    |\n",
            "|    policy_loss        | 0.0446   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0106   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.72    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84300    |\n",
            "|    time_elapsed       | 5939     |\n",
            "|    total_timesteps    | 1686000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84299    |\n",
            "|    policy_loss        | 0.0552   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0217   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.02    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84400    |\n",
            "|    time_elapsed       | 5947     |\n",
            "|    total_timesteps    | 1688000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.99     |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84399    |\n",
            "|    policy_loss        | 0.0962   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00439  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.94    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84500    |\n",
            "|    time_elapsed       | 5954     |\n",
            "|    total_timesteps    | 1690000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84499    |\n",
            "|    policy_loss        | -0.76    |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0287   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.96    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84600    |\n",
            "|    time_elapsed       | 5962     |\n",
            "|    total_timesteps    | 1692000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84599    |\n",
            "|    policy_loss        | -0.112   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0132   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.23    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84700    |\n",
            "|    time_elapsed       | 5968     |\n",
            "|    total_timesteps    | 1694000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.982    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84699    |\n",
            "|    policy_loss        | -0.576   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0457   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.3     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84800    |\n",
            "|    time_elapsed       | 5976     |\n",
            "|    total_timesteps    | 1696000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.86     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84799    |\n",
            "|    policy_loss        | -0.174   |\n",
            "|    std                | 0.0247   |\n",
            "|    value_loss         | 0.0173   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.46    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 84900    |\n",
            "|    time_elapsed       | 5982     |\n",
            "|    total_timesteps    | 1698000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84899    |\n",
            "|    policy_loss        | -0.449   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0225   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.84    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85000    |\n",
            "|    time_elapsed       | 5991     |\n",
            "|    total_timesteps    | 1700000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 84999    |\n",
            "|    policy_loss        | -0.0874  |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0125   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.07    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85100    |\n",
            "|    time_elapsed       | 5997     |\n",
            "|    total_timesteps    | 1702000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.979    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85099    |\n",
            "|    policy_loss        | 0.756    |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0463   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.94    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85200    |\n",
            "|    time_elapsed       | 6005     |\n",
            "|    total_timesteps    | 1704000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.979    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85199    |\n",
            "|    policy_loss        | -0.0735  |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00961  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.31    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85300    |\n",
            "|    time_elapsed       | 6012     |\n",
            "|    total_timesteps    | 1706000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.923    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85299    |\n",
            "|    policy_loss        | 0.214    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0069   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.14    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85400    |\n",
            "|    time_elapsed       | 6020     |\n",
            "|    total_timesteps    | 1708000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.975    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85399    |\n",
            "|    policy_loss        | 0.235    |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00357  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.85    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85500    |\n",
            "|    time_elapsed       | 6027     |\n",
            "|    total_timesteps    | 1710000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85499    |\n",
            "|    policy_loss        | 0.362    |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0205   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.88    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85600    |\n",
            "|    time_elapsed       | 6034     |\n",
            "|    total_timesteps    | 1712000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85599    |\n",
            "|    policy_loss        | 0.66     |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0324   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.98    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85700    |\n",
            "|    time_elapsed       | 6042     |\n",
            "|    total_timesteps    | 1714000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85699    |\n",
            "|    policy_loss        | -0.0309  |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.000941 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.87    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85800    |\n",
            "|    time_elapsed       | 6048     |\n",
            "|    total_timesteps    | 1716000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85799    |\n",
            "|    policy_loss        | -0.0888  |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0137   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.77    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 85900    |\n",
            "|    time_elapsed       | 6056     |\n",
            "|    total_timesteps    | 1718000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.984    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85899    |\n",
            "|    policy_loss        | 0.0751   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00326  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.85    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86000    |\n",
            "|    time_elapsed       | 6061     |\n",
            "|    total_timesteps    | 1720000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 85999    |\n",
            "|    policy_loss        | 0.175    |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00602  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.85    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86100    |\n",
            "|    time_elapsed       | 6069     |\n",
            "|    total_timesteps    | 1722000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86099    |\n",
            "|    policy_loss        | -0.143   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00263  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.79    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86200    |\n",
            "|    time_elapsed       | 6075     |\n",
            "|    total_timesteps    | 1724000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86199    |\n",
            "|    policy_loss        | 0.433    |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.016    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.83    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86300    |\n",
            "|    time_elapsed       | 6081     |\n",
            "|    total_timesteps    | 1726000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86299    |\n",
            "|    policy_loss        | -0.264   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00293  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.9     |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86400    |\n",
            "|    time_elapsed       | 6088     |\n",
            "|    total_timesteps    | 1728000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.949    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86399    |\n",
            "|    policy_loss        | -0.164   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00481  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.75    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86500    |\n",
            "|    time_elapsed       | 6093     |\n",
            "|    total_timesteps    | 1730000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86499    |\n",
            "|    policy_loss        | 0.185    |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00404  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.58    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86600    |\n",
            "|    time_elapsed       | 6100     |\n",
            "|    total_timesteps    | 1732000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.985    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86599    |\n",
            "|    policy_loss        | -0.115   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0021   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.57    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86700    |\n",
            "|    time_elapsed       | 6106     |\n",
            "|    total_timesteps    | 1734000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86699    |\n",
            "|    policy_loss        | -0.458   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00778  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.7     |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 86800    |\n",
            "|    time_elapsed       | 6112     |\n",
            "|    total_timesteps    | 1736000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.971    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86799    |\n",
            "|    policy_loss        | -0.269   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.012    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.87    |\n",
            "| time/                 |          |\n",
            "|    fps                | 283      |\n",
            "|    iterations         | 86900    |\n",
            "|    time_elapsed       | 6119     |\n",
            "|    total_timesteps    | 1738000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86899    |\n",
            "|    policy_loss        | -0.05    |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00397  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.99    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87000    |\n",
            "|    time_elapsed       | 6124     |\n",
            "|    total_timesteps    | 1740000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 86999    |\n",
            "|    policy_loss        | 0.198    |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00301  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.89    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87100    |\n",
            "|    time_elapsed       | 6131     |\n",
            "|    total_timesteps    | 1742000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.86     |\n",
            "|    explained_variance | 0.676    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87099    |\n",
            "|    policy_loss        | 0.109    |\n",
            "|    std                | 0.0247   |\n",
            "|    value_loss         | 0.00457  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.67    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87200    |\n",
            "|    time_elapsed       | 6137     |\n",
            "|    total_timesteps    | 1744000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.87     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87199    |\n",
            "|    policy_loss        | -0.36    |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.00768  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.6     |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87300    |\n",
            "|    time_elapsed       | 6142     |\n",
            "|    total_timesteps    | 1746000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.87     |\n",
            "|    explained_variance | 0.983    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87299    |\n",
            "|    policy_loss        | -0.314   |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.00618  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.61    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87400    |\n",
            "|    time_elapsed       | 6150     |\n",
            "|    total_timesteps    | 1748000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.9      |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87399    |\n",
            "|    policy_loss        | 0.102    |\n",
            "|    std                | 0.0244   |\n",
            "|    value_loss         | 0.0123   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.66    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87500    |\n",
            "|    time_elapsed       | 6155     |\n",
            "|    total_timesteps    | 1750000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.92     |\n",
            "|    explained_variance | 0.983    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87499    |\n",
            "|    policy_loss        | 0.149    |\n",
            "|    std                | 0.0242   |\n",
            "|    value_loss         | 0.0042   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.7     |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87600    |\n",
            "|    time_elapsed       | 6161     |\n",
            "|    total_timesteps    | 1752000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.94     |\n",
            "|    explained_variance | 0.964    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87599    |\n",
            "|    policy_loss        | -0.213   |\n",
            "|    std                | 0.024    |\n",
            "|    value_loss         | 0.0042   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.59    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87700    |\n",
            "|    time_elapsed       | 6168     |\n",
            "|    total_timesteps    | 1754000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.94     |\n",
            "|    explained_variance | 0.974    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87699    |\n",
            "|    policy_loss        | 0.318    |\n",
            "|    std                | 0.024    |\n",
            "|    value_loss         | 0.00633  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.53    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87800    |\n",
            "|    time_elapsed       | 6173     |\n",
            "|    total_timesteps    | 1756000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.96     |\n",
            "|    explained_variance | 0.972    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87799    |\n",
            "|    policy_loss        | 0.0168   |\n",
            "|    std                | 0.0239   |\n",
            "|    value_loss         | 0.00302  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.54    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 87900    |\n",
            "|    time_elapsed       | 6181     |\n",
            "|    total_timesteps    | 1758000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.97     |\n",
            "|    explained_variance | 0.978    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87899    |\n",
            "|    policy_loss        | -0.0702  |\n",
            "|    std                | 0.0238   |\n",
            "|    value_loss         | 0.00398  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.84    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88000    |\n",
            "|    time_elapsed       | 6187     |\n",
            "|    total_timesteps    | 1760000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.95     |\n",
            "|    explained_variance | 0.969    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 87999    |\n",
            "|    policy_loss        | 0.875    |\n",
            "|    std                | 0.0239   |\n",
            "|    value_loss         | 0.0372   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.01    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88100    |\n",
            "|    time_elapsed       | 6193     |\n",
            "|    total_timesteps    | 1762000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.96     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88099    |\n",
            "|    policy_loss        | 0.178    |\n",
            "|    std                | 0.0239   |\n",
            "|    value_loss         | 0.00389  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.05    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88200    |\n",
            "|    time_elapsed       | 6200     |\n",
            "|    total_timesteps    | 1764000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.95     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88199    |\n",
            "|    policy_loss        | -0.176   |\n",
            "|    std                | 0.024    |\n",
            "|    value_loss         | 0.00479  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.83    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88300    |\n",
            "|    time_elapsed       | 6205     |\n",
            "|    total_timesteps    | 1766000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.92     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88299    |\n",
            "|    policy_loss        | -0.12    |\n",
            "|    std                | 0.0242   |\n",
            "|    value_loss         | 0.00157  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.66    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88400    |\n",
            "|    time_elapsed       | 6213     |\n",
            "|    total_timesteps    | 1768000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.94     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88399    |\n",
            "|    policy_loss        | -0.0855  |\n",
            "|    std                | 0.0241   |\n",
            "|    value_loss         | 0.0033   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.44    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88500    |\n",
            "|    time_elapsed       | 6218     |\n",
            "|    total_timesteps    | 1770000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.93     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88499    |\n",
            "|    policy_loss        | -0.0196  |\n",
            "|    std                | 0.0241   |\n",
            "|    value_loss         | 0.00241  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.47    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88600    |\n",
            "|    time_elapsed       | 6225     |\n",
            "|    total_timesteps    | 1772000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.92     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88599    |\n",
            "|    policy_loss        | 0.169    |\n",
            "|    std                | 0.0242   |\n",
            "|    value_loss         | 0.00368  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.47    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88700    |\n",
            "|    time_elapsed       | 6232     |\n",
            "|    total_timesteps    | 1774000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.93     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88699    |\n",
            "|    policy_loss        | -0.0638  |\n",
            "|    std                | 0.0241   |\n",
            "|    value_loss         | 0.00711  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.38    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88800    |\n",
            "|    time_elapsed       | 6237     |\n",
            "|    total_timesteps    | 1776000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.93     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88799    |\n",
            "|    policy_loss        | 0.478    |\n",
            "|    std                | 0.0241   |\n",
            "|    value_loss         | 0.00916  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.48    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 88900    |\n",
            "|    time_elapsed       | 6244     |\n",
            "|    total_timesteps    | 1778000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.95     |\n",
            "|    explained_variance | 0.959    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88899    |\n",
            "|    policy_loss        | 0.0479   |\n",
            "|    std                | 0.024    |\n",
            "|    value_loss         | 0.00585  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.6     |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 89000    |\n",
            "|    time_elapsed       | 6250     |\n",
            "|    total_timesteps    | 1780000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.92     |\n",
            "|    explained_variance | 0.906    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 88999    |\n",
            "|    policy_loss        | 0.196    |\n",
            "|    std                | 0.0242   |\n",
            "|    value_loss         | 0.00718  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.69    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 89100    |\n",
            "|    time_elapsed       | 6256     |\n",
            "|    total_timesteps    | 1782000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.93     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89099    |\n",
            "|    policy_loss        | 0.216    |\n",
            "|    std                | 0.0241   |\n",
            "|    value_loss         | 0.00427  |\n",
            "------------------------------------\n",
            "-------------------------------------\n",
            "| rollout/              |           |\n",
            "|    ep_len_mean        | 50        |\n",
            "|    ep_rew_mean        | -3.6      |\n",
            "| time/                 |           |\n",
            "|    fps                | 284       |\n",
            "|    iterations         | 89200     |\n",
            "|    time_elapsed       | 6263      |\n",
            "|    total_timesteps    | 1784000   |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | 6.93      |\n",
            "|    explained_variance | 0.995     |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 89199     |\n",
            "|    policy_loss        | -0.000369 |\n",
            "|    std                | 0.0241    |\n",
            "|    value_loss         | 0.00381   |\n",
            "-------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.43    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 89300    |\n",
            "|    time_elapsed       | 6268     |\n",
            "|    total_timesteps    | 1786000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.94     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89299    |\n",
            "|    policy_loss        | 0.191    |\n",
            "|    std                | 0.0241   |\n",
            "|    value_loss         | 0.00234  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.23    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 89400    |\n",
            "|    time_elapsed       | 6276     |\n",
            "|    total_timesteps    | 1788000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.95     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89399    |\n",
            "|    policy_loss        | -0.0956  |\n",
            "|    std                | 0.0239   |\n",
            "|    value_loss         | 0.00139  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.1     |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 89500    |\n",
            "|    time_elapsed       | 6282     |\n",
            "|    total_timesteps    | 1790000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.94     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89499    |\n",
            "|    policy_loss        | -0.165   |\n",
            "|    std                | 0.0241   |\n",
            "|    value_loss         | 0.00344  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.08    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 89600    |\n",
            "|    time_elapsed       | 6288     |\n",
            "|    total_timesteps    | 1792000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.94     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89599    |\n",
            "|    policy_loss        | -0.202   |\n",
            "|    std                | 0.0241   |\n",
            "|    value_loss         | 0.0042   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.19    |\n",
            "| time/                 |          |\n",
            "|    fps                | 284      |\n",
            "|    iterations         | 89700    |\n",
            "|    time_elapsed       | 6294     |\n",
            "|    total_timesteps    | 1794000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.92     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89699    |\n",
            "|    policy_loss        | -0.315   |\n",
            "|    std                | 0.0242   |\n",
            "|    value_loss         | 0.00431  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.15    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 89800    |\n",
            "|    time_elapsed       | 6300     |\n",
            "|    total_timesteps    | 1796000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.91     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89799    |\n",
            "|    policy_loss        | -0.015   |\n",
            "|    std                | 0.0243   |\n",
            "|    value_loss         | 0.000312 |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.11    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 89900    |\n",
            "|    time_elapsed       | 6307     |\n",
            "|    total_timesteps    | 1798000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.9      |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89899    |\n",
            "|    policy_loss        | 0.24     |\n",
            "|    std                | 0.0244   |\n",
            "|    value_loss         | 0.00801  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.27    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90000    |\n",
            "|    time_elapsed       | 6313     |\n",
            "|    total_timesteps    | 1800000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.91     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 89999    |\n",
            "|    policy_loss        | -0.188   |\n",
            "|    std                | 0.0243   |\n",
            "|    value_loss         | 0.00351  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.65    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90100    |\n",
            "|    time_elapsed       | 6319     |\n",
            "|    total_timesteps    | 1802000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.9      |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90099    |\n",
            "|    policy_loss        | -0.226   |\n",
            "|    std                | 0.0244   |\n",
            "|    value_loss         | 0.0118   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.77    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90200    |\n",
            "|    time_elapsed       | 6326     |\n",
            "|    total_timesteps    | 1804000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.9      |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90199    |\n",
            "|    policy_loss        | -0.0058  |\n",
            "|    std                | 0.0244   |\n",
            "|    value_loss         | 0.0108   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.9     |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90300    |\n",
            "|    time_elapsed       | 6331     |\n",
            "|    total_timesteps    | 1806000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90299    |\n",
            "|    policy_loss        | 0.323    |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 0.00816  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.76    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90400    |\n",
            "|    time_elapsed       | 6339     |\n",
            "|    total_timesteps    | 1808000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.89     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90399    |\n",
            "|    policy_loss        | 0.646    |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 0.0281   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.72    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90500    |\n",
            "|    time_elapsed       | 6345     |\n",
            "|    total_timesteps    | 1810000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90499    |\n",
            "|    policy_loss        | 0.265    |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 0.00933  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.56    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90600    |\n",
            "|    time_elapsed       | 6351     |\n",
            "|    total_timesteps    | 1812000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.9      |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90599    |\n",
            "|    policy_loss        | -0.00662 |\n",
            "|    std                | 0.0244   |\n",
            "|    value_loss         | 0.00112  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.54    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90700    |\n",
            "|    time_elapsed       | 6358     |\n",
            "|    total_timesteps    | 1814000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90699    |\n",
            "|    policy_loss        | 0.0959   |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 0.00489  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.52    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90800    |\n",
            "|    time_elapsed       | 6363     |\n",
            "|    total_timesteps    | 1816000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90799    |\n",
            "|    policy_loss        | 0.0401   |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 0.00401  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.68    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 90900    |\n",
            "|    time_elapsed       | 6371     |\n",
            "|    total_timesteps    | 1818000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90899    |\n",
            "|    policy_loss        | -0.345   |\n",
            "|    std                | 0.0245   |\n",
            "|    value_loss         | 0.00443  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.73    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91000    |\n",
            "|    time_elapsed       | 6376     |\n",
            "|    total_timesteps    | 1820000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.88     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 90999    |\n",
            "|    policy_loss        | 0.0627   |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.00237  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.74    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91100    |\n",
            "|    time_elapsed       | 6382     |\n",
            "|    total_timesteps    | 1822000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.87     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91099    |\n",
            "|    policy_loss        | -0.0483  |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.00379  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -3.67    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91200    |\n",
            "|    time_elapsed       | 6389     |\n",
            "|    total_timesteps    | 1824000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.87     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91199    |\n",
            "|    policy_loss        | 0.0768   |\n",
            "|    std                | 0.0246   |\n",
            "|    value_loss         | 0.00796  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.21    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91300    |\n",
            "|    time_elapsed       | 6394     |\n",
            "|    total_timesteps    | 1826000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91299    |\n",
            "|    policy_loss        | -0.391   |\n",
            "|    std                | 0.0247   |\n",
            "|    value_loss         | 0.0149   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.73    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91400    |\n",
            "|    time_elapsed       | 6402     |\n",
            "|    total_timesteps    | 1828000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91399    |\n",
            "|    policy_loss        | -0.645   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0163   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.82    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91500    |\n",
            "|    time_elapsed       | 6408     |\n",
            "|    total_timesteps    | 1830000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.96     |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91499    |\n",
            "|    policy_loss        | 0.151    |\n",
            "|    std                | 0.0247   |\n",
            "|    value_loss         | 0.0223   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.49    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91600    |\n",
            "|    time_elapsed       | 6414     |\n",
            "|    total_timesteps    | 1832000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91599    |\n",
            "|    policy_loss        | -0.211   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00505  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.21    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91700    |\n",
            "|    time_elapsed       | 6421     |\n",
            "|    total_timesteps    | 1834000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91699    |\n",
            "|    policy_loss        | 0.128    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00748  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.16    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91800    |\n",
            "|    time_elapsed       | 6426     |\n",
            "|    total_timesteps    | 1836000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.981    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91799    |\n",
            "|    policy_loss        | -0.191   |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0375   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.76    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 91900    |\n",
            "|    time_elapsed       | 6434     |\n",
            "|    total_timesteps    | 1838000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.943    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91899    |\n",
            "|    policy_loss        | 0.0542   |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0389   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.05    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 92000    |\n",
            "|    time_elapsed       | 6440     |\n",
            "|    total_timesteps    | 1840000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.989    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 91999    |\n",
            "|    policy_loss        | 0.191    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0318   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.1     |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 92100    |\n",
            "|    time_elapsed       | 6446     |\n",
            "|    total_timesteps    | 1842000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.94     |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92099    |\n",
            "|    policy_loss        | -0.25    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0682   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.15    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 92200    |\n",
            "|    time_elapsed       | 6453     |\n",
            "|    total_timesteps    | 1844000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92199    |\n",
            "|    policy_loss        | 0.33     |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0301   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.24    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 92300    |\n",
            "|    time_elapsed       | 6458     |\n",
            "|    total_timesteps    | 1846000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92299    |\n",
            "|    policy_loss        | -0.375   |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00944  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.91    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 92400    |\n",
            "|    time_elapsed       | 6466     |\n",
            "|    total_timesteps    | 1848000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92399    |\n",
            "|    policy_loss        | 0.0909   |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.00739  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.93    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 92500    |\n",
            "|    time_elapsed       | 6471     |\n",
            "|    total_timesteps    | 1850000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92499    |\n",
            "|    policy_loss        | 0.134    |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00947  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.08    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 92600    |\n",
            "|    time_elapsed       | 6477     |\n",
            "|    total_timesteps    | 1852000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92599    |\n",
            "|    policy_loss        | 0.223    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0213   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.13    |\n",
            "| time/                 |          |\n",
            "|    fps                | 285      |\n",
            "|    iterations         | 92700    |\n",
            "|    time_elapsed       | 6484     |\n",
            "|    total_timesteps    | 1854000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92699    |\n",
            "|    policy_loss        | 0.0974   |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00499  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.13    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 92800    |\n",
            "|    time_elapsed       | 6489     |\n",
            "|    total_timesteps    | 1856000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92799    |\n",
            "|    policy_loss        | -0.774   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0306   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.94    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 92900    |\n",
            "|    time_elapsed       | 6496     |\n",
            "|    total_timesteps    | 1858000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.959    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92899    |\n",
            "|    policy_loss        | 0.255    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00829  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.04    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93000    |\n",
            "|    time_elapsed       | 6501     |\n",
            "|    total_timesteps    | 1860000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 92999    |\n",
            "|    policy_loss        | 0.447    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0194   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.71    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93100    |\n",
            "|    time_elapsed       | 6506     |\n",
            "|    total_timesteps    | 1862000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93099    |\n",
            "|    policy_loss        | 0.613    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0489   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.47    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93200    |\n",
            "|    time_elapsed       | 6514     |\n",
            "|    total_timesteps    | 1864000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93199    |\n",
            "|    policy_loss        | -0.0248  |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0255   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.4     |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93300    |\n",
            "|    time_elapsed       | 6519     |\n",
            "|    total_timesteps    | 1866000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93299    |\n",
            "|    policy_loss        | -0.0944  |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00725  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.52    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93400    |\n",
            "|    time_elapsed       | 6525     |\n",
            "|    total_timesteps    | 1868000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.982    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93399    |\n",
            "|    policy_loss        | 0.305    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0188   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.7     |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93500    |\n",
            "|    time_elapsed       | 6532     |\n",
            "|    total_timesteps    | 1870000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93499    |\n",
            "|    policy_loss        | 0.291    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00616  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.52    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93600    |\n",
            "|    time_elapsed       | 6537     |\n",
            "|    total_timesteps    | 1872000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93599    |\n",
            "|    policy_loss        | -0.341   |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00598  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.01    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93700    |\n",
            "|    time_elapsed       | 6545     |\n",
            "|    total_timesteps    | 1874000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.982    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93699    |\n",
            "|    policy_loss        | 0.397    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.01     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.11    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93800    |\n",
            "|    time_elapsed       | 6551     |\n",
            "|    total_timesteps    | 1876000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93799    |\n",
            "|    policy_loss        | -0.131   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00499  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.49    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 93900    |\n",
            "|    time_elapsed       | 6557     |\n",
            "|    total_timesteps    | 1878000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93899    |\n",
            "|    policy_loss        | 0.488    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00788  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.84    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94000    |\n",
            "|    time_elapsed       | 6564     |\n",
            "|    total_timesteps    | 1880000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 93999    |\n",
            "|    policy_loss        | 0.112    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00789  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.5     |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94100    |\n",
            "|    time_elapsed       | 6569     |\n",
            "|    total_timesteps    | 1882000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.8      |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94099    |\n",
            "|    policy_loss        | -0.213   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0026   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.16    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94200    |\n",
            "|    time_elapsed       | 6577     |\n",
            "|    total_timesteps    | 1884000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94199    |\n",
            "|    policy_loss        | 0.013    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0147   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.28    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94300    |\n",
            "|    time_elapsed       | 6582     |\n",
            "|    total_timesteps    | 1886000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94299    |\n",
            "|    policy_loss        | 0.0833   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00366  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.55    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94400    |\n",
            "|    time_elapsed       | 6589     |\n",
            "|    total_timesteps    | 1888000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94399    |\n",
            "|    policy_loss        | -0.279   |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0077   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.96    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94500    |\n",
            "|    time_elapsed       | 6596     |\n",
            "|    total_timesteps    | 1890000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94499    |\n",
            "|    policy_loss        | 0.34     |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.033    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.12    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94600    |\n",
            "|    time_elapsed       | 6601     |\n",
            "|    total_timesteps    | 1892000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94599    |\n",
            "|    policy_loss        | 0.44     |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0144   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.48    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94700    |\n",
            "|    time_elapsed       | 6609     |\n",
            "|    total_timesteps    | 1894000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94699    |\n",
            "|    policy_loss        | 0.18     |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0121   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.32    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94800    |\n",
            "|    time_elapsed       | 6615     |\n",
            "|    total_timesteps    | 1896000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94799    |\n",
            "|    policy_loss        | -0.0115  |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0145   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.97    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 94900    |\n",
            "|    time_elapsed       | 6622     |\n",
            "|    total_timesteps    | 1898000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94899    |\n",
            "|    policy_loss        | -0.352   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.00676  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.54    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95000    |\n",
            "|    time_elapsed       | 6628     |\n",
            "|    total_timesteps    | 1900000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 94999    |\n",
            "|    policy_loss        | -0.168   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00572  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.65    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95100    |\n",
            "|    time_elapsed       | 6633     |\n",
            "|    total_timesteps    | 1902000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.987    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95099    |\n",
            "|    policy_loss        | -0.145   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0149   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.89    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95200    |\n",
            "|    time_elapsed       | 6641     |\n",
            "|    total_timesteps    | 1904000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.986    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95199    |\n",
            "|    policy_loss        | -0.00883 |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.011    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.02    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95300    |\n",
            "|    time_elapsed       | 6646     |\n",
            "|    total_timesteps    | 1906000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95299    |\n",
            "|    policy_loss        | 0.0851   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0129   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.26    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95400    |\n",
            "|    time_elapsed       | 6653     |\n",
            "|    total_timesteps    | 1908000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95399    |\n",
            "|    policy_loss        | -0.264   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0112   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.36    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95500    |\n",
            "|    time_elapsed       | 6659     |\n",
            "|    total_timesteps    | 1910000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.963    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95499    |\n",
            "|    policy_loss        | -1.04    |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0535   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.03    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95600    |\n",
            "|    time_elapsed       | 6665     |\n",
            "|    total_timesteps    | 1912000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95599    |\n",
            "|    policy_loss        | 0.29     |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0215   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.68    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95700    |\n",
            "|    time_elapsed       | 6672     |\n",
            "|    total_timesteps    | 1914000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.981    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95699    |\n",
            "|    policy_loss        | 0.117    |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.0169   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.69    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95800    |\n",
            "|    time_elapsed       | 6678     |\n",
            "|    total_timesteps    | 1916000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95799    |\n",
            "|    policy_loss        | 0.0337   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00571  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.87    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 95900    |\n",
            "|    time_elapsed       | 6683     |\n",
            "|    total_timesteps    | 1918000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.993    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95899    |\n",
            "|    policy_loss        | -0.0122  |\n",
            "|    std                | 0.0247   |\n",
            "|    value_loss         | 0.071    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.71    |\n",
            "| time/                 |          |\n",
            "|    fps                | 286      |\n",
            "|    iterations         | 96000    |\n",
            "|    time_elapsed       | 6690     |\n",
            "|    total_timesteps    | 1920000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.85     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 95999    |\n",
            "|    policy_loss        | -0.0866  |\n",
            "|    std                | 0.0247   |\n",
            "|    value_loss         | 0.0015   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.65    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96100    |\n",
            "|    time_elapsed       | 6695     |\n",
            "|    total_timesteps    | 1922000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.99     |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96099    |\n",
            "|    policy_loss        | 0.667    |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0194   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.61    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96200    |\n",
            "|    time_elapsed       | 6703     |\n",
            "|    total_timesteps    | 1924000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96199    |\n",
            "|    policy_loss        | 0.535    |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.0162   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.64    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96300    |\n",
            "|    time_elapsed       | 6708     |\n",
            "|    total_timesteps    | 1926000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96299    |\n",
            "|    policy_loss        | -0.0492  |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00396  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.6     |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96400    |\n",
            "|    time_elapsed       | 6713     |\n",
            "|    total_timesteps    | 1928000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96399    |\n",
            "|    policy_loss        | 1.15     |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.0357   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.6     |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96500    |\n",
            "|    time_elapsed       | 6721     |\n",
            "|    total_timesteps    | 1930000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96499    |\n",
            "|    policy_loss        | -0.223   |\n",
            "|    std                | 0.0248   |\n",
            "|    value_loss         | 0.00677  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.55    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96600    |\n",
            "|    time_elapsed       | 6726     |\n",
            "|    total_timesteps    | 1932000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96599    |\n",
            "|    policy_loss        | -0.281   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00474  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.3     |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96700    |\n",
            "|    time_elapsed       | 6733     |\n",
            "|    total_timesteps    | 1934000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.83     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96699    |\n",
            "|    policy_loss        | -0.261   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00276  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.13    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96800    |\n",
            "|    time_elapsed       | 6739     |\n",
            "|    total_timesteps    | 1936000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96799    |\n",
            "|    policy_loss        | 0.133    |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00864  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.14    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 96900    |\n",
            "|    time_elapsed       | 6744     |\n",
            "|    total_timesteps    | 1938000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.84     |\n",
            "|    explained_variance | 1        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96899    |\n",
            "|    policy_loss        | -0.117   |\n",
            "|    std                | 0.0249   |\n",
            "|    value_loss         | 0.00212  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.06    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97000    |\n",
            "|    time_elapsed       | 6752     |\n",
            "|    total_timesteps    | 1940000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.82     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 96999    |\n",
            "|    policy_loss        | -0.261   |\n",
            "|    std                | 0.025    |\n",
            "|    value_loss         | 0.011    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.18    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97100    |\n",
            "|    time_elapsed       | 6757     |\n",
            "|    total_timesteps    | 1942000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97099    |\n",
            "|    policy_loss        | -0.061   |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0105   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.26    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97200    |\n",
            "|    time_elapsed       | 6763     |\n",
            "|    total_timesteps    | 1944000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97199    |\n",
            "|    policy_loss        | -0.154   |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.0147   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.41    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97300    |\n",
            "|    time_elapsed       | 6769     |\n",
            "|    total_timesteps    | 1946000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97299    |\n",
            "|    policy_loss        | 0.372    |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.0288   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.36    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97400    |\n",
            "|    time_elapsed       | 6774     |\n",
            "|    total_timesteps    | 1948000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97399    |\n",
            "|    policy_loss        | 0.161    |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00225  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.36    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97500    |\n",
            "|    time_elapsed       | 6781     |\n",
            "|    total_timesteps    | 1950000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97499    |\n",
            "|    policy_loss        | 0.147    |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.0233   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.96    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97600    |\n",
            "|    time_elapsed       | 6787     |\n",
            "|    total_timesteps    | 1952000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.77     |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97599    |\n",
            "|    policy_loss        | -0.27    |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.0512   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.05    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97700    |\n",
            "|    time_elapsed       | 6792     |\n",
            "|    total_timesteps    | 1954000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.959    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97699    |\n",
            "|    policy_loss        | 0.42     |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.0395   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.24    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97800    |\n",
            "|    time_elapsed       | 6800     |\n",
            "|    total_timesteps    | 1956000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97799    |\n",
            "|    policy_loss        | 0.169    |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.0429   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.68    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 97900    |\n",
            "|    time_elapsed       | 6805     |\n",
            "|    total_timesteps    | 1958000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.973    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97899    |\n",
            "|    policy_loss        | -0.199   |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.02     |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.15    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 98000    |\n",
            "|    time_elapsed       | 6811     |\n",
            "|    total_timesteps    | 1960000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.973    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 97999    |\n",
            "|    policy_loss        | 0.00688  |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00773  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.17    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 98100    |\n",
            "|    time_elapsed       | 6817     |\n",
            "|    total_timesteps    | 1962000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98099    |\n",
            "|    policy_loss        | -0.274   |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00519  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.25    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 98200    |\n",
            "|    time_elapsed       | 6823     |\n",
            "|    total_timesteps    | 1964000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.81     |\n",
            "|    explained_variance | 0.979    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98199    |\n",
            "|    policy_loss        | -0.0905  |\n",
            "|    std                | 0.0251   |\n",
            "|    value_loss         | 0.00236  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.49    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 98300    |\n",
            "|    time_elapsed       | 6830     |\n",
            "|    total_timesteps    | 1966000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98299    |\n",
            "|    policy_loss        | -0.0867  |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.00571  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.75    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 98400    |\n",
            "|    time_elapsed       | 6835     |\n",
            "|    total_timesteps    | 1968000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98399    |\n",
            "|    policy_loss        | 0.108    |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.00442  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.85    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 98500    |\n",
            "|    time_elapsed       | 6841     |\n",
            "|    total_timesteps    | 1970000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98499    |\n",
            "|    policy_loss        | -0.599   |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.0336   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.15    |\n",
            "| time/                 |          |\n",
            "|    fps                | 287      |\n",
            "|    iterations         | 98600    |\n",
            "|    time_elapsed       | 6848     |\n",
            "|    total_timesteps    | 1972000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98599    |\n",
            "|    policy_loss        | -0.168   |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.00751  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.34    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 98700    |\n",
            "|    time_elapsed       | 6853     |\n",
            "|    total_timesteps    | 1974000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98699    |\n",
            "|    policy_loss        | -0.238   |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.00583  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.42    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 98800    |\n",
            "|    time_elapsed       | 6860     |\n",
            "|    total_timesteps    | 1976000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.977    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98799    |\n",
            "|    policy_loss        | -0.402   |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.019    |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.3     |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 98900    |\n",
            "|    time_elapsed       | 6866     |\n",
            "|    total_timesteps    | 1978000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.8      |\n",
            "|    explained_variance | 0.988    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98899    |\n",
            "|    policy_loss        | 0.357    |\n",
            "|    std                | 0.0252   |\n",
            "|    value_loss         | 0.0156   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -5.09    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99000    |\n",
            "|    time_elapsed       | 6871     |\n",
            "|    total_timesteps    | 1980000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 98999    |\n",
            "|    policy_loss        | 0.373    |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.00975  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.77    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99100    |\n",
            "|    time_elapsed       | 6878     |\n",
            "|    total_timesteps    | 1982000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99099    |\n",
            "|    policy_loss        | 0.0485   |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.00103  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.41    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99200    |\n",
            "|    time_elapsed       | 6883     |\n",
            "|    total_timesteps    | 1984000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.77     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99199    |\n",
            "|    policy_loss        | -0.213   |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.00438  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.11    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99300    |\n",
            "|    time_elapsed       | 6889     |\n",
            "|    total_timesteps    | 1986000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.999    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99299    |\n",
            "|    policy_loss        | 0.00546  |\n",
            "|    std                | 0.0256   |\n",
            "|    value_loss         | 0.0038   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.01    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99400    |\n",
            "|    time_elapsed       | 6896     |\n",
            "|    total_timesteps    | 1988000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.76     |\n",
            "|    explained_variance | 0.992    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99399    |\n",
            "|    policy_loss        | -0.175   |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.00408  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.14    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99500    |\n",
            "|    time_elapsed       | 6901     |\n",
            "|    total_timesteps    | 1990000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.75     |\n",
            "|    explained_variance | 0.998    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99499    |\n",
            "|    policy_loss        | -0.183   |\n",
            "|    std                | 0.0257   |\n",
            "|    value_loss         | 0.0037   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.27    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99600    |\n",
            "|    time_elapsed       | 6908     |\n",
            "|    total_timesteps    | 1992000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.995    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99599    |\n",
            "|    policy_loss        | 0.317    |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.0125   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.48    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99700    |\n",
            "|    time_elapsed       | 6914     |\n",
            "|    total_timesteps    | 1994000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.77     |\n",
            "|    explained_variance | 0.991    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99699    |\n",
            "|    policy_loss        | -0.0671  |\n",
            "|    std                | 0.0255   |\n",
            "|    value_loss         | 0.00865  |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.79    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99800    |\n",
            "|    time_elapsed       | 6919     |\n",
            "|    total_timesteps    | 1996000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.78     |\n",
            "|    explained_variance | 0.997    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99799    |\n",
            "|    policy_loss        | -0.0962  |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.0082   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.96    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 99900    |\n",
            "|    time_elapsed       | 6926     |\n",
            "|    total_timesteps    | 1998000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.77     |\n",
            "|    explained_variance | 0.996    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99899    |\n",
            "|    policy_loss        | 0.0935   |\n",
            "|    std                | 0.0254   |\n",
            "|    value_loss         | 0.0142   |\n",
            "------------------------------------\n",
            "------------------------------------\n",
            "| rollout/              |          |\n",
            "|    ep_len_mean        | 50       |\n",
            "|    ep_rew_mean        | -4.86    |\n",
            "| time/                 |          |\n",
            "|    fps                | 288      |\n",
            "|    iterations         | 100000   |\n",
            "|    time_elapsed       | 6931     |\n",
            "|    total_timesteps    | 2000000  |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | 6.79     |\n",
            "|    explained_variance | 0.994    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 99999    |\n",
            "|    policy_loss        | -0.279   |\n",
            "|    std                | 0.0253   |\n",
            "|    value_loss         | 0.00562  |\n",
            "------------------------------------\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<stable_baselines3.a2c.a2c.A2C at 0x7fd90a466a90>"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 6\n",
        "model_name = \"a2c-PandaReachDense-v2\"; \n",
        "model.save(model_name)\n",
        "env.save(\"vec_normalize.pkl\")\n",
        "\n",
        "# 7\n",
        "from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize\n",
        "\n",
        "# Load the saved statistics\n",
        "eval_env = DummyVecEnv([lambda: gym.make(\"PandaReachDense-v2\")])\n",
        "eval_env = VecNormalize.load(\"vec_normalize.pkl\", eval_env)\n",
        "\n",
        "#  do not update them at test time\n",
        "eval_env.training = False\n",
        "# reward normalization is not needed at test time\n",
        "eval_env.norm_reward = False\n",
        "\n",
        "# Load the agent\n",
        "model = A2C.load(model_name)\n",
        "\n",
        "mean_reward, std_reward = evaluate_policy(model, eval_env)\n",
        "\n",
        "print(f\"Mean reward = {mean_reward:.2f} +/- {std_reward:.2f}\")\n",
        "\n",
        "# 8\n",
        "package_to_hub(\n",
        "    model=model,\n",
        "    model_name=f\"a2c-{env_id}\",\n",
        "    model_architecture=\"A2C\",\n",
        "    env_id=env_id,\n",
        "    eval_env=eval_env,\n",
        "    repo_id=f\"Ryukijano/a2c-{env_id}\", # TODO: Change the username\n",
        "    commit_message=\"Initial commit\",\n",
        ")"
      ],
      "metadata": {
        "id": "-UnlKLmpg80p",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 838,
          "referenced_widgets": [
            "e26e2fa08e2448b599ae19b47423d050",
            "6de3ea06cb6d4d50be950012e4b98a55",
            "fae640cc5a8c462884a9cd615e3c2682",
            "e924ae7a5954423980fbc8751c1bd900",
            "e5bd6c7f5f7a4c30ac45cfb605ad4303",
            "20538c22adec46a68cf61e4b3a86950f",
            "5c81f0c24c554b29ae0becefa688d355",
            "16a9e0378aa54d6c8f6b6bd2eb35fd94",
            "51ba58f98d904d868246cbe1dff5a565",
            "96f7d176addb4e63aaad69f86454c743",
            "1a2dfcd55d1f44cdbe98af3b6ba52b87",
            "41ca82d652664ea18f7573ce4f55ab1f",
            "40dfffa0bd634c10832c62ea606df622",
            "e0c9825ac17540a6b8b0320631923f6e",
            "fa020e55ffd94116afe3ce6f303a7518",
            "7399aa7a4e8f442696db78a76f61a830",
            "4ef9bcc993794a0cbcc6ba57b055ea2d",
            "dfe61677995e41d4a132a0fccce05e1a",
            "968b4e38f8d84ab29fc1bb802a762ce9",
            "8b4520ad34fe4922a19aae8e64bd47aa",
            "79b06a88db2d45bb9adafb479f1328af",
            "ee426146660649d6b0f7b89c141f7a64",
            "cc2dce85d39940c89dffdaffbf2c91fb",
            "6e2f0398430a4bffa7f048f83884d711",
            "6d64726e416c44b4bd9d195f426f573d",
            "9697c4a45c92434086342c0e750267cc",
            "5ae0765a12dc4c5686fc84947c293257",
            "2a5b7d631c8747c28b2f88e062b6c23a",
            "a27612ae7b044a2ca392af8a14b2b491",
            "ceff407b85254fb4898dfe2e74f10c95",
            "67b5ba536b254a81bf3f9501c477b0c0",
            "afae0e81ffb04c7e9d497b731207e681",
            "2eccf98cc0544b9c8cbfe938356ed745",
            "82dc7446550048c1a962119ff1d0097e",
            "451fd2eed21c475894e218c6e8b74bd3",
            "0122668d4f07427aac6f0f2c7d543686",
            "8dc3f46be2f94cdbb1baffc60211edfc",
            "a36a7df9f04d48339845a8c4c266c1fc",
            "0ca9d512574b4d329969b7b6fcf13b36",
            "612ec11319814c279e55438997d129b1",
            "873488239dc445e1ae139e06040d7eeb",
            "0e1c0a01b61043549528bdbd62b153b4",
            "ead13d07941c497884baa0a617ccee9c",
            "5f8ba321ce0e4426af1ef0fd0314f4c8",
            "6216c51a7f074b088bc9fcc3c08e0df6",
            "49e10565811f41919c484d1d1fda3e67",
            "24c76f9d966c48788fd9da971c353aa6",
            "bcc4ed12941c4540a699967177865252",
            "00f030e66abb4b7bbcf773408802ab6b",
            "95dac9ad006f4aefa9ce69ab170dea24",
            "dc4852a69c9c4bc688d084b117129add",
            "d9b2b0d38aef42ef81ba6bd70e6dccf7",
            "bf1d8aa8635e4d0c97a32ad2037f0250",
            "ea326412469948518cbc4f3afc2c6311",
            "e87b3b26439349b88b1c37a28553dc98",
            "43e09e204dd146a88b6717837ebf7f72",
            "2296f0e560d74ad3b09558b27a74963b",
            "6fffcdd8ace14a17bfe65453980652bc",
            "a17e56d7eaf44e35ad79e71a0cb3e773",
            "1332380fd3374eacb9a6059d6aa3b0c0",
            "d65e7f78d8b94028822a66cb71b98efc",
            "37562c17b5f54e4aa417092348543490",
            "d887c679a8864a008d7e09fd51e5abc9",
            "d32667a445014cc787536fd36e78c7f6",
            "e9afd0ead8824c76817ae5c9ec6df20e",
            "2960f3f6f09b4f49a52ce45ad3131deb"
          ]
        },
        "outputId": "d491f77b-10a3-4989-8a1b-efbdd964fd6a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/evaluation.py:67: UserWarning: Evaluation environment is not wrapped with a ``Monitor`` wrapper. This may result in reporting modified episode lengths and rewards, if other wrappers happen to modify these. Consider wrapping environment first with ``Monitor`` wrapper.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mean reward = -4.96 +/- 0.89\n",
            "\u001b[38;5;4mℹ This function will save, evaluate, generate a video of your agent,\n",
            "create a model card and push everything to the hub. It might take up to 1min.\n",
            "This is a work in progress: if you encounter a bug, please open an issue.\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/panda_gym/pybullet.py:97: UserWarning: The use of the render method is not recommended when the environment has not been created with render=True. The rendering will probably be weird. Prefer making the environment with option `render=True`. For example: `env = gym.make('PandaReach-v2', render=True)`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.8/dist-packages/panda_gym/pybullet.py:97: UserWarning: The use of the render method is not recommended when the environment has not been created with render=True. The rendering will probably be weird. Prefer making the environment with option `render=True`. For example: `env = gym.make('PandaReach-v2', render=True)`.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving video to /tmp/tmpwffyrvlm/-step-0-to-step-1000.mp4\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Exception ignored in: <function VecVideoRecorder.__del__ at 0x7fd9115ad820>\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/vec_env/vec_video_recorder.py\", line 113, in __del__\n",
            "    self.close()\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/vec_env/vec_video_recorder.py\", line 109, in close\n",
            "    VecEnvWrapper.close(self)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/vec_env/base_vec_env.py\", line 279, in close\n",
            "    return self.venv.close()\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/vec_env/base_vec_env.py\", line 279, in close\n",
            "    return self.venv.close()\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/vec_env/dummy_vec_env.py\", line 80, in close\n",
            "    env.close()\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/gym/core.py\", line 298, in close\n",
            "    return self.env.close()\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/panda_gym/envs/core.py\", line 279, in close\n",
            "    self.sim.close()\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/panda_gym/pybullet.py\", line 59, in close\n",
            "    self.physics_client.disconnect()\n",
            "pybullet.error: Not connected to physics server.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[38;5;4mℹ Pushing repo Ryukijano/a2c-PandaReachDense-v2 to the Hugging Face\n",
            "Hub\u001b[0m\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "a2c-PandaReachDense-v2.zip:   0%|          | 0.00/108k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e26e2fa08e2448b599ae19b47423d050"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vec_normalize.pkl:   0%|          | 0.00/3.06k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "41ca82d652664ea18f7573ce4f55ab1f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "policy.pth:   0%|          | 0.00/46.0k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "cc2dce85d39940c89dffdaffbf2c91fb"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "pytorch_variables.pth:   0%|          | 0.00/431 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "82dc7446550048c1a962119ff1d0097e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Upload 5 LFS files:   0%|          | 0/5 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "6216c51a7f074b088bc9fcc3c08e0df6"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "policy.optimizer.pth:   0%|          | 0.00/44.7k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "43e09e204dd146a88b6717837ebf7f72"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[38;5;4mℹ Your model is pushed to the Hub. You can view your model here:\n",
            "https://huggingface.co/Ryukijano/a2c-PandaReachDense-v2/tree/main/\u001b[0m\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'https://huggingface.co/Ryukijano/a2c-PandaReachDense-v2/tree/main/'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Some additional challenges 🏆\n",
        "The best way to learn **is to try things by your own**! Why not trying  `HalfCheetahBulletEnv-v0` for PyBullet and `PandaPickAndPlace-v1` for Panda-Gym?\n",
        "\n",
        "If you want to try more advanced tasks for panda-gym, you need to check what was done using **TQC or SAC** (a more sample-efficient algorithm suited for robotics tasks). In real robotics, you'll use a more sample-efficient algorithm for a simple reason: contrary to a simulation **if you move your robotic arm too much, you have a risk of breaking it**.\n",
        "\n",
        "PandaPickAndPlace-v1: https://huggingface.co/sb3/tqc-PandaPickAndPlace-v1\n",
        "\n",
        "And don't hesitate to check panda-gym documentation here: https://panda-gym.readthedocs.io/en/latest/usage/train_with_sb3.html\n",
        "\n",
        "Here are some ideas to achieve so:\n",
        "* Train more steps\n",
        "* Try different hyperparameters by looking at what your classmates have done 👉 https://huggingface.co/models?other=https://huggingface.co/models?other=AntBulletEnv-v0\n",
        "* **Push your new trained model** on the Hub 🔥\n"
      ],
      "metadata": {
        "id": "G3xy3Nf3c2O1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "See you on Unit 7! 🔥\n",
        "## Keep learning, stay awesome 🤗"
      ],
      "metadata": {
        "id": "usatLaZ8dM4P"
      }
    }
  ]
}