Add LibreNMS Integration

README.md

@@ -35,6 +35,7 @@ This Nautobot application framework includes the following integrations:
 - Infoblox
 - IPFabric
 - Itential
+- LibreNMS
 - Cisco Meraki
 - ServiceNow
 - Slurpit
@@ -92,6 +93,7 @@ The SSoT framework includes a number of integrations with external Systems of Re
 * Cisco DNA Center
 * Infoblox
 * Itential
+* LibreNMS
 * Cisco Meraki
 * ServiceNow
 * Slurpit

changes/636.added

@@ -0,0 +1 @@
+Added LibreNMS integration.

development/development.env

@@ -115,4 +115,8 @@ IPFABRIC_TIMEOUT=15
 NAUTOBOT_SSOT_ENABLE_ITENTIAL="True"
 
 NAUTOBOT_SSOT_ENABLE_SLURPIT="False"
-SLURPIT_HOST="https://sandbox.slurpit.io"
+SLURPIT_HOST="https://sandbox.slurpit.io"
+
+NAUTOBOT_SSOT_ENABLE_LIBRENMS="False"
+NAUTOBOT_SSOT_LIBRENMS_SYSTEM_OF_RECORD="LibreNMS"
+NAUTOBOT_SSOT_LIBRENMS_HOSTNAME_FIELD="sysName" # hostname or sysName

development/nautobot_config.py

@@ -229,6 +229,7 @@
         "enable_infoblox": is_truthy(os.getenv("NAUTOBOT_SSOT_ENABLE_INFOBLOX")),
         "enable_ipfabric": is_truthy(os.getenv("NAUTOBOT_SSOT_ENABLE_IPFABRIC")),
         "enable_itential": is_truthy(os.getenv("NAUTOBOT_SSOT_ENABLE_ITENTIAL")),
+        "enable_librenms": is_truthy(os.getenv("NAUTOBOT_SSOT_ENABLE_LIBRENMS", "false")),
         "enable_meraki": is_truthy(os.getenv("NAUTOBOT_SSOT_ENABLE_MERAKI")),
         "enable_servicenow": is_truthy(os.getenv("NAUTOBOT_SSOT_ENABLE_SERVICENOW")),
         "enable_slurpit": is_truthy(os.getenv("NAUTOBOT_SSOT_ENABLE_SLURPIT")),

docs/admin/install.md

@@ -97,6 +97,7 @@ Set up each integration using the specific guides:
 - [Infoblox](./integrations/infoblox_setup.md)
 - [IPFabric](./integrations/ipfabric_setup.md)
 - [Itential](./integrations/itential_setup.md)
+- [LibreNMS](./integrations/librenms_setup.md)
 - [Cisco Meraki](./integrations/meraki_setup.md)
 - [ServiceNow](./integrations/servicenow_setup.md)
 - [Slurpit](./integrations/slurpit_setup.md)

docs/admin/integrations/index.md

@@ -11,6 +11,7 @@ This Nautobot app supports the following integrations:
 - [Infoblox](./infoblox_setup.md)
 - [IPFabric](./ipfabric_setup.md)
 - [Itential](./itential_setup.md)
+- [LibreNMS](./librenms_setup.md)
 - [Cisco Meraki](./meraki_setup.md)
 - [ServiceNow](./servicenow_setup.md)
 - [Slurpit](./slurpit_setup.md)

docs/admin/integrations/librenms_setup.md

@@ -0,0 +1,46 @@
+# LibreNMS
+
+## Description
+
+This App will sync data from the LibreNMS API into Nautobot to create Device and IPAM inventory items. Most items will receive a custom field associated with them called "System of Record", which will be set to "LibreNMS" (or whatever you set the `NAUTOBOT_SSOT_LIBRENMS_SYSTEM_OF_RECORD` environment variable to). These items are then the only ones managed by the LibreNMS SSoT App. Other items within the Nautobot instance will not be affected unless there's items with overlapping names. If an item exists in Nautobot by it's identifiers but it does not have the "System of Record" custom field on it, the item will be updated with "LibreNMS" (or `NAUTOBOT_SSOT_LIBRENMS_SYSTEM_OF_RECORD` environment variable value) when the App runs. This way no duplicates are created, and the App will not delete any items that are not defined in the LibreNMS API data but were manually created in Nautobot.
+
+## Installation
+
+Before configuring the integration, please ensure, that `nautobot-ssot` app was [installed with LibreNMS integration extra dependencies](../install.md#install-guide).
+
+```shell
+pip install nautobot-ssot[librenms]
+```
+
+## Configuration
+
+Once the SSoT package has been installed you simply need to enable the integration by setting `enable_librenms` to True.
+
+```python
+PLUGINS = ["nautobot_ssot"]
+
+PLUGINS_CONFIG = {
+  "nautobot_ssot": {
+        # Other nautobot_ssot settings ommitted.
+        "enable_librenms": is_truthy(os.getenv("NAUTOBOT_SSOT_ENABLE_LIBRENMS", "true")),
+  }
+}
+```
+
+### External Integrations
+
+#### LibreNMS as DataSource
+
+The way you add your LibreNMS server instance is through the "External Integrations" objects in Nautobot. First, create a secret in Nautobot with your LibreNMS API token using an Environment Variable (or sync via secrets provider). Then create a SecretsGroup object and select the Secret you just created and set the Access Type to `HTTP(S)` and the Secret Type to `Token`.
+
+Once this is created, go into the Extensibility Menu and select `External Integrations`. Add an External Intergration with the Remote URL being your LibreNMS server URL (including http(s)://), set the method to `GET`, and select any other headers/settings you might need for your specific instance. Select the secrets group you created as this will inject the API token. Once created, you will select this External Integration when you run the LibreNMS to Nautobot SSoT job.
+
+![LibreNMS External Integration](../../images/librenms-external-integration.png)
+
+#### LibreNMS as DataTarget
+
+NotYetImplemented
+
+### LibreNMS API
+
+An API key with global read-only permissions is the minimum needed to sync information from LibreNMS.

docs/user/integrations/index.md

@@ -11,6 +11,7 @@ This Nautobot app supports the following integrations:
 - [Infoblox](./infoblox.md)
 - [IPFabric](./ipfabric.md)
 - [Itential](./itential.md)
+- [LibreNMS](./librenms.md)
 - [Cisco Meraki](./meraki.md)
 - [ServiceNow](./servicenow.md)
 - [Slurpit](./slurpit.md)

docs/user/integrations/librenms.md

@@ -0,0 +1,35 @@
+## Usage
+
+## Process
+
+### LibreNMS as DataSource
+
+The LibreNMS SSoT integration is built as part of the [Nautobot Single Source of Truth (SSoT)](https://github.com/nautobot/nautobot-app-ssot) app. the SSoT app enables Nautobot to be the aggregation point for data coming from multiple systems of record (SoR).
+
+#### Job Options
+
+- Debug: Additional Logging
+- Librenms Server: External integration object pointing to the required LibreNMS instance.
+- hostname_field: Which LibreNMS field to use as the hostname in Nautobot. sysName or hostanme.
+- sync_location_parents: Whether to lookup City and State to add parent locations for geo locations.
+- tenant: This is used as a filter for objects synced with Nautobot and LibreNMS. This can be used to sync multiple LibreNMS instances into different tenants, like in an MSP environment. This affects which devices are loaded from Nautobot during the sync. It does not affect which devices are loaded from LibreNMS
+
+From LibreNMS into Nautobot, the app synchronizes devices, their interfaces, associated IP addresses, and Locations. Here is a table showing the data mappings when syncing from LibreNMS.
+
+| LibreNMS objects        | Nautobot objects             |
+| ----------------------- | ---------------------------- |
+| geo location            | Location                     |
+| device                  | Device                       |
+| interface               | Interface                    |
+| device os               | Platform/Manufacturer `*`    |
+| os version              | Software/SoftwareImage       |
+| ip address              | IPAddress                    |
+| hardware                | DeviceType                   |
+
+
+`*` Device OS from LibreNMS is not standardized and therefore there is a mapping that can be updated in the `constants.py` file for the integration as more device manufacturers and platforms need to be added.
+
+### LibreNMS as DataTarget
+
+NotYetImplemented
+

mkdocs.yml

@@ -117,6 +117,7 @@ nav:
           - Infoblox: "user/integrations/infoblox.md"
           - IPFabric: "user/integrations/ipfabric.md"
           - Itential: "user/integrations/itential.md"
+          - LibreNMS: "user/integrations/librenms.md"
           - Cisco Meraki: "user/integrations/meraki.md"
           - ServiceNow: "user/integrations/servicenow.md"
           - Slurpit: "user/integrations/slurpit.md"
@@ -137,6 +138,7 @@ nav:
           - Infoblox: "admin/integrations/infoblox_setup.md"
           - IPFabric: "admin/integrations/ipfabric_setup.md"
           - Itential: "admin/integrations/itential_setup.md"
+          - LibreNMS: "admin/integrations/librenms_setup.md"
           - Cisco Meraki: "admin/integrations/meraki_setup.md"
           - ServiceNow: "admin/integrations/servicenow_setup.md"
           - Slurpit: "admin/integrations/slurpit_setup.md"

nautobot_ssot/__init__.py

@@ -102,11 +102,13 @@ class NautobotSSOTAppConfig(NautobotAppConfig):
         "dna_center_show_failures": True,
         "enable_aci": False,
         "enable_aristacv": False,
+        "enable_bootstrap": False,
         "enable_device42": False,
         "enable_dna_center": False,
         "enable_citrix_adm": False,
         "enable_infoblox": False,
         "enable_ipfabric": False,
+        "enable_librenms": False,
         "enable_servicenow": False,
         "enable_slurpit": False,
         "enable_itential": False,

nautobot_ssot/integrations/librenms/constants.py

@@ -0,0 +1,27 @@
+"""Constants for LibreNMS SSoT."""
+
+from django.conf import settings
+
+# Import config vars from nautobot_config.py
+PLUGIN_CFG = settings.PLUGINS_CONFIG["nautobot_ssot"]
+
+librenms_status_map = {
+    0: "Offline",
+    1: "Active",
+    True: "Active",
+    False: "Offline",
+}
+
+os_manufacturer_map = {
+    "ping": "Generic",
+    "linux": "Linux",
+    "routeros": "Mikrotik",
+    "unifi": "Ubiquiti",
+    "airos": "Ubiquiti",
+    "proxmox": "Proxmox",
+    "hpe-ilo": "HP",
+    "cyberpower": "Cyberpower",
+    "opnsense": "Opnsense",
+    "epmp": "Cambium",
+    "tachyon": "Tachyon Networks",
+}

nautobot_ssot/integrations/librenms/diffsync/adapters/__init__.py

@@ -0,0 +1 @@
+"""Adapter classes for loading DiffSyncModels with data from LibreNMS or Nautobot."""

nautobot_ssot/integrations/librenms/diffsync/adapters/librenms.py

@@ -0,0 +1,164 @@
+"""Nautobot Ssot Librenms Adapter for LibreNMS SSoT app."""
+
+import os
+
+from diffsync import DiffSync
+from diffsync.exceptions import ObjectNotFound
+from django.contrib.contenttypes.models import ContentType
+from nautobot.dcim.models import Device, Location, LocationType
+from nautobot.extras.models import Status
+
+from nautobot_ssot.integrations.librenms.constants import (
+    librenms_status_map,
+    os_manufacturer_map,
+)
+from nautobot_ssot.integrations.librenms.diffsync.models.librenms import (
+    LibrenmsDevice,
+    LibrenmsLocation,
+)
+from nautobot_ssot.integrations.librenms.utils import (
+    is_running_tests,
+    normalize_gps_coordinates,
+)
+from nautobot_ssot.integrations.librenms.utils.librenms import LibreNMSApi
+
+
+class LibrenmsAdapter(DiffSync):
+    """DiffSync adapter for LibreNMS."""
+
+    location = LibrenmsLocation
+    device = LibrenmsDevice
+
+    top_level = ["location", "device"]
+
+    def __init__(self, *args, job=None, sync=None, librenms_api: LibreNMSApi, **kwargs):
+        """Initialize LibreNMS.
+
+        Args:
+            job (object, optional): LibreNMS job. Defaults to None.
+            sync (object, optional): LibreNMS DiffSync. Defaults to None.
+            client (object): LibreNMS API client connection object.
+        """
+        super().__init__(*args, **kwargs)
+        self.job = job
+        self.sync = sync
+        self.lnms_api = librenms_api
+
+    def load_location(self, location: dict):
+        """Load Location objects from LibreNMS into DiffSync models."""
+        if self.job.debug:
+            self.job.logger.debug(f'Loading LibreNMS Location {location["location"]}')
+
+        try:
+            self.get(self.location, location["location"])
+        except ObjectNotFound:
+            _latitude = None
+            _longitude = None
+            if location["lat"]:
+                _latitude = normalize_gps_coordinates(location["lat"])
+            if location["lng"]:
+                _longitude = normalize_gps_coordinates(location["lng"])
+            new_location = self.location(
+                name=location["location"],
+                status="Active",
+                location_type="Site",
+                latitude=_latitude,
+                longitude=_longitude,
+                system_of_record=os.getenv("NAUTOBOT_SSOT_LIBRENMS_SYSTEM_OF_RECORD", "LibreNMS"),
+            )
+            self.add(new_location)
+
+    def load_device(self, device: dict):
+        """Load Device objects from LibreNMS into DiffSync models."""
+        if self.job.debug:
+            self.job.logger.debug(f'Loading LibreNMS Device {device["sysName"]}')
+
+        if device["os"] != "ping":
+            try:
+                self.get(self.device, device["sysName"])
+            except ObjectNotFound:
+                if device["disabled"] == 1:
+                    _status = "Offline"
+                else:
+                    _status = librenms_status_map[device["status"]]
+                new_device = self.device(
+                    name=device[self.hostname_field],
+                    device_id=device["device_id"],
+                    location=(device["location"] if device["location"] is not None else "Unknown"),
+                    role=device["type"] if device["type"] is not None else None,
+                    serial_no=device["serial"] if device["serial"] is not None else "",
+                    status=_status,
+                    manufacturer=(
+                        os_manufacturer_map.get(device["os"])
+                        if os_manufacturer_map.get(device["os"]) is not None
+                        else "Unknown"
+                    ),
+                    device_type=(device["hardware"] if device["hardware"] is not None else "Unknown"),
+                    platform=device["os"] if device["os"] is not None else "Unknown",
+                    os_version=(device["version"] if device["version"] is not None else "Unknown"),
+                    system_of_record=os.getenv("NAUTOBOT_SSOT_LIBRENMS_SYSTEM_OF_RECORD", "LibreNMS"),
+                )
+                self.add(new_device)
+        else:
+            self.job.logger.info(f'Device {device[self.hostname_field]} is "ping-only". Skipping.')
+
+    def load(self):
+        """Load data from LibreNMS into DiffSync models."""
+        self.hostname_field = (
+            os.getenv("NAUTOBOT_SSOT_LIBRENMS_HOSTNAME_FIELD", "sysName")
+            if self.job.hostname_field == "env_var"
+            else self.job.hostname_field or "sysName"
+        )
+
+        print("Running tests:", is_running_tests())
+        if is_running_tests():
+            load_source = "file"
+            self.job.sync_locations = True
+            self.hostname_field = "sysName"
+        else:
+            load_source = self.job.load_type
+
+        if load_source != "file":
+            all_devices = self.lnms_api.get_librenms_devices()
+        else:
+            all_devices = self.lnms_api.get_librenms_devices_from_file()
+
+        self.job.logger.info(f'Loading {all_devices["count"]} Devices from LibreNMS.')
+
+        if is_running_tests():
+            print(f"All devices fetched: {all_devices}")
+
+        for _device in all_devices["devices"]:
+            self.load_device(device=_device)
+
+        if self.job.sync_locations:
+            _site, _created = LocationType.objects.get_or_create(name="Site")
+            if _created:
+                _site.content_types.add(ContentType.objects.get(app_label="dcim", model="device"))
+            if is_running_tests():
+                _status = Status.objects.get_or_create(name="Active")[0]
+                _status.content_types.add(ContentType.objects.get_for_model(Device))
+                _status.content_types.add(ContentType.objects.get_for_model(Location))
+                _status.validated_save()
+            else:
+                _status = Status.objects.get(name="Active")
+            Location.objects.get_or_create(
+                name="Unknown",
+                location_type=_site,
+                status=_status,
+            )
+
+            if load_source != "file":
+                all_locations = self.lnms_api.get_librenms_locations()
+            else:
+                all_locations = self.lnms_api.get_librenms_locations_from_file()
+
+            self.job.logger.info(f'Loading {all_locations["count"]} Locations from LibreNMS.')
+
+            if is_running_tests():
+                print(f"All locations fetched: {all_locations}")
+
+            for _location in all_locations["locations"]:
+                self.load_location(location=_location)
+        else:
+            self.job.logger.info("Location Sync Disabled. Skipping loading locations.")