Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Necessary improvements to the shell code #4

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 41 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,46 @@
These scripts are designed to be run via a cron job with the MAILTO variable set to generate an email(or any other notification method) when output occurs.
# Description

Output will occur when an IP is added/removed or changed for the service you are checking.
The supplied bash scripts are supposed to be run via a cron job and assume that the MAILTO variable in the cron is set to send out a notification email when any IP address change happens.

- `check_events.sh` - Checks events.pagerduty.com for changed IP addressses.
# Prerequisites to run this script

- `check_mailservers.sh` - Checks acme.pagerduty.com for changed MX record IP addressses.
(Used when you utilize an email integrated service and need to whitelist outgoing traffic to PagerDuty)
The scripts depend on the external `dig` command to get the list of IP addresses from the PagerDuty servers. You should be able to install the `dig` command using the following steps:

- `check_webhooks.sh` - Checks webhooks.pagerduty.com for changed IP addressses.
## On RedHat-based GNU/Linux servers

- `check_webhooks_and_alert.sh` - Same as `check_webhooks.sh` but triggers a PagerDuty incident.
Install it from the `bind-utils` package available in the repositories.

Command: `dnf install bind-utils`

You will require root permissions to install the package.

## On Debian-based GNU/Linux servers

Install it from the `dnsutils` package available in the repositories.

Command: `apt install dnsutils`

You will require root permissions to install the package.

# Output

It is assumed that there are two scenario's to each script run, which are:

## When the IP addresses change

The script(s) will display a list of all the new IP addresses fetched from the server on the stdout and will send out a notification email to the email address set in the MAILTO variable in the cron.

This notification email may then be captured to initiate an incident in PagerDuty.

## When the IP addresses do not change

There is no change in the files or output on the stdout and no notification email is sent out.

# Description of the files in the repository

| File name | Remarks |
|-|-|
| `check_events.sh` | Checks events.pagerduty.com for changed IP addressses. |
| `check_mailservers.sh` | Checks `acme.pagerduty.com` for changed MX record IP addressses. (Used when you utilize an email integrated service and need to whitelist outgoing traffic to PagerDuty) |
| `check_webhooks.sh` | Checks `webhooks.pagerduty.com` for changed IP addressses. |
| `check_webhooks_and_alert.sh` | Checks `webhooks.pagerduty.com` for changed IP addressses, same as `check_webhooks.sh`, but also triggers a PagerDuty incident. |
35 changes: 23 additions & 12 deletions check_events.sh
Original file line number Diff line number Diff line change
@@ -1,18 +1,29 @@
#!/bin/bash
#!/usr/bin/env bash

## This script is designed to be run via a cron job with the MAILTO variable set to generate an email when output occurs (an IP is added/removed/changed)
# More information on PagerDuty Events - https://support.pagerduty.com/docs/event-management
# PagerDuty Status Page - https://status.pagerduty.com

if [ -f "events_result.txt" ]; then
mv events_result.txt events_result.txt.old
fi
## This script is designed to be run via a cron job with the MAILTO variable
## set to generate an email when output occurs (an IP is added/removed/changed)

dig +short events.pagerduty.com | sort > events_result.txt
# get the list of IP addresses from the PagerDuty Events URL. Storing in a memory variable to reduce disk access
events_results_current = $(dig +short events.pagerduty.com | sort)

if [ -f "events_result.txt.old" ]; then
DIFF=$(diff -q 'events_result.txt.old' 'events_result.txt' > /dev/null)
if [ -f "events_results.txt" ]; then
# we check for the diff and supress any outputs on the stdout
DIFF=$(echo "${events_results_current}" | diff -q 'events_results.txt' - > /dev/null)

# diff command returns a status code 0 if no change has been detected
if [ $? -ne 0 ]; then
echo "Changes detected! New IPs are:"
cat events_result.txt
# The script has detected that the list of IP addresses has changed.
# we overwrite the existing file with the changed IP addresses.
echo ${events_results_current} > events_results.txt
# display the output on stdout. MAILTO cron variable takes over the job of sending out an email.
echo -e "\nThe script has detected a change in PagerDuty's Events IP addresses. The new IP addresses are:\n\n${events_results_current}"
fi
rm events_result.txt.old
fi
else
# we fall in this condition when we run the script for the first time
# or if the events_results.txt has been deleted
# we create a new one and spit the output on the stdout too
echo ${events_results_current} | tee events_results.txt
fi
34 changes: 22 additions & 12 deletions check_mailservers.sh
Original file line number Diff line number Diff line change
@@ -1,18 +1,28 @@
#!/bin/bash
#!/usr/bin/env bash

## This script is designed to be run via a cron job with the MAILTO variable set to generate an email when output occurs (an IP is added/removed/changed)
# PagerDuty Status Page - https://status.pagerduty.com

if [ -f "mailservers_result.txt" ]; then
mv mailservers_result.txt mailservers_result.txt.old
fi
## This script is designed to be run via a cron job with the MAILTO variable
## set to generate an email when output occurs (an IP is added/removed/changed)

dig +short mx acme.pagerduty.com | sed 's/.$//g' | sed 's/^[0-9][0-9]* //g' | xargs dig +short | sort > mailservers_result.txt
# get the list of IP addresses from the PagerDuty acme URL. Storing in a memory variable to reduce disk access
mailservers_results_current = $(dig +short mx acme.pagerduty.com | sed 's/.$//g' | sed 's/^[0-9][0-9]* //g' | xargs dig +short | sort)

if [ -f "mailservers_result.txt.old" ]; then
DIFF=$(diff -q 'mailservers_result.txt.old' 'mailservers_result.txt' > /dev/null)
if [ -f "mailservers_results.txt" ]; then
# we check for the diff and supress any outputs on the stdout
DIFF=$(echo "${mailservers_results_current}" | diff -q 'mailservers_results.txt' - > /dev/null)

# diff command returns a status code 0 if no change has been detected
if [ $? -ne 0 ]; then
echo "Changes detected! New IPs are:"
cat mailservers_result.txt
# The script has detected that the list of IP addresses has changed.
# we overwrite the existing file with the changed IP addresses.
echo ${mailservers_results_current} > mailservers_results.txt
# display the output on stdout. MAILTO cron variable takes over the job of sending out an email.
echo -e "\nThe script has detected a change in PagerDuty Mailserver's IP addresses. The new IP addresses are:\n\n${mailservers_results_current}"
fi
rm mailservers_result.txt.old
fi
else
# we fall in this condition when we run the script for the first time
# or if the mailservers_results.txt has been deleted
# we create a new one and spit the output on the stdout too
echo ${mailservers_results_current} | tee mailservers_results.txt
fi
35 changes: 23 additions & 12 deletions check_webhooks.sh
Original file line number Diff line number Diff line change
@@ -1,18 +1,29 @@
#!/bin/bash
#!/usr/bin/env bash

## This script is designed to be run via a cron job with the MAILTO variable set to generate an email when output occurs (an IP is added/removed/changed)
# PagerDuty Webhooks Documentation - https://support.pagerduty.com/docs/webhooks
# PagerDuty Status Page - https://status.pagerduty.com

if [ -f "webhooks_result.txt" ]; then
mv webhooks_result.txt webhooks_result.txt.old
fi
## This script is designed to be run via a cron job with the MAILTO variable
## set to generate an email when output occurs (an IP is added/removed/changed)

curl -s https://app.pagerduty.com/webhook_ips | tr -d \[\]\" | tr , '\n' | sort > webhooks_result.txt
# get the list of IP addresses from the PagerDuty Webhooks URL. Storing in a memory variable to reduce disk access
webhooks_results_current = $(curl -s https://app.pagerduty.com/webhook_ips | tr -d \[\]\" | tr , '\n' | sort)

if [ -f "webhooks_result.txt.old" ]; then
DIFF=$(diff -q 'webhooks_result.txt.old' 'webhooks_result.txt' > /dev/null)
if [ -f "webhooks_results.txt" ]; then
# we check for the diff and supress any outputs on the stdout
DIFF=$(echo "${webhooks_results_current}" | diff -q 'webhooks_results.txt' - > /dev/null)

# diff command returns a status code 0 if no change has been detected
if [ $? -ne 0 ]; then
echo "Changes detected! New IPs are:"
cat webhooks_result.txt
# The script has detected that the list of IP addresses has changed.
# we overwrite the existing file with the changed IP addresses.
echo ${webhooks_results_current} > webhooks_results.txt
# display the output on stdout. MAILTO cron variable takes over the job of sending out an email.
echo -e "\nThe script has detected a change in PagerDuty Webhook's IP addresses. The new IP addresses are:\n\n${webhooks_results_current}"
fi
rm webhooks_result.txt.old
fi
else
# we fall in this condition when we run the script for the first time
# or if the webhooks_results.txt has been deleted
# we create a new one and spit the output on the stdout too
echo ${webhooks_results_current} | tee webhooks_results.txt
fi
2 changes: 1 addition & 1 deletion check_webhooks_and_alert.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash
#!/usr/bin/env bash
# Webhook delivery service IP address change notification script
#
# Same as check_webhooks.sh, but triggers a PagerDuty incident if there are changes.
Expand Down