Skip to content
This repository has been archived by the owner on Feb 8, 2019. It is now read-only.

Test #98

Open
wants to merge 36 commits into
base: master
Choose a base branch
from
Open

Test #98

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
9423366
docker
Mar 12, 2017
f794080
apply django security release, also kombu pin
Mar 12, 2017
140bed5
make notification related env vars non mandatory
Mar 12, 2017
0214390
updating README
May 10, 2017
463992f
add optionable send of notification for resolved incidents
May 10, 2017
40ab5ad
add HipChat notification method
May 10, 2017
c471874
add better notification logging and result keeping
May 10, 2017
dba9d0c
add unacknowledge backend code
May 10, 2017
2359793
update requirements.txt for hipchat, django-debug-toolbar and securit…
May 10, 2017
d45c1af
add incident endpoint for API as an alias to create_event
May 10, 2017
30a1442
add django-debug-toolbar to settings and urls if DEBUG==True
May 10, 2017
1b2eaea
add better error management for hipchat and default settings.py config
May 10, 2017
b6fe592
fix email notifier to be able to use custom SMTP servers; add default…
May 10, 2017
8b7d61c
modify user notif_method template for better visibility
May 10, 2017
116cf33
front-end: add cosmetic glyphicons to menus; add confirmation before …
May 10, 2017
bde6341
add optional TLS for SMTP email notifier
May 11, 2017
1b54188
add UNACK support in incidents.py
May 12, 2017
fb706dd
notification tasks return result for easier debugging with Flower
May 12, 2017
aae607b
add more verbose errors from incidents.py and send resolve notifications
May 12, 2017
bab0a4a
add escalate_to_service: the possibility to escalate(basically send, …
May 12, 2017
db2c490
fix extra space in openduty/models.py
May 12, 2017
3dd4786
add the option to notify groups of people in escalation rules, events…
May 12, 2017
77f1382
add the option to search by incident_key with autosuggest for the lat…
May 12, 2017
1644473
add example systemd service files
May 12, 2017
5dddc86
add example nginx conf for reverse rpoxy to gunicorn
May 12, 2017
f8468b0
Fix markdown in README file.
christ66 May 18, 2017
7d4320e
Add proxy support
jbarascut Jun 16, 2017
d76ce6b
Update requirement for slacker lib
jbarascut Jun 16, 2017
06ff7ef
Merge pull request #1 from christ66/patch-1
deathowl Sep 24, 2017
f77db22
Merge pull request #1 from jbarascut/slacker
jbarascut Nov 10, 2017
9813d7b
Merge pull request #4 from johnseekins/master
sysrex Dec 13, 2017
f8da9d9
merged pull request
sysrex Apr 23, 2018
27a7d56
Merge branch 'jbarascut-master'
sysrex Apr 23, 2018
1de25a1
merge pull requrest #2
sysrex Apr 23, 2018
1ed627d
Merge branch 'Oxalide-master'
sysrex Apr 23, 2018
9abf4e7
Update the readme file
sysrex Apr 23, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 22 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@
[![Requirements Status](https://requires.io/github/openduty/openduty/requirements.svg?branch=master)](https://requires.io/github/openduty/openduty/requirements/?branch=master)
# What is this?
**Openduty** is an incident escalation tool, just like [Pagerduty](http://pagerduty.com) . It has a Pagerduty compatible API too. It's the result of the first [Ustream Hackathon](http://www.ustream.tv/blog/2014/03/27/hackathon-recap-21-ideas-11-teams-one-goal/). We enjoyed working on it.
#Integrations
# Integrations
Has been tested with Nagios, works well for us. Any Pagerduty Notifier using the Pagerduty API should work without a problem.
[Icinga2 config](https://github.com/deathowl/OpenDuty-Icinga2) for openduty integration

# Notifications
XMPP, email, SMS, Phone(Thanks Twilio for being awesome!), and Push notifications(thanks Pushover also),and Slack are supported at the moment.
# Current status
#Notifications
XMPP, email, SMS, Phone(Thanks Twilio for being awesome!), Push notifications(thanks Pushover, Prowl as well!)and Slack, HipChat, Rocket.chat are supported at the moment.
#Current status
Openduty is in Beta status, it can be considered stable at the moment, however major structural changes can appear anytime (not affecting the API, or the Notifier structure)

# Contribution guidelines
Expand Down Expand Up @@ -79,7 +79,24 @@ python manage.py runserver
```
now, you can start hacking on it.

# After models you've changed your models please run:
# Running as a service with systemd
*OpenDuty can be ran as a service with the help of gunicorn and systemd*
```
cp -r systemd/gunicorn.service.* /etc/systemd/system/

cp -r systemd/celery.service* /etc/systemd/system/

// EDIT VARIABLES IN *.service.d/main.conf TO REFLECT YOUR ENV
vi /etc/systemd/system/gunicorn.service.d/main.conf
vi /etc/systemd/system/celery.service.d/main.conf

systemctl daemon-reload
sudo systemctl start gunicorn
sudo systemctl enable gunicorn

```

# After you've changed your models please run:
```
./manage.py schemamigration openduty --auto
./manage.py schemamigration notification --auto
Expand Down
15 changes: 15 additions & 0 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
FROM ubuntu:14.04
MAINTAINER Balint Csergo <[email protected]>
RUN apt-get update
RUN apt-get -y upgrade
RUN apt-get -y install nginx supervisor
RUN apt-get -y install git python-pip python-dev build-essential g++ libbz2-dev libncurses5-dev libreadline-dev libsqlite3-dev libssl-dev libxml2-dev libxslt-dev make zlib1g-dev libmysqlclient-dev libldap2-dev libsasl2-dev
RUN git clone https://github.com/openduty/openduty.git /opt/openduty
RUN cd /opt/openduty && pip install -r requirements.txt
RUN cd /opt/openduty && pip install gunicorn

ADD supervisord.conf /etc/supervisor/conf.d/supervisord.conf
ADD nginx.conf /etc/nginx/nginx.conf

EXPOSE 80 8000 3306
CMD ["/usr/bin/supervisord"]
73 changes: 73 additions & 0 deletions docker/nginx.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
user www-data;
worker_processes 2;
pid /run/nginx.pid;
daemon off;

events {
worker_connections 768;
# multi_accept on;
}

http {

##
# Basic Settings
##

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;

# server_names_hash_bucket_size 64;
# server_name_in_redirect off;

include /etc/nginx/mime.types;
default_type application/octet-stream;

##
# Logging Settings
##

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

##
# Gzip Settings
##

gzip on;
gzip_disable "msie6";

# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

upstream app {
server 127.0.0.1:8000;
}

server {
listen 80 default_server;
server_name _;
root /opt/openduty/openduty;

location @backend {
proxy_pass http://app;
}

location ~ / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
try_files $uri @backend;
}
}
}
11 changes: 11 additions & 0 deletions docker/supervisord.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[supervisord]
nodaemon=true

[program:openduty]
command=bash -c "cd /opt/openduty && export DJANGO_SETTINGS_MODULE=openduty.settings_docker && exec python manage.py runserver 0.0.0.0:8000"

[program:nginx]
command=/usr/sbin/nginx

[program:celeryd]
command=bash -c "cd /opt/openduty && DJANGO_SETTINGS_MODULE=openduty.settings_docker C_FORCE_ROOT=true python manage.py celery worker"
15 changes: 15 additions & 0 deletions nginx.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
server {
listen 80;
server_name _;
access_log /var/log/openduty/nginx/access.log;
error_log /var/log/openduty/nginx/error.log;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://localhost:8080;
}
}


42 changes: 32 additions & 10 deletions notification/helper.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
__author__ = 'deathowl'

from datetime import datetime, timedelta

from django.db import transaction
from notification.tasks import send_notifications
from openduty.escalation_helper import get_escalation_for_service
from openduty.models import Incident
from django.utils import timezone
from notification.models import ScheduledNotification
from django.conf import settings
Expand All @@ -28,21 +29,41 @@ def notify_user_about_incident(incident, user, delay=None, preparedmsg = None):
@staticmethod
def generate_notifications_for_incident(incident):
now = timezone.make_aware(datetime.now(), timezone.get_current_timezone())
duty_officers = get_escalation_for_service(incident.service_key)

duty_officers = []
# if incident has been escalated, notify according to the escalated service's escalation rule
if hasattr(incident, 'service_to_escalate_to') and incident.service_to_escalate_to is not None:
print "escalation rule in place to " + incident.service_to_escalate_to.name
duty_officers = get_escalation_for_service(incident.service_to_escalate_to)
with transaction.atomic():
incident.description = "[escalated] " + incident.description
incident.save()
else:
duty_officers = get_escalation_for_service(incident.service_key)
current_time = now

notifications = []
group_index = {}
user_method_index = {}

for officer_index, duty_officer in enumerate(duty_officers):
escalation_time = incident.service_key.escalate_after * (officer_index + 1)
if incident.event_type == Incident.RESOLVE and not duty_officer.profile.send_resolve_enabled:
print "Skipping notification for %s because type is RESOLVE and user %s has send_resolve_enabled OFF" % (incident.incident_key, duty_officer.username)
continue
index = 0
if hasattr(duty_officer ,'came_from_group' ):
if not duty_officer.came_from_group in group_index:
group_index[duty_officer.came_from_group] = officer_index
index = group_index[duty_officer.came_from_group]
else:
index = officer_index
escalation_time = incident.service_key.escalate_after * (index + 1)
escalate_at = current_time + timedelta(minutes=escalation_time)

user_method_index[duty_officer.username] = 0
methods = duty_officer.notification_methods.order_by('position').all()
method_index = 0

for method in methods:
notification_time = incident.service_key.retry * method_index + incident.service_key.escalate_after * officer_index
notification_time = incident.service_key.retry * user_method_index[duty_officer.username] + incident.service_key.escalate_after * index
notify_at = current_time + timedelta(minutes=notification_time)
if notify_at < escalate_at:
notification = ScheduledNotification()
Expand All @@ -51,14 +72,14 @@ def generate_notifications_for_incident(incident):
notification.notifier = method.method
notification.send_at = notify_at
uri = settings.BASE_URL + "/incidents/details/" + str(incident.id)
notification.message = "A Service is experiencing a problem: " + incident.incident_key + " " + incident.description + ". Handle at: " + uri + " Details: " + incident.details
notification.message = incident.description + ". Handle at: " + uri + " Details: " + incident.details

notifications.append(notification)

print "Notify %s at %s with method: %s" % (duty_officer.username, notify_at, notification.notifier)
print "[%s] Notify %s about %s at %s with method: %s" % (notification.incident.event_type, duty_officer.username, notification.incident.incident_key, notify_at, notification.notifier)
else:
break
method_index += 1
user_method_index[duty_officer.username] += 1

# todo: error handling

Expand Down Expand Up @@ -90,7 +111,8 @@ def generate_notifications_for_user(incident, user, delay=None, preparedmsg = No
else:
notification.message = preparedmsg
notifications.append(notification)
print "Notify %s at %s with method: %s" % (user.username, notify_at, notification.notifier)
if notification.incident:
print "[%s] Notify %s at %s with method: %s" % (notification.incident.event_type, user.username, notify_at, notification.notifier)
method_index += 1

# todo: error handling
Expand Down
3 changes: 2 additions & 1 deletion notification/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@ class UserNotificationMethod(models.Model):
METHOD_SLACK = 'slack'
METHOD_PROWL = 'prowl'
METHOD_ROCKET = 'rocket'
METHOD_HIPCHAT = 'hipchat'


methods = [METHOD_XMPP, METHOD_PUSHOVER, METHOD_EMAIL, METHOD_TWILIO_SMS, METHOD_TWILIO_CALL, METHOD_SLACK, METHOD_PROWL, METHOD_ROCKET]
methods = [METHOD_XMPP, METHOD_PUSHOVER, METHOD_EMAIL, METHOD_TWILIO_SMS, METHOD_TWILIO_CALL, METHOD_SLACK, METHOD_PROWL, METHOD_ROCKET, METHOD_HIPCHAT]

user = models.ForeignKey(User, related_name='notification_methods')
position = models.IntegerField()
Expand Down
14 changes: 9 additions & 5 deletions notification/notifier/email.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,10 @@ def __init__(self, config):
self.__config = config
def notify(self, notification):

gmail_user = self.__config['user']
gmail_pwd = self.__config['password']
host = self.__config['host']
port = self.__config['port']
user = self.__config['user']
password = self.__config['password']
truncate_length = int(self.__config.get('max_subject_length', 100))
FROM = self.__config['user']
TO = [notification.user_to_notify.email]
Expand All @@ -25,10 +27,12 @@ def notify(self, notification):
message = """\From: %s\nTo: %s\nSubject: %s\n\n%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)
try:
server = smtplib.SMTP("smtp.gmail.com", 587)
server.starttls()
server = smtplib.SMTP(host, int(port))
if self.__config['tls']:
server.starttls()
server.ehlo()
server.login(gmail_user, gmail_pwd)
if user and password:
server.login(user, password)
server.sendmail(FROM, TO, message)
server.close()
print 'successfully sent the mail'
Expand Down
68 changes: 68 additions & 0 deletions notification/notifier/hipchat.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
from hypchat import HypChat
import json
import requests

class HipchatNotifier:

def __init__(self, config):
self.__config = config

def start(self):
if not self.__config['token'] or not self.__config['endpoint']:
print "HipChat configuration is missing %s" % self.__config
raise
elif self.__config['endpoint']:
return HypChat(self.__config['token'], endpoint=self.__config['endpoint'])
else:
return HypChat(self.__config['token'])

def notify(self, notification):
hc = self.start()
try:
description = notification.incident.description
details = notification.incident.details
except :
description = notification.message
details = ""
try:
message = description + " " + details
colour = "yellow"
if "CRITICAL" in message:
colour = "red"
elif "UNKNOWN" in message:
colour = "gray"
if notification.user_to_notify.profile.hipchat_room_name:
print "Notifying HipChat via API v2"
response = hc.get_room(notification.user_to_notify.profile.hipchat_room_name).notification(message, color=colour, notify="True", format="html")
elif notification.user_to_notify.profile.hipchat_room_url:
print "Notifying HipChat via a simple POST"
headers = {"content-type": "application/json"}
hip_msg = '{"color": "' + colour + '", "message": "' + message + '", "notify": true, "message_format": "html"}'
response = requests.post(notification.user_to_notify.profile.hipchat_room_url,headers=headers,data=hip_msg)
print response.content
else:
print "HipChat message send failed"
return
print "HipChat message sent"
except Exception, e:
try:
resp = json.loads(str(e))
print "Failed to send HipChat message %s " % (e, str(resp['error']['code']) + " " + resp['error']['message'])
raise
except ValueError, e2:
print "Failed to send HipChat message and failed to get it's error %s ; %s" % (e, e2)
raise

def get_all_rooms(self):
try:
hc = self.start()
if not hc:
return [""]
else:
rooms = hc.rooms().contents()
names = []
for room in rooms:
names.append(room['name'])
return names
except:
return [""]
2 changes: 1 addition & 1 deletion notification/notifier/slack.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ def __init__(self, config):
self.__config = config

def notify(self, notification):
slack = Slacker(self.__config['apikey'])
slack = Slacker(self.__config['apikey'], http_proxy=self.__config['slack_http_proxy'], https_proxy=self.__config['slack_https_proxy'])
response = slack.chat.post_message(notification.user_to_notify.profile.slack_room_name, notification.message,
username="Openduty", icon_url="https://slack.global.ssl.fastly.net/1937/img/services/pagerduty_48.png")
if not response.error:
Expand Down
Loading