First public release.

.gitignore

@@ -0,0 +1,24 @@
+# See http://help.github.com/ignore-files/ for more about ignoring files.
+#
+# If you find yourself ignoring temporary files generated by your text editor
+# or operating system, you probably want to add a global ignore instead:
+#   git config --global core.excludesfile ~/.gitignore_global
+
+# Ignore bundler config
+/.bundle
+
+# Ignore the default SQLite database.
+/db/*.sqlite3
+
+# Ignore all logfiles and tempfiles.
+/log/*.log
+/tmp
+.DS_Store
+
+# Ignore generated files.
+/Gemfile.lock
+
+# Ignore private configuration files.
+/config/auth.yml
+/config/database.yml
+/config/initializers/secret_token.rb

Gemfile

@@ -0,0 +1,45 @@
+source 'https://rubygems.org'
+gem 'rails', '4.0.1'
+
+# Rails 4.0 dropped this, we'll put it back for now.
+gem 'protected_attributes'
+
+# Remote workers
+gem 'sidekiq'
+gem 'slim'
+# if you require 'sinatra' you get the DSL extended to Object
+gem 'sinatra', '>= 1.3.0', :require => nil
+
+# Tags, necessary in both environments because of the helpers.
+gem 'acts-as-taggable-on'
+
+# XML parsing
+gem 'nokogiri'
+
+group :remote do
+  # Needed for the in-memory SQLite stub. Pointless, but easier than patching
+  #  ActiveRecord out of everything, or Sidekiq to not load models.
+  gem 'sqlite3'
+end
+
+group :local do
+  # We'll just standardize on MySQL.
+  gem 'mysql2'
+
+  # Bootstrap
+  gem 'therubyracer'
+  gem 'less-rails' #Sprockets (what Rails 3.1 uses for its asset pipeline) supports LESS
+  gem 'twitter-bootstrap-rails', '~> 2.2.6'
+
+  # Network gems
+  gem 'netaddr'
+
+  # Pagination
+  gem 'kaminari'
+
+  gem 'jquery-rails'
+  gem 'jquery-ui-rails'
+  gem 'sass-rails'
+  gem 'coffee-rails'
+  gem 'uglifier'
+end

HACKING.md

@@ -0,0 +1,29 @@
+# Hacking Nepenthes
+
+# So you want to write a worker
+
+Writing another worker is designed to be easy. (It may or may not fall short, but that's the goal.) If you want to write a new worker, you'll probably want to split it into two parts: the part that actually needs to reach a server, and a part for processing the data and storing results into the database. (This separation is good because occasionally scans need to be run on a client site across a VPN, or on several remote, partially-trusted machines without access to your database.)
+
+## The setup
+
+I recommend copying `ssl_worker.rb` and `ssl_results.rb` to files for your own worker. For simplicity, name one `[type]_worker.rb` and `[type]_results.rb`, so people know where to look. 
+
+## foo_worker.rb
+
+The worker you use shouldn't require any access to the database or any ActiveRecord models. Pass in any address you want to use as an actual parameter, rather than trying to synthesize it in the worker. Perform your work/scan/etc., and then pass the results back to your results worker. Be sure to send any relevant ID to your results worker, so it can update the database.
+
+## foo_results.rb
+
+Your results worker should be run on the results queue, and can interact with the database. It should process the data from the worker (if necessary), update the database, and quit. Results workers tend to be fairly quick to execute, compared to the normal worker. (Keep in mind that large netpens may have dozens or hundreds of normal worker threads running, but only one or two result worker threads running.)
+
+# Extending models
+
+Feel free to extend models as necessary, but note that both IpAddress and Port have JSON-serialized `.settings` properties, where you can stash results if necessary. This may be appropriate for extra data that will not need to be queried directly, rather than adding another column to the database. On the other hand, never underestimate the usefulness of being able to query based on the results of your worker.
+
+Most workers will likely not need their own model for results, but some may. Anything that can be assumed to always be the same for a given object (such as the SSL details on a given port, or the reverse DNS for an IP address) should be set on that object. Anything that may have multiple results for its parent should consider adding a model. For example, scans have their own model, as it is reasonable to run multiple scans on a given IP address, and screenshots have their own model, as it's possible that screenshotting subdirectories would be desirable.
+
+# Displaying results
+
+You should expose the results of your worker to the user in the user interface. For scans that operate on ports, this should be relatively straightforward: modify the `/app/views/ports/_port.html.erb` template to display your worker's results if they exist and are relevant.
+
+If your result display is likely to be large and/or not frequently accessed, you may consider hiding it in some fashion (for example, under a toggle display). You may also consider making a new page in the interface (either under an existing controller if it fits, or as a new controller), particularly if your results are likely to be large or otherwise unwieldy, or if you want to browse them in a different fashion.

README.md

@@ -0,0 +1,80 @@
+# Nepenthes
+
+# This README is actually informative. Please read it before starting to use Nepenthes.
+
+## Install
+* `brew install phantomjs coreutils` or `sudo apt-get install phantomjs`, depending on your OS.
+* `bundle install`
+* `cp config/database.yml.example config/database.yml` 
+* `cp config/auth.yml.example config/auth.yml`
+* Edit config/database.yml. Set it up with MySQL or MariaDB, please. You'll be happy you did. SQLite, in particular, is problematic.
+* Edit config/auth.yml. Pick a username and password. You'll be using HTTP Basic auth, with the same username/password for everybody.
+* `echo "Nepenthes::Application.config.secret_token = \"``rake secret``\"" > config/initializers/secret_token.rb` (Note that you need backticks around "rake secret", but our GH markdown doesn't seem to tolerate double backticks well.)
+* Make sure that your database is running (MySQL - `mysql.server start` - or SQLite)
+* Edit the `config/database.yml` to have the correct connection info for your database
+* `rake db:create` to create the netpen database.
+* `rake db:migrate`
+* If you're not already running Redis, run it. (`redis-server` in a new terminal window is fine, as is running it as a daemon.) Be warned that Redis listens on all interfaces by default. Nepenthes only needs to access it via localhost, so feel free to lock down Redis' configuration.
+* `rails s`
+* In another terminal window on your local computer, run `sidekiq -c 1 -r . -q results -v`. (If you are using SQLite, you *must not* use more than one thread here. Other databases can use more, but it won't help much: this isn't a very slow step.)
+* Visit http://localhost:3000/regions
+
+If you want to run Nepenthes workers locally:
+* First, reconsider. You *should not* do this from inside a NAT if you're scanning anything public. If you're fully inside a VPN, it *might* work, but may also crash the VPN. The reasoning here is that you're running a lot of nmap scans and will quickly fill up NAT tables and related resources. Doing this will make your sysadmin sad.
+* Second, if you've decided to go for it anyway and not blame any Nepenthes authors, make sure you have local copies of whatever tools you might use. This includes nmap for scans, phantomjs for screenshots, and nikto for Nikto scans.
+* Finally, you will want to run the two `sidekiq` commands from the second below on your local machine. You may need to adjust the `~/nepenthes` directory to suit your environment. You do not need to play with editing the database.yml file, installing the bundle without the local group, or anything else. Your machine likely has more than 0.5 GB of RAM, so consider increasing the thread count if you have sufficient bandwidth to your target.
+
+For remote workers (highly recommended, VPSes are cheap and many allow scanning if you only scan ranges you have permission to scan):
+
+* Get something running *Ubuntu 12.10 or newer* (older versions have severely outdated copies of some packages, and errors will result).
+* From your local Nepenthes directory, run `rsync -a --exclude log --exclude log --exclude config/database.yml --exclude .bundle --exclude Gemfile.lock . tendril:~/nepenthes` (where `tendril` is your host, consider adding it to your `.ssh/config`)
+* `ssh -R 127.0.0.1:6379:127.0.0.1:6379 tendril` (log in to your remote VM, and forward your local Redis connection)
+* The rest is on the remote VM:
+* `sudo apt-get install ruby1.9.1 ruby1.9.1-dev libsqlite3-dev libxslt1-dev nmap phantomjs nikto`
+* `sudo gem install --no-rdoc --no-ri bundler`
+* `cd ~/nepenthes`
+* `cp config/database.yml.example config/database.yml` (you don't need to edit it this time, the in-memory SQLite3 is fine for remote workers: they don't store data.)
+* `bundle install --without local`
+* You'll probably want to run these in two `screen` windows (or at the very least, two terminal windows, as they should be run concurrently):
+* `sidekiq -e sidekiq -c 2 -r ~/nepenthes -q himem_fast -q himem_slow -v` and `sidekiq -e sidekiq -c 20 -r ~/nepenthes -q lomem_fast -q lomem_slow -v`. These settings work well for 512 MB of RAM. You can add more threads (under the -c parameter) as desired, and you will want way more of the "lomem" workers for large sets of scans. Consider getting more RAM or more VMs if this is the case, or the OS will OOM kill your nmap processes, and nothing will get done. One user noted that 150 lomem threads worked for him with 1 GB of RAM.
+
+Feel free to repeat the "for remote workers" section on as many VMs as you want. You will get more mileage out of additional RAM before you get help from multiple VMs, but multiple VMs isn't a bad thing.
+
+## Usage
+* Add a region via http://localhost:3000/regions . The start and end test times must be numbers, but don't actually matter at the moment.
+* Go to http://localhost:3000/ip_addresses and add IP addresses. You can use single IP addresses (one per line, don't comma-separate them) or ranges (192.168.1.0 - 192.168.5.255). CIDR support doesn't currently work, but I think that's a matter of changing a regex. If you want to tag all of the ranges you're entering at a time in some way (hosting facility, country, whatever), you can add tags for all of them (space-separated) in the appropriate field. To tag just addresses in a specific range, you can put them space-separated after the range, on the same line. Adding thousands of IP addresses will be a bit slow.
+
+## Scanning
+
+There is now a web interface for this. It isn't quite as configurable for some scans, but *check it out first*. That's just at http://localhost:3000/ . The instructions below assume you aren't using the web interface, so pick and choose as you go if you want more power than you can extract from the web interface.
+
+### Again, use the web interface home page first. If you need configurability that it doesn't give you, here's a way to run them manually.
+
+* `rails c`
+* Specify some nmap options, such as `opts = ['-Pn', '-p', '80,443,22,25,21,8080,23,3306,143,53', '-sV', '--version-light']`
+** Note: Spaces in the options aren't treated they way you might expect; if the commandline would be `--scan-delay 250ms`, you need to add it as `['--scan-delay', '250ms']`
+* `IpAddress.includes(:scans).where(:scans => {:ip_address_id => nil}).each {|ip| ip.queue_scan!(opts) }`
+* Wait for a bit while every IP address has a scan queued.
+* You can follow progress on http://localhost:3000/sidekiq/ .
+
+For full scans:
+
+* After you've done your lighter scans (and ideally after they've actually returned results - full scans are queued first for hosts with open ports), schedule full scans in the console: `IpAddress.queue_full_scans!`
+
+Once your scans are done:
+
+* To check whether ports are using *SSL* or not: `Port.check_all_ssl!`.
+* To get *screenshots* of applicable webpages (including on all ports), do `Port.take_all_screenshots!`. This is on the `screenshot` queue, and requires PhantomJS on the worker. (Packages exist in most OS's package managers, any recent version is fine.) `sidekiq -c [number of threads] -r . -q screenshot -v` will get it running.
+* To process the results, you'll want to keep a results processor going. `sidekiq -c 1 -r . -q results -v` if you didn't still have it running.
+
+## Results
+
+* You can view results while scans are still going on. http://localhost:3000/ports will give a listing of ports found and the number of each, you can click on a port to get a list of hosts, click on a host to get a list of ports for that host, etc.
+* There are a bunch of features lurking around that need better documentation. You can add .csv to the end of the URL for any(?) /ports page and get a .csv of the applicable hosts, ports, versions (if applicable), and such.
+* http://localhost:3000/ip_addresses.xml will give a combined XML output as if all of the scans were done in one nmap run with XML output. This is handy for importing to Nessus.
+
+## Extending
+
+You can add your own workers to Nepenthes to gather additional information, scan other things, or do whatever you need. Check out `HACKING.md` for information on writing your own worker.
+
+If you have any problems, feel free to submit an issue. Pull requests welcome.

Rakefile

@@ -0,0 +1,7 @@
+#!/usr/bin/env rake
+# Add your own tasks in files placed in lib/tasks ending in .rake,
+# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.
+
+require File.expand_path('../config/application', __FILE__)
+
+Nepenthes::Application.load_tasks

TODO.md

@@ -0,0 +1,3 @@
+* We should have a facility for detecting HTTP servers like we do for SSL servers. This would be useful for Nikto, as well as potentially for other services.
+
+* Full support for domain names would also be useful. For example, Nikto scans could be run on domains rather than ports. SSL scan results could be used to add domain names. We could look up IP addresses for given domains, and add those. We could brute-force subdomains (admin.client.com) and see which ones exist.

app/assets/javascripts/application.js

@@ -0,0 +1,43 @@
+// This is a manifest file that'll be compiled into application.js, which will include all the files
+// listed below.
+//
+// Any JavaScript/Coffee file within this directory, lib/assets/javascripts, vendor/assets/javascripts,
+// or vendor/assets/javascripts of plugins, if any, can be referenced here using a relative path.
+//
+// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
+// the compiled file.
+//
+// WARNING: THE FIRST BLANK LINE MARKS THE END OF WHAT'S TO BE PROCESSED, ANY BLANK LINE SHOULD
+// GO AFTER THE REQUIRES BELOW.
+//
+//= require jquery
+//= require jquery_ujs
+//= require jquery.ui.all
+//= require twitter/bootstrap
+//= require_tree .
+
+$(function() {
+    $( "#dialog-form" ).dialog({
+        autoOpen: false,
+        height: 350,
+        width: 350,
+        modal: true,
+        buttons: {
+            "Save": function() {
+                $( "#dialog-form-tag").submit();
+                $( this ).dialog( "close" );
+},
+Cancel: function() {
+    $( this ).dialog( "close" );
+    }
+},
+close: function() {
+    }
+});
+});
+
+function show_notes_dialog(id) {
+    $( "#dialog-form-tag").attr('action', "/ports/" + id);
+    $( "#dialog-form" ).dialog( "open" );
+}
+

app/assets/javascripts/bootstrap.js.coffee

@@ -0,0 +1,4 @@
+jQuery ->
+  $("a[rel=popover]").popover()
+  $(".tooltip").tooltip()
+  $("a[rel=tooltip]").tooltip()

app/assets/javascripts/domains.js.coffee

@@ -0,0 +1,3 @@
+# Place all the behaviors and hooks related to the matching controller here.
+# All this logic will automatically be available in application.js.
+# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/

app/assets/javascripts/dynamic.js.coffee

@@ -0,0 +1,3 @@
+# Place all the behaviors and hooks related to the matching controller here.
+# All this logic will automatically be available in application.js.
+# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/

app/assets/javascripts/home.js.coffee

@@ -0,0 +1,3 @@
+# Place all the behaviors and hooks related to the matching controller here.
+# All this logic will automatically be available in application.js.
+# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/

app/assets/javascripts/ip_addresses.js.coffee

@@ -0,0 +1,3 @@
+# Place all the behaviors and hooks related to the matching controller here.
+# All this logic will automatically be available in application.js.
+# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/

app/assets/javascripts/ports.js.coffee

@@ -0,0 +1,3 @@
+# Place all the behaviors and hooks related to the matching controller here.
+# All this logic will automatically be available in application.js.
+# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/

app/assets/javascripts/regions.js.coffee

@@ -0,0 +1,3 @@
+# Place all the behaviors and hooks related to the matching controller here.
+# All this logic will automatically be available in application.js.
+# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/

app/assets/stylesheets/application.css

@@ -0,0 +1,14 @@
+/*
+ * This is a manifest file that'll be compiled into application.css, which will include all the files
+ * listed below.
+ *
+ * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets,
+ * or vendor/assets/stylesheets of plugins, if any, can be referenced here using a relative path.
+ *
+ * You're free to add application-wide styles to this file and they'll appear at the top of the
+ * compiled file, but it's generally better to create a new file per style scope.
+ *
+ *= require_self
+ *= require jquery.ui.all
+ *= require_tree .
+ */

app/assets/stylesheets/bootstrap_and_overrides.css.less

@@ -0,0 +1,44 @@
+@import "twitter/bootstrap/bootstrap";
+@import "twitter/bootstrap/responsive";
+
+// Set the correct sprite paths
+@iconSpritePath: asset-path("twitter/bootstrap/glyphicons-halflings.png");
+@iconWhiteSpritePath: asset-path("twitter/bootstrap/glyphicons-halflings-white.png");
+
+// Set the Font Awesome (Font Awesome is default. You can disable by commenting below lines)
+// Note: If you use asset_path() here, your compiled bootstrap_and_overrides.css will not
+//       have the proper paths. So for now we use the absolute path.
+@fontAwesomeEotPath: asset-path("fontawesome-webfont.eot?v=3.0.2");
+@fontAwesomeEotPath_iefix: asset-path("fontawesome-webfont.eot?#iefix&v=3.0.2");
+@fontAwesomeWoffPath: asset-path("fontawesome-webfont.woff?v=3.0.2");
+@fontAwesomeTtfPath: asset-path("fontawesome-webfont.ttf?v=3.0.2");
+
+// Font Awesome
+//@import "fontawesome";
+
+// Glyphicons
+//@import "twitter/bootstrap/sprites.less";
+
+// Your custom LESS stylesheets goes here
+//
+// Since bootstrap was imported above you have access to its mixins which
+// you may use and inherit here
+//
+// If you'd like to override bootstrap's own variables, you can do so here as well
+// See http://twitter.github.com/bootstrap/customize.html#variables for their names and documentation
+//
+// Example:
+// @linkColor: #ff0000;
+
+textarea {
+  width: 400px;
+}
+
+form {
+  margin: 0;
+}
+
+// http://stackoverflow.com/a/10688485 
+.table-nonfluid {
+  width: auto;
+}

app/assets/stylesheets/domains.css.scss

@@ -0,0 +1,3 @@
+// Place all the styles related to the Domains controller here.
+// They will automatically be included in application.css.
+// You can use Sass (SCSS) here: http://sass-lang.com/

app/assets/stylesheets/dynamic.css.scss

@@ -0,0 +1,3 @@
+// Place all the styles related to the dynamic controller here.
+// They will automatically be included in application.css.
+// You can use Sass (SCSS) here: http://sass-lang.com/

app/assets/stylesheets/home.css.scss

@@ -0,0 +1,3 @@
+// Place all the styles related to the Home controller here.
+// They will automatically be included in application.css.
+// You can use Sass (SCSS) here: http://sass-lang.com/

app/assets/stylesheets/ip_addresses.css.scss

@@ -0,0 +1,3 @@
+// Place all the styles related to the IpAddresses controller here.
+// They will automatically be included in application.css.
+// You can use Sass (SCSS) here: http://sass-lang.com/

app/assets/stylesheets/ports.css.scss

@@ -0,0 +1,3 @@
+// Place all the styles related to the Ports controller here.
+// They will automatically be included in application.css.
+// You can use Sass (SCSS) here: http://sass-lang.com/

app/assets/stylesheets/regions.css.scss

@@ -0,0 +1,3 @@
+// Place all the styles related to the Regions controller here.
+// They will automatically be included in application.css.
+// You can use Sass (SCSS) here: http://sass-lang.com/