Skip to content

Commit

Permalink
Terminate Riemann::Tools::HttpCheck threads between tests
Browse files Browse the repository at this point in the history
Under normal operation, we have a single instance of the
`Riemann::Tools::HttpCheck` class for the lifetime of the monitoring
process.

But when testing, we create a new instance for each test, each with its
resolvers and worker thread pools.  The test process will have more and
more threads, until it eventually hit the OS limit of the maximum number
of threads for a process and cause an exception:

> ThreadError: can't create Thread: Resource temporarily unavailable

Make sure that the threads are terminated at the end of each test to
avoid running out of resources if they are limited.
  • Loading branch information
smortex committed Jan 21, 2024
1 parent 73a4b16 commit c4f0df4
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 2 deletions.
28 changes: 26 additions & 2 deletions lib/riemann/tools/http_check.rb
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,15 @@ def initialize
@resolve_queue = Queue.new
@work_queue = Queue.new

@resolvers = []
@workers = []

opts[:resolvers].times do
Thread.new do
@resolvers << Thread.new do
loop do
uri = @resolve_queue.pop
Thread.exit unless uri

host = uri.host

addresses = Resolv::DNS.new.getaddresses(host)
Expand All @@ -59,9 +64,11 @@ def initialize
end

opts[:workers].times do
Thread.new do
@workers << Thread.new do
loop do
uri, addresses = @work_queue.pop
Thread.exit unless uri

test_uri_addresses(uri, addresses)
end
end
Expand All @@ -70,6 +77,23 @@ def initialize
super
end

# Under normal operation, we have a single instance of this class for the
# lifetime of the process. But when testing, we create a new instance
# for each test, each with its resolvers and worker threads. The test
# process may end-up with a lot of running threads, hitting the OS limit
# of max threads by process and being unable to create more thread:
#
# ThreadError: can't create Thread: Resource temporarily unavailable
#
# To avoid this situation, we provide this method.
def shutdown
@resolve_queue.close
@resolvers.map(&:join)

@work_queue.close
@workers.map(&:join)
end

def tick
report(
service: 'riemann http-check resolvers utilization',
Expand Down
4 changes: 4 additions & 0 deletions spec/riemann/tools/http_check_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,10 @@ def protected!
@server&.shutdown
end

after(:each) do
subject.shutdown
end

let(:test_webserver_port) { @server.config[:Port] }

let(:reported_uri) { uri }
Expand Down

0 comments on commit c4f0df4

Please sign in to comment.