The listen gem breaks my laptop

[benaubin]

Steps to reproduce

1. Create a new rails app
2. Run it on a Mac with a bad processor

Expected behavior

My laptop should work fine - processes should stay under the 700-something limit that Apple set.

Actual behavior

When running my Rails app on a Macbook Air with 1.8 ghz of cpu (I know... getting a new computer when they release the new Pro's), I have been getting a "System ran out of resources, can't fork any processes" error in all of my applications (or, they just quietly failed). Finally, I opened Activity Monitor and was able to find more then 500 fsevent_watch processes running (my computer has a maximum of 700 processes total). Doing a Google Search, I stumbled upon the [rb-fsevent gem]. Looking at my Gemfile.lock, I was able to track it down to the listen gem.

I'm unable to tell if it is an issue with the listen gem itself, or @fnx's implementation in Rails (see #22254), but it seems like it's more likely the implementation is broken.

Suggested Fix

I'm unable to tell if it is an issue with the listen gem itself, or @fnx's implementation in Rails (see #22254), but it seems like it's more likely the implementation is broken.

I would make sure that the listen gem is opt-in, especially on systems with low process limits (IIRC, there's a command that allows you to check this).

Temporary Workaround

If your computer is currently showing a "System ran out of resources, can't fork any processes" error, reboot it (logging in and out might work, but I haven't tested that). You probably won't be able to save most of your work (if your software forks a process in order to save).

To remove the gem for a more permanent fix

1. Remove the listen and spring-watcher-listen gems.
2. In your environments/development.rb file, comment out this line:

# Use an evented file watcher to asynchronously detect changes in source code,
# routes, locales, etc. This feature depends on the listen gem.
# config.file_watcher = ActiveSupport::EventedFileUpdateChecker

System configuration

Rails version:
Rails 5.0.0
Ruby version:
ruby 2.3.0p0

[vipulnsward]

Can you create a sample application replicating this for debugging?

[amree]

Just want to chip in.

I'm probably having the same issue. I noticed my fsevent_watch can go up to more than 500 and I can't start new process after that. I can confirm it happens whenever I close my laptop without turning off rails server. So, when I opened my laptop and refresh the localhost, new fsevent_watch somehow will be started even though they already exists. Killing rails server will only kill the new fsevent_watch, not those from before I close my laptop. So, opening and closing the laptop will gradually increase fsevent_watch processes.

System Configuration

Rails version:
Rails 5.0.0.1
Ruby Version
2.3.1p112

[benaubin]

@vipulnsward Yes. In fact, there's a command to do so:

rails new thisbreaksmylaptop

[rvirani1]

All 3 devs on our team are having this issue as well.

[benaubin]

Ok, I came up with a workaround. Looks like it's not only the spring-watch-listen gem, and while removing it helps slightly (I think it might have just been a placebo), it doesn't fully solve the problem.

First I removed the listen and spring-watcher-listen gems.
Then, in my environments/development.rb file, I commented out this line:

 # Use an evented file watcher to asynchronously detect changes in source code,
 # routes, locales, etc. This feature depends on the listen gem.
 # config.file_watcher = ActiveSupport::EventedFileUpdateChecker

Then, Rails seems to work fine.

I very much recommend making using the listen gem opt-in, not opt-out, as it's very hard to debug. Also, laptop breaking probably turns many developers away from rails (it almost did for me).

The listen gem was added in #22254.

[benaubin]

I'm tagging @fnx, as it looks the listen implementation in Rails which he wrote is what contributes to this, not just the spring-watcher-listen gem.

[metacortex]

+1

Now I'm killing manually.
pkill -9 fsevent_wat*

[ahoward]

i'm seeing this too. it is way too easy to orphan processes. the listen gem is a piece of shit and should be removed.

[ahoward]

amateur hour: https://github.com/thibaudgg/rb-fsevent/blob/d803637/lib/rb-fsevent/fsevent.rb#L55

[ahoward]

the guys that are working on that gem know next to nothing about process management. a disaster.

[tenderlove]

@ahoward it might not be great, but lets try to stay civil. 🙇

[benaubin]

@tenderlove Plan on removing listen as a default?

[tenderlove]

@penne12 I don't think so. We need to either fix listen, or find an alternative. If we can find a suitable alternative then we'd remove listen as default. This seems like it might have potential

[benaubin]

@tenderlove I believe active support has something, and while it's not ideal, it doesn't break people's laptops. Literally, it took me almost a week to be able to figure out how to use Rails. Almost completely changed frameworks, even. Rails works fine (esp in development) without listen.

[simi]

@tenderlove especially https://github.com/facebook/watchman/tree/master/ruby/ruby-watchman

[e2]

Hi everyone, Listen maintainer here...

TL;DR - skip to end..

First, I haven't used a Mac for over 5 years. I only use Linux, so unless I can reproduce a problem on the Mac - at least on Travis (OSX builds), I can't fix it.

I probably can't even reasonably run OSX in a Virtual Machine (AFAIK), so if I can't reproduce it or write a unit/acceptance test for it, I won't even know how something is broke.

Especially if nothing is ever reported.

Otherwise, I do release bugfixed-versions of Listen VERY quickly.

With that said ...

Listen supports multiple backends. rb-fsevent is the problem child here.

Basically, most of Listen is a giant workaround for OSX-related issues - by design. So surprising issues on OSX are ... not surprising. And they're often OSX version-dependent.

You can blame Apple for a lack of decent file/dir monitoring.

One solution I'd suggest is replacing the current Darwin backend with fs-event listener from https://github.com/burke/zeus.

I'd do that myself, but 1) I don't have time nowadays (but PRs are encouraged!) 2) I wouldn't be able to test it, since I don't own a Mac.

One thing that Listen does that's special for OSX is: it runs each watched directory in a separate thread. (This was mostly to allow mapping a file back to a given watcher). I don't know if that's the issue here, but I'd be glad to fix+release. There are acceptance tests for OSX on Travis, so if you can reproduce it somehow, it can likely be added to the build to be resolved forever.

There are lots of hideous details (many related to OSX), so feel free to ask, open issues, etc.

As for alternatives I once researched and created a list of "watch-and-do-something" tools here: https://github.com/guard/guard/wiki/Guard-alternatives

Meanwhile, there's polling mode, which sounds bad, but actually may perform much better than expected.

One good/bad thing about Listen is that it's "directory based" and not "file based" (like many other file watching implementations). This means Listen has better support of removing/adding whole trees of files and editor support (especially useful for Rails users).

The downside is: it needs to scan whole trees of changes on OSX, because the current backend has no hints about which files were changed. And HFS filesystem has 1-second resolution, which makes things worse. Especially when combined with file upper/lower case OSX-specific issues.

I really considered facebook/watchman, but it didn't solve all the problems that Listen was designed to solve (but it was pretty close). I also considered the listeners from Zeus, but again, I don't have/use/want OSX. (I only use Linux).

TL;DR

OSX file monitoring is (was?) shitty. It also was the most problematic and often the slowest.

I doubt those things can be fixed by "a gem" instead of fixing OSX.

Trying to switch implementations is like trading one can of "known" worms with another can of "unknown" worms.

My best bet would be to transplant this:

https://github.com/burke/zeus/blob/master/go/filemonitor/filemonitor_darwin.go or this: https://github.com/burke/zeus/blob/v0.15.5/ext/fsevents/main.m to replace this: https://github.com/guard/listen/blob/master/lib/listen/adapter/darwin.rb

Feel free to mention me (@e2) if I can help/advise/etc.

[e2]

One thing: please don't spread groundless FUD about Listen. (At least first make sure Listen really is the culprit).

TL;DR - saying that Listen is shitty because of this issue (without knowing the root cause here) is like saying Rails is shitty because of a Listen issue. It only obscures the problem and discourages maintainers from putting in more personal time to try and improve things.

@ahoward - while personally I'm not offended by your comments, they're just misleading. Listen is just a layer of workarounds that makes file monitoring portable, so if this really was a Listen issue, it would likely be reproduceable on Linux, Windows and BSD. If it isn't, it likely isn't a Listen issue.

Also, Listen works as reliably as the backends let it, e.g. rb-fsevent - which I'm not even a contributor of.

Sure, Listen does need another overhaul (for 4.x) - but that's mostly unrelated to this.

Ironically though, Listen's acceptance tests are the best way to test backends like rb-fsevent. So if there's something messed up, I can maybe just replicate it in Listen and detect bad scenarios and blacklist certain gem version combinations (which I've done before).

(But I'd need some way to turn it into a Travis tests - e.g. with shell code to create the problem and detect orphaned processes).

My point: removing Listen would just mean that some poor contributors would have to waste hours to reinvent the same fixes/workarounds that Listen already has.

So it's not constructive to create a movement of "lets drop Listen". Simply because it's not the root problem here. Listen is more like an API interface, so dropping it wouldn't make sense unless the API was bad.

And just to give a sense:

Both Listen and the backend drivers started as hacks as early as 2010: https://github.com/thibaudgg/rb-fsevent/commits/d803637f19d3d1450105b5e0d4f8ed1060f98acb/lib/rb-fsevent/fsevent.rb - and there's a long history of OSX nightmares that brave contributors have been trying to debug and patch ever since.

And many original contributors have simply moved on with their lives. If anyone wants to make the code "less shitty" - there are probably no other obstacles than being unable to contact the maintainers.

I'm not the maintainer of rb-fsevent, but I'd be glad to add a better OSX backend implementation in Listen. Or even just bumping the Listen dependency on rb-fsevent (if a fix becomes available).

I know it's easy to jump to conclusions when we're frustrated - sometimes it's better to just reach out and ask first. If something works shitty, there may be a bigger reason other than "incompetence".

Sorry for the length - just felt like a wrong I had to fix.

[matthewd]

@tenderlove already addressed the tone of the previous feedback; I don't think it's productive to continue discussing same.

saying that Listen is shitty because of this issue (without knowing the root cause here) is like saying Rails is shitty because of a Listen issue

For the record: Rails is shitty because of this issue, regardless of which gem is the root cause -- and we will consequently explore any reasonable option to make the problem go away.

[vipulnsward]

FYI: I myself face this issue now 😕

[padi]

It's been months that I encountered this issue and it's only now that I found out that it's related to my new Rails project. :| I <3 Rails but I reckon this can be a dealbreaker for new people coming into Rails.

With that said, I'm not knowledgeable enough yet to fix it, but here's a workaround that doesn't have to disable listen/rb-fsevent/spring altogether: we can throttle down applications (including Rails-related processes) in OSX using renice:

Fire and forget way:

$ sudo su -
$ `while true; do renice +15 -p `ps ax | grep 'rb-fsevent' | grep -v grep | awk '{print $1}' | tr 'n' ' '`; done`

... or a command line alternative: make an executable renice-rails in your $PATH with the following content:

#!/bin/sh
# filename: renice-rails
renice +15 -p `ps ax | grep 'rb-fsevent' | grep -v grep | awk '{print $1}' | tr 'n' ' '`

I quickly put up a 1 minute blog about it https://padi.github.io/articles/throttle-down-rails-processes/

[MatthiasWinkelmann]

As a note that may help some experiencing problems: I noticed that the rails file watcher was watching the project root, not just directories with source files below it.

I happened to have a data directory with a few million small image files within a few hundred thousand nested directories. I experienced extreme slowdowns across the board: rails console would take a minute to start, and any model instantiation would take 10 seconds or so.

I tried to dive into the source and find a way to configure evented_file_watcher, but my attempts with Rails.application.config.watched_dirs didn't make a difference. As far as I can tell, Rails adds /app, /lib, etc. separately, but I believe http://www.github.com/rails//rails/activesupport/lib/active_support/evented_file_update_checker.rb#L51 then tries to be smart and combines into their common ancestor.

The listen gem prevents the worst by filtering a set of known prefixes in https://github.com/guard/listen/blob/master/lib/listen/silencer.rb, where you could 'hide' non-source data. Since I didn't feel like having actual data in /tmp, and I couldn't find a way to easily configure this behaviour, I moved the data out of the project tree.

[rails-bot]

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-1-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[benaubin]

Still a problem!

[thibaudgg]

@benaubin could you please give a try to guard/rb-fsevent#81, that could maybe solve it.

[thibaudgg]

@here rb-fsevent 0.10.3 is out with a potential fix for this issue.

[salex]

After spending most of the day trying to figure out why jQuery was not loading on a demo app I found this page. I was running in development mode with nginx proxy_pass and tried a bunch of nginx configs, none of which solved the problem. I then discovered the same code ran fine an a RaspberryPI but failed on a MacMini! Then I remembered the error when I started the Rails server as a daemon, which led me to this page.

developer$ rails s -p 8603 -b 0.0.0.0 -d
=> Booting Puma
=> Rails 5.1.6 application starting in development 
=> Run `rails server -h` for more startup options
me:icash developer$ 
FATAL: prematurely zombied 

I upgraded to 10.12.6 (from .5) and jQuery is now running, but I still got the FATAL error. I'll assume the server startup process got a little further. The rails version is using rb-fsevent 0.10.3 so it was not a fix.

The server command works on my laptop (air) but fails on the MacMini, which has server software installed but not used. Not sure where to go except High Sierra.

[rails-bot]

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-2-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[rails-bot]

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-2-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[tbsvttr]

I had problems similar to those described here on macOS Mojave (10.14.4), Ruby 2 (2.6.2p47) Rails 5 (5.2.3). I could solve them by reinstalling the gems listen and rb-fsevent:

gem uninstall listen rb-fsevent && gem install listen rb-fsevent

[tbsvttr]

This issue has been automatically marked as stale because it has not been commented on for at least three months.
The resources of the Rails team are limited, and so we are asking for your help.
If you can still reproduce this error on the 5-2-stable branch or on master, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

I have problems similar to those described here on macOS Mojave (10.14.4), Ruby 2 (ruby 2.6.3p62) Rails 5 (5.2.3). I could solve them by reinstalling the gems listen and rb-fsevent:

gem uninstall listen rb-fsevent
gem install listen rb-fsevent

However, I need to do this on a regular basis...

[tgxworld]

A fix has been merged into the Listen gem to resolve this issue.

[ldonis]

I'm facing this issue since a while.
ruby 2.7.2
rails 6.1.0
macos 11.1

In my CPU monitor I can see
ruby 99%
fseventsd 0.1%

remove config.file_watcher = ActiveSupport::EventedFileUpdateChecker helps a lot. after this change ruby process goes up to 60% maximum while the browser is loading, then it goes below 10%