Make datapusher uwsgi concurrent #201
Conversation
By default, uwsgi runs with a single process and a single thread.
After extensive testing under heavy load, added "lazy-apps" option. Even though pscopg2 is thread-safe, its not "fork-safe", and was getting this error: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac Lazy-apps took care of the problem by making sure each worker loads a copy of the app, and not using a fork, where the psycopg2 connection is made BEFORE the fork, causing the problem. See links below for more info https://www.psycopg.org/docs/usage.html#thread-safety https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html Also used `workers` synonym for `processes` and dialed down workers/threads to more conservative values.
|
After extensive testing under heavy load, added "lazy-apps" option. Even though pscopg2 is thread-safe, its not fully "fork-safe", and was getting this error: This happens because the connection was created BEFORE the fork, which is a problem per psycopg2 docs: https://www.psycopg.org/docs/usage.html#thread-safety Lazy-apps took care of the problem by making sure each worker loads a copy of the app, and not using a fork, so each worker gets its own connection pool. For more info on 'lazy-apps', see Also used |
|
@jqnatividad This looks good, and as discussed in #199, we should document to use single thread when using sqlite and use this config when using different backends like postgres. An idea would be to have the default be single thread and have the multi-thread options commented out and documented in the readme for people that want to enable that if they choose to use the postgres backend. |
|
@jqnatividad @mbocevski please check #207, particularly the new section that describes this setup |
Fixes #200
By default, uwsgi runs with a single process and a single thread.
This changes it to 6 processes/workers with each process/worker having 15 threads, reflecting the same tweak @metaodi did in #147 (comment) for the old Apache based deployment.
Along with using PostgreSQL instead of sqlite (PR #199), this has the added benefit of making datapusher throughput much higher as its now fully concurrent!