Discussions

Although Celery is the most used solution to execute distributed asynchronous tasks in python and django-channels is the new hype, this project offers a solution based on uWSGI spooler, which requires no additional components, is particularly easy to setup, and has a straight learning curve.

Pre-requisites

uWSGI is normally used ad an application server, to accept requests, transfer control to the python web application using the wsgi protocol, and send the response back.

If configured as shown in this documentation, it can spawn some processes to handle asynchronous tasks, reading the queue from a specified spool directory.

The role of the spooler

The following snippet of code starts a uWSGI server able to process both HTTP requests and asynchronous tasks 1:

uwsgi --check-static=./static --http=:8000 --master \
  --module=wsgi --callable=application \
  --pythonpath=./ \
  --processes=4 --spooler=./uwsgi-spooler --spooler-processes=2

  • 4 processes will accept HTTP requests and send HTTP responses;

  • 2 processes will check the spooler and execute tasks there;

  • 1 master process will superintend all other processes.

  • the ./uwsgi-spooler path is the physical location on disk where the spooled tasks will be kept

Footnotes

1

Setting up uWSGI in production usually involves some sort of frontend proxy, but this is not the place to discuss it.