You have a static HTML page with a form on it. And you want this form to be processed by a Python script when submitted. And you serve your page with NGINX. To make all that work you will also need an application server such as uWSGI.

NGINX and uWSGI

Let’s see, how it’s done.

Unsolved: I couldn't make the CGI-mode work

As you might know, web-servers usually cannot run scripts themselves, so they pass such requests to some other backend server. I think, the most common example of that would be PHP-based CMSes like WordPress.

In our case, we don’t have a full-blown CMS - we merely want to hook a Python script to some simple HTML form. And as I mentioned in the beginning, with NGINX such a setup can be implemented using uWSGI.

uWSGI is exactly a kind of “backend server” I mentioned above. It can process requests, which NGINX will pass to it, and execute your Python scripts.

For initial understanding of bare grounds I used this article. It has nice explanations for most of the steps.

Environment

$ lsb_release -d
Description:	Linux Mint 19.2 Tina

$ apt install python3 python3-dev python3-pip
$ pip install setuptools
$ pip install uwsgi

$ python --version
Python 3.6.8

$ pip --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

$ uwsgi --version
2.0.18

$ which uwsgi
/home/vasya/.local/bin/uwsgi

“Website” structure:

/var/www/html/
├── cgi-bin
   ├── info.py
   ├── some.py
   └── uwsgi.ini
└── index.html

NGINX server config

Our server config (/etc/nginx/sites-available/default):

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    root /var/www/html;

    index index.html;

    server_name _;

    location / {
        try_files $uri $uri/ =404;
    }

    # any request ending with .py will go here
    location ~ \.py$ {
        # the full list is in /etc/nginx/uwsgi_params
        include    uwsgi_params;
        # where requests will be passed - a socket shared with uWSGI
        uwsgi_pass unix:/var/www/html/cgi-bin/wsgi.sock;
    }
}

uWSGI

uWSGI can be run with a bitch-ass-long line of arguments, or you can put those into a config file (/var/www/html/cgi-bin/uwsgi.ini):

[uwsgi]
module = info:application

master = true
processes = 5

socket = wsgi.sock
chmod-socket = 664
vacuum = true

die-on-term = true

Here info:application means that uWSGI will be looking for info.py file and function named application inside it.

And socket points to the socket, which will be used for communicating with NGINX. If you are running uWSGI from /var/www/html/cgi-bin/ folder, then you don’t need to specify the full path.

Now you can run uWSGI like this:

sudo -u www-data uwsgi --ini uwsgi.ini --need-app

The --need-app option is here just to make the application fail if it won’t find the script or function to call. I’ve put it here so you could immediately see if your configuration is fine.

Most likely you’ll want to create a (systemd) service for that, as naturally uWSGI will need to be running all the time in order for NGINX to reach it.

A simple script

First let’s test that it all works. Create a simple script (/var/www/html/cgi-bin/info.py)

def application(environ, start_response):
    start_response('200 damn', [('Content-Type', 'text/html')])
    return [ "Ololo!".encode("utf-8") ]

Parameters environ and start_response come from the specification.

Opening http://localhost/any.py in a web-browser will get you a web-page with this line of text (“Ololo!”). Note that you will get it by opening absolutely any request to localhost that ends with .py.

If you are getting an empty response body, make sure you have .encode("utf-8") in the returned value. At first I didn’t have it there and I was pretty puzzled by getting the right response status (200 damn) but not getting a response body. This answer and documentation revealed the reason.

If you are getting 502 Bad Gateway error, then check if NGINX user (www-data) has access to /var/www/html/cgi-bin/wsgi.sock, and obviously uWSGI user also has to have access to it (or you can just run it from www-data user too):

sudo chown -R www-data:www-data /var/www/html

So returning a simple line of text works. Let’s now try to generate a proper HTML document with some useful information, for instance a list of uWSGI parameters (environ variable) passed from NGINX:

def application(environ, start_response):
    htmlResponse = [ "<!DOCTYPE html>" ]
    htmlResponse.append("<html>")
    htmlResponse.append("<body>")
    htmlResponse.append("<h1>environ variables</h1>")
    for k, v in environ.items():
        htmlResponse.append("<p>environ[{}] = {}</p>".format(k, v))
    htmlResponse.append("</body>")
    htmlResponse.append("</html>")
    start_response('200 OK', [('Content-Type', 'text/html')])
    return [ line.encode("utf-8") for line in htmlResponse ]

Opening a .py page now will give you the following:

uWSGI environ variables

A script behind an HTML form

Since simple scripts work, which means that NGINX and uWSGI can communicate with each other, we can now proceed with calling a Python script when submitting a web-form.

So, here’s a page (/var/www/html/index.html) with a form:

<!DOCTYPE html>
<html>
    <body>
        <h1>Some form</h1>
        <!-- that's where the form will be submitted to -->
        <form action="/cgi-bin/some.py" method="POST">
            <label>Enter some value</label>
            <!-- "name" attribute marks the variable -->
            <input type="text" name="someVal" />
            <!-- by default buttons have "submit" role -->
            <button>Send</button>
        </form>
    </body>
</html>

And here’s the script (/var/www/html/cgi-bin/some.py):

from cgi import parse_qs, escape

def application(environ, start_response):
    try:
        request_body_size = int(environ.get('CONTENT_LENGTH', 0))
    except (ValueError):
        request_body_size = 0

    request_body = environ['wsgi.input'].read(request_body_size)
    vals = parse_qs(request_body)

    #print(vals)

    # escape user unput
    someVal = vals.get(b"someVal", [b"[ you haven't entered shit ]"])[0]
    someVal = escape(someVal.decode("utf-8"))

    htmlResponse = [ "<!DOCTYPE html>" ]
    htmlResponse.append("<html>")
    htmlResponse.append("<body>")
    htmlResponse.append("<h1>Result</h1>")
    htmlResponse.append(f"<p>Your value: <span style='color:blue;'>{someVal}</span></p>")
    htmlResponse.append("</body>")
    htmlResponse.append("</html>")
    start_response("200", [("Content-Type", "text/html")])
    return [ line.encode("utf-8") for line in htmlResponse ]

Modify /var/www/html/cgi-bin/uwsgi.ini for uWSGI to use some.py:

[uwsgi]
module = some:application
...

And run/restart it:

sudo -u www-data uwsgi --ini uwsgi.ini

Now open the form:

HTML form

Enter some value and submit it:

HTML form input

Here’s the result:

HTML form result

So, web-form submits someVal value to /cgi-bin/some.py endpoint, which gets caught by NGINX and passed to uWSGI, which in turn calls the some.py script, and there we finally get the someVal value.

In this particular example the form’s action can have - you guessed it right - absolutely any path, as long as it ends with .py. But below you will find a CGI example, in which the path actually matters.

CGI mode

CGI is more powerful than running a single script. Shortly saying, running uWSGI in CGI mode allows you to have a whole bunch of Python scripts, which are called depending on the path requested.

Sadly, I couldn’t make it work. Nevertheless, I’ll share my attempts, and hopefully someone will point me in the right direction.

To enable CGI you need to have uWSGI with CGI plugin. As I understood, by default (being installed with pip) it comes without this plugin. Following instruction from the documentation I did the following:

$ curl http://uwsgi.it/install | bash -s cgi /tmp/uwsgi
$ sudo mv /tmp/uwsgi /usr/bin

So now I have two uWSGI binaries in the system:

  1. ~/.local/bin/uwsgi - installed from pip;
  2. /usr/bin/uwsgi - installed from script.

First thing I noticed is that the new one (installed from script) cannot run with a single application/module.

Here’s the output from uWSGI installed with pip:

$ sudo -u www-data ~/.local/bin/uwsgi --ini uwsgi.ini --need-app
[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Sun Oct 20 16:14:09 2019] ***
compiled with version: 7.4.0 on 20 October 2019 14:07:07
os: Linux-4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019
nodename: vasya-Parallels
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /var/www/html
detected binary path: /home/vasya/.local/bin/uwsgi
your processes number limit is 15501
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address wsgi.sock fd 3
Python version: 3.6.8 (default, Oct  7 2019, 12:59:55)  [GCC 8.3.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x560fd8ee5a60
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 437424 bytes (427 KB) for 5 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x560fd8ee5a60 pid: 28714 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 28714)
spawned uWSGI worker 1 (pid: 28715, cores: 1)
spawned uWSGI worker 2 (pid: 28716, cores: 1)
spawned uWSGI worker 3 (pid: 28717, cores: 1)
spawned uWSGI worker 4 (pid: 28718, cores: 1)
spawned uWSGI worker 5 (pid: 28719, cores: 1)

And here’s the output from uWSGI installed from script:

$ sudo -u www-data /usr/bin/uwsgi --ini uwsgi.ini --need-app
[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Sun Oct 20 16:13:30 2019] ***
compiled with version: 7.4.0 on 18 October 2019 12:14:16
os: Linux-4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019
nodename: vasya-Parallels
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /var/www/html
detected binary path: /usr/bin/uwsgi
your processes number limit is 15501
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address wsgi.sock fd 3
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 437424 bytes (427 KB) for 5 cores
*** Operational MODE: preforking ***
*** no app loaded. GAME OVER ***
VACUUM: unix socket wsgi.sock removed.

But okay, that’s not what we’re after anyway, so let’s just ignore it.

Running uWSGI in CGI mode requires NGINX site config to have uwsgi_modifier1 9:

location ~ \.py$ {
    include         uwsgi_params;
    uwsgi_modifier1 9;
    uwsgi_pass      unix:/var/www/html/cgi-bin/wsgi.sock;
}

You also need to make the following modifications in the uWSGI config (/var/www/html/cgi-bin/uwsgi.ini):

[uwsgi]
plugin-dir = /usr/lib/uwsgi/plugins
plugins = cgi
cgi = /var/www/html/cgi-bin
cgi-allowed-ext = .py
cgi-helper = .py=python

master = true
processes = 5

socket = wsgi.sock
chmod-socket = 664
vacuum = true

die-on-term = true

Note that even though this new uWSGI from script is supposed to be having CGI plugin, I had to explicitly specify where to find it (/usr/lib/uwsgi/plugins). Without this line I was getting the following error:

$ sudo -u www-data /usr/bin/uwsgi --ini uwsgi.ini
[uWSGI] getting INI configuration from uwsgi.ini
open("./cgi_plugin.so"): No such file or directory [core/utils.c line 3724]
!!! UNABLE to load uWSGI plugin: ./cgi_plugin.so: cannot open shared object file: No such file or directory !!!

Weird, but okay. Like I said, explicitly providing plugin-dir path fixes that.

But the next problem I wasn’t able to solve. Every time I want to run some script I get 502 Bad Gateway error from NGINX, because uWSGI fails with unknown 500 error. At the same time, if I try to open some non-existent script, then I get 404 Not Found error from uWSGI.

For example, here’re 3 URLs:

  1. http://localhost/another.py
  2. http://localhost/info.py
  3. http://localhost/some.py

And here’re results I am getting from trying to open those:

[pid: 9606|app: -1|req: -1/1] 127.0.0.1 () {42 vars in 624 bytes} [Fri Oct 18 15:40:50 2019] GET /another.py => generated 9 bytes in 0 msecs (HTTP/1.1 404) 2 headers in 71 bytes (0 switches on core 0)
[pid: 9604|app: -1|req: -1/2] 127.0.0.1 () {42 vars in 618 bytes} [Fri Oct 18 15:41:01 2019] GET /info.py => generated 0 bytes in 15 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
[pid: 9606|app: -1|req: -1/3] 127.0.0.1 () {42 vars in 618 bytes} [Fri Oct 18 15:41:05 2019] GET /some.py => generated 0 bytes in 15 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)

So obviously it finds the files (info.py and some.py), but why then it fails with 500 error?

What I also noticed is that I get the same result from uWSGI installed with pip (~/.local/bin/uwsgi). So what was the point in installing the one from script (/usr/bin/uwsgi), how is this one different?

Anyway, I would be happy if someone could help me to understand what’s wrong here.