You have a static HTML page with a form on it. And you want this form to be processed by a Python script when submitted. And you serve your page with NGINX. To make all that work you will also need an application server such as uWSGI.

Let’s see, how it’s done.

As you might know, web-servers usually cannot run scripts themselves, so they pass such requests to some other backend server. I think, the most common example of that would be PHP-based CMSes like WordPress.

In our case, we don’t have a full-blown CMS - we merely want to hook a Python script to some simple HTML form. And as I mentioned in the beginning, with NGINX such a setup can be implemented using uWSGI.

uWSGI is exactly a kind of “backend server” I mentioned above. It can process requests, which NGINX will pass to it, and execute your Python scripts.

For initial understanding of bare grounds I used this article. It has nice explanations for most of the steps.

Environment

$ lsb_release -d
Description:	Linux Mint 19.2 Tina

$ apt install python3 python3-dev python3-pip
$ pip install setuptools
$ pip install uwsgi

$ python --version
Python 3.6.8

$ pip --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

$ uwsgi --version
2.0.18

$ which uwsgi
/home/vasya/.local/bin/uwsgi

“Website” structure:

/var/www/html/
├── cgi-bin
│   ├── info.py
│   ├── some.py
│   └── uwsgi.ini
└── index.html

NGINX server config

Our server config (/etc/nginx/sites-available/default):

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    root /var/www/html;

    index index.html;

    server_name _;

    location / {
        try_files $uri $uri/ =404;
    }

    # any request ending with .py will go here
    location ~ \.py$ {
        # the full list is in /etc/nginx/uwsgi_params
        include    uwsgi_params;
        # where requests will be passed - a socket shared with uWSGI
        uwsgi_pass unix:/var/www/html/cgi-bin/wsgi.sock;
    }
}

uWSGI

uWSGI can be run with a bitch-ass-long line of arguments, or you can put those into a config file (/var/www/html/cgi-bin/uwsgi.ini):

[uwsgi]
module = info:application

master = true
processes = 5

socket = wsgi.sock
chmod-socket = 664
vacuum = true

die-on-term = true

Here info:application means that uWSGI will be looking for info.py file and function named application inside it.

And socket points to the socket, which will be used for communicating with NGINX. If you are running uWSGI from /var/www/html/cgi-bin/ folder, then you don’t need to specify the full path.

Now you can run uWSGI like this:

sudo -u www-data uwsgi --ini uwsgi.ini --need-app

The --need-app option is here just to make the application fail if it won’t find the script or function to call. I’ve put it here so you could immediately see if your configuration is fine.

Most likely you’ll want to create a (systemd) service for that, as naturally uWSGI will need to be running all the time in order for NGINX to reach it.

A simple script

First let’s test that it all works. Create a simple script (/var/www/html/cgi-bin/info.py)

def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/html')])
    return [ "Ololo!".encode("utf-8") ]

Parameters environ and start_response come from the specification.

Opening http://localhost/any.py in a web-browser will get you a web-page with this line of text (“Ololo!”). Note that you will get it by opening absolutely any request to localhost that ends with .py.

If you are getting an empty response body, make sure you have .encode("utf-8") in the returned value. At first I didn’t have it there and I was pretty puzzled by getting the right response status (200 OK) but not getting a response body. This answer and documentation revealed the reason.

If you are getting 502 Bad Gateway error, then check if NGINX user (www-data) has access to /var/www/html/cgi-bin/wsgi.sock, and obviously uWSGI user also has to have access to it (or you can just run it from www-data user too):

sudo chown -R www-data:www-data /var/www/html

So returning a simple line of text works. Let’s now try to generate a proper HTML document with some useful information, for instance a list of uWSGI parameters (environ variable) passed from NGINX - replace your current info.py with the following:

def application(environ, start_response):
    htmlResponse = [ "<!DOCTYPE html>" ]
    htmlResponse.append("<html>")
    htmlResponse.append("<body>")
    htmlResponse.append("<h1>environ variables</h1>")
    for k, v in environ.items():
        htmlResponse.append("<p>environ[{}] = {}</p>".format(k, v))
    htmlResponse.append("</body>")
    htmlResponse.append("</html>")
    start_response('200 OK', [('Content-Type', 'text/html')])
    return [ line.encode("utf-8") for line in htmlResponse ]

Opening any .py page in browser now (after restarting uWSGI) will give you the following:

A script behind an HTML form

Since simple scripts work, which means that NGINX and uWSGI can communicate with each other, we can now proceed with calling a Python script when submitting a web-form.

So, here’s a page (/var/www/html/index.html) with a form:

<!DOCTYPE html>
<html>
    <body>
        <h1>Some form</h1>
        <!-- that's where the form will be submitted to -->
        <form action="/absolutely/any/path.py" method="POST">
            <label>Enter some value</label>
            <!-- "name" attribute marks the variable -->
            <input type="text" name="someVal" />
            <!-- by default buttons have "submit" role -->
            <button>Send</button>
        </form>
    </body>
</html>

And here’s the script (/var/www/html/cgi-bin/some.py):

from cgi import parse_qs, escape

def application(environ, start_response):
    try:
        request_body_size = int(environ.get('CONTENT_LENGTH', 0))
    except (ValueError):
        request_body_size = 0

    request_body = environ['wsgi.input'].read(request_body_size)
    vals = parse_qs(request_body)

    #print(vals)

    # escape user unput
    someVal = vals.get(b"someVal")
    if someVal:
        someVal = escape(someVal[0].decode("utf-8"))

    htmlResponse = [ "<!DOCTYPE html>" ]
    htmlResponse.append("<html>")
    htmlResponse.append("<body>")
    htmlResponse.append("<h1>Result</h1>")
    if someVal:
        htmlResponse.append(f"<p>Your value: <span style='color:blue;'>{someVal}</span></p>")
    else:
        htmlResponse.append("<p style='color:red;'>You haven't entered shit!</p>")
    htmlResponse.append("</body>")
    htmlResponse.append("</html>")
    start_response("200", [("Content-Type", "text/html")])
    return [ line.encode("utf-8") for line in htmlResponse ]

Starting with Python 3.8 you’ll get this error:

ImportError: cannot import name 'parse_qs' from 'cgi' (/usr/lib/python3.8/cgi.py)

To fix it replace the import statement with this:

#from cgi import parse_qs, escape
from urllib.parse import parse_qs
from html import escape

Now modify /var/www/html/cgi-bin/uwsgi.ini for uWSGI to use some.py:

[uwsgi]
module = some:application
...

And run/restart it:

sudo -u www-data uwsgi --ini uwsgi.ini --need-app

Now open the form:

Enter some value and submit it:

Here’s the result:

So, web-form submits someVal value to /cgi-bin/some.py endpoint, which gets caught by NGINX and passed to uWSGI, which in turn calls the some.py script, and there we finally get the someVal value.

In this particular example the form’s action can have - you guessed it right - absolutely any path, as long as it ends with .py. But below you will find a CGI example, in which the path actually matters.

If you’d like to be able to call different scripts based on the incoming URL without using CGI mode, then I guess this can be achieved with uWSGI routing, although I haven’t investigated that option. If I understood it correctly, you will still have one main entry-point-kind-of script, and it will be calling other scripts based on URL parsing.

CGI mode

Running uWSGI in CGI mode allows you to have a whole bunch of Python scripts, which can be called depending on the URL path requested.

~~Sadly, I couldn’t make it work. Nevertheless, I’ll share my attempts, and hopefully someone will point me in the right direction.~~

To enable CGI you need to have uWSGI with CGI plugin. As I understood, by default (being installed with pip) it comes without this plugin. Following instruction from the documentation I did the following:

$ curl http://uwsgi.it/install | bash -s cgi /tmp/uwsgi
$ sudo mv /tmp/uwsgi /usr/bin

So now I have two uWSGI binaries in the system:

~/.local/bin/uwsgi - installed from pip;
/usr/bin/uwsgi - installed from script.

First thing I noticed is that the new one (installed from script) cannot run with a single application/module.

Here’s the output from uWSGI installed with pip:

$ sudo -u www-data ~/.local/bin/uwsgi --ini uwsgi.ini --need-app
[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Sun Oct 20 16:14:09 2019] ***
compiled with version: 7.4.0 on 20 October 2019 14:07:07
os: Linux-4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019
nodename: vasya-Parallels
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /var/www/html
detected binary path: /home/vasya/.local/bin/uwsgi
your processes number limit is 15501
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address wsgi.sock fd 3
Python version: 3.6.8 (default, Oct  7 2019, 12:59:55)  [GCC 8.3.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x560fd8ee5a60
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 437424 bytes (427 KB) for 5 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x560fd8ee5a60 pid: 28714 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 28714)
spawned uWSGI worker 1 (pid: 28715, cores: 1)
spawned uWSGI worker 2 (pid: 28716, cores: 1)
spawned uWSGI worker 3 (pid: 28717, cores: 1)
spawned uWSGI worker 4 (pid: 28718, cores: 1)
spawned uWSGI worker 5 (pid: 28719, cores: 1)

And here’s the output from uWSGI installed from script:

$ sudo -u www-data /usr/bin/uwsgi --ini uwsgi.ini --need-app
[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.18 (64bit) on [Sun Oct 20 16:13:30 2019] ***
compiled with version: 7.4.0 on 18 October 2019 12:14:16
os: Linux-4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019
nodename: vasya-Parallels
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /var/www/html
detected binary path: /usr/bin/uwsgi
your processes number limit is 15501
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address wsgi.sock fd 3
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 437424 bytes (427 KB) for 5 cores
*** Operational MODE: preforking ***
*** no app loaded. GAME OVER ***
VACUUM: unix socket wsgi.sock removed.

But okay, that’s not what we’re after anyway, so let’s just ignore it.

Running uWSGI in CGI mode requires NGINX site config to have uwsgi_modifier1 9:

location ~ \.py$ {
    include         uwsgi_params;
    uwsgi_modifier1 9;
    uwsgi_pass      unix:/var/www/html/cgi-bin/wsgi.sock;
}

If you won’t set it, then you’ll get the following error when launching uWSGI (and your scripts won’t execute):

-- unavailable modifier requested: 0 --

You also need to make the following modifications in the uWSGI config (/var/www/html/cgi-bin/uwsgi.ini):

[uwsgi]
plugin-dir = /usr/lib/uwsgi/plugins
plugins = cgi
cgi = /var/www/html/cgi-bin
cgi-allowed-ext = .py
cgi-helper = .py=python

master = true
processes = 5

socket = wsgi.sock
chmod-socket = 664
vacuum = true

die-on-term = true

Note that even though this new uWSGI from script is supposed to be having CGI plugin, I had to explicitly specify where to find it (/usr/lib/uwsgi/plugins). Without this line I was getting the following error:

$ sudo -u www-data /usr/bin/uwsgi --ini uwsgi.ini
[uWSGI] getting INI configuration from uwsgi.ini
open("./cgi_plugin.so"): No such file or directory [core/utils.c line 3724]
!!! UNABLE to load uWSGI plugin: ./cgi_plugin.so: cannot open shared object file: No such file or directory !!!

Weird, but okay. Like I said, explicitly providing plugin-dir path fixes that.

But the next problem I wasn’t able to solve. Every time I want to run some script I get 502 Bad Gateway error from NGINX, because uWSGI fails with unknown 500 error. At the same time, if I try to open some non-existent script, then I get 404 Not Found error from uWSGI.

For example, here’re 3 URLs:

http://localhost/another.py - I don’t have this script
http://localhost/info.py - this script I do have
http://localhost/some.py - and this script I do have

And here’re results I am getting from trying to open those URLs:

[pid: 9606|app: -1|req: -1/1] 127.0.0.1 () {42 vars in 624 bytes} [Fri Oct 18 15:40:50 2019] GET /another.py => generated 9 bytes in 0 msecs (HTTP/1.1 404) 2 headers in 71 bytes (0 switches on core 0)
[pid: 9604|app: -1|req: -1/2] 127.0.0.1 () {42 vars in 618 bytes} [Fri Oct 18 15:41:01 2019] GET /info.py => generated 0 bytes in 15 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
[pid: 9606|app: -1|req: -1/3] 127.0.0.1 () {42 vars in 618 bytes} [Fri Oct 18 15:41:05 2019] GET /some.py => generated 0 bytes in 15 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)

So obviously it finds the files (info.py and some.py), but why then it fails with 500 error?

Updates

04.08.2021 | CGI is not WSGI

…As it suddenly turned out (thanks a lot, mate!) almost two years later, CGI is not the same thing as WSGI (yeah, breaking news), and so uWSGI was rightfully returning 500 errors, as the scripts were not meant or rather not compatible for CGI mode.

With the very first simple script as an example, to make it work in CGI mode, it needs to be changed from:

def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/html')])
    return [ "Ololo!".encode("utf-8") ]

to:

htmlResponse = [ "Content-Type: text/html" ]
htmlResponse.append("Status: 200 OK")
htmlResponse.append("") # this empty line in the response is important
htmlResponse.append("Ololo!")
for line in htmlResponse:
    print(line)

or simply:

print("Content-Type: text/html")
print("Status: 200 OK")
print() # this empty line in the response is important
print("Ololo!")

So the main point here is that in CGI mode it should be “just scripts” without start_response object/function and they should write plain-text response right to the output.

It’s also worth to mention here that environ object is no longer available, so scripts for handling forms actions will work differently too:

import cgi

form = cgi.FieldStorage()
someVal = form.getvalue("someVal")

htmlResponse = [ "Content-Type: text/html" ]
htmlResponse.append("Status: 200 OK")
htmlResponse.append("")
htmlResponse.append("<!DOCTYPE html>")
htmlResponse.append("<html>")
htmlResponse.append("<body>")
htmlResponse.append("<h1>Result</h1>")
if someVal:
    htmlResponse.append(f"<p>Your value: <span style='color:blue;'>{someVal}</span></p>")
else:
    htmlResponse.append("<p style='color:red;'>You haven't entered shit!</p>")
htmlResponse.append("</body>")
htmlResponse.append("</html>")
for line in htmlResponse:
    print(line)

As you can see, the form data is still available, but now it comes from cgi.FieldStorage (here’s a short example). It is (should be) also possible to get it from environment variables, as it is described in this answer.

Newer versions

While I was reviving my setup, I discovered that online installer is no longer available, even though it is still mentioned in the documentation:

To build a single binary with CGI support:
curl http://uwsgi.it/install | bash -s cgi /tmp/uwsgi

There are a couple of issues reported in their repository about that, and actually these lines were already deleted from sources, but apparently updated documentation was never deployed to the actual website.

Either way, this option is no longer available, so to get CGI mode we’ll need to build the uWSGI from sources.

To save you some time, if you’ll clone the uWSGI repository and build the latest version (for me it was 2989143935775a21d6e8d30c7166c473bd1f78bd), then in the end you will get it failing with the following error for no good reason:

*** Operational MODE: preforking+threaded ***
initialized CGI path: /var/www/html/cgi-bin
*** no app loaded. GAME OVER ***

So instead of going with the latest commit just take the latest released version (for me it was 2.0.19.1).

The build process is rather simple (but it wasn’t fucking simple to figure that out):

$ cd /tmp
$ git clone --depth 1 --branch 2.0.19.1 git@github.com:unbit/uwsgi.git
$ cd uwsgi
$ python uwsgiconfig.py --build core
$ python uwsgiconfig.py --plugin plugins/python core
$ python uwsgiconfig.py --plugin plugins/cgi core

That way you will get a so-called “core” build, extended with the plugins you’ll need: python and cgi:

$ ls -L1 | grep -e 'uwsgi$' -e '_plugin.so$'
cgi_plugin.so
python_plugin.so
uwsgi

The updated uwsgi.ini looks like this:

[uwsgi]
plugin-dir = /tmp/uwsgi
plugins = python,cgi
cgi = /var/www/html/cgi-bin
cgi-allowed-ext = .py
cgi-helper = .py=python

master = true
processes = 5
threads = 10

socket = wsgi.sock
chmod-socket = 664
vacuum = true

As you can see, this time we need to load both cgi and python plugins. If you won’t build/load python plugin, you’ll get the following warning when running uWSGI:

!!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!
no request plugin is loaded, you will not be able to manage requests.
you may need to install the package for your language of choice, or simply load it with --plugin.
!!!!!!!!!!! END OF WARNING !!!!!!!!!!

Although my scripts were still working fine even without it.

If everything’s good, then having run uWSGI you should get the following output:

$ cd /var/www/html/cgi-bin/
$ sudo -u www-data /tmp/uwsgi/uwsgi --ini ./uwsgi.ini
[uWSGI] getting INI configuration from ./uwsgi.ini
*** Starting uWSGI 2.0.19.1 (64bit) on [Wed Aug  4 18:10:08 2021] ***
compiled with version: 9.3.0 on 04 August 2021 16:01:16
os: Linux-5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021
nodename: kubuntu2004vm
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /var/www/html/cgi-bin
detected binary path: /tmp/uwsgi/uwsgi
your processes number limit is 15371
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address wsgi.sock fd 3
Python version: 3.8.10 (default, Jun  2 2021, 10:49:15)  [GCC 9.4.0]
Python main interpreter initialized at 0x56416da808b0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 1002144 bytes (978 KB) for 50 cores
*** Operational MODE: preforking+threaded ***
initialized CGI path: /var/www/html/cgi-bin
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 23277)
spawned uWSGI worker 1 (pid: 23278, cores: 10)
spawned uWSGI worker 2 (pid: 23279, cores: 10)
spawned uWSGI worker 3 (pid: 23280, cores: 10)
spawned uWSGI worker 4 (pid: 23281, cores: 10)
spawned uWSGI worker 5 (pid: 23282, cores: 10)

I can now open http://localhost/info.py, and then info.py will be executed; if I open http://localhost/some.py, then some.py will be executed; and if I open a page with a form and submit it, then the script specified in action attribute will get executed (note that this time it should be not /absolutely/any/path.py but an actual path to the script).

So this is it, CGI mode works, yay.

Surely, now uwsgi executable, .so plugins and uwsgi.ini config need to be moved to a proper place (/usr/local/bin and so on), a service to launch uWSGI needs to be created, etc, but that’s just routine tasks, I won’t cover it here.

11.03.2022 | Django with uWSGI

Here’s also instructions for running a Django project with uWSGI behind NGINX.

Declaration of VAR

NGINX, uWSGI and Python scripts

Environment

NGINX server config

uWSGI

A simple script

A script behind an HTML form

CGI mode

Updates

04.08.2021 | CGI is not WSGI

Newer versions

11.03.2022 | Django with uWSGI

Feeds

Stuff

Tags

Declaration of VAR

Environment

NGINX server config

uWSGI

A simple script

A script behind an HTML form

CGI mode

Updates

04.08.2021 | CGI is not WSGI

Newer versions

11.03.2022 | Django with uWSGI

Feeds

Stuff

Social networks

Tags