Running Python as a Linux Service

By tom, April 28, 2017

This is a rather long post going into the detail of making a Linux service with Python. It’s long, but hopefully because it contains enough detail to be helpful! Let me know on Twitter in the usual place if it’s helped…

Aims

This post is going to cover how to get a Linux computer to run your Python code when it starts up. Why might you want to do this? Well, perhaps you’re building a robot and want to start up a control menu when you turn it on, or a weather station which should start capturing data when it’s connected to power (including after a power cut!). I’m using the Raspberry Pi 3 running Raspbian but these instructions should apply to any similar system.

Specifically, we want to:

1. Have code run on startup, because that’s the entire point of this post.
2. Be able to disable and enable your service easily. Most scenarios will have you create the service, enable it, and forget about it, but you may find you want to prevent it running later on so this needs to be easy.
3. Be able to manually stop, start and restart a running service. When you’re debugging in particular, or if you’ve disabled the service running on startup, you want to have easy control over the service lifecycle.
4. Run in a Python environment dedicated to this particular piece of code. We don’t want our service to be at the mercy of what other users do with the Python installation, and if we have more than one service we want them to be able to have their own libraries installed, or even different versions of Python itself.
5. Run as an arbitrary user. Sometimes you need to run as root, but you should try to avoid this, in particular for services that receive data from the network. Running your service as a lower privilege user reduces the potential impact of a security flaw in your service.

Overview

Our solution consists of three components:

1. We set up a virtual environment with the Python version and libraries we want to use for the service. See my previous post, ‘Isolating your Python with VirtualEnv’, for more details.
2. We write a Python script that contains the code we want to run, plus a bit of extra logic to manage changing the active user (if we ran as root before) and to handle shutdown events.
3. Finally we need a shell script which is managed by the service infrastructure in your Linux installation and which runs our Python code. We then tell the operating system that it should use this script as a service.

Virtual Environment

When you’re creating a service, you really don’t want to accidentally change something elsewhere which prevents it from running or which changes its behaviour. Python provides a mechanism to handle this, and in ‘Isolating your Python with VirtualEnv’ I outline why and how you can do this (see that post for more detail). For now we’ll create a new virtual environment in the default user’s home directory. Open a shell, and, as the ‘pi’ user, run the following:

cd /home/pi
virtualenv venv

You should see a new directory ‘venv’ created in your home directory, and if you look inside that directory you’ll see it contains a full Python installation. At this point you can activate the virtual environment and install any libraries you might need for your code. For example, I might want to install the picamera module because my code needs access to the camera module on the Pi – remember that this environment is completely separate from the system Python, so you’ll have to re-install any libraries you need even if they’re included in the default operating system installation of Python!

By way of example, here’s how I’d install picamera:

cd /home/pi
source venv/bin/activate
pip install picamera

We’ve now got our very own Python installation, which we’ll use exclusively for our new service, and we’ve installed any libraries we’re going to need. Next we need the actual Python code we want to run!

Python Code

The following code is a skeleton – it doesn’t do anything interesting but it handles all the plumbing your service needs to respond to shutdown signals, change users etc. Create a new file ‘myservice.py’ in the ‘pi’ user’s home directory and copy this into it:

#!/home/pi/venv/bin/python

import grp
import os
import pwd
from signal import signal, SIGINT, SIGTERM
from sys import exit

def drop_privileges(uid_name='nobody', gid_name='nogroup'):
    if os.getuid() != 0:
        # We're not root so, like, whatever dude
        return

    # Get the uid/gid from the name
    running_uid = pwd.getpwnam(uid_name).pw_uid
    running_gid = grp.getgrnam(gid_name).gr_gid

    # Reset group access list
    os.initgroups(uid_name, running_gid)

    # Try setting the new uid/gid
    os.setgid(running_gid)
    os.setuid(running_uid)

    # Ensure a very conservative umask
    old_umask = os.umask(077)

def get_shutdown_handler(message=None):
    """
    Build a shutdown handler, called from the signal methods
    :param message:
        The message to show on the second line of the LCD, if any. Defaults to None
    """

    def handler(signum, frame):
        # If we want to do anything on shutdown, such as stop motors on a robot,
        # you can add it here.
        print(message)
        exit(0)

    return handler

signal(SIGINT, get_shutdown_handler('SIGINT received'))
signal(SIGTERM, get_shutdown_handler('SIGTERM received'))

# Do anything you need to do before changing to the 'pi' user (our service
# script will run as root initially so we can do things like bind to low
# number network ports or memory map GPIO pins)

# Become 'pi' to avoid running as root
drop_privileges(uid_name='pi', gid_name='pi')

# Do the rest of your service after this point, you'll be running as 'pi'
while True:
    # This is where the main work of your service should go
    pass

Let’s go through this from the top:

The very first line is a comment, but it’s a special kind of comment. Specifically, a comment line which starts with #! will tell the operating system that the rest of the line is the command which should be run to run this script. So, putting this line there is saying that when Linux runs this Python script, it’s equivalent to saying /home/pi/venv/bin/python myservice.py. You’ll notice that this is the absolute path to the Python interpreter in our virtual environment – by specifying that we use this we’ll get any libraries we installed into that environment in the previous step.

Next we import some basic bits and pieces from the standard Python language so we can use them later.

Lines 10 to 27 define a function, drop_privileges which changes the current user. We’re going to configure the service to start running as root, but we don’t want to actually run the main meat of the service as the root user, that’s a terrible idea, so we create a function which will drop all our root privileges as soon as we don’t need them.

Lines 30 to 43 define a function, which itself returns a function (take a moment to look at the code if that sounds crazy!). The function it returns can be customised with a message, we’ll use this to create appropriate handler functions for different system events – the version here doesn’t really do anything, but if you had code you needed to run when your service shuts down this is the place to put it.

Lines 46 and 47 use our previously defined function and associates functions it produces with signals. Signals are a mechanism used by Linux to tell processes to terminate, stop, resume etc. The service infrastructure will send these signals to our script when we tell the service to stop, these lines will bind those signals to the functions we want to run when they’re received.

At this point you should put any code that needs root access. This might include opening a low number network port (for example, starting up a web server on port 80), or any other operation which requires a high level of access to the system. On my most recent robot I used a library which required root access to initialise but which then worked fine as a regular user, so I put the init functions for that library here.

Line 54 calls our drop_privileges function to become the ‘pi’ user. From this point on we are running as that user rather than root. This matters, because it means that any potential security flaw in your service (such as failing to be careful enough when processing input from a web server) can only give the attacker access to a relatively unprivileged user. If they could gain root access they could do literally anything they wanted.

Line 57 is your ‘real’ service code. By this point you’ll be running as the appropriate user, you’ll be configured to respond to shutdown messages and can get on with whatever it is your service actually does! In my robot example, here’s where I look for a controller and start responding to button presses, show my control menus on the LCD display etc etc.

Shell Script

Before you go any further, you need to decide on a name for your service! This is more than just cosmetic, because you’ll need to make certain things match up; the names of these files does matter. In this example my service is called ‘viridia’, as that’s the name of my last PiWars robot. Obviously where you see that word in this script and the following explanation you’ll need to replace it with your service name.

Create a new file, named the same as your service, in /etc/init.d/. You’ll need to run as root to do this, so e.g.

sudo nano /etc/init.d/viridia

Copy the following into that file:

#!/bin/sh

### BEGIN INIT INFO
# Provides:          viridia
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Viridia Robot Service
# Description:       Starts up the environment and top level menu for Viridia
### END INIT INFO

# Change the next 2 lines to suit where you install your script
# and what you want to call it
DAEMON=/home/pi/myservice.py
DAEMON_NAME=viridia

# Add any command line options for your daemon here
DAEMON_OPTS=""

# This next line determines what user the script runs as.
# Root generally not recommended but could be required for some libraries.
DAEMON_USER=root

# The process ID of the script when it runs is stored here:
PIDFILE=/var/run/$DAEMON_NAME.pid

. /lib/lsb/init-functions

do_start () {
    log_daemon_msg "Starting system $DAEMON_NAME daemon"
    start-stop-daemon --start --background --pidfile $PIDFILE --make-pidfile --user $DAEMON_USER --chuid $DAEMON_USER --startas $DAEMON -- $DAEMON_OPTS
    log_end_msg $?
}
do_stop () {
    log_daemon_msg "Stopping system $DAEMON_NAME daemon"
    start-stop-daemon --stop --pidfile $PIDFILE --retry 10
    log_end_msg $?
}

case "$1" in

    start|stop)
        do_${1}
        ;;

    restart|reload|force-reload)
        do_stop
        do_start
        ;;

    status)
        status_of_proc "$DAEMON_NAME" "$DAEMON" && exit 0 || exit $?
        ;;

    *)
        echo "Usage: /etc/init.d/$DAEMON_NAME {start|stop|restart|status}"
        exit 1
        ;;

esac
exit 0

Again, line by line from the top:

Line 1 tells Linux that we should run this file using the shell.

Lines 3 to 11 are comments, but (again) a special kind of comment. These lines contain the information the service infrastructure needs to determine when your service should run, its name, description and suchlike. The Default-Start and Default-Stop sections contain ‘run levels’. When Linux start up and shuts down it moves through a set of states, and these lines are specifying that we should start up when entering some of those states and shut down on entering others. The effect of the lines as used here is to start on boot after the core of the system has been initialised, and to shutdown when we shutdown the operating system.

Lines 14 to 26 set up some variables we’ll use later. The comments should show what they’re for – DAEMON is your Python script from the previous step, DAEMON_NAME is the name of your service (see above!), DAEMON_OPTS is any extra stuff you need passed on the command line to your script, and DAEMON_USER is the user we’ll start as. In this case we start as root, but our Python code will quickly relinquish root access. If you never need root access in your Python code you can change the user here and not bother with any of the privilege management in the actual Python code.

Line 28 pulls in some premade functions we’ll then use.

The rest of the script isn’t something you have to edit – it uses the variables defined in lines 14 to 26 to run your code, setting up a file containing the process identifier so it can find it again later to stop it etc.

Enabling your Service

Once we’ve got our file in /etc/init.d/myservice (assuming we’re calling it ‘myservice’) you need to make it executable, and then tell the service infrastructure about it. This will read those comments at the top of the script containing things like run-levels and use the information there to configure the service framework such that your service will be started and stopped at the appropriate times.

This also makes your Python script executable.

As root, do the following:

chmod a+x /etc/init.d/myservice
chmod a+x /home/pi/myservice.py
update-rc.d myservice defaults

Your service is now installed! Next we can look at how to manage it.

Managing your new service

You can now use the service infrastructure provided by Linux to start, stop, restart, disable, enable and get the status of your service:

Disabling and enabling your service

If you’ve followed the instructions so far your service will be installed and enabled to run on startup. If you ever want to disable the service (you’re not using it for now, or for whatever reason you don’t want it to run automatically) you can do:

sudo update-rc.d -f myservice remove

To re-enable the service:

sudo update-rc.d myservice defaults

Starting, stopping and querying your service

You may, especially during development, want to manually control your service. For example, you might disable it from running automatically using the command above, but then once you’ve made some changes to the code you don’t really want to have to re-enable and reboot to have it run. For those cases you can use

sudo service myservice stop

… to stop a running service,

sudo service myservice start

… to start the service manually, and

service myservice status

… to see any messages about the service, check whether it’s running etc. (You can run this command as a regular user, you only need root access when you’re actually changing things).

Alternatives

You may find that some people recommend the cron tool to accomplish some of the same functionality. This works because cron, the unix scheduler, has an ‘on startup’ trigger, but it won’t get you the monitoring or management you can get by using the service infrastructure. Conversely, cron is absolutely the right option if you need your code to run once every Tuesday or similar.

Further reading

Viridia’s service script is the linux service script used by my PiWars robot, and viridia_service.py the Python code to run the robot service.

James Coyle’s Cheat Sheet is a quick reference to update-rc.d and service commands.

The Debian wiki page for LSB init scripts has more information about how to specify exactly how and when your script gets run by the service infrastructure.