Repeated Tasks with Systemd Service/Timers on NixOS

Jun 16, 2021 22:10 · 1097 words · 6 minute read NixOS Systemd Automation

Systemd provides an excellent suite of tools for automating repeated tasks. It handles logging, dependency management, and scheduling. When combined with NixOS, you get the added benefits of package isolation and declarative configuration. I’ve used this as a robust framework for automating everything from web scraping to offsite backups. Let’s walk through how you might be able to utilize this with your existing scripts. We’ll start by create a script called myscript.sh with the contents:

echo "The date is $(date)"

We’ll update our /etc/nixos/configuration.nix to create our service and timer to call the script:

systemd.services.my-repeating-task= {
  serviceConfig.Type = "oneshot";
  path = with pkgs; [ bash ];
  script = ''
    bash /path/to/myscript.sh
  '';
};
systemd.timers.my-repeating-task = {
  wantedBy = [ "timers.target" ];
  partOf = [ "my-repeating-task.service" ];
  timerConfig = {
    OnCalendar = "*:0/1";
    Unit = "my-repeating-task.service";
  };
};

It’s a good idea to keep my-repeating-task the same between the timer and the service. If they differ, additional config options are required so they can find each other.

We create a oneshot service to call our script and exit since we’re not spawning a process. The service will be called every minute by our timer. If you want to test right away, you can run the following after rebuilding to check the logs:

sudo systemctl start my-repeating-task.timer
sudo journalctl -fu my-repeating-task

That’s a good start. The thing about having to read logs, though, is that it’s tedious. We don’t want to monitor; we want results! We can send notifications utilizing ssmtp and a dummy email. You can easily use a Gmail or yahoo account but be aware that some hosts will require you to use an app password instead of your regular password.

services.ssmtp = {
  enable = true;
  root = "dummyuser@domain.com";
  domain = "domain.com";
  hostName = "smtp.mail.domain.com:587";
  authUser = "dummyuser@domain.com";
  # This needs to be an app password not your regular one
  authPassFile = "/path/to/.ssmtp-authpass";
  useTLS = true;
  useSTARTTLS = true;
};

Then use sendmail to test that your SMTP settings are correct:

# If you don't have sendmail installed you can run temporarily install with
nix-shell -p system-sendmail
echo -e "To:me@domain.com\nFrom:dummyuser@domain.com\nSubject: hi\n\nbody text\n" | sendmail -t

Now the service can pipe the results of our script to sendmail to notify us. But what about service failures? Systemd can declare a service to run in the event of a failure. We’ll design a generic service that will receive the failed service’s name so it can email us with some relevant information:

systemd.services."notify-email@" = {
  serviceConfig.Type = "oneshot";
  path = with pkgs; [ systemd system-sendmail ];
  scriptArgs = "%I";
  script = ''
    UNIT=$(systemd-escape $1)
    TO="me@domain.com"
    FROM="dummyuser@domain.com"
    SUBJECT="$UNIT Failed"
    HEADERS="To:$TO\nFrom:$FROM\nSubject: $SUBJECT\n"
    BODY=$(systemctl status --no-pager $UNIT || true)
    echo -e "$HEADERS\n$BODY" | sendmail -t
  '';
};
systemd.services.my-repeating-service = {
  serviceConfig.Type = "oneshot";
  path = with pkgs; [ bash system-sendmail ]
  script = ''
    TO="me@domain.com"
    FROM="dummyuser@domain.com"
    SUBJECT="My first script"
    HEADERS="To:$TO\nFrom:$FROM\nSubject: $S
    BODY=$(bash /home/thorny/myscript.sh)
    echo -e "$HEADERS\n$BODY" | sendmail -t
  '';
  onFailure = [ "notify-email@%n.service" ];
};

Our updated service will now call our notify service if anything fails during execution. To test both scenarios you can:

  • execute the service as is. This will email the results of the script as expected.
  • add exit 1 to the end of myscript.sh and then execute the service. This will send the error email.

That covers the basics of the automation setup. The final part is managing the packages for each service. NixOS will run the script in isolation, and it won’t have access to the packages you’ve installed. With the power of nix’s package management, we can specify different versions of each service’s packages. If you need different versions of PostgreSQL for two different backups you could do the following:

systemd.services.db1-backup= {
  serviceConfig.Type = "oneshot";
  path = with pkgs; [ postgresql_10 ]
  script = ''
    # command to backup
  '';
  onFailure = [ "notify-email@%n.service" ];
};
systemd.services.db2-backup= {
  serviceConfig.Type = "oneshot";
  path = with pkgs; [ postgresql_13 ]
  script = ''
    # command to backup
  '';
  onFailure = [ "notify-email@%n.service" ];
};

We can even utilize the packages of other channels such as unstable if we want to use the latest code:

systemd.services.job-using-unstable =
let
  unstable = import <unstable> { config = config.nixpkgs.config; };
in
{
  serviceConfig.Type = "oneshot";
  path = with unstable; [ pkg1 ];
  script = ''
    # run stuff
  '';
  onFailure = [ "notify-email@%n.service" ];
};

TLDR; gimmie the code

services.ssmtp = {
  enable = true;
  root = "dummyuser@domain.com";
  domain = "domain.com";
  hostName = "smtp.mail.domain.com:587";
  authUser = "dummyuser@domain.com";
  # This needs to be an app password not your regular one
  authPassFile = "/path/to/.ssmtp-authpass";
  useTLS = true;
  useSTARTTLS = true;
};
systemd.services."notify-email@" = {
  serviceConfig.Type = "oneshot";
  path = with pkgs; [ systemd system-sendmail ];
  scriptArgs = "%I";
  script = ''
    UNIT=$(systemd-escape $1)
    TO="me@domain.com"
    FROM="dummyuser@domain.com"
    SUBJECT="$UNIT Failed"
    HEADERS="To:$TO\nFrom:$FROM\nSubject: $SUBJECT\n"
    BODY=$(systemctl status --no-pager $UNIT || true)
    echo -e "$HEADERS\n$BODY" | sendmail -t
  '';
};
systemd.services.my-repeating-service = {
  serviceConfig.Type = "oneshot";
  path = with pkgs; [ bash system-sendmail ]
  script = ''
    TO="me@domain.com"
    FROM="dummyuser@domain.com"
    SUBJECT="My first script"
    HEADERS="To:$TO\nFrom:$FROM\nSubject: $S
    BODY=$(bash /home/thorny/myscript.sh)
    echo -e "$HEADERS\n$BODY" | sendmail -t
  '';
  onFailure = [ "notify-email@%n.service" ];
};
systemd.timers.my-repeating-service = {
  wantedBy = [ "timers.target" ];
  partOf = [ "my-repeating-service.service" ];
  timerConfig = {
    OnCalendar = "daily";
    Unit = "my-repeating-service.service";
  };
};

This framework has now become the basis for all the personal tasks I automate. Adding new scripts is as easy as copying a few lines of code, as is moving an existing job to a different server. Using minimal external dependencies also means that I don’t have to worry about this solution becoming outdated for a long time. If you love automating all the small things, I highly recommend giving NixOS a spin.

Update (2022-08-03)

I’ve just updated from NixOS 21.11 to 22.05 and services.ssmtp has been replaced by programs.msmtp. There are some minor tweaks from above. Most notably, you can include the from address in the accounts section. If you do not, sendmail -t errors with envelope-from address is missing which can be solved by using sendmail -t --read-envelope-from instead. It’s up to you.

programs.msmtp = {
  enable = true;
  defaults = {
    tls = true;
    port = 587;
  };
  accounts = {
    default = {
      auth = true;
      from = "dummyuser@domain.vom";
      host = "smtp.mail.domain.com:587";
      passwordeval = "cat /path/to/.ssmtp-authpass";
      user = "dummyuser@domain.com";
    };
  };
};
# Then the systemd parts don't require the FROM anymore
systemd.services."notify-email@" = {
  serviceConfig.Type = "oneshot";
  path = with pkgs; [ systemd system-sendmail ];
  scriptArgs = "%I";
  script = ''
    UNIT=$(systemd-escape $1)
    TO="me@domain.com"
    SUBJECT="$UNIT Failed"
    HEADERS="To:$TO\nSubject: $SUBJECT\n"
    BODY=$(systemctl status --no-pager $UNIT || true)
    echo -e "$HEADERS\n$BODY" | sendmail -t
  '';
};