Programmatically Start Long-Running Command via SSH in Background and Disconnect (and get the PID)

UPDATE: Turns out, I googled the wrong thing/keywords. There is a helpful answer on StackOverflow.

It was surprisingly difficult to programmatically start a long-running command via SSH in background and disconnect the SSH session immediately. So, in case you need help, here’s how I did it. The final solution is ridiculously simple.

Goal

My goal is to programmatically (i.e., from a script) start a command on a remote server. So far so good, let’s use ssh user@host some_command where we don’t need an interactive session.

However, the ssh invocation should terminate as soon as the command has been dispatched without killing the command on the receiver side!

Solution

TL;DR: Just disconnect stdout and stderr (EOF) from ssh and ssh closes the connection while the process keeps running!

There are multiple solutions to disown processes out there, but most tutorials focus on interactive sessions. Here is the simplest solution I came up with for a non-interactive session that is flexible enough for my use-cases. No nohup logic, no disown, or other magic needed!

# This is a bash script where one task is to start a control on a remote 
# server. However, the command should just be dispatched, but no SSH session
# should blockingly wait for it.
#
# To have the flexibility to locate the process later, we also fetch the PID
# of that background task.

# `sleep 30`: the actual task
# `>/dev/null 2>/dev/null`: I/O redirection; disconnect client SSH stream (EOF)
# `&`: put process in background 
BG_PID=$(ssh testuser@192.168.124.42 "sleep 30 >/dev/null 2>/dev/null & echo \$\!")
echo "PID of background process on remote is $BG:PID"

I hope this helps! I’m not 100% sure where this behavior is specified, especially that ownership of the remote (and supposed to be long-running) process gets transferred to the init process instead of just being killed. But it works reliably!

I’m on NixOS 24.11 systems and OpenSSH_9.9p2.