Parallel Recursive FTPS Mirroring (Directory Download) from Command Line

Published by Philipp Schuster on 2024-04-172024-04-17

TL;DR

In this blog post, I show you the best option I found that you can use in 2024 to perform a parallel recursive download (backup) of an FTP directory via FTPS (explicit FTP via TLS) from the command line. lftp is the best command line tool (for Linux, macOS and other UNIX-like systems), but needs to be properly configured. For GUI-focused users, I recommend FileZilla.

Introduction

FTP is a quite outdated protocol these days. Apart from accessing web storage on various simple web hosting providers, I’m not aware of any real and justified use of this technology in 2024. “Modern” alternatives such as SFTP are usually better suited, but you need SSH access to a server for that.

The Problem

Not every (old-fashioned) web hosting provider gives you an SSH/SFTP access (at least in the lower paying tiers). I’m also “trapped” in such a situation. I’m having various (10+ year) old web projects that still live on such a web hosting provider, and I just didn’t migrate them yet onto something I have more control over.

Downloading the website files onto a locally owned storage, especially when having PHP-centric software, such as WordPress or Matomo, is crucial as backup to be independent of the provider. As finding information on how to download files recursively and parallel via FTPS (explicit FTP over TLS) was not as easy or as compact to find as hoped, here’s my solution. In the end, it is quite simple.

The Solution: lftp

lftp is an open-source CLI tool available in most Linux distributions and package managers. All it takes is the following bash command:

lftp -u $USER,$PASS -e "\
	set ftp:ssl-force true \
	set ftp:ssl-protect-data true \
	" $HOST <<EOF

mirror --continue --parallel=10 --verbose $REMOTE_DIR $LOCAL_DIR

quit
EOF

It performs a

• TLS-secured FTP transmission (no credentials are transferred in plain text)
• recursive mirroring of the specified remote directory
• 10 parallel connections (FTP is slow: so parallel connections mitigate that to a certain degree)

Convenient Wrapper Script

I’ve created a convenient wrapper script acting as high-level CLI tool around lftp. You can find it as part of my NixOS configs and consume it like this (Nix required):

• $ nix shell github:phip1611/nixos-configs#ftp-backup.
• $ ftp-backup --help