Python-based utility for backing up files on Windows systems
MattCASmith.net | @MattCASmith
- Backutil is a learning/hobby project and some aspects of its code may not follow best
- practices. While you're welcome to use it, you do so at your own risk. Make sure you take
- a manual backup of your files before trying it out, and don't go relying on it to back
- up your production servers.Backutil is a simple, Python-based utility for backing up files from Windows systems to compressed, password-protected local archives. It has features for performing incremental backups and automatically rotating backup files. This is achieved using robocopy and 7-Zip, which must be installed.
To back up your files, simply ensure you have configured Backutil (see below) and run backutil.exe from the Command Prompt or PowerShell. The utility will report on its progress until the backup is successfully completed. More detail can also be found in backutil_log.csv.
When the utility is finished, you should find your complete backup files in your designated backup folder. The number and size of these backup files can be configured using the incremental backup and rotation settings, which are set in the configuration file or as command line options.
As Backutil automatically manages your backup files, it can be configured to run automatically at the desired interval using the Windows Task Scheduler. Backutil's features can be used to generate rolling full or incremental backups as required by your backup objectives and disk size.
Aside from all the testing that comes naturally during the development process, I have been using Backutil to back up my personal files since the start of 2021 using a Windows scheduled task to run the utility on a weekly basis. My configuration performs incremental backups on a five-file rotation and so far has worked without a hitch, to a level where I occasionally even forgot it was running.
Backutil can be configured via three main means: a configuration file, a file containing a list of directories to be backed up, and a series of command line options that override other settings. If a configuration file and backup list file are present, Backutil can be run using the following simple command.
.\backutil.exe
In terms of an installation directory, I put the executable and configuration files in C:\backutil\bin\ and use C:\backutil\ as the staging folder for the temporary files and records. However, you can put these files wherever you like as long as your settings are configured accordingly.
Backutil automatically loads settings from a file named config.ini, including the location of the list of directories to back up, folders for backups and temporary files, and incremental backup and rotation options. The configuration file should be located in the same folder as backutil.exe.
The contents of an example configuration file are shown below.
[LOCAL]
computer_name = matts-pc
backup_list = backup-list.txt
staging_folder = C:\backutil\
archive_pass = supersecretpassword
incremental = True
rotation = True
retained = 5
max_threads = 6
[SERVER]
server_directory = D:\backups\
The table below sets out what each option in the config.ini configuration file does. Note that all directories supplied via the configuration file must include the trailing backslash.
| Section | Key | Purpose |
|---|---|---|
| LOCAL | computer_name | Sets backup folder/record name |
| LOCAL | backup_list | Sets the backup list filename |
| LOCAL | staging_folder | Sets folder for temporary file storage |
| LOCAL | archive_pass | Sets 7-Zip backup file password |
| LOCAL | incremental | Turns incremental backups on/off (True/False) |
| LOCAL | rotation | Turns backup rotation on/off (True/False) |
| LOCAL | retained | Sets number of backups to retain if rotation is on |
| LOCAL | max_threads | Sets maximum number of threads for multiprocessing |
| SERVER | server_directory | Sets folder for backup storage |
The backup list file is a text file containing a list of directories. When Backutil is run, it will automatically generate a list of files to back up by scanning the contents of these directories and all subdirectories. The format of the backup list file should look something like the example below.
C:/Users/Matt/Desktop
C:/Users/Matt/Downloads
C:/Users/Matt/Music/iTunes/iTunes Media/Music
C:/Users/Matt/Pictures
C:/Users/Matt/Videos
Backutil also supports several options if you wish to set certain configuration parameters manually from the Command Prompt or PowerShell. Note that any parameters set via command line options will override the respective parameters in the config.ini configuration file.
| Short | Long | Purpose |
|---|---|---|
| -h | --help | Displays the help file |
| -n <name> | --name <name> | Manually sets the backup folder/record name |
| -l <file> | --list <file> | Manually sets the backup list file |
| -i | --incremental | Manually turns on incremental backups |
| -r <no> | --rotate <no> | Manually turns on backup rotation and sets number of backups |
| -t <no> | --threads <no> | Manually sets max threads for multiprocessing |
The following command shows an example of how the command line options may be used.
.\backutil.exe -n matts-pc -l locations.txt -i -r 5
Running Backutil with the options above will save backup files to a folder called matts-pc (note that this folder name is also how previous backups are tracked). The list of directories to back up files from will be retrieved from locations.txt. Backups will be incremental (only changed files will be backed up each time Backutil runs) and five previous backups will be retained. A maximum of six threads will be used for hash generation and file copy operations.
| Date | Version | Changes |
|---|---|---|
| 03/05/2021 | v0.70 | Implemented multiprocessing and other improvements: - Faster hash generation and file copying with multiprocessing - Improved terminal output and logging - General code improvements/tidying |
| 26/03/2021 | v0.61 | Implemented SQLite and other speed improvements: - All data processed using SQLite - Hashes generated using bigger file chunks - File size cut by 80 per cent due to Pandas removal |
| 19/02/2021 | v0.52 | Small bug fixes and improvements from v0.51: - 7-Zip file now generated directly in destination folder - Hash file now only generated after successful backup - Blank line at end of backup list file no longer required - Help page consistent with online documentation - Fixed --help and --incremental arguments |
My determination to build a minimum viable product before the end of 2020 means that I have a backlog of bug fixes and new features to add during 2021. These include:
-
Remote backups - You'll notice that some parts of Backutil use terminology associated with remote backups (for example, the Server section in the configuration file). This is because Backutil could originally be configured to use WinSCP to send backup files to a remote server. This has been removed for the initial release, but I hope to reinstate it in a future version.
-
Graphical user interface (GUI) - I've played around with Python GUIs a couple of times before, but have never had a script worth implementing one for. Depending on time limitations, I might develop a GUI for Backutil to increase ease of use for less experienced users.
-
Inconsistencies and bug fixes - The more time you spend with a piece of code, the more flaws you find in it. I'm sure I'll spot plenty to fix along the way.
Have you got more ideas for new Backutil features? Or have you found bugs that you think I haven't? Please send me an email to let me know so I can add them to the development backlog.
If you're interested in the project, check back regularly to see if there are new releases. I'll also announce any updates on my Twitter account and on my blog.