Tdarr V2: Distributed Transcoding System (Patreon)

Hello!

As promised, Tdarr V2 is now out. You can find the setup instructions here:

http://tdarr.io/docs/

For those on Docker, from 31st Jan onwards, the 'latest' tag will contain Tdarr V2. Until then it will contain V1. Giving a 1-week heads-up.

It's been a pretty intense couple of months of working on V2 and I know that this release won't work for everyone but, as long as the Server and Nodes can read/write from your libraries and transcode cache, it's running pretty well. I already have a version in development with some new stats for Patreon supporters but for the next couple of weeks, I'll be focusing purely on bug fixes from whatever weird situations come up.

Here is a quick rundown of the features:

• A new modularised system whereby Tdarr Nodes can be placed on multiple machines (cross-platform) which work together with Tdarr Server to process your files.
• 'Click- to-run' binaries for Windows, Linux, macOS and Arm (no need to manually install extra dependencies)
• An updater module which makes upgrading to the latest version much easier.
• Embedded database with live backup (no need for MongoDB)
• GPU and CPU transcode workers so you can maximise hardware usage (there's an option to allow GPU workers to do CPU tasks too which pretty much turns the worker into the old 'General' worker type from V1)
• If using multiple plugins, files are fully processed in the transcode cache before being moved back to the library or output folder (instead of the 'back and forth at each plugin step' method of V1 )
• New files are kept in the cache until you accept them so you can review them (there's an auto-accept option).
• Plugins can now auto-install any required external dependencies
• New UI and themes
• New API endpoints (see the docs for more details)
• A tonne of bug fixes which plagued V1 (see the changelog for more info).

Due to a large number of changes, V1 is not compatible with V2, sorry about that.

I have worked on this every day since mid-November. Here are some insights:

~ 400 hours (would like to have done more but I have to work around my full-time job)

~ 900 commits

~ 70 PRs

-15 iterations/versions

- What seems like a tonne of pizza

Building a cross-platform application like this is incredibly time-consuming - you implement a feature, it works for you but not someone else, you make fix which then breaks something else on a platform/environment which you don't have etc etc so thanks for being patient.

A huge thanks to the following on Discord who have helped a lot with testing:

Jabartik

Vodka

Jegler

supersnellehenk

HenkaN

morpheus2n2

Tronyx

If you have any issues/questions then feel free to message (preferably on Discord or Reddit as Patreons messaging system is somewhat problematic).

Welcome to the new members and once again thanks for the support!

HGG

Changelog

- Beta v1.99.14 release [23rd January 2021]:

- Node option to allow GPU workers to do CPU tasks

- More UI optimisations

- api/v2/status endpoint added in

- Remove force processing option (Use video/audio transcode settings instead)

- Fix UI plugins bug when creating a plugin

- Fix local plugin bug (auto install import-fresh)

- Beta v1.99.13 release [20th January 2021]:

- Button to requeue all staged files

- Move failed items straight to the Transcode Error tab if no completed cache file

- Fix transcode bug when filename has quotes in

- Improve node path translation

- Beta v1.99.12 release [17th January 2021]:

- UI performance improvements including reduced lag when lots of workers running

- Properly end subprocesses (ccextractor, ffmpeg etc)

- Increase time allowed for newly scanned files to be written to DB

- Reduce number of threads when reading from DB to prevent thread timeout errors when running on an HDD

- Increase RAM limit from 1.7GB default to 16GB for those with huge libraries

- Post-processing plugins now run both after transcoding and if transcoding is not required

- Can specify external dependencies in plugins, will be installed automatically

- Create backup of config JSON files before overwriting with defaults when config is invalid

- Faster health checks due to reduced file scanning

- Updated API routes

- 'Copy if conditions met' now works on output folder option

- Fix file scanner bug if file can't be accessed

- Fix large RAM usage bug introduced in v1.99.11

- Fix large worker error log bug

- Beta v1.99.11 release [10th January 2021]:

- Vastly improved non-blocking database with automatic live backup, UI statuses and multi-threaded reading (up to 50 threads)

- Multi-threaded non-blocking queue builder

- Multi-threaded non-blocking search functionality

- Server now checks if node is same version

- Verbose logs now tagged with 'verbose'.Moved multiple logs to verbose

- Allow retry copying file from cache if error (e.g. if drive is unplugged)

- Absolute backup path is now shown on backups tab

- Auto-scroll to results at top when cliking on pie section on stats tab

- Fix worker hanging bug in some situations (error log too large)

- Fix 'Toggle all' schedule button

- Fix text not showing when editing plugin in WebUI

- Fix cache copy error when accepting a cache file after a plugin has failed

- Fix cache copy error crashing cache copy service

- Fix plugin update bug if default community plugin repo can't be fetched

- Fix library prioritisation not working

- Fix node limit bug whereby updating limits on one node would update others

- Fix worker task request issue if node is not yet registered

- Fix items in Transcode Success/Error and Health check Success/Error not being sorted newest first

- Fix scenario whereby newly transcoded files were being scanned with extra scanners instead of just FFprobe

- Fix folder watcher adding cache files in some situations

- Beta v1.99.10 release [2nd January 2021]:

- Fix folder watcher not detecting new files

- Fix on-demand file scan not showing extra file details in UI

- Fix path bug when using a windows Node with a unix Server

- Workers will check all plugins to see if they can process any of them. Previously, if a GPU transcode plugin was after a CPU transcode plugin, a GPU worker would requeue a file with 'queued:requireCPU' as it would only check up to the CPU plugin.

- Add Node option to specify what type of GPU Node should use. This means that if you have, for example, a QSV and NVENC plugin in the plugin stack, you can set one Node to process with QSV and another (on the same machine) to process with NVENC, maximising hardware usage.

- Stats now record transcode and health check counts + space saved

- Updated startup tests. Group tests together

- Fix Windows HandBrake startup test

- Tdarr will now cleanup any SRTs which CCExtractor makes while scanning.

- Staging section is now saved so the server can be restarted while Nodes are still running (although that is not recommended)

- Option added to requeue files in staging section

- Staging section is more organised and grouped.

- Fix Node path translators bug on Windows

- Fix cache copy service loop bug if error

- Beta v1.99.09 release [31st December 2020]:

- Add basic Node schedules

- Fix toPrecision bug on Tdarr tab

- Fix worker bug - invalid cache file

- Beta v1.99.08 release [30th December 2020]:

- Tdarr Server + WebUI for Arm and MacOS

- Add cache copy service to fix copy bug

- Save worker limits

- Use active instead of used RAM in resource stats

- Option to set plugin repository

- Add history to 'folder to folder conversion'

- Fix bug when cancelling remote workers

- Prevent Node registering if ID is already taken

- chmod binaries on start

- Put Nodes into tabs

- Show plugin stack status in workers section

- Beta v1.99.07 release [27th December 2020]:

- Add error log to failed items

- Add dark, medium theme

- Use memory db for all databases

- Fix GPU health check scan bug

- Fix WebUI URL bug

- Beta v1.99.06 release [26th December 2020]:

- Docker containers

- Add verbose logs to scanner

- Option to set file scanner thread count

- Fix health check worker bug (not processing items)

- Add folder to folder conversion

- Add progress bar to DB read on startup

- Add serverPort config option

- Improved Plugin tab (quicker and more reliable when loading plugins)

- New worker UI

- Option to set webUIPort

- Fix folder browser bug if at root folder

- Fix Search tab bug

- Fix Discord login

- Update default Node IPs to 0.0.0.0

- Beta v1.99.05 release [20th December 2020]:

- Option to auto accept successful transcodes

- Update HandBrake on Windows to 1.3.3

- Improve default paths for binaries

- Add transcode and scanner binary tests on start-up

- Move all module configs + logs into same folder

- Beta v1.99.04 release [17th December 2020]:

- v2 Preview base. New modular system comprised of Tdarr Updater, Server, Node, WebUI and DesktopUI