Scalelite (scalelite-recording-importer) losing recordings

[pielonet]

Hi,

We faced an issue last week with a customer's recordings storage :

• The end storage disk was full
• Recordings tar archives were deposited in directory /var/bigbluebutton/spool
• Scalelite should have failed to decompress the archives to the full disk. Instead it did "as if" everything had succeeded : It deleted the source archive and wrote a new entry in its database to tell that the recording had properly been imported.
• This resulted in many lost recordings : Scalelite API tells the recording is present but it is actually not. The user receives a 404
  unrecoverable error

What we would have expected :

• Scalelite fails to decompress the recording tar archive to the full disk, detects it and keeps the source archive... and emits an alert !

I do not mean Scalelite (scalelite-recording-importer) should check if the final storage was full but more generally check if uncompressing the recording archive and moving it to the final storage succeeded or not. If not, Scalelite should not delete the archive but possibly mark it as undone and emit an alert. It should also not insert data into the database in that case.

Kind regards,
Thierry

[lonesomewalker]

In general: the idea for catching these nasty things is okay.

But:

• you should have a monitoring system for that
• the transfered recordings should be still on your BBB servers, so a simple re-transfer fixes everything
• how should Scalelite detect that after decompression the server is full?

Again: i agree it would be VERY nice to have a mechanism to prevent things like this.

[pielonet]

Hi,
@lonesomewalker
You're perfectly right : we do have a monitoring system for disk filling up. But we reacted to the alert too late.

Whatmore our BBB servers are only transient : when recordings have been transfered to Scalelite they are definitely destroyed on the BBB server and the BBB server itself may be destroyed at any time after every sessions have ended on it.

I guess scalelite-recording-importer service could decompress the archive locally in /var/bigbluebutton/spooland transfer the files via rsync to the final storage. In case the rsync transfer would fail - for any reason (disk full, network issue...) - it would keep the archive and emit an alert.

Kind regards,
Thierry