-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RSDK-9344] Use snappy for robot configuration compression #366
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,7 +2,14 @@ | |
use bytes::Bytes; | ||
use hyper::{http::uri::InvalidUri, Uri}; | ||
use prost::{DecodeError, Message}; | ||
use std::{cell::RefCell, ffi::CString, num::NonZeroI32, rc::Rc}; | ||
use snap::{read::FrameDecoder, write::FrameEncoder}; | ||
use std::{ | ||
cell::RefCell, | ||
ffi::CString, | ||
io::{Read, Write}, | ||
num::NonZeroI32, | ||
rc::Rc, | ||
}; | ||
use thiserror::Error; | ||
|
||
use crate::{ | ||
|
@@ -34,6 +41,8 @@ pub enum NVSStorageError { | |
NVSValueDecodeError(#[from] DecodeError), | ||
#[error(transparent)] | ||
NVSUriParseError(#[from] InvalidUri), | ||
#[error(transparent)] | ||
NVSRobotConfigCompressionError(#[from] std::io::Error), | ||
} | ||
|
||
#[derive(Clone)] | ||
|
@@ -246,13 +255,44 @@ impl RobotConfigurationStorage for NVSStorage { | |
} | ||
|
||
fn store_robot_configuration(&self, cfg: &RobotConfig) -> Result<(), Self::Error> { | ||
self.set_blob(NVS_ROBOT_CONFIG_KEY, cfg.encode_to_vec().into())?; | ||
// Normally, we would just rely on blob comparison in | ||
// `NVSStorage::set_blob` to dedup the write, but if snappy | ||
// compression of the robot config isn't entirely | ||
// deterministic, then that protection would be defeated. So, | ||
// check the preimages here instead, and if they are | ||
// equivalent, skip writing. | ||
if self.has_robot_configuration() { | ||
if let Ok(cached_cfg) = self.get_robot_configuration() { | ||
if cached_cfg == *cfg { | ||
return Ok(()); | ||
} | ||
} | ||
} | ||
|
||
let encoded_cfg = cfg.encode_to_vec(); | ||
let mut compressor = FrameEncoder::new(Vec::new()); | ||
compressor | ||
.write_all(&encoded_cfg[..]) | ||
.map_err(NVSStorageError::NVSRobotConfigCompressionError)?; | ||
let compressed = compressor | ||
.into_inner() | ||
.map_err(|e| NVSStorageError::NVSRobotConfigCompressionError(e.into_error()))?; | ||
log::info!( | ||
"robot configuration compressed for NVS storage: {} bytes before, {} bytes after", | ||
encoded_cfg.len(), | ||
compressed.len() | ||
); | ||
self.set_blob(NVS_ROBOT_CONFIG_KEY, compressed.into())?; | ||
Ok(()) | ||
} | ||
|
||
fn get_robot_configuration(&self) -> Result<RobotConfig, Self::Error> { | ||
let robot_config = self.get_blob(NVS_ROBOT_CONFIG_KEY)?; | ||
RobotConfig::decode(&robot_config[..]).map_err(NVSStorageError::NVSValueDecodeError) | ||
let mut decompressed_robot_config = vec![]; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One thing we could do to harden this here would be that if we fail to decompress the buffer with snappy, that we then try to just feed the raw blob |
||
FrameDecoder::new(&self.get_blob(NVS_ROBOT_CONFIG_KEY)?[..]) | ||
.read_to_end(&mut decompressed_robot_config) | ||
.map_err(NVSStorageError::NVSRobotConfigCompressionError)?; | ||
RobotConfig::decode(&decompressed_robot_config[..]) | ||
.map_err(NVSStorageError::NVSValueDecodeError) | ||
} | ||
|
||
fn reset_robot_configuration(&self) -> Result<(), Self::Error> { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing to contemplate here is what it might mean if we ever wanted to change this compression algorithm, or not do compression anymore. What would be the implications for upgrade? OTA?