Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hardware JPEG decoder #4576

Merged
merged 5 commits into from
Feb 11, 2025
Merged

Hardware JPEG decoder #4576

merged 5 commits into from
Feb 11, 2025

Conversation

cepetr
Copy link
Contributor

@cepetr cepetr commented Feb 4, 2025

This PR implements a hardware JPEG decoder for T3W1.

  • Implemented a new gfx/jpegdec module that wraps the STM32 hardware JPEG decoder.
  • Implemented a new trezorhal/jpegdec.rs module that provides the JpegDecoder API for Rust.
  • Modified shape::JpegImage to support the hardware jpeg decoder.
  • Additionally, two new features have been introduced - hw_jpeg_decoder and ui_jpeg - which replace jpeg and ui_jpeg_decoder.
  • Implemented missing blurring for Rgba8888 canvas
  • Implemented simple downscaling for Rgba8888->Rgba8888 bitblt copy operation

The simulator is still using the original JPEG decoder implemented in Rust.
All other models (except T3W1) should not be affected by this PR.

@cepetr cepetr added the T3W1 label Feb 4, 2025
@cepetr cepetr requested a review from TychoVrahe February 4, 2025 13:12
@cepetr cepetr self-assigned this Feb 4, 2025
Copy link

github-actions bot commented Feb 4, 2025

core UI changes device test click test persistence test
T2T1 Model T test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
T3B1 Safe 3 test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
T3T1 Safe 5 test(screens) main(screens) test(screens) main(screens) test(screens) main(screens)
All main(screens)

@prusnak
Copy link
Member

prusnak commented Feb 4, 2025

Can you post the results of the benchmark? I.e. how much faster is it than SW decoding on the same chip?

Also are there any limitations of the HW decoder when compared to SW encoding?

@cepetr
Copy link
Contributor Author

cepetr commented Feb 4, 2025

Can you post the results of the benchmark? I.e. how much faster is it than SW decoding on the same chip?

Also are there any limitations of the HW decoder when compared to SW encoding?

For 380×520 color images, software decoding takes approximately 600 ms, while hardware decoding takes about 10 ms (both include YCbCr to RGB conversion and display presentation).

According to the STM datasheet, the hardware decoder should comply with the JPEG standard and parse all standard JPEG headers. Unfortunately, it cannot decode progressive JPEGs (though the same applies to the software decoder we currently use).

One current limitation of the hardware decoder, compared to the software decoder we use, is that it cannot scale JPEG images. As a result, we cannot display a small preview of the uploaded image. However, this is not a problem if we slightly change the UI.

@cepetr cepetr force-pushed the cepetr/hw-jpeg-decoding branch 2 times, most recently from 852b6d6 to b565601 Compare February 4, 2025 17:35
@cepetr cepetr marked this pull request as ready for review February 4, 2025 17:35
@cepetr cepetr requested a review from prusnak as a code owner February 4, 2025 17:35
@matejcik
Copy link
Contributor

matejcik commented Feb 5, 2025

One current limitation of the hardware decoder, compared to the software decoder we use, is that it cannot scale JPEG images. As a result, we cannot display a small preview of the uploaded image. However, this is not a problem if we slightly change the UI.

we should be able to scale the image while streaming it -- unless the jpeg decoder is barfing pixels directly into the framebuffer?

@cepetr
Copy link
Contributor Author

cepetr commented Feb 5, 2025

One current limitation of the hardware decoder, compared to the software decoder we use, is that it cannot scale JPEG images. As a result, we cannot display a small preview of the uploaded image. However, this is not a problem if we slightly change the UI.

we should be able to scale the image while streaming it -- unless the jpeg decoder is barfing pixels directly into the framebuffer?

Yes, it would definitely be possible - and it wouldn’t be too slow if we simply skipped pixels and used (1/2, 1/4, …) scaling as before. However, it would still be much slower than just copying with DMA2D. You also have to consider that the image quality wouldn’t be perfect for some images due to aliasing.

@cepetr cepetr force-pushed the cepetr/hw-jpeg-decoding branch from 4019044 to 86c2993 Compare February 7, 2025 10:52
@cepetr
Copy link
Contributor Author

cepetr commented Feb 8, 2025

One current limitation of the hardware decoder, compared to the software decoder we use, is that it cannot scale JPEG images. As a result, we cannot display a small preview of the uploaded image. However, this is not a problem if we slightly change the UI.

we should be able to scale the image while streaming it -- unless the jpeg decoder is barfing pixels directly into the framebuffer?

Finally, I implemented simple downscaling so that JpegShape now works as it did before, without any limitations.

@cepetr cepetr force-pushed the cepetr/hw-jpeg-decoding branch from f3208bb to 052c85a Compare February 10, 2025 07:14
@TychoVrahe
Copy link
Contributor

One more thing i noticed about blurring on T3W1. Currently the corners of the display are bit miscolored, as accessing pixels out of display area actually returns white per GFXMMU settings. I changed that in this PR #4602

@cepetr
Copy link
Contributor Author

cepetr commented Feb 10, 2025

One more thing i noticed about blurring on T3W1. Currently the corners of the display are bit miscolored, as accessing pixels out of display area actually returns white per GFXMMU settings. I changed that in this PR #4602

Yes, I didn’t realize that. 0 as GFXMMU is much better, but still, it would be slightly better to change how we draw the blurred JPEG to eliminate these minor artifacts at the corners.

@cepetr cepetr force-pushed the cepetr/hw-jpeg-decoding branch from 7410636 to e39deba Compare February 11, 2025 07:44
@cepetr cepetr merged commit 7c7e8c8 into main Feb 11, 2025
94 checks passed
@cepetr cepetr deleted the cepetr/hw-jpeg-decoding branch February 11, 2025 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Approved
Development

Successfully merging this pull request may close these issues.

4 participants