Skip to content

Commit

Permalink
refactor: Rename vith to vignet
Browse files Browse the repository at this point in the history
Signed-off-by: Vincent Boutour <[email protected]>
  • Loading branch information
ViBiOh committed Sep 17, 2024
1 parent 9830cd1 commit d637f4d
Show file tree
Hide file tree
Showing 12 changed files with 76 additions and 80 deletions.
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,17 +39,17 @@ Fibr creates a `.fibr` folder in _root folder_ for storing its metadata: shares'

### Sidecars

Fibr generates thumbnails of images, PDF and videos when these [mime-types are detected](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types) and sidecars are provided. Sidecars are [ViBiOh/vith](https://github.com/vibioh/vith) and [ViBiOh/exas](https://github.com/vibioh/exas). Thumbnails are generated in [WebP](https://developers.google.com/speed/webp/) format, in their animated format for video thumbnail.
Fibr generates thumbnails of images, PDF and videos when these [mime-types are detected](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types) and sidecars are provided. Sidecars are [ViBiOh/vignet](https://github.com/vibioh/vignet) and [ViBiOh/exas](https://github.com/vibioh/exas). Thumbnails are generated in [WebP](https://developers.google.com/speed/webp/) format, in their animated format for video thumbnail.

You can refer to these projects for installing and configuring them and set `-thumbnailURL` and `-exifURL` options.

Sidecars may have constraints regarding concurrent work (e.g. HLS conversion is a CPU-intensive task) or rate limit (e.g. geocoding can have rate-limiting). Call to these sidecars can be made with HTTP, which is not fault tolerant but easy to setup, or with an AMQP messaging, which is more resilient but more complex to setup. An easy-to-setup AMQP messaging instance can be done with [CloudAMQP](https://www.cloudamqp.com) (I have no affiliation of any kind to this company, just a happy customer). When AMQP connection URI is provided, Fibr will use it as default communication protocol instead of HTTP.

#### HTTP Live Streaming

Fibr has a special treatment for videos, that can be very large sometimes. With the help of the `vith` sidecar, it can convert a video to its [HLS version](https://en.wikipedia.org/wiki/HTTP_Live_Streaming). It keeps the original video as is, and stores streamable version in the metadatas directory. It's a basic conversion into the appropriate format: no resolution, frame-per-second or any quality specifications are changed. Conversion is done where this two requirements are met altogether:
Fibr has a special treatment for videos, that can be very large sometimes. With the help of the `vignet` sidecar, it can convert a video to its [HLS version](https://en.wikipedia.org/wiki/HTTP_Live_Streaming). It keeps the original video as is, and stores streamable version in the metadatas directory. It's a basic conversion into the appropriate format: no resolution, frame-per-second or any quality specifications are changed. Conversion is done where this two requirements are met altogether:

- `vith` is configured with direct access to the filesystem (see [`vith`documentation about configuring `WorkDir`](https://github.com/vibioh/vith#usage) and [`fibr` configuration](#usage) for enabling it). Direct access disable large file transfer in the network.
- `vignet` is configured with direct access to the filesystem (see [`vignet`documentation about configuring `WorkDir`](https://github.com/vibioh/vignet#usage) and [`fibr` configuration](#usage) for enabling it). Direct access disable large file transfer in the network.
- the video bitrate is above [`thumbnailMinBitrate (default 80000000)`](#usage)

### Chunk upload
Expand Down Expand Up @@ -289,13 +289,13 @@ Usage of fibr:
--thumbnailAmqpExchange string [thumbnail] AMQP Exchange Name ${FIBR_THUMBNAIL_AMQP_EXCHANGE} (default "fibr")
--thumbnailAmqpStreamRoutingKey string [thumbnail] AMQP Routing Key for stream ${FIBR_THUMBNAIL_AMQP_STREAM_ROUTING_KEY} (default "stream")
--thumbnailAmqpThumbnailRoutingKey string [thumbnail] AMQP Routing Key for thumbnail ${FIBR_THUMBNAIL_AMQP_THUMBNAIL_ROUTING_KEY} (default "thumbnail")
--thumbnailDirectAccess [thumbnail] Use Vith with direct access to filesystem (no large file upload, send a GET request, Basic Auth recommended) ${FIBR_THUMBNAIL_DIRECT_ACCESS} (default false)
--thumbnailDirectAccess [thumbnail] Use Vignet with direct access to filesystem (no large file upload, send a GET request, Basic Auth recommended) ${FIBR_THUMBNAIL_DIRECT_ACCESS} (default false)
--thumbnailLargeSize uint [thumbnail] Size of large thumbnail for story display (thumbnail are always squared). 0 to disable ${FIBR_THUMBNAIL_LARGE_SIZE} (default 800)
--thumbnailMaxSize int [thumbnail] Maximum file size (in bytes) for generating thumbnail (0 to no limit). Not used if DirectAccess enabled. ${FIBR_THUMBNAIL_MAX_SIZE} (default 209715200)
--thumbnailMinBitrate uint [thumbnail] Minimal video bitrate (in bits per second) to generate a streamable version (in HLS), if DirectAccess enabled ${FIBR_THUMBNAIL_MIN_BITRATE} (default 80000000)
--thumbnailPassword string [thumbnail] Vith Thumbnail Basic Auth Password ${FIBR_THUMBNAIL_PASSWORD}
--thumbnailURL string [thumbnail] Vith Thumbnail URL ${FIBR_THUMBNAIL_URL} (default "http://vith:1080")
--thumbnailUser string [thumbnail] Vith Thumbnail Basic Auth User ${FIBR_THUMBNAIL_USER}
--thumbnailPassword string [thumbnail] Vignet Thumbnail Basic Auth Password ${FIBR_THUMBNAIL_PASSWORD}
--thumbnailURL string [thumbnail] Vignet Thumbnail URL ${FIBR_THUMBNAIL_URL} (default "http://vignet:1080")
--thumbnailUser string [thumbnail] Vignet Thumbnail Basic Auth User ${FIBR_THUMBNAIL_USER}
--title string Application title ${FIBR_TITLE} (default "fibr")
--url string [alcotest] URL to check ${FIBR_URL}
--userAgent string [alcotest] User-Agent for check ${FIBR_USER_AGENT} (default "Alcotest")
Expand Down
50 changes: 25 additions & 25 deletions compose.yaml
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
---
version: '3.9'
version: "3.9"

services:
service:
image: vibioh/fibr
environment:
FIBR_AUTH_USERS: '${BASIC_USERS}'
FIBR_AMQP_URI: 'amqp://guest:guest@rabbit:5672/'
FIBR_AMQP_WAIT_TIMEOUT: '30s'
FIBR_CHUNK_UPLOAD: 'true'
FIBR_EXIF_DIRECT_ACCESS: 'true'
FIBR_AUTH_USERS: "${BASIC_USERS}"
FIBR_AMQP_URI: "amqp://guest:guest@rabbit:5672/"
FIBR_AMQP_WAIT_TIMEOUT: "30s"
FIBR_CHUNK_UPLOAD: "true"
FIBR_EXIF_DIRECT_ACCESS: "true"
FIBR_REDIS_ADDRESS: redis:6379
FIBR_THUMBNAIL_DIRECT_ACCESS: 'true'
FIBR_THUMBNAIL_DIRECT_ACCESS: "true"
ports:
- '1080:1080/tcp'
- "1080:1080/tcp"
volumes:
- '${DATA_DIR}:/data'
- '/tmp:/tmp'
- "${DATA_DIR}:/data"
- "/tmp:/tmp"
entrypoint:
- /wait
- -timeout
Expand All @@ -30,7 +30,7 @@ services:
depends_on:
- redis
- rabbit
user: '${DATA_USER_ID}'
user: "${DATA_USER_ID}"
read_only: true

redis:
Expand All @@ -46,12 +46,12 @@ services:
exas:
image: vibioh/exas
environment:
EXAS_AMQP_URI: 'amqp://guest:guest@rabbit:5672/'
EXAS_AMQP_WAIT_TIMEOUT: '30s'
EXAS_GEOCODE_URL: 'https://nominatim.openstreetmap.org'
EXAS_STORAGE_FILE_SYSTEM_DIRECTORY: '/data'
EXAS_AMQP_URI: "amqp://guest:guest@rabbit:5672/"
EXAS_AMQP_WAIT_TIMEOUT: "30s"
EXAS_GEOCODE_URL: "https://nominatim.openstreetmap.org"
EXAS_STORAGE_FILE_SYSTEM_DIRECTORY: "/data"
volumes:
- '${DATA_DIR}:/data'
- "${DATA_DIR}:/data"
entrypoint:
- /wait
- -timeout
Expand All @@ -62,27 +62,27 @@ services:
- /exas
depends_on:
- rabbit
user: '${DATA_USER_ID}'
user: "${DATA_USER_ID}"
read_only: true

vith:
image: vibioh/vith
vignet:
image: vibioh/vignet
environment:
VITH_AMQP_URI: 'amqp://guest:guest@rabbit:5672/'
VITH_AMQP_WAIT_TIMEOUT: '30s'
VITH_STORAGE_FILE_SYSTEM_DIRECTORY: '/data'
VIGNET_AMQP_URI: "amqp://guest:guest@rabbit:5672/"
VIGNET_AMQP_WAIT_TIMEOUT: "30s"
VIGNET_STORAGE_FILE_SYSTEM_DIRECTORY: "/data"
volumes:
- '${DATA_DIR}:/data'
- "${DATA_DIR}:/data"
entrypoint:
- /wait
- -timeout
- 30s
- -address
- tcp:service:1080
- -next
- /vith
- /vignet
depends_on:
- rabbit
- image
user: '${DATA_USER_ID}'
user: "${DATA_USER_ID}"
read_only: true
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ require (
github.com/ViBiOh/exas v0.8.1
github.com/ViBiOh/flags v1.6.0
github.com/ViBiOh/httputils/v4 v4.80.6
github.com/ViBiOh/vith v0.6.0
github.com/ViBiOh/vignet v0.0.1
github.com/rabbitmq/amqp091-go v1.10.0
github.com/redis/go-redis/v9 v9.6.1
github.com/rs/xid v1.6.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ github.com/ViBiOh/flags v1.6.0 h1:f69V4ROK5XHAV71cL1yVP2UC/8xKiKJvunvgrtW/qsg=
github.com/ViBiOh/flags v1.6.0/go.mod h1:sbFyRQ7u5xzCssqGHn/hLSTlxmMYKe09WaPneJxHveA=
github.com/ViBiOh/httputils/v4 v4.80.6 h1:jhguNnarSEevpHZbP93Ix+qPfIDlue8HAdM7iAZnq2U=
github.com/ViBiOh/httputils/v4 v4.80.6/go.mod h1:uEOrzMewNPhnks4jcuPgjV+OkbrFMSscAMODZ2ysj2s=
github.com/ViBiOh/vith v0.6.0 h1:bdLmwgqUEWi3ivShtAhtM9gOZlWQzvZi2Z5b0BRSB1U=
github.com/ViBiOh/vith v0.6.0/go.mod h1:eRqtDU2uB8g85vll4spbnCV93DHyHNAYwXCMIBH1GuQ=
github.com/ViBiOh/vignet v0.0.1 h1:gKTh9FJBJ7+ZWZFvAXsefcPoi9QmBoaZXUiL5DKNK1M=
github.com/ViBiOh/vignet v0.0.1/go.mod h1:DCeiKdWswm6bhWIs3WAeuKIcs5v85Y06skW1R+kW5Vs=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
Expand Down
2 changes: 1 addition & 1 deletion infra/web.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ spec:
FIBR_THUMBNAIL_DIRECT_ACCESS: "true"
FIBR_THUMBNAIL_LARGE_SIZE: "1000"
FIBR_THUMBNAIL_MIN_BITRATE: "30000000"
FIBR_THUMBNAIL_URL: http://vith
FIBR_THUMBNAIL_URL: http://vignet
OTEL_RESOURCE_ATTRIBUTES: env=production,git.repository_url=github.com/ViBiOh/fibr
secrets:
FIBR_AMQP_URI: AgA9cvt+WxhWbE0BhP+LdNOgUWVNaLr8BBpi1BSVmHByUsUkRwM2rrN6wncgPTfJGW2nRzah+tYfgs2A31d3eNpi1UuG4ndH6IRSxrUJqA5FUXha8NlN5jzpqALFcIaEP0zavkef5U9YmuO3UdUT9o/thHwwJzy+au0DI44qtImIsEtOsVzq4fwBGIt2A+z5QqmeZK0TMcao630QqwLoksY0C0ZR+XomxuaGUCdR7xgiuQnHKeSp9idXoQc5sa84EEuoix+FCXuMwKn26dSytqLgeYpND2Ryj0WeRX6zl4Mf5V65PDBMqPxNvn9l+QxoKbM4HB++6N8oAvfkHQTCIqKH8lsrznpguYt1dYDEZwkMK6ho1Xy/ZVPGdKAELKdt+SYWoaZ9DV4A/sj4SkRFuGvPkoAc8raS/bxyWGbLWtwvCYSr0P99M7zTSDiNzEqy+3zjS0Q6vYE+JPKvAHwMrpAjWzB2hT0en10LPPnjNDNcSRPo118mzpopik752t6Ejx3JzAVU2OI08eSxxoejh/ZabuPocaZIseBRvWxHYoXVbLMgbmuEmvYc0PXm+/FNUsj/d7gs7aEsl5OUDwgpjDUtaq90eW5aUZb6OrGp2sKc3OmXDGq4IWDdqbBV7mzt/sjTvULBe9LwunneZN9OUJvYj5+r5107qaxzOQYmwYx8Qvs7GECbxcXofKg1lfWKUzbnFn9ekbqPegOSTQ5XD/y3LHimvaKtnpKHRXkAW5Si9krBExtljW8WHpTsW2nP9Rhf
Expand Down
4 changes: 2 additions & 2 deletions pkg/provider/provider.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ var (
// ThumbnailExtensions contains extensions of file eligible to thumbnail
ThumbnailExtensions = map[string]bool{".jpg": true, ".jpeg": true, ".png": true, ".gif": true, ".tiff": true, ".webp": true, ".pdf": true, ".mp4": true, ".mov": true, ".avi": true, ".ogg": true, ".mkv": true, ".heic": true}

// VithExtensions contains extensions of file eligible to thumbnail generation by vith
VithExtensions = map[string]bool{".jpg": true, ".jpeg": true, ".png": true, ".gif": true, ".tiff": true, ".webp": true, ".mp4": true, ".mov": true, ".avi": true, ".ogg": true, ".mkv": true, ".heic": true}
// VignetExtensions contains extensions of file eligible to thumbnail generation by vignet
VignetExtensions = map[string]bool{".jpg": true, ".jpeg": true, ".png": true, ".gif": true, ".tiff": true, ".webp": true, ".mp4": true, ".mov": true, ".avi": true, ".ogg": true, ".mkv": true, ".heic": true}
)

func MetadataDirectory(item absto.Item) string {
Expand Down
4 changes: 2 additions & 2 deletions pkg/thumbnail/amqp.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@ import (

absto "github.com/ViBiOh/absto/pkg/model"
"github.com/ViBiOh/httputils/v4/pkg/telemetry"
vith "github.com/ViBiOh/vith/pkg/model"
vignet "github.com/ViBiOh/vignet/pkg/model"
amqp "github.com/rabbitmq/amqp091-go"
)

func (s Service) AMQPHandler(ctx context.Context, message amqp.Delivery) (err error) {
ctx, end := telemetry.StartSpan(ctx, s.tracer, "amqp")
defer end(&err)

var req vith.Request
var req vignet.Request
if err := json.Unmarshal(message.Body, &req); err != nil {
return fmt.Errorf("decode: %w", err)
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/thumbnail/event.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
)

func (s Service) EventConsumer(ctx context.Context, e provider.Event) {
if s.vithRequest.IsZero() && s.amqpClient == nil {
if s.vignetRequest.IsZero() && s.amqpClient == nil {
return
}

Expand Down
14 changes: 7 additions & 7 deletions pkg/thumbnail/generate.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import (
absto "github.com/ViBiOh/absto/pkg/model"
"github.com/ViBiOh/fibr/pkg/provider"
"github.com/ViBiOh/httputils/v4/pkg/request"
"github.com/ViBiOh/vith/pkg/model"
vith "github.com/ViBiOh/vith/pkg/model"
"github.com/ViBiOh/vignet/pkg/model"
vignet "github.com/ViBiOh/vignet/pkg/model"
)

const (
Expand All @@ -26,7 +26,7 @@ func (s Service) generate(ctx context.Context, item absto.Item, scale uint64) (e
itemType := typeOfItem(item)

var resp *http.Response
resp, err = s.requestVith(ctx, item, itemType, scale)
resp, err = s.requestVignet(ctx, item, itemType, scale)
if err != nil {
s.increaseMetric(ctx, itemType.String(), "error")
return fmt.Errorf("request thumbnailer: %w", err)
Expand Down Expand Up @@ -54,19 +54,19 @@ func (s Service) generate(ctx context.Context, item absto.Item, scale uint64) (e
return err
}

func (s Service) requestVith(ctx context.Context, item absto.Item, itemType model.ItemType, scale uint64) (*http.Response, error) {
func (s Service) requestVignet(ctx context.Context, item absto.Item, itemType model.ItemType, scale uint64) (*http.Response, error) {
outputName := s.PathForScale(item, scale)

if s.amqpClient != nil {
s.increaseMetric(ctx, itemType.String(), "publish")
return nil, s.amqpClient.PublishJSON(ctx, vith.NewRequest(item.Pathname, outputName, itemType, scale), s.amqpExchange, s.amqpThumbnailRoutingKey)
return nil, s.amqpClient.PublishJSON(ctx, vignet.NewRequest(item.Pathname, outputName, itemType, scale), s.amqpExchange, s.amqpThumbnailRoutingKey)
}

s.increaseMetric(ctx, itemType.String(), "request")

if s.directAccess {
return s.vithRequest.Method(http.MethodGet).Path("%s?type=%s&scale=%d&output=%s", item.Pathname, itemType, scale, outputName).Send(ctx, nil)
return s.vignetRequest.Method(http.MethodGet).Path("%s?type=%s&scale=%d&output=%s", item.Pathname, itemType, scale, outputName).Send(ctx, nil)
}

return provider.SendLargeFile(ctx, s.storage, item, s.vithRequest.Method(http.MethodPost).Path("?type=%s&scale=%d", itemType, scale))
return provider.SendLargeFile(ctx, s.storage, item, s.vignetRequest.Method(http.MethodPost).Path("?type=%s&scale=%d", itemType, scale))
}
20 changes: 10 additions & 10 deletions pkg/thumbnail/stream.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ import (

absto "github.com/ViBiOh/absto/pkg/model"
"github.com/ViBiOh/httputils/v4/pkg/request"
"github.com/ViBiOh/vith/pkg/model"
"github.com/ViBiOh/vignet/pkg/model"
)

func (s Service) HasStream(ctx context.Context, item absto.Item) bool {
_, err := s.Info(ctx, getStreamPath(item))
return err == nil
}

func (s Service) handleVithResponse(ctx context.Context, err error, body io.ReadCloser) error {
func (s Service) handleVignetResponse(ctx context.Context, err error, body io.ReadCloser) error {
if err != nil {
s.increaseMetric(ctx, "stream", "error")
return fmt.Errorf("send request: %w", err)
Expand All @@ -39,13 +39,13 @@ func (s Service) shouldGenerateStream(ctx context.Context, item absto.Item) (boo

s.increaseMetric(ctx, "stream", "bitrate")

resp, err := s.vithRequest.Method(http.MethodHead).Path("%s?type=%s", item.Pathname, typeOfItem(item)).Send(ctx, nil)
resp, err := s.vignetRequest.Method(http.MethodHead).Path("%s?type=%s", item.Pathname, typeOfItem(item)).Send(ctx, nil)
if err != nil {
s.increaseMetric(ctx, "stream", "error")
return false, fmt.Errorf("retrieve metadata: %w", err)
}

rawBitrate := resp.Header.Get("X-Vith-Bitrate")
rawBitrate := resp.Header.Get("X-Vignet-Bitrate")
if len(rawBitrate) == 0 {
return false, nil
}
Expand Down Expand Up @@ -84,20 +84,20 @@ func (s Service) generateStream(ctx context.Context, item absto.Item) error {

s.increaseMetric(ctx, "stream", "request")

resp, err := s.vithRequest.Method(http.MethodPut).Path("%s?output=%s", input, url.QueryEscape(output)).Send(ctx, nil)
return s.handleVithResponse(ctx, err, resp.Body)
resp, err := s.vignetRequest.Method(http.MethodPut).Path("%s?output=%s", input, url.QueryEscape(output)).Send(ctx, nil)
return s.handleVignetResponse(ctx, err, resp.Body)
}

func (s Service) renameStream(ctx context.Context, old, new absto.Item) error {
s.increaseMetric(ctx, "stream", "rename")

resp, err := s.vithRequest.Method(http.MethodPatch).Path("%s?to=%s&type=%s", getStreamPath(old), url.QueryEscape(getStreamPath(new)), typeOfItem(old)).Send(ctx, nil)
return s.handleVithResponse(ctx, err, resp.Body)
resp, err := s.vignetRequest.Method(http.MethodPatch).Path("%s?to=%s&type=%s", getStreamPath(old), url.QueryEscape(getStreamPath(new)), typeOfItem(old)).Send(ctx, nil)
return s.handleVignetResponse(ctx, err, resp.Body)
}

func (s Service) deleteStream(ctx context.Context, item absto.Item) error {
s.increaseMetric(ctx, "stream", "delete")

resp, err := s.vithRequest.Method(http.MethodDelete).Path("%s?type=%s", getStreamPath(item), typeOfItem(item)).Send(ctx, nil)
return s.handleVithResponse(ctx, err, resp.Body)
resp, err := s.vignetRequest.Method(http.MethodDelete).Path("%s?type=%s", getStreamPath(item), typeOfItem(item)).Send(ctx, nil)
return s.handleVignetResponse(ctx, err, resp.Body)
}
Loading

0 comments on commit d637f4d

Please sign in to comment.