CHANGELOG.md

Changelog

v0.7.2 (2023-03-13)

• 1196a24 Update manifests to v0.7.2
• a6f64f8 feat: use randomized shuffle using vertex name as the seed (#601)
• 16b20a3 fix: ack the dropped messages as well (#603)
• 4cdef17 feat: enable controller HA (#602)
• d2e0513 feat: expose dnspolicy and dnsconfig to pod template (#598)
• 6002c82 Chore: tickgen changes to test reduce pipelines (#587)
• b1aee94 feat: bidirectional streaming (#553)
• 67fc688 feat: use customized binary serde for nats message payload (#585)
• cc10419 fix: Idle watermark fix for read batch size > 0 and partial idle outgoing edges (#575)
• df1574d feat: implement watermark propagation for source data transformer (#557)
• 45d5c39 feat: namespace search (#559)
• b14d470 fix: refine log for buffer validation. Fixes #185 (#573)
• a8e8bb1 feat: add readiness and liveness check for daemon server. Fixes #543 (#571)
• fd6acb6 feat: marshal/unmarshal binary for read message (#565)
• d1032b4 chore(deps): bump golang.org/x/net from 0.0.0-20220722155237-a158d28d115b to 0.7.0 (#568)
• fd00ebd chore(deps): bump golang.org/x/text from 0.3.7 to 0.3.8 (#567)
• 05ec77f feat: expose image pull policy to user defined containers (#563)
• ecbe3a0 fix: typos in reduce examples (#556)
• 0dc3f5c feat: edge-watermark (#537)
• 77298c8 feat: enable envFrom for user defined containers (#554)

Contributors

• Derek Wang
• Dillen Padhiar
• Juanlu Yu
• Julie Vogelman
• Keran Yang
• Vedant Gupta
• Vigith Maurice
• Yashash H L
• ashwinidulams
• dependabot[bot]

v0.7.1 (2023-02-14)

• 92925c1 Update manifests to v0.7.1
• 2f8e147 feat: remove secret watch privilege dependency (#542)
• f8e7daa fix: Use a copied object to update (#541)
• 98de245 chore(deps): bump github.com/emicklei/go-restful from 2.9.5+incompatible to 2.16.0+incompatible (#539)
• 0df812c feat: improve reduce performance (#501)
• ab49de6 feat: Offset time idle watermark put (#529)
• c0aa7c1 fix: securityContext not applied to container templates (#528)
• ac33fb0 feat: idle watermark v0 (#520)
• 2844cfb feat: Reduce UI Support (#500)
• e701180 feat: enable RuntimeClassName for vertex pod (#519)
• bb94f63 feat: add builtin filter and event time extractor for source transformer (#517)
• 4562196 chore(deps): bump ua-parser-js from 0.7.32 to 0.7.33 in /ui (#507)
• 764cefd Add an e2e test for source data transformer (#505)
• 7665d6c feat: Implement source data transformer and apply to all existing sources (#487)
• d022608 fix: -ve metrics and return early if isLate (#495)

Contributors

• Derek Wang
• Juanlu Yu
• Keran Yang
• Vedant Gupta
• Vigith Maurice
• ashwinidulams
• dependabot[bot]

v0.7.0 (2023-01-13)

• 734e5d3 Update manifests to v0.7.0
• 5d6c533 fix: JetStream context KV store/watch fix (#460)
• d6152e7 doc: reduce persistent store (#458)
• ac77656 doc: reduce documentation (#448)
• 257356a chore(deps): bump json5 from 1.0.1 to 1.0.2 in /ui (#454)
• 7752db4 refactor: simplify http request construction in test cases (#444)
• 1a10af4 refactor: use exact matching instead of regex to perform e2e data validation. (#443)
• 2777e27 doc: windowing fixed and sliding (#439)
• 70fc008 refactor: move redis sink resources creation to E2ESuite (#437)
• 6c078b4 refactor: a prototype for enhancing E2E test framework (#424)
• e7021c9 feat: pipeline watermark (#416)

Contributors

• Derek Wang
• Juanlu Yu
• Keran Yang
• Vedant Gupta
• Vigith Maurice
• dependabot[bot]

v0.7.0-rc1 (2022-12-16)

• 71887db Update manifests to v0.7.0-rc1
• dda4835 feat: reduce metrics. Closes #313 (#414)
• 85dbe4d feat: udsink grpc stream (#421)
• fa07587 chore(doc): scope UDF under a dir (#426)
• 0a911da feat: sliding window. closes #339 (#354)
• a46fb96 refactor: nats/jetstream testing (#418)
• 13d95c4 feat: nats as source (#411)
• f1e7c73 fix: adding lock while discovering partitions, Closes #412 (#413)
• 3b64d67 fix(test): e2e-api-pod can not start on M1 mac (#410)
• 6504a56 fix: getWatermark to return-1 if any processor returns -1 (#402)
• d4d2204 fix: e2e testing for PBQ WAL with reduce pipeline (#393)
• 80e9785 feat: add Grafana instruction and a dashboard template. Closes #287 (#381)
• 2f94a91 fix: unit tests for replay. Closes #373 (#377)
• 8f367ab chore(docs): update docs (#380)
• efe4d41 fix: best effort processing during SIGTERM. Closes #371 (#372)
• 7e041d8 feat(wal): First pass to implement WAL and hook to PBQ store. (#344)
• 256e66b feat: watermark otwatcher enhancement (#364)
• f817057 refactor(docs): group docs in categories (#362)
• 1a5d424 chore(deps): bump loader-utils from 2.0.3 to 2.0.4 in /ui (#356)
• 6c8f03f fix(controller): vertex nil check for edge listing. Fixes #352 (#353)
• f254c28 fix: data race in pbq manager. Closes #348 (#349)
• bc35945 Chore: Windower interface. closes #234 (#340)
• 3206bd1 feat: add minikube, kind and podman support (#206)
• d40ecda refactor: Close watermark fetcher and publisher correctly (#336)
• 0d8f659 passing window information inside the context (#341)
• 8951627 feat: timestamp in UI to display milliseconds. closes #280 (#337)
• 5c43f5a Simple reduce pipeline. Fixes #289 (#317)
• 7f5d86c feat: add blackhole sink. Closes #329 (#330)
• 10f355c fix: move watermark based on the head of the read batch (#332)
• b2b975f feat: configurable jetstream storage (#328)
• 3fcf637 feat: support adding sidecars in vertex pods. Closes #323 (#325)
• 6eab1b5 feat: populate watermark settings to vertex spec. Closes #320 (#321)
• 2355978 doc: add few use cases (#318)
• bfc1eb6 Chore: run in sdks-e2e tests, python-udsink log check before go-udsink (#315)
• cda41ec fix: jetstream build watermark progressors bug (#316)
• bfab8f1 feat: update watermark offset bucket implementation (#307)
• 1d86aa5 feat: shuffling support (#306)
• b817920 feat: customize init-container resources. Closes #303 (#304)
• 0548d4d feat: watermark - remove non-share OT bucket option (#302)
• cc44875 feat: customization for batch jobs. Closes #259 (#300)
• d16015f refactor: abstract pod template (#296)
• 4550f45 feat: customization for daemon deployment. Closes #223 (#290)
• d61377a feat: add pvc support for reduce vertex PBQ (#292)
• b0e3f94 fix(doc): hyperlink for security doc (#288)
• 6c61728 feat: support adding init containers to vertices. Closes #284 (#285)
• 88cf272 fix: retry when getting EOF error at E2E test (#281)
• 1436071 feat: Watermark millisecond. Fixes #201 (#278)
• 7a7e794 feat: add pipeline node counts and age to printcolumn. Closes #267 (#282)
• 5883e97 feat: introduce reduce UDF. Closes #246 (#262)
• a0dc17f feat: add pandoc to required tools development doc. Closes #276 (#277)
• 284be2d feat: add isbsvc type and age to printcolumn. Closes #268 (#275)
• 7bb689b fix: watermark consumer fix (#273)
• 8ff9e28 refactor: generalize watermark fetching as an interface of ISB service. Fixes #252 (#263)
• 8e038d1 fix: set default property values for minimal CRD installation (#264)
• 57df392 fix: validate only one isbsvc implementation type is defined. Fixes #269 (#271)
• 21378a3 fix: main branch make build failure: math.MaxInt64 for int type (#265)
• 3d9997d fix: nil pointer deref when running example with minimal CRD. Fixes #260 (#261)
• 4b0cbc3 fix: retry only the failed offsets (#255)
• 27e6a87 fix: re-enable build constraint on processor manager test. Fixes #256 (#257)
• 98b3ec4 fix: container resource for jetstream isbsvc. Fixes #253 (#254)
• e615e16 fix: update vertex watermark fetching logic. Fixes: #134 (#245)
• 30c734b fix: watermark watcher leak (#242)
• 3d29f79 fix(docs): fix a typo (#241)
• 0370fd6 feat: Support running UX server with namespace scope. Fixes #248 (#249)
• 29f15d5 fix(manifests): Include ServiceAccount in namespace scoped install (#240)
• 6870d2a fix: Watermark close fix and removed the nil check (#238)
• 998e398 fix: skip publishing watermarks to unexpected vertices. Fixes #235 (#236)
• fff05f3 fix: update default watermark to -1. Fixes #133 (#218)
• a23e359 feat: support disabling TLS and changing port for UX server (#228)
• 5a4387c feat: reducer for stream aggregation without fault tolerance (#208)
• fc2ba4e feat: in-memory watermark store for better testing (#216)
• c89aef3 Add USERS.md (#221)
• 2377c4c fix(watermark): generator should not publish wm for every message (#217)

Contributors

• David Seapy
• Derek Wang
• Ed Lee
• Juanlu Yu
• Keran Yang
• Shay Dratler
• SianLoong
• Vedant Gupta
• Vigith Maurice
• Yashash H L
• ashwinidulams
• dependabot[bot]
• xdevxy

v0.6.5 (2022-12-07)

• 845c959 Update manifests to v0.6.5
• 676ea1c fix: adding lock while discovering partitions, Closes #412 (#413)
• c439a6a fix(test): e2e-api-pod can not start on M1 mac (#410)
• 115a69d fix: getWatermark to return-1 if any processor returns -1 (#402)
• e6e24ee fix: e2e testing for PBQ WAL with reduce pipeline (#393)
• 7ef3d47 feat: add Grafana instruction and a dashboard template. Closes #287 (#381)
• 13ce4d2 fix: unit tests for replay. Closes #373 (#377)

Contributors

• Derek Wang
• Keran Yang
• Yashash H L
• xdevxy

v0.6.4 (2022-11-28)

• ad9719a Update manifests to v0.6.4
• c5e8217 chore(docs): update docs (#380)
• 1b244c1 fix: best effort processing during SIGTERM. Closes #371 (#372)
• 9bb8ebd feat(wal): First pass to implement WAL and hook to PBQ store. (#344)

Contributors

• Derek Wang
• Vigith Maurice
• xdevxy

v0.6.3 (2022-11-18)

• 3cf391b Update manifests to v0.6.3
• bec020b feat: watermark otwatcher enhancement (#364)
• 2b5478f refactor(docs): group docs in categories (#362)
• 6d9e129 chore(deps): bump loader-utils from 2.0.3 to 2.0.4 in /ui (#356)
• 77364a4 fix(controller): vertex nil check for edge listing. Fixes #352 (#353)
• 7db4fe5 fix: data race in pbq manager. Closes #348 (#349)
• 6f5e83a Chore: Windower interface. closes #234 (#340)
• 24ba515 feat: add minikube, kind and podman support (#206)
• 12c6ca5 refactor: Close watermark fetcher and publisher correctly (#336)
• 2a8b97e passing window information inside the context (#341)
• d52a5a7 feat: timestamp in UI to display milliseconds. closes #280 (#337)
• de9059c Simple reduce pipeline. Fixes #289 (#317)
• 3d936a5 feat: add blackhole sink. Closes #329 (#330)
• 4590547 fix: move watermark based on the head of the read batch (#332)
• 049e5c6 feat: configurable jetstream storage (#328)
• ee5cd64 feat: support adding sidecars in vertex pods. Closes #323 (#325)

Contributors

• David Seapy
• Derek Wang
• Juanlu Yu
• Shay Dratler
• Vedant Gupta
• Vigith Maurice
• Yashash H L
• ashwinidulams
• dependabot[bot]

v0.6.2 (2022-11-07)

• 99be6c0 Update manifests to v0.6.2
• dc733da feat: populate watermark settings to vertex spec. Closes #320 (#321)
• 2b247ca doc: add few use cases (#318)
• 07ffa16 Chore: run in sdks-e2e tests, python-udsink log check before go-udsink (#315)
• 7b3285b fix: jetstream build watermark progressors bug (#316)
• 1198a60 feat: update watermark offset bucket implementation (#307)
• 34a6d70 feat: shuffling support (#306)
• 448127f feat: customize init-container resources. Closes #303 (#304)
• 61cf227 feat: watermark - remove non-share OT bucket option (#302)
• 51c9ff4 feat: customization for batch jobs. Closes #259 (#300)
• afbe255 refactor: abstract pod template (#296)
• 2408946 feat: customization for daemon deployment. Closes #223 (#290)
• bedf567 feat: add pvc support for reduce vertex PBQ (#292)
• 2341614 fix(doc): hyperlink for security doc (#288)
• 6c05190 feat: support adding init containers to vertices. Closes #284 (#285)
• dc96b87 fix: retry when getting EOF error at E2E test (#281)
• f5db937 feat: Watermark millisecond. Fixes #201 (#278)
• c153536 feat: add pipeline node counts and age to printcolumn. Closes #267 (#282)

Contributors

• David Seapy
• Derek Wang
• Juanlu Yu
• Keran Yang
• Vigith Maurice

v0.6.1 (2022-10-26)

• 32b284f Update manifests to v0.6.1
• 9684e16 fix(manifests): Include ServiceAccount in namespace scoped install (#240)
• fe83918 fix(docs): fix a typo (#241)
• f2094b4 feat: introduce reduce UDF. Closes #246 (#262)
• e19a1e7 feat: add pandoc to required tools development doc. Closes #276 (#277)
• 9a93711 feat: add isbsvc type and age to printcolumn. Closes #268 (#275)
• f25e303 fix: watermark consumer fix (#273)
• d2a3d90 refactor: generalize watermark fetching as an interface of ISB service. Fixes #252 (#263)
• 5ffcadc fix: set default property values for minimal CRD installation (#264)
• 17a9956 fix: validate only one isbsvc implementation type is defined. Fixes #269 (#271)
• 2272a1f fix: main branch make build failure: math.MaxInt64 for int type (#265)
• 02c31d2 fix: nil pointer deref when running example with minimal CRD. Fixes #260 (#261)
• 391b53e fix: retry only the failed offsets (#255)
• 7b42dc8 fix: re-enable build constraint on processor manager test. Fixes #256 (#257)
• 3436049 fix: container resource for jetstream isbsvc. Fixes #253 (#254)
• 33ce742 fix: update vertex watermark fetching logic. Fixes: #134 (#245)
• fd219a5 fix: watermark watcher leak (#242)
• 979a3a3 feat: Support running UX server with namespace scope. Fixes #248 (#249)
• 5e9d1c1 fix: Watermark close fix and removed the nil check (#238)
• 340bd82 fix: skip publishing watermarks to unexpected vertices. Fixes #235 (#236)
• 904b2cd fix: update default watermark to -1. Fixes #133 (#218)
• 321e285 feat: support disabling TLS and changing port for UX server (#228)
• d0d74e1 feat: reducer for stream aggregation without fault tolerance (#208)
• 06a9b58 feat: in-memory watermark store for better testing (#216)
• f25cc58 Add USERS.md (#221)
• a37cece fix(watermark): generator should not publish wm for every message (#217)

Contributors

• David Seapy
• Derek Wang
• Ed Lee
• Juanlu Yu
• Keran Yang
• SianLoong
• Yashash H L
• ashwinidulams

v0.6.0 (2022-10-12)

• 48aad5f Update manifests to v0.6.0
• 09ce54f fix(autoscaling): Ack pending should be included in total pending calculation (#212)
• 9922787 fix(autoscaling): Skip autoscaling if vertex is not in running phase (#207)
• bc2380a feat: ISBSVC add support for redis cluster mode (#195)
• 72a96a5 refactor: move controllers package to pkg/reconciler (#192)
• b1b78fa fix: update udf fetchWatermark and publishWatermark initial values (#193)
• d491260 fix(docs): readme for UI development (#181)
• 6b121c6 feat: grpc udsink (#174)
• 567da7b fix: numaflow-go udf example & docs (#177)
• 4652f80 fix: use scale.max if it is set (#179)
• 900314b fix broken path (#176)
• 3b02f2a feat: Shuffle implementation (#169)
• 021bb9d feat: windowing operations (#157)
• 7d41129 feat: watermark for sources (#159)
• 5f5b2df fix: daemon service client memory leak (#161)
• bfe9669 pbq implementation (#155)
• 8dfedd8 feat: check if udf is running in liveness probe (#156)
• 81e76d8 feat: Add udf grpc support Fixes #145 (#146)
• 511faff refactor: some refactor on watermark (#149)
• 7fe40c4 fix: Fixed JS bug (#144)
• 24a16a0 bug: watermark needs nil check
• f4ed831 fix: pipeline UI broken when vertex scaling down to 0 (#132)
• 0ae0377 feat: JetStream auto-reconnection (#127)
• 2fc04eb feat: Add watermark for sink vertex (#124)
• d958ee6 feat: autoscaling with back pressure factor (#123)
• b1f7768 feat: add watermark to UI (#122)
• 7feeaa8 feat: add processing rate to UI (#121)
• 43fae93 feat: Expose watermark over HTTP (#120)
• ec02304 fix: daemon service rest api not working (#119)
• f3da56d chore(deps): bump terser from 5.14.1 to 5.14.2 in /ui (#117)
• e2e63c8 feat: Numaflow autoscaling (#115)
• e5da3f5 feat: watermark for headoffset (#116)
• a45b2ee feat: support namespace scope installation (#112)
• ce39199 feat: Expose ReadTimeoutSeconds on Vertex (#110)
• 18ad1c5 fix: imagepullpocily for local testing (#113)
• 469849b feat: removed udfWorkers from limits and added some docs (#103)
• 3fada66 feat: Add icon and other minor changes (#94)
• a81838d feat: end to end tickgen watermark validation (#98)
• d7d9317 fix: Broken hyperlink (#96)
• a2e0792 add no-op KV Store (#91)
• 45c8cb6 feat: no operation watermark progressor (#90)
• 448c229 feat: kafka source pending messages (#80)
• 1aa3930 feat: Interface for Watermark (#82)
• be78c52 feat: expose pending messages and processing rate (#79)
• df30f2a feat: Added the right way to decipher from and to vertex (#78)
• 639c459 feat: define buffer limits on edges (#70)
• 41fdd38 feat: Merge UX server code (#67)
• ced9907 feat: auto-scaling (part 1) (#59)
• fd5b374 Added name to service spec (#58)
• dc2badf feat: introduce source buffer and sink buffer (#53)
• 4ed83a2 feat: async publishing for kafka sink (#51)
• 9f9f5ba fix spelling errors (#48)
• f423002 feat: controller to create buckets (#47)
• 8328739 turn on watermark only if ENV value is true (#46)
• 46f72e2 minimal end to end line-graph watermark integration (#43)
• 1f8203f Fixed spelling error (#44)
• f1e99ea Exponential buckets (#42)
• dfcfdeb fix: different behavior for time.After in go 1.18 (#39)

Contributors

• Chrome
• Derek Wang
• Juanlu Yu
• Krithika3
• Qianbo Huai
• Saravanan Balasubramanian
• Sidhant Kohli
• Vigith Maurice
• Yashash H L
• dependabot[bot]

v0.5.6 (2022-09-19)

• ac15d22 Update manifests to v0.5.6
• f236375 feat: grpc udsink (#174)
• 2650c2d fix: numaflow-go udf example & docs (#177)
• c44f733 fix: use scale.max if it is set (#179)
• 39e92d0 fix broken path (#176)
• 46ce0f8 feat: Shuffle implementation (#169)
• 71ca00a feat: windowing operations (#157)
• ca00b78 feat: watermark for sources (#159)
• 91e21ce pbq implementation (#155)
• 6542400 feat: check if udf is running in liveness probe (#156)
• 79dce0b feat: Add udf grpc support Fixes #145 (#146)

Contributors

• Chrome
• Derek Wang
• Juanlu Yu
• Vigith Maurice
• Yashash H L

v0.5.5 (2022-09-07)

• 9aae638 Update manifests to v0.5.5
• 3241432 fix: daemon service client memory leak (#161)
• be47a26 refactor: some refactor on watermark (#149)
• 857cce7 fix: Fixed JS bug (#144)
• da16abc bug: watermark needs nil check
• c9998a1 fix: pipeline UI broken when vertex scaling down to 0 (#132)

Contributors

• Derek Wang
• Krithika3
• Vigith Maurice

v0.5.4 (2022-08-05)

• 57513b4 Update manifests to v0.5.4
• 94cdb82 feat: JetStream auto-reconnection (#127)
• 8d83540 feat: Add watermark for sink vertex (#124)
• 228ba32 feat: autoscaling with back pressure factor (#123)
• 9833efd feat: add watermark to UI (#122)
• 0dab55d feat: add processing rate to UI (#121)
• ffd38a1 feat: Expose watermark over HTTP (#120)
• c09502a fix: daemon service rest api not working (#119)
• ebc10f4 chore(deps): bump terser from 5.14.1 to 5.14.2 in /ui (#117)
• 84490ca feat: Numaflow autoscaling (#115)
• 32b9848 feat: watermark for headoffset (#116)
• 283dae9 feat: support namespace scope installation (#112)
• 8e612b1 feat: Expose ReadTimeoutSeconds on Vertex (#110)
• d95d41b fix: imagepullpocily for local testing (#113)

Contributors

• Derek Wang
• Krithika3
• Saravanan Balasubramanian
• Sidhant Kohli
• Vigith Maurice
• dependabot[bot]

v0.5.3 (2022-07-08)

• efee544 Update manifests to v0.5.3
• 5895fac feat: removed udfWorkers from limits and added some docs (#103)
• 0b75495 feat: Add icon and other minor changes (#94)
• 7eb08f5 feat: end to end tickgen watermark validation (#98)
• 3338e65 fix: Broken hyperlink (#96)
• e311222 add no-op KV Store (#91)
• 5d2f90e feat: no operation watermark progressor (#90)
• f58d0f4 feat: kafka source pending messages (#80)
• cbb16ca feat: Interface for Watermark (#82)
• 5592bb1 feat: expose pending messages and processing rate (#79)
• 06a3df2 feat: Added the right way to decipher from and to vertex (#78)
• a0908ad feat: define buffer limits on edges (#70)
• a1d3639 feat: Merge UX server code (#67)
• 571c48e feat: auto-scaling (part 1) (#59)
• 1e0384b Added name to service spec (#58)

Contributors

• Derek Wang
• Krithika3
• Sidhant Kohli
• Vigith Maurice

v0.5.2 (2022-06-13)

• 2f2d10c Update manifests to v0.5.2
• cedd0d1 feat: introduce source buffer and sink buffer (#53)
• d3301aa feat: async publishing for kafka sink (#51)
• 2474eb8 fix spelling errors (#48)
• c4a12f8 feat: controller to create buckets (#47)
• eb97dc3 turn on watermark only if ENV value is true (#46)
• f189ba3 minimal end to end line-graph watermark integration (#43)

Contributors

• Derek Wang
• Qianbo Huai
• Vigith Maurice

v0.5.1 (2022-06-02)

• bb9be80 Update manifests to v0.5.1
• 912747e Fixed spelling error (#44)
• 3aeb33a Exponential buckets (#42)
• 1d65682 fix: different behavior for time.After in go 1.18 (#39)

Contributors

• Derek Wang
• Krithika3