feat: support blpop/brpop #1548
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…to new_unstable # Conflicts: # src/net/src/dispatch_thread.cc # src/pika_command.cc # src/pstd/include/env.h
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
wanghenshui
reviewed
May 24, 2023
|
还需要补充driver测试，用redis-py redisgo都行 |
好的，不过这几天有些事，可能要过个几天再加上 |
|
@cheniujh please fix the file confliction. |
![cr-gpt[bot]](https://avatars.githubusercontent.com/in/292659?s=60&v=4){kind=small}
wanghenshui
previously approved these changes
Jul 6, 2023
AlexStocks
reviewed
Jul 11, 2023
AlexStocks
reviewed
Jul 11, 2023
cheniujh
force-pushed
the
new_add_blpop_brpop
branch
from
7d90230 to
24d123e
Compare
AlexStocks
approved these changes
Jul 19, 2023
bigdaronlee163
pushed a commit
to bigdaronlee163/pika
that referenced
this pull request
Jun 8, 2024
* working on BlpopCmd::Doinitial * working on : making the maps and new monitoring thread * added bLRPop_blocking_info_ in pika_server.cc * adding functions * adding functions02 * temp save * data structure done * working * revised data structure * working on moving * working * next step is to add a timeout scan * added TimerTaskManager * modified TimerTaskManager(v2) * scan added * tiny fix * changed the task of unblocking conn from sync to async * removed some code for testing * change from partition to db * temp save for testing * handle binlog of blr/pop(when conn get served, write a binlog of lpop/rpop) * improved sanitizer options in CMakeLists * temp save * add unit test * modified cmakelists * temp save * improved code based on reviwer's opinion * renamed some variables * 1. added record lock in ServeAndUnblockConns(void* args) 2. deleted testing code 3. formated code using clang-format * added multi-db concurrency test * renamed a flag from kCmdFlagsMayDfferWrite to kCmdFlagsMayDfferWriteBinlog * 1. added record lock in ServeAndUnblockConns(void* args) 2. deleted testing code 3. formated code using clang-format 4. removed kCmdFlagsMayDeferWriteBinlog and override dobinlog in blpop and brpop * removed unsed code * revised code based on opinion of reviewer * add the unit test to github workflow * revised unit test file * removed time counting * clear github action cache * revised based on reviwer's opinions * Modify to adapt to Mac. * Modify to adapt to Mac 2. --------- Co-authored-by: cjh <[email protected]>
cheniujh
added a commit
to cheniujh/pika
that referenced
this pull request
Sep 24, 2024
* working on BlpopCmd::Doinitial * working on : making the maps and new monitoring thread * added bLRPop_blocking_info_ in pika_server.cc * adding functions * adding functions02 * temp save * data structure done * working * revised data structure * working on moving * working * next step is to add a timeout scan * added TimerTaskManager * modified TimerTaskManager(v2) * scan added * tiny fix * changed the task of unblocking conn from sync to async * removed some code for testing * change from partition to db * temp save for testing * handle binlog of blr/pop(when conn get served, write a binlog of lpop/rpop) * improved sanitizer options in CMakeLists * temp save * add unit test * modified cmakelists * temp save * improved code based on reviwer's opinion * renamed some variables * 1. added record lock in ServeAndUnblockConns(void* args) 2. deleted testing code 3. formated code using clang-format * added multi-db concurrency test * renamed a flag from kCmdFlagsMayDfferWrite to kCmdFlagsMayDfferWriteBinlog * 1. added record lock in ServeAndUnblockConns(void* args) 2. deleted testing code 3. formated code using clang-format 4. removed kCmdFlagsMayDeferWriteBinlog and override dobinlog in blpop and brpop * removed unsed code * revised code based on opinion of reviewer * add the unit test to github workflow * revised unit test file * removed time counting * clear github action cache * revised based on reviwer's opinions * Modify to adapt to Mac. * Modify to adapt to Mac 2. --------- Co-authored-by: cjh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
新增对redis的blpop、brpop命令的支持。
功能说明：blpop/brpop是lpop/rpop的阻塞版本
(redis官方文档示例：)
实现思路：
1.维护了一个map，用于记录key与对应的阻塞客户端，大致如下:
这两个map以及一个对应的std::shared_mutex加到了DispatchThread中作为成员变量。
对这两个map的操作：
阻塞客户端：当blpop/brpop无法从目标list中取得元素时，往maps_from_keys_to_conns中对应key的list中新增Node（也往map_from_conns_to_keys插入一个list of keys)。
解阻塞：当lpush,rpush，rpoplpush命令执行后，用maps_from_keys_to_conns执行一下find方法，看是否有客户端正在等待当前key（刚刚发生push行为的），如果有，则给往线程池塞一个任务，用于向被阻塞的客户端写响应，以及清理两个map中的对应项。
超时扫描：在serverThread的threadmain的epoll里注册定时任务（目前是200ms执行一次），每次扫描整个maps_from_keys_to_conns，查看是否有过期的Node，如果有，回写一个-1(nil)，并且清理两个map中的相关项。
断联清理：当客户端断开连接时，对maps_from_conns_to_keys执行一下find，查看一下该连接是否是从被blpop/brpop阻塞的状态中退出的，如果是，则清理map。
连接保活：在检查客户端keep_alive的部分（默认是60s超时就会断开），加了一个判断：如果在maps_from_conns_to_keys中能find到该客户端的对应记录，则不断开此连接，且更新其last_interaction属性
命令的行为：
TODO:
blpop这种阻塞式命令对binlog有影响，需要调整
何时写binlog：在blpop，brpop中重写了doBinlog方法，只有真正发生写行为的时候才写一条binlog
binlog写什么：对于blpop/brpop来说，当用户真正得到元素时，往binlog写一条对应的lpop/rpop，如果用户在得到元素前就主动断开，则不写binlog（因为pop行为实际上也没有发生）。
产生的binlog：命令部分由小写字母组成（lpop/rpop), 而用户执行Lpop/Rpop命令产生的binlog中的命令都是大写（LPOP/RPOP），大小写不影响从节点消费，同时也可以作为一种区别方式（区分该条lpop/rpop binlog究竟是blpop/brpop产生的，还是用户执行Lpop/Rpop命令产生的）
单测：已集成到Github Workflow中，每次pr都会触发。
代码整理：
- 把方法体从.h移动到对应的.cpp
- 删除测试用的打印输出
- 使用clang-format整理格式(只整理了这个pr涉及到的文件）
Fixes: #114