[feat]tools-v2: bs clean-recycle #2401

lng2020 · 2023-04-14T03:38:48Z

What problem does this PR solve?

Issue Number: #2032

Problem Summary: support clean-recycle command in curve tool

What is changed and how it works?

What's Changed: add /tools-v2/pkg/cli/command/curvebs/clean_recycle.go, modify /tools-v2/pkg/config/bs.go and /tools-v2/README.md

How it Works:
command help

root@5bf3486a47c4:/# /root/curve bs clean-recycle -h
Usage:  curve bs clean-recycle [flags]

clean recycle bin

Flags:
  -c, --conf string            config file (default is $HOME/.curve/curve.yaml or
                               /etc/curve/curve.yaml)
      --expiredtime string     expire time (default 0s)
  -f, --format string          output format (json|plain) (default "plain")
  -h, --help                   print help
      --mdsaddr string         mds address, should be like
                               127.0.0.1:6700,127.0.0.1:6701,127.0.0.1:6702
      --password string        user password
      --recycleprefix string   recycle prefix (default "")
      --rpcretrytimes int32    rpc retry times (default 1)
      --rpctimeout duration    rpc timeout (default 10s)
      --showerror              display all errors in command
      --user string            user name
      --verbose                show some log

Examples:
$ curve bs clean-recycle --expiredtime=1h --recycleprefix=/test

command usage

root@75e1ef28f110:/# curve_ops_tool create --fileName=/test --userName=root
root@75e1ef28f110:/# /root/curve bs delete file --filename=/test --user=root
+---------+--------+
| RESULT  | REASON |
+---------+--------+
| success |        |
+---------+--------+
root@75e1ef28f110:/# /root/curve bs list dir --dir=/RecycleBin
+----+-------------------------------+----------+----------------+-------+---------------------+---------------+----------+
| ID |           FILENAME            | PARENTID |    FILETYPE    | OWNER |        CTIME        | ALLOCATEDSIZE | FILESIZE |
+----+-------------------------------+----------+----------------+-------+---------------------+---------------+----------+
| 5  | /RecycleBin/test-5-1681442546 | 1        | INODE_PAGEFILE | root  | 2023-04-14 11:22:21 | 0 B           | 20 GiB   |
+----+-------------------------------+----------+----------------+-------+---------------------+---------------+----------+
root@75e1ef28f110:/# /root/curve bs clean-recycle
+---------+--------+
| RESULT  | REASON |
+---------+--------+
| success |        |
+---------+--------+
root@75e1ef28f110:/# /root/curve bs list dir --dir=/RecycleBin

Side effects(Breaking backward compatibility? Performance regression?):

Check List

Relevant documentation/comments is changed or added
I acknowledge that all my contributions will be made under the project's license

Cyber-SiKu · 2023-04-17T02:43:20Z

Please fix the conflict

Cyber-SiKu · 2023-04-17T07:03:33Z

I just merged into a PR, seems to be a bit of a conflict?

lng2020 · 2023-04-17T07:20:05Z

Just some configurations in tools-v2/pkg/config/bs.go. I seem to be unable to resolve them locally (perhaps due to parentheses). Could you kindly help me resolve them?

Cyber-SiKu · 2023-04-17T07:16:12Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+var _ basecmd.FinalCurveCmdFunc = (*CleanRecycleCommand)(nil) // check interface
+
+// DeleteCertainFileRpc
+type DeleteCertainFileRpc struct {


You can add a wrapper call to delete file, like this.

curve/tools-v2/pkg/cli/command/curvebs/query/seginfo/segment.go

Lines 143 to 160 in efcbe1c

func GetSegment(caller *cobra.Command) (*nameserver2.GetOrAllocateSegmentResponse, *cmderror.CmdError) {

getCmd := NewQuerySegmentCommand()

config.AlignFlagsValue(caller, getCmd.Cmd, []string{

config.RPCRETRYTIMES, config.RPCTIMEOUT, config.CURVEBS_MDSADDR,

config.CURVEBS_PATH, config.CURVEBS_USER, config.CURVEBS_PASSWORD,

config.CURVEBS_OFFSET,

})

getCmd.Cmd.SilenceErrors = true

getCmd.Cmd.SilenceUsage = true

getCmd.Cmd.SetArgs([]string{"--format", config.FORMAT_NOOUT})

err := getCmd.Cmd.Execute()

if err != nil {

retErr := cmderror.ErrBsCreateFileOrDirectoryType()

retErr.Format(err.Error())

return getCmd.Response, retErr

}

return getCmd.Response, cmderror.Success()

}

Cyber-SiKu · 2023-04-17T07:16:56Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+type ListDirRpc struct {
+	Info          *basecmd.Rpc
+	Request       *nameserver2.ListDirRequest
+	curveFSClient nameserver2.CurveFSServiceClient
+}


Cyber-SiKu · 2023-04-17T07:18:24Z

tools-v2/pkg/config/bs.go

+	VIPER_CURVEBS_OFFSET        = "curvebs.offset"
+	CURVEBS_SIZE                = "size"
+	VIPER_CURVEBS_SIZE          = "curvebs.size"
+	CURVEBS_RECYCLE_PREFIX      = "recycleprefix"


add default value

Cyber-SiKu · 2023-04-17T07:18:29Z

tools-v2/pkg/config/bs.go

+	CURVEBS_RECYCLE_PREFIX      = "recycleprefix"
+	VIPER_RECYCLE_PREFIX        = "curvebs.recycleprefix"
+	CURVE_EXPIRED_TIME          = "expiredtime"
+	VIPER_CURVE_EXPIRED_TIME    = "curvebs.expiredtime"


Cyber-SiKu · 2023-04-17T07:27:33Z

Just some configurations in tools-v2/pkg/config/bs.go. I seem to be unable to resolve them locally (perhaps due to parentheses). Could you kindly help me resolve them?

keep you edit.

keep master.

Cyber-SiKu · 2023-04-17T07:28:48Z

It is best to combine your commits into one.

lng2020 · 2023-04-17T07:32:50Z

It is best to combine your commits into one.

Thank you for your help! I just noticed that the error handling has changed in the master branch, and I haven't addressed that in my pr. I will take your review suggestions and refactor the code accordingly. I expect to submit the changes tomorrow and hopefully resolve the issue.

Cyber-SiKu · 2023-04-17T07:46:38Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+	RECYCLEBINDIRNAME      = "RecycleBin"
+	RECYCLEBINDIR          = "/" + RECYCLEBINDIRNAME


merge to one?

Cyber-SiKu · 2023-04-17T07:48:44Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+
+// method of CleanRecycleCommand struct
+func (crCmd *CleanRecycleCommand) Init(cmd *cobra.Command, args []string) error {
+	crCmd.recyclePrefix = config.GetBsRecyclePrefix(crCmd.Cmd)


What is this for?

What is this for?

Delete the files in the recycle bin that its original filename has the prefix. I'm following the code in old tool of curve.

Cyber-SiKu · 2023-04-17T09:19:00Z

what is your email or wechat?

lng2020 · 2023-04-17T09:55:07Z

what is your email or wechat?

Already in the wechat group. My nick name is lng and wechat id is lng3075707637.

Cyber-SiKu · 2023-04-19T02:18:45Z

cicheck

Cyber-SiKu · 2023-04-19T02:27:23Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+// CleanRecycleCommand
+type CleanRecycleCommand struct {
+	basecmd.FinalCurveCmd
+	recyclePrefix string


What is this for?

Cyber-SiKu · 2023-04-24T02:37:42Z

tools-v2/pkg/config/bs.go

-	CURVEBS_DIR                  = "dir"
-	VIPER_CURVEBS_DIR            = "curvebs.dir"
+	CURVEBS_FORCE                = "force"
+	CURVEBS_DEFAULT_FORCE        = false


add VIPER_CURVEBS_FORCE

Cyber-SiKu · 2023-04-24T02:40:03Z

tools-v2/pkg/config/bs.go

@@ -77,6 +75,10 @@ const (
 	CURVEBS_STRIPE_COUNT         = "stripecount"
 	VIPER_CURVEBS_STRIPE_COUNT   = "curvebs.stripecount"
 	CURVEBS_DEFAULT_STRIPE_COUNT = uint64(32)
+	CURVEBS_RECYCLE_PREFIX       = "recycleprefix"
+	VIPER_RECYCLE_PREFIX         = "curvebs.recycleprefix"


option flag should be given default value

Cyber-SiKu · 2023-04-24T02:45:57Z

tools-v2/pkg/config/bs.go

+	AddBsStringOptionFlag(cmd, CURVEBS_RECYCLE_PREFIX, "recycle prefix (default \"\")")
+}
+
+func AddBsExpireTimeFlag(cmd *cobra.Command) {


Suggested change

func AddBsExpireTimeFlag(cmd *cobra.Command) {

func AddBsExpireTimeOptionFlag(cmd *cobra.Command) {

Cyber-SiKu · 2023-04-24T02:46:28Z

tools-v2/pkg/config/bs.go

@@ -400,3 +391,20 @@ func GetBsFlagInt32(cmd *cobra.Command, flagName string) int32 {
 	}
 	return value
 }
+
+// flag for clean recycle bin
+func AddBsRecyclePrefixFlag(cmd *cobra.Command) {


Suggested change

func AddBsRecyclePrefixFlag(cmd *cobra.Command) {

func AddBsRecyclePrefixOptionFlag(cmd *cobra.Command) {

Cyber-SiKu · 2023-04-24T02:49:21Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+)
+
+const (
+	cleanRecycleBinExample = `$ curve bs clean-recycle --expiredtime=1h --recycleprefix=/test`


curve bs delete recycle maybe better?

Cyber-SiKu · 2023-04-24T02:52:36Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+
+		filename := RECYCLEBINDIR + "/" + fileInfo.GetFileName()
+		crCmd.Cmd.Flags().Set(config.CURVEBS_PATH, filename)
+		crCmd.Cmd.Flags().Set(config.CURVEBS_FORCE, "true")


Add a constant definition in tools-v2/internal/utils/string.go?

Cyber-SiKu · 2023-04-24T02:54:42Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+	}
+
+	out := make(map[string]string)
+	out[cobrautil.ROW_RESULT] = "success"


Suggested change

out[cobrautil.ROW_RESULT] = "success"

out[cobrautil.ROW_RESULT] = cobrautil.ROW_VALUE_SUCCESS

Cyber-SiKu · 2023-04-24T02:55:11Z

tools-v2/pkg/cli/command/curvebs/clean_recycle/clean_recycle.go

+	list := cobrautil.Map2List(out, []string{cobrautil.ROW_RESULT})
+	crCmd.TableNew.Append(list)
+
+	crCmd.Result = "success"


Suggested change

crCmd.Result = "success"

crCmd.Result = out

Cyber-SiKu · 2023-04-24T07:39:35Z

cicheck

Cyber-SiKu · 2023-04-26T06:21:34Z

cicheck

Signed-off-by: lng2020 <[email protected]>

Cyber-SiKu · 2023-04-26T07:34:24Z

cicheck

lng2020 force-pushed the feature/curve-bs-clean-recycle branch from c983f10 to f1c7b79 Compare April 14, 2023 03:41

lng2020 mentioned this pull request Apr 14, 2023

[curve tool]: implement clean-recyle command #2032

Closed

lng2020 force-pushed the feature/curve-bs-clean-recycle branch from 392c041 to 0cd94e4 Compare April 14, 2023 06:43

lng2020 mentioned this pull request Apr 15, 2023

[curve/toos-v2]: add check-consistency #2351

Open

lng2020 force-pushed the feature/curve-bs-clean-recycle branch from 7227f6b to a4fd1c9 Compare April 17, 2023 07:01

Cyber-SiKu requested changes Apr 17, 2023

View reviewed changes

Cyber-SiKu reviewed Apr 17, 2023

View reviewed changes

lng2020 force-pushed the feature/curve-bs-clean-recycle branch from 0d0062c to 681850b Compare April 18, 2023 11:36

Cyber-SiKu reviewed Apr 19, 2023

View reviewed changes

lng2020 force-pushed the feature/curve-bs-clean-recycle branch 3 times, most recently from 6c55fb1 to 426d432 Compare April 23, 2023 09:47

Cyber-SiKu reviewed Apr 24, 2023

View reviewed changes

lng2020 force-pushed the feature/curve-bs-clean-recycle branch 2 times, most recently from bb259c1 to 35885f0 Compare April 24, 2023 07:38

Cyber-SiKu approved these changes Apr 24, 2023

View reviewed changes

code for clean-recycle

a8ee8dc

Signed-off-by: lng2020 <[email protected]>

lng2020 force-pushed the feature/curve-bs-clean-recycle branch from 35885f0 to a8ee8dc Compare April 26, 2023 07:18

wu-hanqing approved these changes Apr 26, 2023

View reviewed changes

Cyber-SiKu merged commit 61c07b3 into opencurve:master Apr 26, 2023

lng2020 deleted the feature/curve-bs-clean-recycle branch April 26, 2023 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat]tools-v2: bs clean-recycle #2401

[feat]tools-v2: bs clean-recycle #2401

lng2020 commented Apr 14, 2023

Cyber-SiKu commented Apr 17, 2023

Cyber-SiKu commented Apr 17, 2023

lng2020 commented Apr 17, 2023

Cyber-SiKu Apr 17, 2023

Cyber-SiKu Apr 17, 2023

Cyber-SiKu Apr 17, 2023

Cyber-SiKu Apr 17, 2023

Cyber-SiKu commented Apr 17, 2023

Cyber-SiKu commented Apr 17, 2023

lng2020 commented Apr 17, 2023

Cyber-SiKu Apr 17, 2023

lng2020 Apr 17, 2023

Cyber-SiKu Apr 17, 2023

lng2020 Apr 17, 2023

Cyber-SiKu commented Apr 17, 2023 •

edited

Loading

lng2020 commented Apr 17, 2023

Cyber-SiKu commented Apr 19, 2023

Cyber-SiKu Apr 19, 2023

Cyber-SiKu Apr 24, 2023

Cyber-SiKu Apr 24, 2023

Cyber-SiKu Apr 24, 2023

Cyber-SiKu Apr 24, 2023

Cyber-SiKu Apr 24, 2023

Cyber-SiKu Apr 24, 2023

Cyber-SiKu Apr 24, 2023

Cyber-SiKu Apr 24, 2023

Cyber-SiKu commented Apr 24, 2023

Cyber-SiKu commented Apr 26, 2023

Cyber-SiKu commented Apr 26, 2023

	func GetSegment(caller cobra.Command) (nameserver2.GetOrAllocateSegmentResponse, *cmderror.CmdError) {
	getCmd := NewQuerySegmentCommand()
	config.AlignFlagsValue(caller, getCmd.Cmd, []string{
	config.RPCRETRYTIMES, config.RPCTIMEOUT, config.CURVEBS_MDSADDR,
	config.CURVEBS_PATH, config.CURVEBS_USER, config.CURVEBS_PASSWORD,
	config.CURVEBS_OFFSET,
	})
	getCmd.Cmd.SilenceErrors = true
	getCmd.Cmd.SilenceUsage = true
	getCmd.Cmd.SetArgs([]string{"--format", config.FORMAT_NOOUT})
	err := getCmd.Cmd.Execute()
	if err != nil {
	retErr := cmderror.ErrBsCreateFileOrDirectoryType()
	retErr.Format(err.Error())
	return getCmd.Response, retErr
	}
	return getCmd.Response, cmderror.Success()
	}

		RECYCLEBINDIRNAME = "RecycleBin"
		RECYCLEBINDIR = "/" + RECYCLEBINDIRNAME

	func AddBsExpireTimeFlag(cmd *cobra.Command) {
	func AddBsExpireTimeOptionFlag(cmd *cobra.Command) {

	func AddBsRecyclePrefixFlag(cmd *cobra.Command) {
	func AddBsRecyclePrefixOptionFlag(cmd *cobra.Command) {

	out[cobrautil.ROW_RESULT] = "success"
	out[cobrautil.ROW_RESULT] = cobrautil.ROW_VALUE_SUCCESS

[feat]tools-v2: bs clean-recycle #2401

[feat]tools-v2: bs clean-recycle #2401

Conversation

lng2020 commented Apr 14, 2023

What problem does this PR solve?

What is changed and how it works?

Check List

Cyber-SiKu commented Apr 17, 2023

Cyber-SiKu commented Apr 17, 2023

lng2020 commented Apr 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cyber-SiKu commented Apr 17, 2023

Cyber-SiKu commented Apr 17, 2023

lng2020 commented Apr 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cyber-SiKu commented Apr 17, 2023 • edited Loading

lng2020 commented Apr 17, 2023

Cyber-SiKu commented Apr 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cyber-SiKu commented Apr 24, 2023

Cyber-SiKu commented Apr 26, 2023

Cyber-SiKu commented Apr 26, 2023

Cyber-SiKu commented Apr 17, 2023 •

edited

Loading