Add go mod

[timvaillancourt]

Description

This is a merge PR of the user-submitted PR: #932

Unfortunately this step is required as our gh-ost testing won't run on external forks

In case this PR introduced Go code changes:

• contributed code is using same conventions as original code
• script/cibuild returns with no formatting errors, build errors or unit test errors.

cc @natewernimont-wk / @shlomi-noach

[tomdeering-wf]

Thanks for being willing to accept this change! Now we'll be able to build gh-ost with Go 1.16 without having to revert go build back to GO111MODULE=auto (the default in Go 1.15).

[shlomi-noach]

Shoot for the stars

[timvaillancourt]

👋 @natewernimont-wk this ran into automated testing failures with the following output:

2021-03-10 04:24:24 FATAL Unable to read osc chunk size: strconv.ParseInt: parsing "": invalid syntax

A rollback to master was able to resolve this. Whatever the issue it is strange it didn't get caught in CI 🤔

I haven't had time to dig into "why", but the command that triggered the problem was:

gh-ost --conf=/etc/mysql/gh-ost.cnf --test-on-replica --database=REDACTED --table=REDACTED \
--verbose --alter=engine=innodb --execute --initially-drop-ghost-table --initially-drop-old-table \
--max-load=Threads_running=30 --critical-load=Threads_running=1000 --critical-load-interval-millis=3000 \
--switch-to-rbr --chunk-size=2500 --cut-over=default --timestamp-old-table --exact-rowcount \
--concurrent-rowcount --serve-socket-file=/tmp/gh-ost.test.sock --panic-flag-file=/tmp/ghost-test-panic.flag \
--hooks-path=REDACTED --hooks-hint=@root --ssl --ssl-allow-insecure --verbose

[timvaillancourt]

@shlomi-noach this is failing on an error I could use some context around:

 go/logic/inspect.go:619:20: m.GetUint64 undefined (type sqlutils.RowMap has no field or method GetUint64)

It appears that function doesn't exist in github.com/outbrain/golib/sqlutils and I can't find a point in the history where it did exist. It does exist in vendor dir on master though 🤔

That lead me to wondering if a patch was committed to the vendor/ dir of this repo but the git history doesn't support that either

I'm curious if you had any ideas how this may have happened. I suppose the options are to add it to github.com/outbrain/golib or make a copy of that logic as a utility-func in go/ in this repo. I think I'm leaning towards the latter as the golib repo doesn't seem very current

[shlomi-noach]

@timvaillancourt right. GetUint64 was added downstream, directly into vendor/ path, in openark@31069ae4

What I'm suggesting:

• I have this better maintained repo: https://github.com/openark/golib
• I'll add GetUint64 functionality in that repo
• you will add replace github.com/outbrain/golib => github.com/openark/golib in go.mod

I'm just trying to think whether there's been more changes local to golib under vendor. I'll either git blame later on, or just confirm the code builds properly with openark/golib.

[shlomi-noach]

Related: outbrain-inc/golib#10 openark/golib#11

[shlomi-noach]

openark/golib#11 is merged, and gh-ost builds well with it.

@timvaillancourt either that replace directive, or just search&replace github.com/outbrain/golib with github.com/openark/golib throughout the code at your discretion.

[timvaillancourt]

openark/golib#11 is merged, and gh-ost builds well with it.

@shlomi-noach thanks for the PR and background ❤️! I'll look out for this if this PR runs into any other problems 👍

[timvaillancourt]

This is passing real-world testing as of commit: 71d27d0

If anyone has time to test this on more environments that would be much appreciated! 🙇

cc @natewernimont-wk / @tomdeering-wf / @shlomi-noach

[timvaillancourt]

This has passed several-100 tests on real tables, so I'll merge this 👍

[d-bytebase]

Thanks for providing go mod for gh-ost! When are we going to have 1.1.5 release for the change?

[dm-2]

Hi @d-bytebase, we're currently investigating a data corruption bug that we believe may have been introduced by an updated dependency in this PR - once we've identified the cause and fixed it, the plan is to put out a new release based on the master branch (1.1.3 and 1.1.4 are based on 1.1.2).

[chenrui333]

Looks like this change was never been released? Can we cut a release for this PR? Thanks!