various MOLT corrections and fixes #19326
Conversation
Files changed:
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
✅ Netlify Preview
To edit notification comments on pull requests, go to your Netlify site configuration. |
| For previous binaries, refer to the [MOLT version manifest](https://molt.cockroachdb.com/molt/cli/versions.html). {% if page.name != "molt.md" %}For release details, see the [MOLT changelog]({% link releases/molt.md %}).{% endif %} | ||
| The following binaries are included: | ||
| - `molt` | ||
| - `replicator`. To use replication features, `replicator` **must** be located either in the same directory as `molt` or in a directory beneath `molt`. |
There was a problem hiding this comment.
@ryanluu12345 We should fix this since it was an issue that was reported recently where the directories beneath didnt seem to work
src/current/molt/molt-fetch.md
Outdated
|
|
||
| {% include_cached copy-clipboard.html %} | ||
| ~~~ | ||
| --bucket-path 's3://migration/data/cockroach' | ||
| --bucket-path 's3://migration/data/cockroach/?AWS_REGION=ap-south-1' |
There was a problem hiding this comment.
@ZhouXing19 @ryanluu12345 Is this the right format for this? I thought we don't respect anything with parsms for the bucket path. Just s3://some-name/path
Thats why the import-region flag was introduced.
There was a problem hiding this comment.
It's true.
We ignore the parameter passed via the --bucket-path.
So in this example we can do
--bucket-path 's3://migration/data/cockroach
--import-region 'ap-south-1'
and in the gcp example it can be
--bucket-path 'gs://migration/data/cockroach
--assume-role='[email protected]'
--use-implicit-auth
There was a problem hiding this comment.
I'm very confused. The flag description says:
Required when AWS_REGION needs to be included in the --bucket-path URL. For example, if the s3 URL has AWS_REGION=ap-south-1, include --import-region=ap-south-1.
Also, for --bucket-path, we have:
Only the URL path is used; query parameters (e.g., credentials) are ignored. To include the AWS_REGION query parameter in an s3 URL, set the --import-region flag.
So basically "include the AWS_REGION in the s3 URL" doesn't mean literally including it in the s3 URL? It just means passing AWS_REGION to the IMPORT command internally? If so, then the current docs are pretty misleading IMO. @ZhouXing19 @Jeremyyang920
There was a problem hiding this comment.
Sorry for the confusion!
So basically "include the AWS_REGION in the s3 URL" doesn't mean literally including it in the s3 URL?
Unfortunately yes. The AWS_REGION passed in the --bucket-path URL will be ignored.
Eventually, the real bucket path used for IMPORT will be this formula:
{bucket-path, stripped with all parameter}{/?AWS_REGION={value passed via --aws-region}}{&ASSUME_ROLE={value passed via --assume-role}}
cc @rohan-joshi @ryanluu12345 @Jeremyyang920 it seems that our current flag syntax is confusing.
For context, I was advocating having the user pass all these parameters via the bucket path, rather than using individual flags. I think this was discussed in one weekly that i wasn't able to make it, and eventually we settled with the individual flags solution. But given the usage seems confusing here, I wonder if it worths us reconsider the one-flag approach.
There was a problem hiding this comment.
I didn't mean to suggest that the flags are inherently confusing, just that I was still confused after all the discussion on how they work. I'd want to make the docs crystal-clear on this. Are the following true?
- No query parameters should be used in any cloud storage URLs defined in
--bucket-path - If you need to pass information that you would normally pass in a query parameter, use the flag instead (
--import-region,--assume-role,--use-implicit-auth)
There was a problem hiding this comment.
No query parameters should be used in any cloud storage URLs defined in --bucket-path
It's more that the parameters used in the bucket path will be ignored. It won't error out if being specified, but it won't take effect in the eventual url used by IMPORT.
If you need to pass information that you would normally pass in a query parameter, use the flag instead (--import-region, --assume-role, --use-implicit-auth)
Yes, only if those parameters are passed via flags, they will take effect in the eventual import url.
There was a problem hiding this comment.
Looks good so far! Not sure if this needs to be addressed now, but it seems that MOLT Verify flags for --table-filter and --schema-filter don't have their default values specified ('.*'), while their MOLT Fetch flags counterparts do.
|
Addressed your feedback @noelcrl @Jeremyyang920 @ZhouXing19, PTAL! |
Co-authored-by: Florence Morris <[email protected]>
DOC-12144
DOC-12174
replicatorbinary