chore: Basic UnixFS sanity checks #10701

PsychoPunkSage · 2025-02-06T05:31:09Z

Changes:

Basic Sanity checks for UnixFS nodes. <Raw data and dag-pb>
Added no-validate flag to cmd.options

Closes #10331

Signed-off-by: Abhinav Prakash <[email protected]>

https://docs.ipfs.tech/install/command-line/#system-requirements states 6 GiB, updating readme to match that

lidel

triage note:

thank you, this is useful, but needs regression test
@PsychoPunkSage mind adding a test that confirms error occurs when someone tries to copy dag-cbor to MFS? (e.g. empty dag-cbor: bafyreigbtj4x7ip5legnfznufuopl4sg4knzc2cof6duas4b3q2fy6swua)

core/commands/files.go

gammazero

See suggested changes and get CI passing first.

Signed-off-by: Abhinav Prakash <[email protected]>

PsychoPunkSage · 2025-02-15T19:19:51Z

Hi @lidel I wrote a test and it is failing ig... currently solving it... Can give some suggestions which might help me

=== RUN   TestFilesCp_DagCborNodeFails
    /home/psychopunk_sage/dev/OpenSource/kubo/core/commands/file_test.go:55: expected error but got nil
--- FAIL: TestFilesCp_DagCborNodeFails (0.01s)
FAIL
FAIL    github.com/ipfs/kubo/core/commands      0.051s

@gammazero Made required changes....

Signed-off-by: Abhinav Prakash <[email protected]>

gammazero · 2025-02-18T23:35:28Z

The test is failing because the type assertion for dag.RawNode succeeds, here.

To test your code, the test needs to create something that is not a dag.RawNode and is also not a dag.ProtoNode or is a dag.ProtoNode that fails when calling FSNodeFromBytes on node.(*dag.ProtoNode).Data()

Signed-off-by: Abhinav Prakash <[email protected]>

PsychoPunkSage · 2025-02-19T10:49:34Z

Hi @gammazero
Thank you for help....

The Test is passing
Output

=== RUN   TestFilesCp_DagCborNodeFails
--- PASS: TestFilesCp_DagCborNodeFails (0.00s)
PASS
ok      github.com/ipfs/kubo/core/commands      0.029s

This is necessary because a "force" option is already used elsewhere, and its use in multiple places is causing CI to report the error: `Error: option name "force" used multiple times`

gammazero · 2025-02-20T08:52:48Z

@PsychoPunkSage I made 3 additional minor changes. See my 3 commits above.

gammazero

LGTM

PsychoPunkSage · 2025-02-20T12:18:04Z

Hi @gammazero
I have gone through the changes... Thanks
GTG

PsychoPunkSage · 2025-02-23T14:35:02Z

hi @lidel I think is PR is ready to be merged.
Please have a look at it

lidel

Thank you, mostly ok, but needs end-to-end tests + small nits inline.

core/commands/files.go

core/commands/files_test.go

Co-authored-by: Marcin Rataj <[email protected]>

Signed-off-by: Abhinav Prakash <[email protected]>

PsychoPunkSage · 2025-02-27T20:05:39Z

test/cli/files_test.go

+		assert.Contains(t, res.Stderr.String(), "source must be a UnixFS")
+	})
+
+	t.Run("files cp with invalid DAG node succeeds with force", func(t *testing.T) {


Hi @lidel
I noticed an interesting behavior here...
This test reveals something important about your implementation - the --force flag skips the initial validation, but there's still a later step (flushing) that can't handle non-UnixFS node types.
filesCpCmd allows copying a non-UnixFS node with --force, but MFS itself might not be able to properly handle or flush that type of node, causing this error during the flush operation.

What should I do?? Maybe create another PR to handle the Flush thing....
Adding the Flush here will make this PR quite big ig.

Discussed this during triage today, and the consensus is to remove --force flag.

We don't have time to go intoo rabbit hole of fleshing out specs for MFS that supports codecs other than UnixfS( dag-pb and raw).

We don't want this actually, because MFS data model is files and directories (which could be mounted over something like FUSE in the future).

This formally limits ipfs files API to files and directories (unixfs).
If someone wants to mix UnixFS with other codecs (like dag-cbor) they can use lower level API at ipfs dag.

@PsychoPunkSage so the ask is to remove --force from this PR 🙏

Disregard, I'm going to do it :)

Signed-off-by: Abhinav Prakash <[email protected]>

lidel

Thank you, we will include this in Kubo 0.34.0-rc1

PsychoPunkSage and others added 3 commits February 6, 2025 10:51

fix::> Added checks for UnixFS + added force flag

f8a2fe5

Signed-off-by: Abhinav Prakash <[email protected]>

docs: update min requirements (ipfs#10687)

9db46b1

https://docs.ipfs.tech/install/command-line/#system-requirements states 6 GiB, updating readme to match that

docs(release): update RELEASE_CHECKLIST.md after v0.33.1 (ipfs#10697)

3031cf3

PsychoPunkSage requested a review from a team as a code owner February 6, 2025 05:31

Merge branch 'master' into issue-10331

b28bce9

lidel requested changes Feb 11, 2025

View reviewed changes

gammazero reviewed Feb 11, 2025

View reviewed changes

core/commands/files.go Outdated Show resolved Hide resolved

gammazero reviewed Feb 15, 2025

View reviewed changes

core/commands/files.go Outdated Show resolved Hide resolved

gammazero requested changes Feb 15, 2025

View reviewed changes

PsychoPunkSage added 5 commits February 15, 2025 23:03

fix::> Removed confusing shorthand

e922ef0

Signed-off-by: Abhinav Prakash <[email protected]>

fix::> impl changes

3297308

Signed-off-by: Abhinav Prakash <[email protected]>

fix::> Basic Setup of Tests

4a8bb62

Signed-off-by: Abhinav Prakash <[email protected]>

fix::> test 1

a687b2b

Signed-off-by: Abhinav Prakash <[email protected]>

fix::> test passing

de94ec6

Signed-off-by: Abhinav Prakash <[email protected]>

fix::> formatting

58e4c90

Signed-off-by: Abhinav Prakash <[email protected]>

PsychoPunkSage requested review from lidel and gammazero February 16, 2025 04:31

PsychoPunkSage added 2 commits February 16, 2025 19:03

fix::> Proper test env ready

8e6796f

Signed-off-by: Abhinav Prakash <[email protected]>

fix::> Proper test ready

fa7c769

Signed-off-by: Abhinav Prakash <[email protected]>

guillaumemichel assigned lidel, guillaumemichel and gammazero and unassigned guillaumemichel and lidel Feb 18, 2025

gammazero added the need/author-input Needs input from the original author label Feb 18, 2025

fix::> Test Passing

1296054

Signed-off-by: Abhinav Prakash <[email protected]>

Merge branch 'master' into issue-10331

888e1e8

gammazero added 3 commits February 19, 2025 22:35

Rename files cp "force" option to "no-validate"

8ab8a83

This is necessary because a "force" option is already used elsewhere, and its use in multiple places is causing CI to report the error: `Error: option name "force" used multiple times`

Use require package for more compact test code

d945049

Rename file_test.go to files_test.go to match the files command

34f05a2

gammazero added skip/changelog This change does NOT require a changelog entry and removed need/author-input Needs input from the original author labels Feb 20, 2025

gammazero approved these changes Feb 20, 2025

View reviewed changes

lidel mentioned this pull request Feb 25, 2025

Release 0.34 #10685

Open

47 tasks

lidel requested changes Feb 26, 2025

View reviewed changes

core/commands/files.go Outdated Show resolved Hide resolved

core/commands/files.go Outdated Show resolved Hide resolved

core/commands/files.go Outdated Show resolved Hide resolved

core/commands/files_test.go Outdated Show resolved Hide resolved

lidel unassigned gammazero Feb 26, 2025

lidel added the need/author-input Needs input from the original author label Feb 26, 2025

PsychoPunkSage and others added 4 commits February 27, 2025 23:51

Update core/commands/files.go

b77142d

Co-authored-by: Marcin Rataj <[email protected]>

Update core/commands/files.go

9f7f8b6

Co-authored-by: Marcin Rataj <[email protected]>

Update core/commands/files.go

9a59b8d

Co-authored-by: Marcin Rataj <[email protected]>

fix::> Added dedicated tests....

7ad6847

Signed-off-by: Abhinav Prakash <[email protected]>

PsychoPunkSage commented Feb 27, 2025

View reviewed changes

fix::> more random tests

548f630

Signed-off-by: Abhinav Prakash <[email protected]>

PsychoPunkSage requested a review from lidel March 2, 2025 16:49

lidel added 3 commits March 5, 2025 19:49

refactor: remove --force, simplify checks

5c93132

Merge branch 'master' into issue-10331

bf04ec1

docs: changelog

23c8b6e

lidel force-pushed the issue-10331 branch from a059b0b to 23c8b6e Compare March 5, 2025 19:15

lidel removed need/author-input Needs input from the original author skip/changelog This change does NOT require a changelog entry labels Mar 5, 2025

lidel self-assigned this Mar 5, 2025

lidel approved these changes Mar 5, 2025

View reviewed changes

lidel merged commit e221e94 into ipfs:master Mar 5, 2025
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Basic UnixFS sanity checks #10701

chore: Basic UnixFS sanity checks #10701

PsychoPunkSage commented Feb 6, 2025 •

edited by gammazero

Loading

lidel left a comment •

edited

Loading

gammazero left a comment

PsychoPunkSage commented Feb 15, 2025 •

edited

Loading

gammazero commented Feb 18, 2025 •

edited

Loading

PsychoPunkSage commented Feb 19, 2025

gammazero commented Feb 20, 2025

gammazero left a comment

PsychoPunkSage commented Feb 20, 2025

PsychoPunkSage commented Feb 23, 2025

lidel left a comment

PsychoPunkSage Feb 27, 2025

lidel Mar 4, 2025 •

edited

Loading

lidel Mar 5, 2025

lidel left a comment

chore: Basic UnixFS sanity checks #10701

chore: Basic UnixFS sanity checks #10701

Conversation

PsychoPunkSage commented Feb 6, 2025 • edited by gammazero Loading

lidel left a comment • edited Loading

Choose a reason for hiding this comment

gammazero left a comment

Choose a reason for hiding this comment

PsychoPunkSage commented Feb 15, 2025 • edited Loading

gammazero commented Feb 18, 2025 • edited Loading

PsychoPunkSage commented Feb 19, 2025

gammazero commented Feb 20, 2025

gammazero left a comment

Choose a reason for hiding this comment

PsychoPunkSage commented Feb 20, 2025

PsychoPunkSage commented Feb 23, 2025

lidel left a comment

Choose a reason for hiding this comment

PsychoPunkSage Feb 27, 2025

Choose a reason for hiding this comment

lidel Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

lidel Mar 5, 2025

Choose a reason for hiding this comment

lidel left a comment

Choose a reason for hiding this comment

PsychoPunkSage commented Feb 6, 2025 •

edited by gammazero

Loading

lidel left a comment •

edited

Loading

PsychoPunkSage commented Feb 15, 2025 •

edited

Loading

gammazero commented Feb 18, 2025 •

edited

Loading

lidel Mar 4, 2025 •

edited

Loading