Expose Utf8LossyChunksIter

[dylni]

Proposal

Problem statement

When Utf8Error::valid_up_to and Utf8Error::error_len are used, their results will almost always be used to get substrings of the original string. However, since Utf8Error does not have a reference to the original string, it cannot have methods to return the substrings.

Utf8LossyChunksIter is also much easier to use.

Motivation, use-cases

This is useful when creating a custom byte string formatter. UTF-8 portions are usually output using the Display implementation for str or str::escape_debug, but invalid portions might require custom formatting.

Example

Code using str::from_utf8 (requires unsafe):

while !string.is_empty() {
    let (valid, invalid) = match str::from_utf8(string) {
        Ok(string) => (string, &[][..]),
        Err(error) => {
            let valid_len = error.valid_up_to();
            let valid = unsafe { str::from_utf8_unchecked(&string[..valid_len]) };
            let mut invalid = &string[valid_len..];
            if let Some(invalid_len) = error.error_len() {
                invalid = &invalid[..invalid_len];
            }
            (valid, invalid)
        }
    };
    // formatting for `valid` and `invalid`
    string = &string[valid.len() + invalid.len()..];
}

Code using the new API:

for chunk in Utf8Chunks::new(string) {
    let valid = chunk.valid();
    let invalid = chunk.invalid();
    // formatting for `valid` and `invalid`
}

Solution sketches

Make the following changes, and change the feature for these structs from str_internals to utf8_chunks.

• Remove Utf8Lossy.
• Rename Utf8LossyChunksIter to Utf8Chunks.
• Rename Utf8LossyChunk to Utf8Chunk.
• Rename Utf8LossyChunk::broken to invalid.
• Change fields of Utf8Chunks into getter methods (i.e., valid and invalid).
• Add:

  impl<'a> Utf8Chunks<'a> {
      fn new(bytes: &'a [u8]) -> Self;
  }

Links and related work

What happens now?

This issue is part of the libs-api team API change proposal process. Once this issue is filed the libs-api team will review open proposals in its weekly meeting. You should receive feedback within a week or two.

[m-ou-se]

The bstr crate has a similar API, which also exposes another field/method on the chunks: incomplete, to check whether the incomplete sequence might still become valid if more bytes are added: https://docs.rs/bstr/latest/bstr/struct.Utf8Chunk.html

[m-ou-se]

Regardless of that minor detail, I'm in favor of working towards exposing this as a public API!

[dylni]

Great! I agree that basing the API off bstr would be sensible, so I have updated the issue. The changes were actually surprisingly close already, but providing methods instead of fields will be better for future compatibility.

I'm leaving out the incomplete method for now, since that can be added later and I don't personally need it. However, I can add it back if it seems useful.

Also, it might be worth discussing if the constructor should be <[u8]>::utf8_chunks or UtfChunks::new. If I understand the process correctly, that discussion can happen on the tracking issue.

[dylni]

@m-ou-se Is this proposal ready for a PR? It still has the "initial-comment-period" label, so I'm not sure if it's been agreed that this should be added.

[m-ou-se]

@dylni Sorry! We don't have this processs automated yet. Yes, feel free to go ahead and open a tracking issue for this new unstable feature and send a PR for its implementation.

[dylni]

Thanks!

Tracking issue: rust-lang/rust#99543

[dylni]

This has been merged: rust-lang/rust#99544