Support preserving unknown fields in ProtoBuf format

docs/formats.md

@@ -25,6 +25,7 @@ stable, these are currently experimental features of Kotlin Serialization.
   * [Oneof field (experimental)](#oneof-field-experimental)
     * [Usage](#usage)
     * [Alternative](#alternative)
+  * [Preserving unknown fields (experimental)](#preserving-unknown-fields-experimental)
   * [ProtoBuf schema generator (experimental)](#protobuf-schema-generator-experimental)
 * [Properties (experimental)](#properties-experimental)
 * [Custom formats (experimental)](#custom-formats-experimental)
@@ -483,7 +484,7 @@ Field #3: 1D Fixed32 Value = 3, Hex = 03-00-00-00
 
 ### Lists as repeated fields
 
-By default, kotlin lists and other collections are representend as repeated fields. 
+By default, kotlin lists and other collections are represented as repeated fields. 
 In the protocol buffers when the list is empty there are no elements in the 
 stream with the corresponding number. For Kotlin Serialization you must explicitly specify a default of `emptyList()`
 for any property of a collection or map type. Otherwise you will not be able deserialize an empty
@@ -642,6 +643,54 @@ is also compatible with the `message Data` given above, which means the same inp
 
 But please note that there are no exclusivity checks. This means that if an instance of `Data2` has both (or none) `homeNumber` and `workNumber` as non-null values and is serialized to protobuf, it no longer complies with the original schema. If you send such data to another parser, one of the fields may be omitted, leading to an unknown issue.
 
+### Preserving unknown fields (experimental)
+
+You may keep updating your schema by adding new fields, but you may not want to break compatibility with the old data.
+
+Kotlin Serialization `ProtoBuf` format supports preserving unknown fields, as described in the [Protocol Buffer-Unknown Fields](https://protobuf.dev/programming-guides/proto3/#unknowns).
+
+To keep the unknown fields, add a property in type `ProtoMessage` with default value `null` or `ProtoMessage.Empty`, and annotation `@ProtoUnknownFields` to your data class.
+
+<!--- INCLUDE
+import kotlinx.serialization.*
+import kotlinx.serialization.protobuf.*
+-->
+
+```kotlin
+@OptIn(ExperimentalSerializationApi::class)
+@Serializable
+data class Data(
+    @ProtoNumber(1) val name: String,
+    @ProtoUnknownFields val unknownFields: ProtoMessage = ProtoMessage.Empty
+)
+
+@OptIn(ExperimentalSerializationApi::class)
+@Serializable
+data class NewData(
+    @ProtoNumber(1) val name: String,
+    @ProtoNumber(2) val age: Int,
+)
+
+@OptIn(ExperimentalSerializationApi::class)
+fun main() {
+  val dataFromNewBinary = NewData("Tom", 25)
+  val hexString = ProtoBuf.encodeToHexString(dataFromNewBinary)
+  val dataInOldBinary = ProtoBuf.decodeFromHexString<Data>(hexString)
+  val hexOfOldData = ProtoBuf.encodeToHexString(dataInOldBinary)
+  println(hexOfOldData)
+  println(hexString)
+  assert(hexOfOldData == hexString)
+}
+```
+
+> You can get the full code [here](../guide/example/example-formats-09.kt).
+
+```text
+0a03546f6d1019
+0a03546f6d1019
+```
+
+<!--- TEST -->
 ### ProtoBuf schema generator (experimental)
 
 As mentioned above, when working with protocol buffers you usually use a ".proto" file and a code generator for your
@@ -676,15 +725,15 @@ fun main() {
   println(schemas)
 }
 ```
-> You can get the full code [here](../guide/example/example-formats-09.kt).
+> You can get the full code [here](../guide/example/example-formats-10.kt).
 
 Which would output as follows.
 
 ```text
 syntax = "proto2";
 
 
-// serial name 'example.exampleFormats09.SampleData'
+// serial name 'example.exampleFormats10.SampleData'
 message SampleData {
   required int64 amount = 1;
   optional string description = 2;
@@ -729,7 +778,7 @@ fun main() {
 }
 ```      
 
-> You can get the full code [here](../guide/example/example-formats-10.kt).
+> You can get the full code [here](../guide/example/example-formats-11.kt).
 
 The resulting map has dot-separated keys representing keys of the nested objects.
 
@@ -814,7 +863,7 @@ fun main() {
 }
 ```                                    
 
-> You can get the full code [here](../guide/example/example-formats-11.kt).
+> You can get the full code [here](../guide/example/example-formats-12.kt).
 
 As a result, we got all the primitive values in our object graph visited and put into a list
 in _serial_ order.
@@ -923,7 +972,7 @@ fun main() {
 }
 ```
 
-> You can get the full code [here](../guide/example/example-formats-12.kt).
+> You can get the full code [here](../guide/example/example-formats-13.kt).
 
 Now we can convert a list of primitives back to an object tree.
 
@@ -1021,7 +1070,7 @@ fun main() {
 }
 -->
 
-> You can get the full code [here](../guide/example/example-formats-13.kt).
+> You can get the full code [here](../guide/example/example-formats-14.kt).
 
 <!--- TEST 
 [kotlinx.serialization, kotlin, 9000]
@@ -1135,7 +1184,7 @@ fun main() {
 }
 ```
 
-> You can get the full code [here](../guide/example/example-formats-14.kt).
+> You can get the full code [here](../guide/example/example-formats-15.kt).
 
 We see the size of the list added to the result, letting the decoder know where to stop. 
 
@@ -1254,7 +1303,7 @@ fun main() {
 
 ```
 
-> You can get the full code [here](../guide/example/example-formats-15.kt).
+> You can get the full code [here](../guide/example/example-formats-16.kt).
 
 In the output we see how not-null`!!` and `NULL` marks are used.
 
@@ -1389,7 +1438,7 @@ fun main() {
 }
 ```
 
-> You can get the full code [here](../guide/example/example-formats-16.kt).
+> You can get the full code [here](../guide/example/example-formats-17.kt).
 
 As we can see, the result is a dense binary format that only contains the data that is being serialized. 
 It can be easily tweaked for any kind of domain-specific compact encoding.
@@ -1590,7 +1639,7 @@ fun main() {
 }
 ```
 
-> You can get the full code [here](../guide/example/example-formats-17.kt).
+> You can get the full code [here](../guide/example/example-formats-18.kt).
 
 As we can see, our custom byte array format is being used, with the compact encoding of its size in one byte. 
 

docs/serialization-guide.md

@@ -161,6 +161,7 @@ Once the project is set up, we can start serializing some classes.
   * <a name='oneof-field-experimental'></a>[Oneof field (experimental)](formats.md#oneof-field-experimental)
     * <a name='usage'></a>[Usage](formats.md#usage)
     * <a name='alternative'></a>[Alternative](formats.md#alternative)
+  * <a name='preserving-unknown-fields-experimental'></a>[Preserving unknown fields (experimental)](formats.md#preserving-unknown-fields-experimental)
   * <a name='protobuf-schema-generator-experimental'></a>[ProtoBuf schema generator (experimental)](formats.md#protobuf-schema-generator-experimental)
 * <a name='properties-experimental'></a>[Properties (experimental)](formats.md#properties-experimental)
 * <a name='custom-formats-experimental'></a>[Custom formats (experimental)](formats.md#custom-formats-experimental)

formats/protobuf/api/kotlinx-serialization-protobuf.api

@@ -30,6 +30,25 @@ public final class kotlinx/serialization/protobuf/ProtoIntegerType : java/lang/E
 	public static fun values ()[Lkotlinx/serialization/protobuf/ProtoIntegerType;
 }
 
+public final class kotlinx/serialization/protobuf/ProtoMessage {
+	public static final field Companion Lkotlinx/serialization/protobuf/ProtoMessage$Companion;
+	public final fun asByteArray ()[B
+	public fun equals (Ljava/lang/Object;)Z
+	public final fun getSize ()I
+	public fun hashCode ()I
+	public final fun merge (Lkotlinx/serialization/protobuf/ProtoMessage;)Lkotlinx/serialization/protobuf/ProtoMessage;
+	public final fun plus (Lkotlinx/serialization/protobuf/ProtoMessage;)Lkotlinx/serialization/protobuf/ProtoMessage;
+}
+
+public final class kotlinx/serialization/protobuf/ProtoMessage$Companion {
+	public final fun getEmpty ()Lkotlinx/serialization/protobuf/ProtoMessage;
+	public final fun serializer ()Lkotlinx/serialization/KSerializer;
+}
+
+public final class kotlinx/serialization/protobuf/ProtoMessageKt {
+	public static final fun merge (Lkotlinx/serialization/protobuf/ProtoMessage;Lkotlinx/serialization/protobuf/ProtoMessage;)Lkotlinx/serialization/protobuf/ProtoMessage;
+}
+
 public abstract interface annotation class kotlinx/serialization/protobuf/ProtoNumber : java/lang/annotation/Annotation {
 	public abstract fun number ()I
 }
@@ -62,6 +81,13 @@ public synthetic class kotlinx/serialization/protobuf/ProtoType$Impl : kotlinx/s
 	public final synthetic fun type ()Lkotlinx/serialization/protobuf/ProtoIntegerType;
 }
 
+public abstract interface annotation class kotlinx/serialization/protobuf/ProtoUnknownFields : java/lang/annotation/Annotation {
+}
+
+public synthetic class kotlinx/serialization/protobuf/ProtoUnknownFields$Impl : kotlinx/serialization/protobuf/ProtoUnknownFields {
+	public fun <init> ()V
+}
+
 public final class kotlinx/serialization/protobuf/schema/ProtoBufSchemaGenerator {
 	public static final field INSTANCE Lkotlinx/serialization/protobuf/schema/ProtoBufSchemaGenerator;
 	public final fun generateSchemaText (Ljava/util/List;Ljava/lang/String;Ljava/util/Map;)Ljava/lang/String;

formats/protobuf/api/kotlinx-serialization-protobuf.klib.api

@@ -28,6 +28,10 @@ open annotation class kotlinx.serialization.protobuf/ProtoType : kotlin/Annotati
         final fun <get-type>(): kotlinx.serialization.protobuf/ProtoIntegerType // kotlinx.serialization.protobuf/ProtoType.type.<get-type>|<get-type>(){}[0]
 }
 
+open annotation class kotlinx.serialization.protobuf/ProtoUnknownFields : kotlin/Annotation { // kotlinx.serialization.protobuf/ProtoUnknownFields|null[0]
+    constructor <init>() // kotlinx.serialization.protobuf/ProtoUnknownFields.<init>|<init>(){}[0]
+}
+
 final enum class kotlinx.serialization.protobuf/ProtoIntegerType : kotlin/Enum<kotlinx.serialization.protobuf/ProtoIntegerType> { // kotlinx.serialization.protobuf/ProtoIntegerType|null[0]
     enum entry DEFAULT // kotlinx.serialization.protobuf/ProtoIntegerType.DEFAULT|null[0]
     enum entry FIXED // kotlinx.serialization.protobuf/ProtoIntegerType.FIXED|null[0]
@@ -49,6 +53,24 @@ final class kotlinx.serialization.protobuf/ProtoBufBuilder { // kotlinx.serializ
         final fun <set-serializersModule>(kotlinx.serialization.modules/SerializersModule) // kotlinx.serialization.protobuf/ProtoBufBuilder.serializersModule.<set-serializersModule>|<set-serializersModule>(kotlinx.serialization.modules.SerializersModule){}[0]
 }
 
+final class kotlinx.serialization.protobuf/ProtoMessage { // kotlinx.serialization.protobuf/ProtoMessage|null[0]
+    final val size // kotlinx.serialization.protobuf/ProtoMessage.size|{}size[0]
+        final fun <get-size>(): kotlin/Int // kotlinx.serialization.protobuf/ProtoMessage.size.<get-size>|<get-size>(){}[0]
+
+    final fun asByteArray(): kotlin/ByteArray // kotlinx.serialization.protobuf/ProtoMessage.asByteArray|asByteArray(){}[0]
+    final fun equals(kotlin/Any?): kotlin/Boolean // kotlinx.serialization.protobuf/ProtoMessage.equals|equals(kotlin.Any?){}[0]
+    final fun hashCode(): kotlin/Int // kotlinx.serialization.protobuf/ProtoMessage.hashCode|hashCode(){}[0]
+    final fun merge(kotlinx.serialization.protobuf/ProtoMessage): kotlinx.serialization.protobuf/ProtoMessage // kotlinx.serialization.protobuf/ProtoMessage.merge|merge(kotlinx.serialization.protobuf.ProtoMessage){}[0]
+    final fun plus(kotlinx.serialization.protobuf/ProtoMessage): kotlinx.serialization.protobuf/ProtoMessage // kotlinx.serialization.protobuf/ProtoMessage.plus|plus(kotlinx.serialization.protobuf.ProtoMessage){}[0]
+
+    final object Companion { // kotlinx.serialization.protobuf/ProtoMessage.Companion|null[0]
+        final val Empty // kotlinx.serialization.protobuf/ProtoMessage.Companion.Empty|{}Empty[0]
+            final fun <get-Empty>(): kotlinx.serialization.protobuf/ProtoMessage // kotlinx.serialization.protobuf/ProtoMessage.Companion.Empty.<get-Empty>|<get-Empty>(){}[0]
+
+        final fun serializer(): kotlinx.serialization/KSerializer<kotlinx.serialization.protobuf/ProtoMessage> // kotlinx.serialization.protobuf/ProtoMessage.Companion.serializer|serializer(){}[0]
+    }
+}
+
 sealed class kotlinx.serialization.protobuf/ProtoBuf : kotlinx.serialization/BinaryFormat { // kotlinx.serialization.protobuf/ProtoBuf|null[0]
     open val serializersModule // kotlinx.serialization.protobuf/ProtoBuf.serializersModule|{}serializersModule[0]
         open fun <get-serializersModule>(): kotlinx.serialization.modules/SerializersModule // kotlinx.serialization.protobuf/ProtoBuf.serializersModule.<get-serializersModule>|<get-serializersModule>(){}[0]
@@ -64,4 +86,5 @@ final object kotlinx.serialization.protobuf.schema/ProtoBufSchemaGenerator { //
     final fun generateSchemaText(kotlinx.serialization.descriptors/SerialDescriptor, kotlin/String? = ..., kotlin.collections/Map<kotlin/String, kotlin/String> = ...): kotlin/String // kotlinx.serialization.protobuf.schema/ProtoBufSchemaGenerator.generateSchemaText|generateSchemaText(kotlinx.serialization.descriptors.SerialDescriptor;kotlin.String?;kotlin.collections.Map<kotlin.String,kotlin.String>){}[0]
 }
 
+final fun (kotlinx.serialization.protobuf/ProtoMessage?).kotlinx.serialization.protobuf/merge(kotlinx.serialization.protobuf/ProtoMessage?): kotlinx.serialization.protobuf/ProtoMessage // kotlinx.serialization.protobuf/merge|[email protected]?(kotlinx.serialization.protobuf.ProtoMessage?){}[0]
 final fun kotlinx.serialization.protobuf/ProtoBuf(kotlinx.serialization.protobuf/ProtoBuf = ..., kotlin/Function1<kotlinx.serialization.protobuf/ProtoBufBuilder, kotlin/Unit>): kotlinx.serialization.protobuf/ProtoBuf // kotlinx.serialization.protobuf/ProtoBuf|ProtoBuf(kotlinx.serialization.protobuf.ProtoBuf;kotlin.Function1<kotlinx.serialization.protobuf.ProtoBufBuilder,kotlin.Unit>){}[0]