Is there an easier way to convert kaitai to raw bytes？ #881
Comments
|
See https://doc.kaitai.io/faq.html#writing and the issue #27 - Kaitai Struct does not yet officially support writing structures back to bytes. However, there are Java PoC branches called The compiler doesn't have writing support for other languages than Java, but there are serialization routines available for some runtime libraries largely made by @jchv: C++ (kaitai-io/kaitai_struct_cpp_stl_runtime#30), C# (kaitai-io/kaitai_struct_csharp_runtime#14), Go ( I wish I could give you a more satisfying answer, but this is the best I can tell you right now. |
|
Thanks for your advises, I'm trying to build the KSC's serialization branch and see how it works, but i'm new to java and scala. I installed Intellij and scala plugin, download serialization branch and open it as a project, when I tried to build the project lots of errors appeared. Is there a specific tutorial to build and run kaitai_struct_compiler？ |
It would help if you posted the console output (messages marked as when I checkout the -sbt.version = 1.2.7
+sbt.version = 1.4.3If this doesn't fix it, please post the console output so that I can help you with that.
Not really (at least not that I'm aware of), but it should be relatively straightforward. Building the KSC only depends on If you have a problem, please don't hesistate to ask. |
|
I successfully build
when i tried to run I think step2 was wrong, can you give me some suggestions? |
The Java runtime library published in the Maven Central repository reflects the 0.9 tag on the I would simply Presuming this flat structure: kaitai-struct-compiler -t java --no-auto-read --read-write my_format.ksy # generates MyFormat.java
git clone --branch serialization --depth 1 https://github.com/kaitai-io/kaitai_struct_java_runtime.git # fills `kaitai_struct_java_runtime/` dir with .java files
mkdir bin/ # for Java .class bytecode files
shopt -s globstar # to find .java files in all subdirectories recursively
javac -encoding UTF-8 -d bin $(ls **/*.java)
java -cp bin App [...] # [...] are command line arguments required by the App.java (if any)This should work. |
It's most likely easier to build the runtime for itself like any other MVN project, which should result in some JAR-files within the directory hierarchy of the runtime. Afterwards, adopt the build of your own project to reference those JAR files in your local file system. Dependencies required by the runtime, if any at all, can easily be added as dependencies of your own project this way, cached like all other dependencies etc. This way you don't need to care about individual source code files and stuff. There are multiple ways to reference those JARs or put them into your local MVN mirror or alike. <dependency>
<groupId>anything</groupId>
<artifactId>anything</artifactId>
<version>anything</version>
<scope>system</scope>
<systemPath>${basedir}/lib/jar_name.jar</systemPath>
</dependency>https://stackoverflow.com/a/51647143/2055163 This is what I'm doing for Gradle: repositories
{
jcenter()
flatDir
{
dirs '../Libs Java 3rd/Parsers/Generators/kaitai_struct/runtime/libs_java_3rd_usage/lib'
}
}
[...]
dependencies
{
implementation ':kaitai-struct-runtime:0.9', [...]
runtimeOnly [...]
} |
|
@generalmimon @ams-tschoening Thanks for your help! I successfully built the program, but there was a problem when parsing the data: The ksy file: The test java script: By the way, if I want to add serialization support for python, which files need to be modified? |
It would be of help to know which line |
It seems to me that the System.out.println(data.enipHeader().enipLength());The problem is that when you enable the So you need to call EnipSegment data = new EnipSegment(new ByteBufferKaitaiStream(hexArray));
+ data._read();
System.out.println(data.enipHeader().enipLength()); |
|
You are my savior! data._read() works! but data._write() will throw a
I tried to enlarge the |
Please show the full interaction of your application code (the I need to see your code so I can help. |
|
my 1. import io.kaitai.struct.ByteBufferKaitaiStream;
2.
3. public class main {
4. public static void main(String[] args) {
5.
6. byte[] hexData = new byte[] {
7. 0x70, 0x00, 0x22, 0x00, 0x00, 0x01, 0x02, 0x12,
8. 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
9. 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
10. 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x02, 0x00,
11. (byte)0xA1, 0x00, 0x04, 0x00, 0x01, 0x2D, (byte)0x94, 0x00,
12. (byte)0xB1, 0x00, 0x0E, 0x00, 0x03, 0x00, 0x03, 0x02,
13. 0x20, 0x64, 0x24, 0x01, 0x02, 0x00, 0x01, 0x00,
14. 0x02, 0x00
15. };
16.
17. EnipSegment data = new EnipSegment(new ByteBufferKaitaiStream(hexData));
18.
19. data._read();
20. data.enipHeader().setEnipCommand(0x55);
21.
22. try {
23. data._check();
24. data._write();
25. }catch (Exception e){
26. System.out.println(e);
27. }
28.
29. System.out.println(data.enipHeader().enipCommand());
30. }
31. }generated module: 1. // This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild
2.
3. import io.kaitai.struct.ByteBufferKaitaiStream;
4. import io.kaitai.struct.KaitaiStruct;
5. import io.kaitai.struct.KaitaiStream;
6. import java.io.IOException;
7. import io.kaitai.struct.ConsistencyError;
8.
9.
10. /**
11. * Rockwell controller use ENIP communicate with RS5000 software
12. */
13. public class EnipSegment extends KaitaiStruct.ReadWrite {
14. @org.jetbrains.annotations.NotNull
15. @org.jetbrains.annotations.Contract("_ -> new")
16. public static EnipSegment fromFile(String fileName) throws IOException {
17. return new EnipSegment(new ByteBufferKaitaiStream(fileName));
18. }
19. public EnipSegment() {
20. this(null, null, null);
21. }
22.
23. public EnipSegment(KaitaiStream _io) {
24. this(_io, null, null);
25. }
26.
27. public EnipSegment(KaitaiStream _io, KaitaiStruct.ReadWrite _parent) {
28. this(_io, _parent, null);
29. }
30.
31. public EnipSegment(KaitaiStream _io, KaitaiStruct.ReadWrite _parent, EnipSegment _root) {
32. super(_io);
33. this._parent = _parent;
34. this._root = _root == null ? this : _root;
35. }
36. public void _read() {
37. this._raw_enipHeader = this._io.readBytes(24);
38. KaitaiStream _io__raw_enipHeader = new ByteBufferKaitaiStream(_raw_enipHeader);
39. this.enipHeader = new EnipHeader(_io__raw_enipHeader, this, _root);
40. this.enipHeader._read();
41. this._raw_commandData = this._io.readBytesFull();
42. KaitaiStream _io__raw_commandData = new ByteBufferKaitaiStream(_raw_commandData);
43. this.commandData = new CommandData(_io__raw_commandData, this, _root);
44. this.commandData._read();
45. }
46.
47. public void _write() {
48. KaitaiStream _io__raw_enipHeader = new ByteBufferKaitaiStream(24);
49. this.enipHeader._write(_io__raw_enipHeader);
50. this._io.writeStream(_io__raw_enipHeader);
51. this.commandData._write(this._io);
52. }
53.
54. public void _check() {
55. }
56. public static class EnipHeader extends KaitaiStruct.ReadWrite {
57. public static EnipHeader fromFile(String fileName) throws IOException {
58. return new EnipHeader(new ByteBufferKaitaiStream(fileName));
59. }
60. public EnipHeader() {
61. this(null, null, null);
62. }
63.
64. public EnipHeader(KaitaiStream _io) {
65. this(_io, null, null);
66. }
67.
68. public EnipHeader(KaitaiStream _io, EnipSegment _parent) {
69. this(_io, _parent, null);
70. }
71.
72. public EnipHeader(KaitaiStream _io, EnipSegment _parent, EnipSegment _root) {
73. super(_io);
74. this._parent = _parent;
75. this._root = _root;
76. }
77. public void _read() {
78. this.enipCommand = this._io.readU2le();
79. this.enipLength = this._io.readU2le();
80. this.sessionHandle = this._io.readU4le();
81. this.enipStatus = this._io.readU4le();
82. this.senderContext = this._io.readBytes(8);
83. this.enipOptions = this._io.readBytes(4);
84. }
85.
86. public void _write() {
87. this._io.writeU2le(this.enipCommand);
88. this._io.writeU2le(this.enipLength);
89. this._io.writeU4le(this.sessionHandle);
90. this._io.writeU4le(this.enipStatus);
91. this._io.writeBytes(this.senderContext);
92. this._io.writeBytes(this.enipOptions);
93. }
94.
95. public void _check() {
96. if (senderContext().length != 8)
97. throw new ConsistencyError("sender_context", senderContext().length, 8);
98. if (enipOptions().length != 4)
99. throw new ConsistencyError("enip_options", enipOptions().length, 4);
100. }
101. private int enipCommand;
102. private int enipLength;
103. private long sessionHandle;
104. private long enipStatus;
105. private byte[] senderContext;
106. private byte[] enipOptions;
107. private EnipSegment _root;
108. private EnipSegment _parent;
109. public int enipCommand() { return enipCommand; }
110. public void setEnipCommand(int _v) { enipCommand = _v; }
111. public int enipLength() { return enipLength; }
112. public void setEnipLength(int _v) { enipLength = _v; }
113. public long sessionHandle() { return sessionHandle; }
114. public void setSessionHandle(long _v) { sessionHandle = _v; }
115. public long enipStatus() { return enipStatus; }
116. public void setEnipStatus(long _v) { enipStatus = _v; }
117. public byte[] senderContext() { return senderContext; }
118. public void setSenderContext(byte[] _v) { senderContext = _v; }
119. public byte[] enipOptions() { return enipOptions; }
120. public void setEnipOptions(byte[] _v) { enipOptions = _v; }
121. public EnipSegment _root() { return _root; }
122. public void set_root(EnipSegment _v) { _root = _v; }
123. public EnipSegment _parent() { return _parent; }
124. public void set_parent(EnipSegment _v) { _parent = _v; }
125. }
126. public static class CommandData extends KaitaiStruct.ReadWrite {
127. public static CommandData fromFile(String fileName) throws IOException {
128. return new CommandData(new ByteBufferKaitaiStream(fileName));
129. }
130. public CommandData() {
131. this(null, null, null);
132. }
133.
134. public CommandData(KaitaiStream _io) {
135. this(_io, null, null);
136. }
137.
138. public CommandData(KaitaiStream _io, EnipSegment _parent) {
139. this(_io, _parent, null);
140. }
141.
142. public CommandData(KaitaiStream _io, EnipSegment _parent, EnipSegment _root) {
143. super(_io);
144. this._parent = _parent;
145. this._root = _root;
146. }
147. public void _read() {
148. this.interfaceHandle = this._io.readU4le();
149. this.timeout = this._io.readU2le();
150. this.itemCount = this._io.readU2le();
151. this._raw_addressItem = this._io.readBytes(8);
152. KaitaiStream _io__raw_addressItem = new ByteBufferKaitaiStream(_raw_addressItem);
153. this.addressItem = new ConnectedAddressItem(_io__raw_addressItem, this, _root);
154. this.addressItem._read();
155. this._raw_dataItem = this._io.readBytes(6);
156. KaitaiStream _io__raw_dataItem = new ByteBufferKaitaiStream(_raw_dataItem);
157. this.dataItem = new DataItem(_io__raw_dataItem, this, _root);
158. this.dataItem._read();
159. this.enipData = this._io.readBytesFull();
160. }
161.
162. public void _write() {
163. this._io.writeU4le(this.interfaceHandle);
164. this._io.writeU2le(this.timeout);
165. this._io.writeU2le(this.itemCount);
166. KaitaiStream _io__raw_addressItem = new ByteBufferKaitaiStream(8);
167. this.addressItem._write(_io__raw_addressItem);
168. this._io.writeStream(_io__raw_addressItem);
169. KaitaiStream _io__raw_dataItem = new ByteBufferKaitaiStream(6);
170. this.dataItem._write(_io__raw_dataItem);
171. this._io.writeStream(_io__raw_dataItem);
172. this._io.writeBytes(this.enipData);
173. }
174.
175. public void _check() {
176. }
177. private long interfaceHandle;
178. private int timeout;
179. private int itemCount;
180. private ConnectedAddressItem addressItem;
181. private DataItem dataItem;
182. private byte[] enipData;
183. private EnipSegment _root;
184. private EnipSegment _parent;
185. private byte[] _raw_addressItem;
186. private byte[] _raw_dataItem;
187. public long interfaceHandle() { return interfaceHandle; }
188. public void setInterfaceHandle(long _v) { interfaceHandle = _v; }
189. public int timeout() { return timeout; }
190. public void setTimeout(int _v) { timeout = _v; }
191. public int itemCount() { return itemCount; }
192. public void setItemCount(int _v) { itemCount = _v; }
193. public ConnectedAddressItem addressItem() { return addressItem; }
194. public void setAddressItem(ConnectedAddressItem _v) { addressItem = _v; }
195. public DataItem dataItem() { return dataItem; }
196. public void setDataItem(DataItem _v) { dataItem = _v; }
197. public byte[] enipData() { return enipData; }
198. public void setEnipData(byte[] _v) { enipData = _v; }
199. public EnipSegment _root() { return _root; }
200. public void set_root(EnipSegment _v) { _root = _v; }
201. public EnipSegment _parent() { return _parent; }
202. public void set_parent(EnipSegment _v) { _parent = _v; }
203. public byte[] _raw_addressItem() { return _raw_addressItem; }
204. public void set_raw_AddressItem(byte[] _v) { _raw_addressItem = _v; }
205. public byte[] _raw_dataItem() { return _raw_dataItem; }
206. public void set_raw_DataItem(byte[] _v) { _raw_dataItem = _v; }
207. }
208. public static class ConnectedAddressItem extends KaitaiStruct.ReadWrite {
209. public static ConnectedAddressItem fromFile(String fileName) throws IOException {
210. return new ConnectedAddressItem(new ByteBufferKaitaiStream(fileName));
211. }
212. public ConnectedAddressItem() {
213. this(null, null, null);
214. }
215.
216. public ConnectedAddressItem(KaitaiStream _io) {
217. this(_io, null, null);
218. }
219.
220. public ConnectedAddressItem(KaitaiStream _io, EnipSegment.CommandData _parent) {
221. this(_io, _parent, null);
222. }
223.
224. public ConnectedAddressItem(KaitaiStream _io, EnipSegment.CommandData _parent, EnipSegment _root) {
225. super(_io);
226. this._parent = _parent;
227. this._root = _root;
228. }
229. public void _read() {
230. this.typeId = this._io.readU2le();
231. this.addressItemLen = this._io.readU2le();
232. this.connectionId = this._io.readU4le();
233. }
234.
235. public void _write() {
236. this._io.writeU2le(this.typeId);
237. this._io.writeU2le(this.addressItemLen);
238. this._io.writeU4le(this.connectionId);
239. }
240.
241. public void _check() {
242. }
243. private int typeId;
244. private int addressItemLen;
245. private long connectionId;
246. private EnipSegment _root;
247. private EnipSegment.CommandData _parent;
248. public int typeId() { return typeId; }
249. public void setTypeId(int _v) { typeId = _v; }
250. public int addressItemLen() { return addressItemLen; }
251. public void setAddressItemLen(int _v) { addressItemLen = _v; }
252. public long connectionId() { return connectionId; }
253. public void setConnectionId(long _v) { connectionId = _v; }
254. public EnipSegment _root() { return _root; }
255. public void set_root(EnipSegment _v) { _root = _v; }
256. public EnipSegment.CommandData _parent() { return _parent; }
257. public void set_parent(EnipSegment.CommandData _v) { _parent = _v; }
258. }
259. public static class DataItem extends KaitaiStruct.ReadWrite {
260. public static DataItem fromFile(String fileName) throws IOException {
261. return new DataItem(new ByteBufferKaitaiStream(fileName));
262. }
263. public DataItem() {
264. this(null, null, null);
265. }
266.
267. public DataItem(KaitaiStream _io) {
268. this(_io, null, null);
269. }
270.
271. public DataItem(KaitaiStream _io, EnipSegment.CommandData _parent) {
272. this(_io, _parent, null);
273. }
274.
275. public DataItem(KaitaiStream _io, EnipSegment.CommandData _parent, EnipSegment _root) {
276. super(_io);
277. this._parent = _parent;
278. this._root = _root;
279. }
280. public void _read() {
281. this.typeId = this._io.readU2le();
282. this.dataItemLen = this._io.readU2le();
283. this.cipSeq = this._io.readU2le();
284. }
285.
286. public void _write() {
287. this._io.writeU2le(this.typeId);
288. this._io.writeU2le(this.dataItemLen);
289. this._io.writeU2le(this.cipSeq);
290. }
291.
292. public void _check() {
293. }
294. private int typeId;
295. private int dataItemLen;
296. private int cipSeq;
297. private EnipSegment _root;
298. private EnipSegment.CommandData _parent;
299. public int typeId() { return typeId; }
300. public void setTypeId(int _v) { typeId = _v; }
301. public int dataItemLen() { return dataItemLen; }
302. public void setDataItemLen(int _v) { dataItemLen = _v; }
303. public int cipSeq() { return cipSeq; }
304. public void setCipSeq(int _v) { cipSeq = _v; }
305. public EnipSegment _root() { return _root; }
306. public void set_root(EnipSegment _v) { _root = _v; }
307. public EnipSegment.CommandData _parent() { return _parent; }
308. public void set_parent(EnipSegment.CommandData _v) { _parent = _v; }
309. }
310. private EnipHeader enipHeader;
311. private CommandData commandData;
312. private EnipSegment _root;
313. private KaitaiStruct.ReadWrite _parent;
314. private byte[] _raw_enipHeader;
315. private byte[] _raw_commandData;
316. public EnipHeader enipHeader() { return enipHeader; }
317. public void setEnipHeader(EnipHeader _v) { enipHeader = _v; }
318. public CommandData commandData() { return commandData; }
319. public void setCommandData(CommandData _v) { commandData = _v; }
320. public EnipSegment _root() { return _root; }
321. public void set_root(EnipSegment _v) { _root = _v; }
322. public KaitaiStruct.ReadWrite _parent() { return _parent; }
323. public void set_parent(KaitaiStruct.ReadWrite _v) { _parent = _v; }
324. public byte[] _raw_enipHeader() { return _raw_enipHeader; }
325. public void set_raw_EnipHeader(byte[] _v) { _raw_enipHeader = _v; }
326. public byte[] _raw_commandData() { return _raw_commandData; }
327. public void set_raw_CommandData(byte[] _v) { _raw_commandData = _v; }
328. }I tried calling _check() before _write() and changing input byte array, it won't work. |
|
@silver-fang1 Thanks for the code. I can finally see the problem - after you You need to import io.kaitai.struct.ByteBufferKaitaiStream;
public class main {
public static void main(String[] args) {
byte[] hexData = new byte[] {
0x70, 0x00, 0x22, 0x00, 0x00, 0x01, 0x02, 0x12,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x02, 0x00,
(byte)0xA1, 0x00, 0x04, 0x00, 0x01, 0x2D, (byte)0x94, 0x00,
(byte)0xB1, 0x00, 0x0E, 0x00, 0x03, 0x00, 0x03, 0x02,
0x20, 0x64, 0x24, 0x01, 0x02, 0x00, 0x01, 0x00,
0x02, 0x00
};
EnipSegment data = new EnipSegment(new ByteBufferKaitaiStream(hexData));
data._read();
+ System.out.println("_io.pos after _read(): " + data._io().pos());
+ data._io().seek(0);
+ System.out.println("_io.pos before _write(): " + data._io().pos());
+
data.enipHeader().setEnipCommand(0x55);
+ data.enipHeader()._check();
+
+ System.out.println("hexData before _write(): " + byteArrayToHex(hexData));
try {
data._check();
data._write();
}catch (Exception e){
System.out.println(e);
}
- System.out.println(data.enipHeader().enipCommand());
+ System.out.println("hexData after _write(): " + byteArrayToHex(hexData));
+ System.out.println("_io.pos after _write(): " + data._io().pos());
+
+ System.out.println("enipCommand: " + data.enipHeader().enipCommand());
+ }
+
+ protected static String byteArrayToHex(byte[] arr) {
+ StringBuilder sb = new StringBuilder("[");
+ for (int i = 0; i < arr.length; i++) {
+ if (i > 0)
+ sb.append(' ');
+ sb.append(String.format("%02x", arr[i]));
+ }
+ sb.append(']');
+ return sb.toString();
}
}BTW, the Output: $ javac -encoding UTF-8 -d bin $(ls **/*.java)
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding="UTF-8"
$ java -cp bin main
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding="UTF-8"
_io.pos after _read(): 58
_io.pos before _write(): 0
hexData before _write(): [70 00 22 00 00 01 02 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 02 00 a1 00 04 00 01 2d 94 00 b1 00 0e 00 03 00 03 02 20 64 24 01 02 00 01 00 02 00]
hexData after _write(): [55 00 22 00 00 01 02 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 02 00 a1 00 04 00 01 2d 94 00 b1 00 0e 00 03 00 03 02 20 64 24 01 02 00 01 00 02 00]
_io.pos after _write(): 58
enipCommand: 85 |
|
@silver-fang1 Also notice that I added a data.enipHeader().setEnipCommand(0x55);
+ data.enipHeader()._check();Calling Therefore, you should call try {
data._check();
data._write();
}catch (Exception e){As you can see in the generated code, /**
* Rockwell controller use ENIP communicate with RS5000 software
*/
public class EnipSegment extends KaitaiStruct.ReadWrite {
// ...
public void _write() {
KaitaiStream _io__raw_enipHeader = new ByteBufferKaitaiStream(24);
this.enipHeader._write(_io__raw_enipHeader);
this._io.writeStream(_io__raw_enipHeader);
this.commandData._write(this._io);
}
public void _check() { // NB: no checks
}
public static class EnipHeader extends KaitaiStruct.ReadWrite {
// ...... but the subtype public static class EnipHeader extends KaitaiStruct.ReadWrite {
// ...
public void _read() {
this.enipCommand = this._io.readU2le();
// ...
this.senderContext = this._io.readBytes(8);
this.enipOptions = this._io.readBytes(4);
}
public void _write() {
this._io.writeU2le(this.enipCommand);
// ...
this._io.writeBytes(this.senderContext);
this._io.writeBytes(this.enipOptions);
}
public void _check() {
if (senderContext().length != 8)
throw new ConsistencyError("sender_context", senderContext().length, 8);
if (enipOptions().length != 4)
throw new ConsistencyError("enip_options", enipOptions().length, 4);
}Calling the However, when you changed a field of data.enipHeader().setEnipCommand(0x55);
+ data.enipHeader()._check();Although there are not necessarily any checks specifically for An answer to your previous question from #881 (comment):
Yes, exactly. As I mentioned in #881 (comment), for Python there is already a WIP pull request with serialization routines:
So I suggest forking from there for a faster start and making it follow the existing Java runtime library code ( class JavaCompiler(val typeProvider: ClassTypeProvider, config: RuntimeConfig)
extends LanguageCompiler(typeProvider, config)
with SingleOutputFile
with UpperCamelCaseClasses
with ObjectOrientedLanguage
with EveryReadIsExpression
with EveryWriteIsExpression
with GenericChecks
with UniversalFooter
with UniversalDoc
with AllocateIOLocalVar
with FixedContentsUsingArrayByteLiteral
with SwitchIfOps
with NoNeedForFullClassPath {Certainly at least |
I'm using kaitai parsing network protocol，it works well. But when I want to develop a client to communicate with the server using a specified protocol, I need to generate raw data packets，copy each field into an array，at the end send the array.
So is there an easier way to do this?
The text was updated successfully, but these errors were encountered: