Finding spark pipeline start time from spline lineage

[yuvanprakash]

Im exploring spline to determine how much time it took for spark to execute a pipeline - Duration of the execution (from initialising spark context till writing the result). I could see

"timestamp":1611397050192

in the Spline lineage file which is actually write time. Is there any option to get Start Time of the pipeline from Spline Lineage Log?

[wajda]

This question is somewhat slippery. It depends on the meaning of the term "start time".
It may sound silly but... the start of what? - the application, the write command, the first read command etc.
For instance, you may have a complex transformation chain with some data frames cached and reused in multiple sub-chains. So in reality some computations would be omitted even though they would still be represented in the logical execution plan (and hence in Spline lineage).
Spark tells us the duration of the write command, but it doesn't tell us what computation nodes were actually included in that duration time. To answer it precisely we'd need to examine the physical plan, but unfortunately there is no trace from the physical level back to the logical level. So we are only left with guessing, and that's probably not what you are after :)

If your pipelines aren't complex and not using caching then you can calculate the start time by simply subtracting the duration from the timestamp.

val startTimestamp = timestamp - durationNs * 1000000` // the duration is captured in nanoseconds while timestamp is in millis