Add session and statement state for all query types

[penghuo]

Description

1. create session and statement for all the queryType.

Issues Resolved

#2401

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[vamsimanohar]

is error written to result index or statement model in request index?
Are these different cases?

[penghuo]

both, Flint spark job,

• write result, state, error, to result index.
• write state and error to request index.

@kaituo please help confirm also.

[kaituo]

yes

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.56%. Comparing base (2f2ecd2) to head (d45376c).
Report is 207 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2413      +/-   ##
============================================
+ Coverage     95.54%   95.56%   +0.01%     
- Complexity     4985     4987       +2     
============================================
  Files           478      478              
  Lines         13883    13919      +36     
  Branches        931      931              
============================================
+ Hits          13265    13301      +36     
  Misses          598      598              
  Partials         20       20              

Flag	Coverage Δ
sql-engine	95.56% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[penghuo]

@kaituo two questions.

1. should we add a new StatementState cancelling and Spark job update it as cancelled?
2. what is the required configuration of REPL job, we have following configurations now, do we need more?
   • config.put(FLINT_JOB_REQUEST_INDEX, DATASOURCE_TO_REQUEST_INDEX.apply(datasourceName));
   • config.put(FLINT_JOB_SESSION_ID, sessionId);

[kaituo]

@kaituo two questions.

1. should we add a new StatementState `cancelling` and Spark job update it as `cancelled`?

2. what is the required configuration of REPL job, we have following configurations now, do we need more?
   
   * config.put(FLINT_JOB_REQUEST_INDEX, DATASOURCE_TO_REQUEST_INDEX.apply(datasourceName));
   * config.put(FLINT_JOB_SESSION_ID, sessionId);

1. Can you just update it as cancelled? If you cancel before repl pick the statement up or after repl finishes the statement, repl doesn't need to do anything else. If you cancel after repl picks it up and before repl finishes, repl may change your state, which I think is fine.

2. I also need these two:

   val dataSource = conf.get("spark.flint.datasource.name", "unknown")
   val wait = conf.get("spark.flint.job.type", "continue")

[kaituo]

does that mean there is only one job id for all DML queries? Would it cause issues if there is a concurrent DML running?

[kaituo]

should the type be streaming?

[Swiddis]

Closing as stale -- reopen if needed