Make ForwardDiff Optional

[ChrisRackauckas]

@time using OrdinaryDiffEqTsit5
# 0.678347 seconds (806.13 k allocations: 60.106 MiB, 3.97% gc time, 7.38% compilation time)

@time begin
  using OrdinaryDiffEqTsit5
  function lorenz(du,u,p,t)
      du[1] = 10.0(u[2]-u[1])
      du[2] = u[1]*(28.0-u[3]) - u[2]
      du[3] = u[1]*u[2] - (8/3)*u[3]
  end
  u0 = [1.0;0.0;0.0]; tspan = (0.0,100.0)
  prob = ODEProblem(lorenz,u0,tspan)
  solve(prob,Tsit5())
end

#  0.765026 seconds (1.29 M allocations: 86.179 MiB, 2.16% gc time, 24.20% compilation time: 2% of which was recompilation)

@time_imports using OrdinaryDiffEqTsit5

#=
               ┌ 0.0 ms DocStringExtensions.__init__()
      1.4 ms  DocStringExtensions
      0.5 ms  Reexport
      7.6 ms  Preferences
      0.6 ms  PrecompileTools
      0.5 ms  FastPower
      1.2 ms  ArrayInterface
      0.9 ms  StaticArraysCore
      0.7 ms  ArrayInterface → ArrayInterfaceStaticArraysCoreExt
      5.6 ms  FunctionWrappers
      0.5 ms  MuladdMacro
      3.2 ms  OrderedCollections
      0.5 ms  UnPack
      0.7 ms  Parameters
      0.8 ms  Statistics
      0.4 ms  IfElse
      0.9 ms  CommonWorldInvalidations
      9.8 ms  Static
      0.8 ms  Compat
      0.5 ms  Compat → CompatLinearAlgebraExt
     12.2 ms  StaticArrayInterface
      1.0 ms  ManualMemory
               ┌ 0.0 ms ThreadingUtilities.__init__()
      4.7 ms  ThreadingUtilities
      0.7 ms  SIMDTypes
      2.0 ms  LayoutPointers
      2.9 ms  CloseOpenIntervals
      8.5 ms  StrideArraysCore
      0.6 ms  BitTwiddlingConvenienceFunctions
               ┌ 0.0 ms CPUSummary.__init__()
      1.6 ms  CPUSummary
               ┌ 0.0 ms PolyesterWeave.__init__()
      2.6 ms  PolyesterWeave
      0.8 ms  Polyester
      1.5 ms  FastBroadcast
      9.4 ms  RecipesBase
      0.8 ms  ExprTools
      1.1 ms  Serialization
      0.9 ms  RuntimeGeneratedFunctions
      7.2 ms  MacroTools
               ┌ 0.0 ms InverseFunctions.__init__()
      1.6 ms  InverseFunctions
      0.7 ms  ConstructionBase
      0.6 ms  ConstructionBase → ConstructionBaseLinearAlgebraExt
      0.6 ms  CompositionsBase
      0.6 ms  CompositionsBase → CompositionsBaseInverseFunctionsExt
      0.6 ms  InverseFunctions → InverseFunctionsDatesExt
               ┌ 0.0 ms Accessors.__init__()
     10.6 ms  Accessors
      0.7 ms  Accessors → LinearAlgebraExt
      2.9 ms  SymbolicIndexingInterface
      0.8 ms  Adapt
      0.7 ms  DataValueInterfaces
      1.0 ms  DataAPI
      0.7 ms  IteratorInterfaceExtensions
      0.7 ms  TableTraits
      8.2 ms  Tables
      0.9 ms  GPUArraysCore
      0.6 ms  ArrayInterface → ArrayInterfaceGPUArraysCoreExt
     17.1 ms  RecursiveArrayTools
      0.8 ms  RecursiveArrayTools → RecursiveArrayToolsFastBroadcastExt
               ┌ 0.0 ms TruncatedStacktraces.__init__()
      0.8 ms  TruncatedStacktraces
      6.6 ms  Setfield
     14.8 ms  IrrationalConstants
      1.2 ms  DiffRules
      1.3 ms  DiffResults
               ┌ 2.1 ms OpenLibm_jll.__init__()
      4.9 ms  OpenLibm_jll
      0.9 ms  NaNMath
      0.9 ms  LogExpFunctions
      0.8 ms  JLLWrappers
               ┌ 12.0 ms CompilerSupportLibraries_jll.__init__()
     15.4 ms  CompilerSupportLibraries_jll
               ┌ 1.3 ms OpenSpecFun_jll.__init__()
      2.3 ms  OpenSpecFun_jll
      5.7 ms  SpecialFunctions
      0.8 ms  CommonSubexpressions
     25.3 ms  ForwardDiff
      0.8 ms  LogExpFunctions → LogExpFunctionsInverseFunctionsExt
      0.7 ms  FastPower → FastPowerForwardDiffExt
      0.7 ms  RecursiveArrayTools → RecursiveArrayToolsForwardDiffExt
      0.9 ms  EnumX
      0.8 ms  ConcreteStructs
      0.9 ms  FastClosures
      1.1 ms  PreallocationTools
      0.8 ms  FunctionWrappersWrappers
      0.9 ms  SciMLStructures
               ┌ 0.0 ms Distributed.__init__()
      9.3 ms  Distributed
      0.8 ms  CommonSolve
      3.1 ms  ADTypes
      1.2 ms  ADTypes → ADTypesConstructionBaseExt
     75.6 ms  MLStyle 20.54% compilation time
      4.6 ms  Expronicon
      8.7 ms  SciMLOperators
      1.0 ms  SciMLOperators → SciMLOperatorsStaticArraysCoreExt
               ┌ 0.0 ms SciMLBase.__init__()
     76.9 ms  SciMLBase
      5.1 ms  DiffEqBase
     38.0 ms  FillArrays
      1.3 ms  FillArrays → FillArraysStatisticsExt
      0.9 ms  SimpleUnPack
     21.6 ms  DataStructures
     31.1 ms  OrdinaryDiffEqCore
     39.0 ms  OrdinaryDiffEqTsit5
=#

This shows that ForwardDiff and its dependencies are a large part of the startup time, and that comes through DiffEqBase. With the DifferentiationInterface changes we will hopefully no longer default to ForwardDiff anyways and instead with Enzyme, and so we should make the stack not require ForwardDiff unless the user/solver wants it.

[ChrisRackauckas]

if isdefined(PreallocationTools, :FixedSizeDiffCache)
    function anyeltypedual(x::PreallocationTools.FixedSizeDiffCache,
            ::Type{Val{counter}} = Val{0}) where {counter}
        Any
    end
end

need to find out what to do with that.