Initialising optimisers with constant parameters.

[axsk]

I cannot find a way to set constant optimisers parameters together with Scheduler.
For example, I would like to setup a AdamW optimiser with exponentially decaying learning rate but also prescribe a constant decay of 1e-2.

I would expect this to work but obtain an error

julia> Optimisers.setup(Scheduler(AdamW, η=Exp(1e-3, 0.99), λ=1e-3), model)

ERROR: MethodError: objects of type Float64 are not callable
Maybe you forgot to use an operator such as *, ^, %, / etc. ?
Stacktrace:
  [1] (::ParameterSchedulers.var"#40#41"{Int64})(s::Float64)
    @ ParameterSchedulers ./none:0
  [2] iterate
    @ ./generator.jl:47 [inlined]
  [3] collect_to!
    @ ./array.jl:892 [inlined]
  [4] collect_to_with_first!
    @ ./array.jl:870 [inlined]
  [5] collect(itr::Base.Generator{@NamedTuple{η::Exp{Float64}, λ::Float64}, ParameterSchedulers.var"#40#41"{Int64}})
    @ Base ./array.jl:844
  [6] _totuple
    @ ./tuple.jl:425 [inlined]
  [7] Tuple
    @ ./tuple.jl:391 [inlined]
  [8] NamedTuple
    @ ./namedtuple.jl:149 [inlined]
  [9] _get_opt(scheduler::Scheduler{@NamedTuple{η::Exp{Float64}, λ::Float64}, typeof(AdamW)}, t::Int64)
    @ ParameterSchedulers ~/.julia/packages/ParameterSchedulers/ebjgq/src/scheduler.jl:46
 [10] init(o::Scheduler{@NamedTuple{η::Exp{Float64}, λ::Float64}, typeof(AdamW)}, x::Matrix{Float32})
    @ ParameterSchedulers ~/.julia/packages/ParameterSchedulers/ebjgq/src/scheduler.jl:51
 [11] #_setup#3
    @ ~/.julia/packages/Optimisers/ywGX8/src/interface.jl:40 [inlined]

I also tried wrapping λ=ParameterSchedulers.Constant(1e-3))
resulting in another error

ERROR: MethodError: Cannot `convert` an object of type Float64 to an object of type Tuple{Float64, Float64}

Closest candidates are:
  convert(::Type{T}, ::T) where T<:Tuple
   @ Base essentials.jl:456
  convert(::Type{T}, ::PyCall.PyObject) where T<:Tuple
   @ PyCall ~/.julia/packages/PyCall/1gn3u/src/conversions.jl:218
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:84
  ...

Stacktrace:
  [1] Adam(eta::Float64, beta::Float64, epsilon::Float64)
    @ Optimisers ~/.julia/packages/Optimisers/ywGX8/src/interface.jl:268

Calling λ decay or leaving out the arguments names completely also did not help.

[axsk]

So I think this works using the parameter names eta and lambda and patching

ParameterSchedulers.jl/src/scheduler.jl

Line 48 in 7b3c081

return scheduler.constructor(kwargs...)

to return constructor(;kwargs...)...

[axsk]

I'd open a PR for this but am unsure whether this is not breaking stuff.
Also, while at it, do we want to allow passing in numbers, i.e. lambda=1e-3 instead of lambda=x->1e-3 or having to use Constant?
Seems that could be done easily by modifying the line above.

[darsnack]

patching

ParameterSchedulers.jl/src/scheduler.jl

Line 48 in 7b3c081

return scheduler.constructor(kwargs...)

to return constructor(;kwargs...)...

This is a bug and can be fixed in a patch.

Also, while at it, do we want to allow passing in numbers, i.e. lambda=1e-3 instead of lambda=x->1e-3 or having to use Constant?

Yeah we can replace numbers with Constant similar to what is done for Sequence.

This is also not breaking and can be released in a patch.

Feel free to submit a PR!