Implementation of Iterative ProductMode()

[LukasBarner]

As discussed on discourse, an iterative solution strategy for ProductMode() would be nice.
I have given this a first (and probably naive) shot.
Changes include:

• added dependency Printf to format output strings in the iterative log
• added a vector of regularization parameters to ProductMode to be iteratively decreased for each solve
• added a vector of constraint references pointing to the complementarity constraints
• added an iterative solution procedure in the optimize!() function, including logging.
  I would have preferred to do the iterative procedure in place (i.e. without passing a copy for each iteration). The commented code would implement this. However, copying seems more appropriate in this case, as in place modification does not work.
  Changes are small, so no old functionality should break.

I would be happy about a brief discussion, especially in case anyone sees a more elegant approach.
Warnings/errors are not yet implemented, also not thoroughly tested on my end. This is more to test the waters and not a final approach :D
It's my first PR, so I would be happy about some feedback!

[LukasBarner]

I edited a bit of the code to make it more concise and a little more performant.

For practical usage, it is important to set the right solver attributes before running the model. I guess this can hardly be done inside BilevelJuMP ex-ante, as code should work independent of the solver.
For example, in Ipopt the following settings should be made:

set_optimizer_attribute(model, "warm_start_init_point", "yes")
set_optimizer_attribute(model, "warm_start_bound_push", 1e-12)
set_optimizer_attribute(model, "warm_start_bound_frac", 1e-12)
set_optimizer_attribute(model, "warm_start_slack_bound_frac", 1e-12)
set_optimizer_attribute(model, "warm_start_slack_bound_push", 1e-12)
set_optimizer_attribute(model, "warm_start_mult_bound_push", 1e-12)

Additionally, not printing the solver log gives a higher level overview.
set_optimizer_attribute(model, "print_level", 0)

In the future, some of the warmstart related settings may be included automatically for popular solvers, but I think leaving this to the user should be sufficient for now :D

[LukasBarner]

Hey @joaquimg,
I have now also implemented a default solution strategy without iteratively using copy_to (this seems to be considerably better from a performance point of view).
I have tested my code for bigger problems on a cluster computer, and it allowed me to solve problems that were not solvable with sufficiently low regularization via the one-shot strategy.
There are still a few points that I'm not really happy with (for example the comp_idxs_in_solver (returned by get_solver_comp_idxs), but after some testing I believe the code works as intended and doesn't break anything existing.
In case you are testing with Ipopt, there are still a few problems though:

• The copy_to way still has some issues in Ipopt and MOI, see copy_to does not copy attributes for variable bound constraints jump-dev/Ipopt.jl#333, [Bridges] fix supports for VariableIndex attributes jump-dev/MathOptInterface.jl#1991, and I believe that Fix ConstraintDualStart with VariableIndex jump-dev/Ipopt.jl#334 breaks even the old implementation of BilevelJuMP.
• Ipopt is missing MOI.set(model::Optimizer, ::MOI.ConstraintSet, ci::MOI.ConstraintIndex{F,S}, set::S,) where {F<:_FUNCTIONS,S<:_SETS}. I am not sure if this may not be on purpose, and my way of doing the _iterative_optimize_solver! is a really dumb idea, but I have opened a PR with Ipopt.jl to discuss this (add functionality to set constraint RHS jump-dev/Ipopt.jl#336).

[LukasBarner]

Think this should be good to go for now.
@joaquimg, could you give some feedback?

[joaquimg]

Will look into it!

[joaquimg]

I like where it is going.
Can you fix the JuMP style?
I will make changes following that

[joaquimg]

in many functions and variable names this PR does nof follow the JuMP style guide: https://jump.dev/JuMP.jl/stable/developers/style/

[LukasBarner]

I'm still fairly inexperienced, so please excuse the question. Is this mostly about naming functions/variables like SetIterPrimalStarts! or IterEps?
Or do I need to consider anything else?

[LukasBarner]

OK, I have now taken care of function and variable names as well as code formatting.

[LukasBarner]

Now includes tests and a mutable log as well. Note that I had to update Ipopt compat (because it did not have methods to alter the rhs prior to jump-dev/Ipopt.jl#336).

[joaquimg]

Formatter was added to master, so you can just run that and it will fix part of the style.

[LukasBarner]

can you check if this is better?

[LukasBarner]

Just realized that the following line is problematic if there exist different types of complements (greater/less) within the same model:
https://github.com/LukasBarner/BilevelJuMP.jl/blob/88561d99bb8045feef1926bf8f86b9733d7acdb7/src/jump.jl#L1060
To keep everything as easy as possible, I flipped the second complementarity constraint for conic problems.
In case this is not a solution you prefer, I could also do something else (like a conditional, or something more type-stable).

Further, the previous attempt did not work correctly with slacks, which is also fixed now.

[LukasBarner]

@joaquimg is there anything I can do/fix atm?