Univariate user-defined functions with the multivariate signature

[odow]

See https://discourse.julialang.org/t/nested-optimization-matrix-multiplication-and-internal-functions/118098/10?u=odow

[odow]

This is needed for my work in https://github.com/lanl-ansi/MathOptAI.jl/issues/90

[odow]

An alternative would be to allow univariate functions to not support second-order derivatives.

[odow]

As a motivating example, we have this working:

    filename = joinpath(dir, "model_Tanh_vector_GrayBox.pt")
    PythonCall.pyexec(
        """
        import torch

        model = torch.nn.Sequential(
            torch.nn.Linear(3, 16),
            torch.nn.Tanh(),
            torch.nn.Linear(16, 2),
        )

        torch.save(model, filename)
        """,
        @__MODULE__,
        (; filename = filename),
    )
    # Full-space
    model = Model(Ipopt.Optimizer)
    set_silent(model)
    @variable(model, x[1:3])
    ml_model = MathOptAI.PytorchModel(filename)
    y = MathOptAI.add_predictor(model, ml_model, x; gray_box = true)

where the PyTorch model is loaded as a user-defined function with automatic memoization for the vector-valued output and we use torch to compute the Jacobian via PythonCall.

But...it doesn't work if the network has a scalar input.

[odow]

I realized that this is no longer trivial, because we've pushed the opt-in part of this into the solvers:
https://github.com/jump-dev/Ipopt.jl/blob/4c156461ef1fda3c9f015520197afda4e8ca3e26/src/MOI_wrapper.jl#L606-L611

So perhaps we just need a way for univariates to not supply second derivatives.