Support for internet gateways

[rcgoodfellow]

• fixes Implement the Internet Gateway concept #2154
• depends on Support for omicron internet gateway model opte#588

[rcgoodfellow]

Something I'm not too sure about with this PR is how it needs to interact with the reconfigurator. If any services move around, we need to trigger the vpc_route_manager background task. @jgallagher do you have some insight here?

[rcgoodfellow]

Calling this comment out as something to discuss in this PR. CC @FelixMcFelix

[FelixMcFelix]

Yeah, I had maybe overlooked the scope of things that could change in connection with an Internet Gateway. That's a big surface area to keep an eye on...

I'm not really sure how we would want to work around this, however. The motivation for using router versions was more for performance rather than correctness. My intent was to prevent each Nexus's copy of the background task from querying every Project's/VPC's/Subnet's state on a minute-by-minute (and per-wakeup) basis, since there are a lot of tables which are referenced during route resolution. This partly stems from the granularity of background task wakeups, which don't AFAIK allow a selective wakeup. So my initial feeling is that having Nexus store resolved tables and then send deltas is orthogonal to that?

It could also, in practice, not be too expensive to just rebuild VPC routing tables periodically! In which case, the places where we perform version bumps and/or background task wakeups would keep the system responsive rather than be crucial for correctness. If cost remains a concern, I wonder if we can just use modified timestamps in connected tables to drive the route update process (which would, I hope, be less error-prone).

[jgallagher]

Something I'm not too sure about with this PR is how it needs to interact with the reconfigurator. If any services move around, we need to trigger the vpc_route_manager background task. @jgallagher do you have some insight here?

Maybe! The bit of reconfigurator that can actually move services around is also an RPW (blueprint_executor). However, it doesn't currently know whether any given execution actually moved services. It's structured as "idempotently send every sled the set of services it should be running", and almost always that does not result in any changes. So I think I have some questions and some options:

• Does the vpc_route_manager already run periodically, so we're only looking for an optimization to tighten the window between "services moved around" and "VPC route manager fixed things"?
• If so, would it make sense to trigger vpc_route_manager after every blueprint_executor run? That would effectively cause it to run at the period of blueprint execution + its own timer, I think; I don't know whether that would be okay.
• I think blueprint_executor could know whether any sleds actually changed, if we modified the sled-agent API it's using to return information about whether it made any changes, at which point it could explicitly trigger vpc_route_manager only if services moved (or maybe only if certain services moved?). There might be wrinkles here I'm not aware of that make this tricky; should we look into this?