[21.05] zebra restart tuning and keepalived integration

[sysvinit]

This change raises the startup backoff limits for zebra. The system default restart settings allow zebra to restart up to five time in 60 seconds, however in some circumstances where zebra has crashed due to the system being overloaded, it may take longer for the conditions causing zebra to crash to clear, so this change raises the limit to 20 times in 120 seconds.

Additionally, when zebra crashes on the primary router in a location, this should ideally cause the router to demote itself and let the secondary take over. This change adds a new check script to the keepalived configuration in all locations to monitor the liveness of zebra.

PL-132950

@flyingcircusio/release-managers

Release process

Impact: internal.

Changelog: none.

PR release workflow (internal)

• PR has internal ticket
• internal issue ID (PL-…) part of branch name
• internal issue ID mentioned in PR description text
• ticket is on Platform agile board
• ticket state set to Pull request ready
• if ticket is more urgent than within the next few days, directly contact a member of the Platform team

Design notes

• Provide a feature toggle if the change might need to be adjusted/reverted quickly depending on context. Consider whether the default should be on or off. Example: rate limiting.
• All customer-facing features and (NixOS) options need to be discoverable from documentation. Add or update relevant documentation such that hosted and guided customers can understand it as well.

Security implications

• Security requirements defined? (WHERE)
  • Improve site reliability and service availability with better integration between FRR and keepalived.
• Security requirements tested? (EVIDENCE)
  • Load test in DEV used to artificially cause zebra crashes and test keepalived demotion behaviour.

[ctheune]

Generally fine, little adjustment: add a 10 sec restartsec option (and increase the startlimitinterval accordingly to 320s)