PoC: BIOS version & settings management #138

aobort · 2024-10-03T09:46:30Z

Proposed Changes

This PR contains PoC implementation for management of server's BIOS version and settings:

API types
Controller
Service managing tasks (scan, apply settings, update version)

This approach fully separate concrete job implementation from the reconciliation flow.

To discuss

BootOrder

Do we need to move boot order field from server to serverBIOS CR? Semantically, yes. As it's one of BIOS settings.

BIOS settings in `.status`

From my perspective, it reasonable to reflect in .status.bios.settings only those settings which are set in .spec.bios.settings. This will make comparison of .spec and .status much easier.

Storing bios settings

Do we need custom type for bios setting:

type SettingsEntry struct {
    // +required
    Name string `json:"name"`
    
    // +required
    Value string `json:"value"`
        
    // +optional
    Unsupported bool `json:"unsupported,omitempty"`
}

to avoid attempting to apply settings which are not supported in the specified BIOS version? I.e.:

apiVersion: metal.ironcore.dev/v1alpha1
kind: ServerBIOS
metadata:
  name: foo
spec:
  scanPeriodMinutes: 30
  serverRef:
    name: bar
  bios:
    # changing both version and 'unsupported' flag in the same time
    version: 1.0.0  --> 2.0.0
    settings:
    - name: legacyboot
      value: enabled
      unsupported: false  --> true

upgrading from version 1.0.0 to 2.0.0 will lead to the bios setting responsible for legacy boot is deprecated/unsupported, thus it is explicitly marked as unsupported and will not be considered during settings applying.

internal/controller/server_controller.go

internal/job/bios_bmc.go

stefanhipfel · 2024-11-20T09:11:35Z

fmi/server_http.go

+func (s *ServerHTTP) registerRoutes() {
+	s.mux.HandleFunc("/scan", s.scanHandler)
+	s.mux.HandleFunc("/settings-apply", s.settingsApplyHandler)
+	s.mux.HandleFunc("/version-update", s.versionUpdateHandler)


idea: would be great to have an endpoint to get all current active tasks from the sync.Map

@stefanhipfel thanks for idea.
Totally agree. Also we might need to implement queues for requests, rate limiting, retrying on client side and tons of other stuff. Apart from that, I'd also like to generate http API from spec without hardcoding endpoints.

But for now, I think the main goal is to come to agreement - will we proceed further with this design or not.

stefanhipfel · 2024-11-20T09:54:44Z

internal/controller/serverbios_controller.go

+	// if referred server is not in Available state - stop reconciliation
+	if server.Status.State != metalv1alpha1.ServerStateAvailable {
+		return ctrl.Result{RequeueAfter: r.RequeueInterval}, nil
+	}


do we need to stop the server reconciliation in the meantime?
e.g.: do not allow serverClaim

No, we do not need to stop server's reconciliation - we do not care what is the state of the server. But we need to stop serverBIOS reconciliation, bc both BIOS version and settings update will lead to server reboot. So for now we decided that we'll work only with servers which are in "Available" state.

If the server is in available state, then we'll need to set it to "maintenance". But the servers' maintenance topic is still open. Thus I decide to not to mention it at all for now.

#76

ok but during a serverBios update, someone could still claim the server. Any available server can be claimed

ok but during a serverBios update, someone could still claim the server. Any available server can be claimed

@stefanhipfel that's a good point. However, I do not really like the idea that the serverBIOS controller would change server's state to exclude it from reconciliation. For number of reasons the server's "owner" might want to postpone the update. Especially in terms of version upgrade.
I could add the "Maintenance" state for server and update the controller, so it will check whether server's state is "Maintenance" instead of "Available".

@aobort yes i think so as well, that the serverBIOS should not change a server's state.

Alternative idea: the server reconciler checks on the serverBIOS state.

stefanhipfel · 2024-11-20T11:04:09Z

overall I think the POC looks ok

stefanhipfel · 2024-11-20T10:03:59Z

internal/controller/serverbios_controller.go

+	// if referred server is not in Available state - stop reconciliation
+	if server.Status.State != metalv1alpha1.ServerStateAvailable {
+		return ctrl.Result{RequeueAfter: r.RequeueInterval}, nil
+	}


ok but during a serverBios update, someone could still claim the server. Any available server can be claimed

stefanhipfel · 2024-11-20T11:17:18Z

api/v1alpha1/serverbios_types.go

+type ServerBIOSStatus struct {
+	// BIOS contains a bios version and settings.
+	// +optional
+	BIOS BIOSSettings `json:"bios,omitempty"`


we could add the current state of the bios task

@stefanhipfel Taking into account:

possible separating of metal-operator and bios/fw operator

possible separation of the bios/fw operator and task runner

I'd suggest to proceed as is for now. Then we could add some endpoint to retrieve active tasks for concrete server or something like this. But it will require to store the references for running tasks etc. So I'd avoid adding it right now due to necessity to design this better.

stefanhipfel · 2024-11-20T14:40:51Z

@aobort overall I think the POC looks good.

Next step would be to gather some improvements and test it with real examples!

stefanhipfel · 2024-12-04T12:45:49Z

@defo89 @aobort @afritzler what do u guys think about taking the bios controller out of the metal operator, same as the boot/ipam operator? We would be more flexible in the future and changes could be implemented independently of the metal operator

aobort · 2024-12-05T05:26:44Z

@defo89 @aobort @afritzler what do u guys think about taking the bios controller out of the metal operator, same as the boot/ipam operator? We would be more flexible in the future and changes could be implemented independently of the metal operator

I proposed it in the beginning of this "epic". Thus, I agree.

Signed-off-by: Artem Bortnikov <[email protected]>

aobort · 2024-12-06T12:16:01Z

@stefanhipfel I've updated the controller: now it checks whether server is in "Maintenance" state. After #76 solved, we'll update server bios controller respectively. Adding the task state into the serverBIOS object's status is prematurely, from my perspective. Let's first run the stuff on real hardware and look how the "instant" operations (like scan and settings apply) work. Also added initial test for BIOS versions retrieval.
I'd say it's ready to be merged.

cc: @afritzler @defo89

Signed-off-by: Artem Bortnikov <[email protected]>

github-actions bot added api-change size/XL labels Oct 3, 2024

aobort force-pushed the poc-firmware-controllers branch from 2ac16c3 to e663a97 Compare October 4, 2024 07:23

github-actions bot added size/XXL and removed size/XL labels Oct 4, 2024

stefanhipfel reviewed Oct 4, 2024

View reviewed changes

internal/controller/server_controller.go Outdated Show resolved Hide resolved

aobort force-pushed the poc-firmware-controllers branch from a6733e4 to b87e059 Compare October 7, 2024 07:18

aobort requested a review from stefanhipfel October 9, 2024 09:20

stefanhipfel reviewed Oct 10, 2024

View reviewed changes

internal/job/bios_bmc.go Outdated Show resolved Hide resolved

stefanhipfel reviewed Oct 10, 2024

View reviewed changes

internal/job/bios_bmc.go Outdated Show resolved Hide resolved

aobort requested a review from stefanhipfel October 11, 2024 07:25

aobort force-pushed the poc-firmware-controllers branch 4 times, most recently from 0aea100 to 958928c Compare November 4, 2024 10:52

github-actions bot added the documentation Improvements or additions to documentation label Nov 7, 2024

aobort force-pushed the poc-firmware-controllers branch 2 times, most recently from b682a57 to 3468011 Compare November 11, 2024 06:26

stefanhipfel reviewed Nov 20, 2024

View reviewed changes

stefanhipfel requested a review from gehoern November 20, 2024 14:41

aobort force-pushed the poc-firmware-controllers branch 2 times, most recently from d06686f to b02ff6e Compare December 5, 2024 08:36

aobort marked this pull request as ready for review December 5, 2024 09:14

aobort force-pushed the poc-firmware-controllers branch from b02ff6e to ed1a9fc Compare December 6, 2024 10:44

aobort added 11 commits December 6, 2024 12:47

serverbios api types and controller

14b4dc6

Signed-off-by: Artem Bortnikov <[email protected]>

naive bios management job implementation

1c25720

Signed-off-by: Artem Bortnikov <[email protected]>

task executor interface

3f9f8fb

Signed-off-by: Artem Bortnikov <[email protected]>

server and client

ad1791f

Signed-off-by: Artem Bortnikov <[email protected]>

separate server and client

0fc8bd2

Signed-off-by: Artem Bortnikov <[email protected]>

local fmi server option

9051120

Signed-off-by: Artem Bortnikov <[email protected]>

cleanup after rebase

33a7b81

Signed-off-by: Artem Bortnikov <[email protected]>

replace available state check with maintenance state check

fa08cd4

Signed-off-by: Artem Bortnikov <[email protected]>

add license

6a019af

Signed-off-by: Artem Bortnikov <[email protected]>

server bios controller tests

90f18c2

Signed-off-by: Artem Bortnikov <[email protected]>

update go.mod

971fffa

Signed-off-by: Artem Bortnikov <[email protected]>

aobort force-pushed the poc-firmware-controllers branch from ed1a9fc to 971fffa Compare December 6, 2024 10:52

aobort added 2 commits December 7, 2024 08:35

embed task runner interface into server interface

2a7c4fe

Signed-off-by: Artem Bortnikov <[email protected]>

rework comparison of desired and observed bios settings

145651e

Signed-off-by: Artem Bortnikov <[email protected]>

aobort closed this Dec 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PoC: BIOS version & settings management #138

PoC: BIOS version & settings management #138

aobort commented Oct 3, 2024 •

edited

Loading

stefanhipfel Nov 20, 2024

aobort Nov 20, 2024

stefanhipfel Nov 20, 2024 •

edited

Loading

aobort Nov 20, 2024 •

edited

Loading

stefanhipfel Nov 20, 2024

aobort Nov 20, 2024

stefanhipfel Nov 20, 2024

stefanhipfel commented Nov 20, 2024

stefanhipfel Nov 20, 2024

stefanhipfel Nov 20, 2024

aobort Dec 5, 2024

stefanhipfel commented Nov 20, 2024

stefanhipfel commented Dec 4, 2024 •

edited

Loading

aobort commented Dec 5, 2024

aobort commented Dec 6, 2024

PoC: BIOS version & settings management #138

PoC: BIOS version & settings management #138

Conversation

aobort commented Oct 3, 2024 • edited Loading

Proposed Changes

To discuss

BootOrder

BIOS settings in .status

Storing bios settings

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanhipfel Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

aobort Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanhipfel commented Nov 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanhipfel commented Nov 20, 2024

stefanhipfel commented Dec 4, 2024 • edited Loading

aobort commented Dec 5, 2024

aobort commented Dec 6, 2024

aobort commented Oct 3, 2024 •

edited

Loading

BIOS settings in `.status`

stefanhipfel Nov 20, 2024 •

edited

Loading

aobort Nov 20, 2024 •

edited

Loading

stefanhipfel commented Dec 4, 2024 •

edited

Loading