Netmiko: OS Upgrade Process Automation

[ktbyers]

Starting Step:

Create an abstract process for IOS/IOS-XE, IOS-XR, NX-OS, JunOS, and Arista.

Basically a series of steps of what an OS upgrade for each of these platforms would consist of. Obviously, there will be a lot of variations here depending on what the hardware is.

High level questions:

1. Is there an auto-detection mechanism(s)? For hardware, for inventory characteristics.

1. What is the control channel? I am assuming that SSH will be used as the general purpose control channel (at least initially). What part of this can be accomplished via APIs?

2. What is the transfer mechanism? I am assuming SCP as the standard transfer mechanism. Does there need to be others. If they are others, I would have a pretty strong preference to have them be initiated from the network device (i.e. they are things that just are executed from the SSH control channel).

   1. Pre-transfer requirements (i.e. gathering information pre-transfer). Space available, current config, current image, boot settings.

3. Post-transfer requirements. MD5 or checksum on file. Is this required/optional, what should the default be?

4. Mechanics of specifying image to be used.

5. Reload mechanics. Default will definitely be off here.

6. Post-transfer requirements (i.e. information gathered post-transfer) and what is done with the information. Are all modules/interfaces still present?

7. High-availability contexts and what are their requirements. Other platform specific idiosyncrasies.

8. Any other caveats like ROM requirements, sanity checking if ROM is supported, memory requirements.

Not saying any of these will/should be included, but just specifying some of the issues.

[lampwins]

One the biggest issues I run into on Junos is space. Either running out of space on the upload or during unpacking/verifying.

I am sure that issue must exist on other platforms so a necessary step would be checking free space and doing remediation if needed/wanted.

[jabelk]

1. Establish standard to be compared against for a given hardware platform, supervisor, and function
2. Gather facts from device (either through SNMP or textfsm/regex parsing), which are, hardware platform, supervisor information, and function of device (assumed from metadata, likely the naming convention of the hostname)
3. Compare facts to the standard and decide if device requires an upgrade or not.
4. If device requires an upgrade, copy the relevant file(s) from a fileserver, change boot statements, removing all but the current running and the new one, check to see if space on primary, and other storage locations, and copy the file if proper space, otherwise clean up space.
5. Verify md5 hash from step 1 standards, of copied file(s)
6. Mark device as ready for reload and service impacting portion
7. When change window is ready, capture a list of show commands into textfsm outputs (pre-upgrade state)
8. Reload machine (if multiple supervisor, reload all of them)
9. Send post-reload commands, note any differences between pre/post, possibly flagging certain commands as more critical than others
10. Verify running new image and mark device as now in standard.

[ktbyers]

From the top of my head, this is similar to what I tested for small IOS and ASA devices. Note, I didn't have any auto detection of the nature of the remote device.

1. Determine size and MD5 of file on local file system.
2. SSH into remote device and verify sufficient space available on the file system.
3. Check if device already has the new image on the file system.
4. Check if the device is already running the new image.
   5a. Archive current device configuration
   5b. Transfer new image to device using SCP
5. Probably will want to make some SCP optimizations on some Cisco devices as file transfers by default are very slow.
6. Verify MD5 of image now on the device.
7. Set boot image in config.

8. Verify boot variable, if appropriate.

   10. Reload.

9. Operational verification of device.

10. Flag any configuration changes
11. Verify device is actually running the new image.

[ktbyers]

@lampwins Since I don't know the Junos OS upgrade process...it would definitely be helpful if you are someone else could sketch out what you would typically do during a manual OS upgrade.

[johnarnold]

@ktbyers for perspective, we use netmiko for device interaction, but there's a higher level device driver. we built code upgrade logic into the layer above netmiko. Some things, like the source of the file and md5 hash are going to be pretty environment specific. Also, regular reload vs fast reload is worth thinking about.