Sharding nornir inventories

I’m looking for thoughts/recommendations on how to partition a large inventory into smaller inventory objects

I have a set of grouped tasks (eg connect, then check ACLs, then check SNMP etc etc) and a large number of hosts, so many that some of my hosts are timing out while doing the first task of connecting to all of them. (obviously I could refactor how my tests/tasks are organized but what fun is that?)

Let’s say I have 10k hosts in one inventory object, how can I run the whole series of grouped tasks on inventory objects consisting of 128 hosts at a time until they’re all done?

Thanks!

Based on your description, I am assuming that you have an explicit task to connect to the devices and then the task that is timing out is the first one after you have manually connected to each one of them. If you want to execute all the tasks on a single device at a time you can group them in a grouped task.

The following doc is a bit outdated but it describes how to do what you want:

https://nornir.readthedocs.io/en/latest/ref/internals/execution_model.html

What I’ve done is to write a set of grouped tasks with sub-tasks that are all logically related:

For example:

  • check_ntp

    • check timezone
    • check sync
    • check acl
  • check_snmp

    • check communities
    • check acl
    • check servers
  • check_tacacs

    • check servers
    • check key

I had not done any manual connection setup.

What I’m discovering is that after the connection is set up (in check_ntp this example) by the time that grouped task is complete and testing moves on to the next grouped task (check_snmp) the connection may have been dropped by the device due to idle timeouts and check_snmp isn’t aware of this and does not attempt to re-establish the connection so everything fails after that point.

My work-around has been to move all of my live testing to the first grouped task so anything that interacts with the device all happens as quickly as possible

However, I thought that perhaps if there was an easy way to split an inventory up into smaller chunks that I could avoid the work around of reorganizing my tests and also potentially easily offload the testing of each chunk to other host machines

an easy way to split an inventory up into smaller chunks

The nr.filter method is your friend then :slight_smile: If you don’t have an easy way to group them you could use a filter_func and a global set or list to keep track of which ones you have processed already:

https://nornir.readthedocs.io/en/latest/tutorials/intro/inventory.html#Filter-functions

Also, grouped tasks can have other grouped tasks so you could easily nest your current ones in a larger one.

I am using the filter functions for some other things (eg by device type. Thanks, by the way, they work great! )

I was just hoping to somehow be able to treat an inventory object like a list or deque

eg

# Get the first 10 hosts
sub_inventory_a = main_inventory.filter(F(host_index=[0,10]))

or

# Get the next 10 hosts
sub_inventory_b = main_inventory[10,20]

etc etc

If there is not a good built-in way to do this I’ll just see what I can rig up

You could achieve that by adding an index yourself to the host (i.e. via a transform_function) and then using the filter_func to select the range.

Thank you, I’ll dig into that and see what I can work out

Regarding the transform_function:

Is it only called on inventory initialization?

For instance, if I do something along the lines of:

index_value = 0

def add_an_index(host):
    # This function receives a Host object for manipulation
    global index_value
    host.index = index_value
    index_value += 1 

# Create the overall inventory
nr: Nornir = InitNornir( "transform_function": add_an_index,)

# Filter down to IOS devices
ios_only_inventory = nr.filter(F(groups__contains="cisco_ios")

# Filter IOS devices down to 2900s
2900_only = ios_only_inventory .filter(F(device__contains="2900")

Then the inventory “2900_only”, which is a filter of a filter, is still going to retain the original index values rather than re-calculating the index each time the inventory is further filtered down?

If so, the indexes will have gaps so another filter like:

first_hundred_2900 = 2900_only.filter(F(index___any=[range(0,100)]))

May actually only return less than 100 devices depending on how many devices were removed by previous filters?

That’s correct. Instead of the manual indexing, another option would be the following:

# we do some prefiltering if needed
some = nr.filter(F(...))

# we create a list of the hosts in the nornir object we want to break down into chunks
hosts_in_some = list(some.inventory.hosts.keys())

# we use that list to narrow down the nornir object
first_hundred = some.filter(F(hostname__in=hosts_in_some[0:100]))
second_hundred = some.filter(F(hostname__in=hosts_in_some[101:200]))
...

I think this should do the trick with almost no added complexity to your original code. You should be able to even create a wrapper to do this for you automatically.

Interesting, thanks for the great idea! I’ll work on this for a bit and share what I come up with

Here’s what I came up with, though I haven’t yet reworked my tests to try it out

def chunks(list, chunk_size) -> list:
    """Yield successive n-sized chunks from l."""
    for i in range(0, len(list), chunk_size):
        yield list[i:i + chunk_size]

# Create a list of the hosts in the nornir object we want to break down into slices
inventory_to_slice = filtered_inventory_all_nxos
host_list = list(inventory_to_slice.inventory.hosts.keys())

# Work through the list of hosts in slices
slice_size = 5
for host_list_slice in chunks(host_list, slice_size):
    inventory_slice = inventory_to_slice.filter(F(hostname__in=host_list_slice))
    print(inventory_slice.inventory.hosts.keys())
1 Like