Deep_merge for nornir (group + host)

One thing that nornir doesn’t does is “deep_merge”. i.e. I have in my group “madrid” the following:

interfaces:
  port-channel2:
    admin_state: up
    channel_id: 2
    description: kubw-201:bond0
    state: present
    suspend_individual: false
    swmode: access
    type: PROD
    spanning_tree: edge
    vlans:
      - 855
    vpc: 2

and in my host file:

interfaces:
  port-channel2:
    members:
      - name: Ethernet1/2
        description: kubw-201:enp94s0f0
        admin_state: up

In Ansible, it does deep_merge and creates a variable with the aggregate:

interfaces:
  port-channel2:
    admin_state: up
    channel_id: 2
    description: kubw-201:bond0
    state: present
    suspend_individual: false
    swmode: access
    type: PROD
    spanning_tree: edge
    vlans:
      - 855
    vpc: 2
    members:
      - name: Ethernet1/2
        description: kubw-201:enp94s0f0
        admin_state: up

But this deep merge is not implemented in Nornir. Here is my recipe (of course, it can be improved) to do it. It used the deep_merge library that can be installed with pip:

import deep_merge
 
def render_configs(task):
    filename = task.host["j2_template_file"]
    merge_config = {}
    for group in task.host.groups.refs:
        if os.path.isfile(f"group_vars/{group}.yaml"):
            group_config = task.run(
                task = load_yaml,
                name = "Load group yaml file",
                file = f"group_vars/{group}.yaml")
            merge_config = deep_merge.merge(merge_config,group_config.result)

    host_config = task.run(
        task = load_yaml,
        name = "Load host yaml file",
        file = f"host_vars/{task.host}.yaml")
    merge_config = deep_merge.merge(merge_config,host_config.result)
    task.host.data = merge_config
    r = task.run(
        task=template_file,
        name="Base Template Configuration",
        template=filename,
        path="templates",
        data=merge_config,
    )
    task.host["config"] = r.result

I hope this can help someone.

2 Likes

@mundofer Thanks for posting your solution/workaround to this.

The underlying item i.e. not doing a deep merge is by design in Nornir.

Thanks for the explanation. I hope that my solution can help the people that need deep merge, like those coming from Ansible.

Just out of curiosity, do you know what led to that design decision?

I don’t have an opinion one way or the other as to whether the deep-merge is a good idea, but perhaps it is worthy of an inventory initialization option that defaults to false?

IIRC–mostly related to keeping it simple (complexity causes a lot of issues/problems including maintenance problems, bugs, and documentation issues).

You can still do a deep-merge as seen above–you just have to write/copy the code to do it.

I definitely don’t want that behavior…I want to maintain the current behavior.

As Kirk said it was a matter of keeping it simple. This is a contentious feature, no matter if you choose one behavior or the other there is going to be people that will prefer the other options. In such cases, I usually implement the simplest thing possible :slight_smile:

For those that need this complexity of merging group and host objects, my original solution didn’t contemplate one specific case: When we have nested groups, i.e. group Europe includes groups Spain, Italy and Portugal and Spain contains routers Madridf1, Madrid2 (just an example, I’m sure you understand the hierarchy). For that case I’ve modified my original solution:

def recurse_merge(task,group_list,merge):
    for group in group_list:
        subgroup_list = group.groups.refs
        merge = recurse_merge(task,subgroup_list,merge)
    for group in group_list:
        if os.path.isfile(f"group_vars/{group}.yaml"):
            group_config = task.run(
                task = load_yaml,
                name = "Load group yaml file",
                file = f"group_vars/{group}.yaml")
            print("merging " + f"group_vars/{group}.yaml")
            merge = deep_merge.merge(merge,group_config.result)
    return merge


def render_configs(task):
    filename = task.host["j2_template_file"]
    merge_config = {}
    merge_config = recurse_merge(task,task.host.groups.refs,merge_config)
    host_config = task.run(
        task = load_yaml,
        name = "Load host yaml file",
        file = f"host_vars/{task.host}.yaml")
    merge_config = deep_merge.merge(merge_config,host_config.result)
    task.host.data = merge_config
    r = task.run(
        task=template_file,
        name="Base Template Configuration",
        template=filename,
        path="templates",
        data=merge_config,
    )
    task.host["config"] = r.result

I hope this can help someone