Hello,
I have a little problem with Nornir scripts and I hope you can help me out.
I’m using Nornir to deploy configurations on a couple of Cisco NCS Router. Everything works fine except sometimes one router fails with the error message “raise ValueError(“Unable to find prompt: {}”.format(prompt))”.
Except from the python trace I don’t see any indication what happened and caused this problem.
What would be the best approach to solve this problem?
Thanks
Dirk
Could you share your code that cause this issue?
Sure. The function deploy_evpn is called from a separate script and use the task create_evpn to deploy a config.
def create_evpn(task, ecx_vpn, ccon_data, basedir):
# function to create and EVPN instance; meant to run on role "evpn_pe" and platform "iosxr"
# Transform data to configuration via a template file
r = task.run(task=text.template_file,
name="Create EVPN config from template file",
template="create-evpn.j2",
path=f"{basedir}/jinja-templates/{task.host.platform}",
ccd=ccon_data,
ecx_vpn = ecx_vpn)
# Save compiled configuration into a host variable
task.host["evpn_config"] = r.result
# Deploy the configuration to the device using NAPALM
task.run(task=networking.napalm_configure,
name="Loading Configuration on the device",
replace=False,
configuration=task.host["evpn_config"])
def deploy_evpn(basedir, ecx_vpn, ccon_data, my_dry_run):
logging.info("deploy_evpn dry_run [{}]".format(my_dry_run))
nr = InitNornir(
core={"num_workers": 20},
inventory={
"plugin": "nornir.plugins.inventory.simple.SimpleInventory",
"options": {
"host_file": "{}/nornir-inventory/hosts.yaml".format(basedir),
"group_file": "{}/nornir-inventory/groups.yaml".format(basedir),
"defaults_file": "{}/nornir-inventory/defaults.yaml".format(basedir)
}
},
logging={"enabled": False},
dry_run=my_dry_run
)
evpn_pe_hosts = nr.filter(F(groups__any=ecx_vpn['locations']) & F(role="evpn_pe") & F(platform="iosxr"))
if ccon_data['action'] == 'create':
logging.info("Playbook to create a new EVPN instance")
task_result = evpn_pe_hosts.run(task=create_evpn, ecx_vpn=ecx_vpn, ccon_data=ccon_data, basedir=basedir)
my_print_result(task_result)
if task_result.failed is True:
logging.error("Something went wrong. Reset Nornir state, wait 30 seconds and try again")
evpn_pe_hosts.data.reset_failed_hosts()
time.sleep(30)
logging.info("Now we can continue...")
task_result2 = evpn_pe_hosts.run(task=create_evpn, ecx_vpn=ecx_vpn, ccon_data=ccon_data, basedir = basedir)
my_print_result(task_result2)
if task_result2.failed is True:
raise RuntimeError("EVPN configuration failed. Failed hosts [{}], processed hosts [{}]".format(", ".join(task_result2.failed_hosts.keys()), ", ".join(task_result2.keys())))
# Inventory
---
router-1:
hostname: 172.16.1.1
port: 22
platform: iosxr
connection_options:
napalm:
extras:
optional_args:
global_delay_factor: 2
groups:
- group_x
data:
site: site-1
role: evpn_pe
type: router
loopback_ip: 172.16.1.1
locations:
location_x:
- some_info
router-2:
hostname: 172.16.1.2
port: 22
platform: iosxr
connection_options:
napalm:
extras:
optional_args:
global_delay_factor: 2
groups:
- group_x
data:
site: site-2
role: evpn_pe
type: router
loopback_ip: 172.16.1.2
locations:
location_x:
- some_info
...
I already implemented a workaround. If the deployment fails the first time it will wait 30 seconds and try again.
So if I understand correctly, it is the function create_evpn that sometimes cause the error? Do you have the full stacktrace, and the output of the print_result of the task when the issue happens?
I do not have experience with napalm but looking at the issue, I am thinking about a timer issue. Could it be that pushing the configuration takes time on these devices and napalm times out? Have you tried to increase the timeout timer?
Also, if you use netmiko instead of napalm to push the configuration, do you have the same issue?
Hello,
thanks for looking into this. Yes, it is the function create_evpn causing the poblem. I also thought about a timeout. However, I didn’t find any indication or reason on that. When I tried again a few seconds later it was working fine. That’s why I implemented the workaround in the script and I set the global_delay_factor in the inventory. Based on the documentation I wasn’t sure if this has any effect but I tried it anyway. The problem is, that the error doesn’t happen every time. That makes it more difficult to troubleshoot. I never try using netmiko instead of napalm.
The config is deployed on Cisco NCS540 router running IOS-XR 6.6.3.
Here is a stactrace from when this problem occured:
Traceback (most recent call last):
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/nornir/core/task.py”, line 85, in start
r = self.task(self, **self.params)
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/nornir/plugins/tasks/networking/napalm_configure.py”, line 27, in napalm_configure
device = task.host.get_connection(“napalm”, task.nornir.config)
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/nornir/core/inventory.py”, line 302, in get_connection
extras=conn.extras,
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/nornir/core/inventory.py”, line 352, in open_connection
configuration=configuration,
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/nornir/plugins/connections/napalm.py”, line 51, in open
connection.open()
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/napalm/iosxr/iosxr.py”, line 80, in open
self.device.open()
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/pyIOSXR/iosxr.py”, line 166, in open
**self.netmiko_kwargs)
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/netmiko/ssh_dispatcher.py”, line 246, in ConnectHandler
return ConnectionClass(*args, **kwargs)
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/netmiko/base_connection.py”, line 317, in init
self._open()
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/netmiko/base_connection.py”, line 323, in _open
self._try_session_preparation()
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/netmiko/base_connection.py”, line 738, in _try_session_preparation
self.session_preparation()
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/netmiko/cisco/cisco_xr.py”, line 13, in session_preparation
self.set_base_prompt()
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/netmiko/base_connection.py”, line 1066, in set_base_prompt
prompt = self.find_prompt(delay_factor=delay_factor)
File “/home/users/ggcninetsdms/ansible-2.8/ansible-2.8/lib64/python3.6/site-packages/netmiko/base_connection.py”, line 1107, in find_prompt
raise ValueError(“Unable to find prompt: {}”.format(prompt))
ValueError: Unable to find prompt: