I have now further investigated this topic increasing the loglevel on the FreeBSD side.
There is a clear pattern which seems to point toward a problem at the MikroTik end.
Looking at the lacpdu transmits, there two cycles can be observed. The previously reported 5h cycle with breaking connection, but also there is a 27 minute cycle with a near-miss break.
During these 27 minute cycles, the MikroTik box takes an ever longer amount of time to answer a lacpdu_transmit, Starting with 0-1 seconds until reaching 30 seconds delay. 30 seconds is the heartbeat interval, i.e. a delay of this duration is considered a HB miss by FreeBSD.
The 27 minutes cycles do NOT break the LACP connection, as the retransmissions manage to re-sync the two devices within the 3 attempts.
Now every 5 hours, the 3rd attempt is also missed and the FreeBSD decides to drop/restart the LACP.
As the MikroTik end is not observable (no logs, no nothing), I have no idea what could cause the ever longer delay in answering that is observed. But until proven otherwise, I hold the switch to be responsible for the break in the protocol.
Here are the cleaned up and arranged logs captured today. The last column shows the ever increasing time that the MikroTik switch takes to answer the LACPDU transmit.
The switch gets fresh again at 09:29:35 AM after an initial retransmission, but then again degrades in answering time. These 27 minute cycles go on and on, with the re-sync something using all 3 attempts. Eventually after 11 cycles, the third attempt is also missed the the LAG destroyed.
Firmware in use is SwOS Lite 2.18.
Unless somebody from MikroTik can elaborate here on this I am tempting to take this as a support problem.
Cheers,
Kei
There is a clear pattern which seems to point toward a problem at the MikroTik end.
Looking at the lacpdu transmits, there two cycles can be observed. The previously reported 5h cycle with breaking connection, but also there is a 27 minute cycle with a near-miss break.
During these 27 minute cycles, the MikroTik box takes an ever longer amount of time to answer a lacpdu_transmit, Starting with 0-1 seconds until reaching 30 seconds delay. 30 seconds is the heartbeat interval, i.e. a delay of this duration is considered a HB miss by FreeBSD.
The 27 minutes cycles do NOT break the LACP connection, as the retransmissions manage to re-sync the two devices within the 3 attempts.
Now every 5 hours, the 3rd attempt is also missed and the FreeBSD decides to drop/restart the LACP.
As the MikroTik end is not observable (no logs, no nothing), I have no idea what could cause the ever longer delay in answering that is observed. But until proven otherwise, I hold the switch to be responsible for the break in the protocol.
Here are the cleaned up and arranged logs captured today. The last column shows the ever increasing time that the MikroTik switch takes to answer the LACPDU transmit.
The switch gets fresh again at 09:29:35 AM after an initial retransmission, but then again degrades in answering time. These 27 minute cycles go on and on, with the re-sync something using all 3 attempts. Eventually after 11 cycles, the third attempt is also missed the the LAG destroyed.
Code:
2024-03-2609:11:00 AMtransmt100:00:272024-03-2609:11:13 AMreceive100:00:3100:00:132024-03-2609:11:30 AMtransmt100:00:302024-03-2609:11:44 AMreceive100:00:3100:00:142024-03-2609:12:00 AMtransmt100:00:302024-03-2609:12:14 AMreceive100:00:3000:00:142024-03-2609:12:30 AMtransmt100:00:302024-03-2609:12:45 AMreceive100:00:3100:00:152024-03-2609:13:01 AMtransmt100:00:312024-03-2609:13:15 AMreceive100:00:3000:00:142024-03-2609:13:31 AMtransmt100:00:302024-03-2609:13:46 AMreceive100:00:3100:00:152024-03-2609:14:01 AMtransmt100:00:302024-03-2609:14:17 AMreceive100:00:3100:00:162024-03-2609:14:31 AMtransmt100:00:302024-03-2609:14:47 AMreceive100:00:3000:00:162024-03-2609:15:01 AMtransmt100:00:302024-03-2609:15:18 AMreceive100:00:3100:00:172024-03-2609:15:31 AMtransmt100:00:302024-03-2609:15:48 AMreceive100:00:3000:00:172024-03-2609:16:02 AMtransmt100:00:312024-03-2609:16:19 AMreceive100:00:3100:00:172024-03-2609:16:32 AMtransmt100:00:302024-03-2609:16:50 AMreceive100:00:3100:00:182024-03-2609:17:02 AMtransmt100:00:302024-03-2609:17:20 AMreceive100:00:3000:00:182024-03-2609:17:32 AMtransmt100:00:302024-03-2609:17:51 AMreceive100:00:3100:00:192024-03-2609:18:02 AMtransmt100:00:302024-03-2609:18:21 AMreceive100:00:3000:00:192024-03-2609:18:32 AMtransmt100:00:302024-03-2609:18:52 AMreceive100:00:3100:00:202024-03-2609:19:02 AMtransmt100:00:302024-03-2609:19:22 AMreceive100:00:3000:00:202024-03-2609:19:32 AMtransmt100:00:302024-03-2609:19:53 AMreceive100:00:3100:00:212024-03-2609:20:03 AMtransmt100:00:312024-03-2609:20:24 AMreceive100:00:3100:00:212024-03-2609:20:33 AMtransmt100:00:302024-03-2609:20:54 AMreceive100:00:3000:00:212024-03-2609:21:03 AMtransmt100:00:302024-03-2609:21:25 AMreceive100:00:3100:00:222024-03-2609:21:33 AMtransmt100:00:302024-03-2609:21:55 AMreceive100:00:3000:00:222024-03-2609:22:03 AMtransmt100:00:302024-03-2609:22:26 AMreceive100:00:3100:00:232024-03-2609:22:33 AMtransmt100:00:302024-03-2609:22:57 AMreceive100:00:3100:00:242024-03-2609:23:03 AMtransmt100:00:302024-03-2609:23:27 AMreceive100:00:3000:00:242024-03-2609:23:33 AMtransmt100:00:302024-03-2609:23:58 AMreceive100:00:3100:00:252024-03-2609:24:03 AMtransmt100:00:302024-03-2609:24:29 AMreceive100:00:3100:00:262024-03-2609:24:33 AMtransmt100:00:302024-03-2609:24:59 AMreceive100:00:3000:00:262024-03-2609:25:04 AMtransmt100:00:312024-03-2609:25:30 AMreceive100:00:3100:00:262024-03-2609:25:34 AMtransmt100:00:302024-03-2609:26:00 AMreceive100:00:3000:00:262024-03-2609:26:04 AMtransmt100:00:302024-03-2609:26:31 AMreceive100:00:3100:00:272024-03-2609:26:34 AMtransmt100:00:302024-03-2609:27:01 AMreceive100:00:3000:00:272024-03-2609:27:04 AMtransmt100:00:302024-03-2609:27:32 AMreceive100:00:3100:00:282024-03-2609:27:34 AMtransmt100:00:302024-03-2609:28:03 AMreceive100:00:3100:00:292024-03-2609:28:04 AMtransmt100:00:302024-03-2609:28:33 AMreceive100:00:3000:00:292024-03-2609:28:34 AMtransmt100:00:302024-03-2609:29:04 AMtransmt000:00:312024-03-2609:29:04 AMreceive100:00:3000:00:002024-03-2609:29:35 AMtransmt100:00:312024-03-2609:29:35 AMreceive100:00:3100:00:002024-03-2609:30:05 AMtransmt100:00:302024-03-2609:30:05 AMreceive100:00:3000:00:002024-03-2609:30:35 AMtransmt100:00:302024-03-2609:30:36 AMreceive100:00:3100:00:012024-03-2609:31:05 AMtransmt100:00:302024-03-2609:31:07 AMreceive100:00:3100:00:022024-03-2609:31:35 AMtransmt100:00:302024-03-2609:31:37 AMreceive100:00:3000:00:022024-03-2609:32:05 AMtransmt100:00:302024-03-2609:32:08 AMreceive100:00:3100:00:032024-03-2609:32:35 AMtransmt100:00:302024-03-2609:32:39 AMreceive100:00:3100:00:042024-03-2609:33:05 AMtransmt100:00:302024-03-2609:33:10 AMreceive100:00:3100:00:052024-03-2609:33:36 AMtransmt100:00:312024-03-2609:33:40 AMreceive100:00:3000:00:042024-03-2609:34:06 AMtransmt100:00:302024-03-2609:34:11 AMreceive100:00:3100:00:052024-03-2609:34:36 AMtransmt100:00:302024-03-2609:34:42 AMreceive100:00:3100:00:062024-03-2609:35:06 AMtransmt100:00:302024-03-2609:35:12 AMreceive100:00:3000:00:062024-03-2609:35:36 AMtransmt100:00:302024-03-2609:35:43 AMreceive100:00:3100:00:072024-03-2609:36:06 AMtransmt100:00:302024-03-2609:36:14 AMreceive100:00:3100:00:082024-03-2609:36:36 AMtransmt100:00:302024-03-2609:36:45 AMreceive100:00:3100:00:092024-03-2609:37:06 AMtransmt100:00:302024-03-2609:37:15 AMreceive100:00:3000:00:092024-03-2609:37:36 AMtransmt100:00:302024-03-2609:37:46 AMreceive100:00:3100:00:102024-03-2609:38:07 AMtransmt100:00:312024-03-2609:38:17 AMreceive100:00:3100:00:102024-03-2609:38:36 AMtransmt100:00:292024-03-2609:38:47 AMreceive100:00:3000:00:112024-03-2609:39:07 AMtransmt100:00:312024-03-2609:39:18 AMreceive100:00:3100:00:112024-03-2609:39:37 AMtransmt100:00:302024-03-2609:39:49 AMreceive100:00:3100:00:122024-03-2609:40:07 AMtransmt100:00:302024-03-2609:40:20 AMreceive100:00:3100:00:132024-03-2609:40:37 AMtransmt100:00:302024-03-2609:40:50 AMreceive100:00:3000:00:132024-03-2609:41:07 AMtransmt100:00:302024-03-2609:41:21 AMreceive100:00:3100:00:142024-03-2609:41:37 AMtransmt100:00:302024-03-2609:41:52 AMreceive100:00:3100:00:152024-03-2609:42:07 AMtransmt100:00:302024-03-2609:42:22 AMreceive100:00:3000:00:152024-03-2609:42:37 AMtransmt100:00:302024-03-2609:42:53 AMreceive100:00:3100:00:162024-03-2609:43:07 AMtransmt100:00:302024-03-2609:43:24 AMreceive100:00:3100:00:172024-03-2609:43:38 AMtransmt100:00:312024-03-2609:43:54 AMreceive100:00:3000:00:162024-03-2609:44:08 AMtransmt100:00:302024-03-2609:44:25 AMreceive100:00:3100:00:172024-03-2609:44:38 AMtransmt100:00:302024-03-2609:44:56 AMreceive100:00:3100:00:182024-03-2609:45:08 AMtransmt100:00:302024-03-2609:45:26 AMreceive100:00:3000:00:182024-03-2609:45:38 AMtransmt100:00:302024-03-2609:45:57 AMreceive100:00:3100:00:192024-03-2609:46:08 AMtransmt100:00:302024-03-2609:46:28 AMreceive100:00:3100:00:202024-03-2609:46:39 AMtransmt100:00:312024-03-2609:46:58 AMreceive100:00:3000:00:192024-03-2609:47:09 AMtransmt100:00:302024-03-2609:47:29 AMreceive100:00:3100:00:202024-03-2609:47:39 AMtransmt100:00:302024-03-2609:48:00 AMreceive100:00:3100:00:212024-03-2609:48:09 AMtransmt100:00:302024-03-2609:48:30 AMreceive100:00:3000:00:212024-03-2609:48:39 AMtransmt100:00:302024-03-2609:49:01 AMreceive100:00:3100:00:222024-03-2609:49:09 AMtransmt100:00:302024-03-2609:49:32 AMreceive100:00:3100:00:232024-03-2609:49:39 AMtransmt100:00:302024-03-2609:50:02 AMreceive100:00:3000:00:232024-03-2609:50:09 AMtransmt100:00:302024-03-2609:50:33 AMreceive100:00:3100:00:242024-03-2609:50:39 AMtransmt100:00:302024-03-2609:51:04 AMreceive100:00:3100:00:252024-03-2609:51:09 AMtransmt100:00:302024-03-2609:51:34 AMreceive100:00:3000:00:252024-03-2609:51:39 AMtransmt100:00:302024-03-2609:52:05 AMreceive100:00:3100:00:262024-03-2609:52:10 AMtransmt100:00:312024-03-2609:52:36 AMreceive100:00:3100:00:262024-03-2609:52:40 AMtransmt100:00:302024-03-2609:53:06 AMreceive100:00:3000:00:262024-03-2609:53:10 AMtransmt100:00:302024-03-2609:53:37 AMreceive100:00:3100:00:272024-03-2609:53:40 AMtransmt100:00:302024-03-2609:54:07 AMreceive100:00:3000:00:272024-03-2609:54:10 AMtransmt100:00:302024-03-2609:54:38 AMreceive100:00:3100:00:282024-03-2609:54:40 AMtransmt100:00:302024-03-2609:55:09 AMreceive100:00:3100:00:292024-03-2609:55:10 AMtransmt100:00:302024-03-2609:55:39 AMreceive100:00:3000:00:292024-03-2609:55:40 AMtransmt100:00:302024-03-2609:56:10 AMreceive100:00:3100:00:302024-03-2609:56:11 AMtransmt100:00:312024-03-2609:56:41 AMtransmt000:00:312024-03-2609:56:41 AMreceive100:00:3000:00:002024-03-2609:57:11 AMtransmt100:00:302024-03-2609:57:11 AMreceive100:00:3000:00:002024-03-2609:57:41 AMtransmt100:00:302024-03-2609:57:42 AMreceive100:00:3100:00:012024-03-2609:58:11 AMtransmt100:00:302024-03-2609:58:12 AMreceive100:00:3000:00:01Firmware in use is SwOS Lite 2.18.
Unless somebody from MikroTik can elaborate here on this I am tempting to take this as a support problem.
Cheers,
Kei
Statistics: Posted by kei — Tue Mar 26, 2024 1:13 pm