Unusual (disruptive) burst of high ping times.

[gearsincorg]

This is annother wird effect I'm seeing with the full A301 system.

Periodicall, out of the blue I will get a burst of high-ping times reported on the Driver Station chart display, and while it occurs I get major arm motor disruptions. This may happen while the arm is actively being used, or it may occur while the robot is just sitting there enabled.

The appearance is very consistan, the arm (that's running in position mode) will drop or jerk arounf dor a couple of seconds and then it will return to it's original position (which is being continually sent).

At the same time there will be a nice bell curve of uncreased Ping times, and sometimes there will be some small packet loss number. I haven't been able to capture an screen grab of this as it scrolls off quickly, but I've drawn what it looks like below:

The drawn size and shape are approximately correct.

This may happen every 30-60 minutes or so.

I have observed this on two different Windows 11 PC,s in two completely different locations. One is my usual development location, and one was completely unrelated to my usual network connections.

I will attempt to capture a screen image, but if it happens again, is there a log I can also save and tag here?

[ThadHouse]

Generally, issues like this are caused by something in the Windows WiFi stack deciding it wants more control over the WiFi, and that causes lag in programs that use it. Its nothing inherently wrong with the system, other than the fairly tight boundaries we have for control timeouts.

Windows update can actually be a big thing that causes this, in addition to essentially any random programs installed. For years, our recommendation is to run a completely separate DS with nothing extra installed on it for FRC uses, specifically because we can't control what other apps or services run in the backround.

This is also why FRC also wants to get a Driver Hub like appliance that is legal as well, to allow working around some of these random issues.

BTW you don't actually need the screenshot. If you see it happen, wait a few seconds and then close the DS. Then you can go into the logs folder (C:\Users\Public\FirstDriverStation\Logs) and grab the latest or 2 latest log files. You can send those, and we can check all the info, or you can load those logs into AdvantageScope and see it all for yourself (theres a bunch of things in there you could look at). You can get a description for all the keys at https://github.com/wpilibsuite/FirstDriverStation-Public/blob/main/docs/LoggingKeys.md

[qwertychouskie]

Generally, issues like this are caused by something in the Windows WiFi stack deciding it wants more control over the WiFi, and that causes lag in programs that use it. Its nothing inherently wrong with the system, other than the fairly tight boundaries we have for control timeouts.

Perhaps consider integrating https://github.com/catid/WLANOptimizer into the DS? (Yay, random knowledge from like 8 years ago comes in handy!)

That said, I am a bit confused as to why DS latency spikes would cause motors holding position to start spazzing. Is a watchdog timer constantly engaging and disengaging? Though I'd expect a watchdog timer to kill the program, not just make the motors momentarily limp; are there like multiple levels of watchdog timers?

I'd argue having motors start spazzing out when latency gets a bit shaky is much more of a safety and damage concern than having only a slightly longer watchdog that actually stops the opmode. Sometimes, you have to work on parts of the robot with it enabled (e.g. dealing with vertical lift stringing and such), and if the robot starts spazzing out even when no one is near the controllers, that could easily cause injury. (Even just an unexpected disable could cause injury, but spazzing definitely sounds way riskier.)

[ThadHouse]

The watchdog timer doesn't stop the program, it just disables outputs, until the next control packet comes in.

Packets are sent by the DS every 20ms, and the watchdog timer is set at 125ms.

[qwertychouskie]

The watchdog timer doesn't stop the program, it just disables outputs, until the next control packet comes in.

Packets are sent by the DS every 20ms, and the watchdog timer is set at 125ms.

Hmm, I definitely find that behavior concerning/risky, but that's perhaps best discussed in another ticket.

For now, @gearsincorg can you try https://github.com/catid/WLANOptimizer/blob/master/tests/stand_alone.exe and see if it fixes the issue while it's running?

[gearsincorg]

OK, so some very cool updates.

1. I was able to recreate the problem on demand. Just the process of popping up the WiFi dialog on the PC (without actually making any changes) will trigger the burst of long ping times (although they really aren't that long compared to bad FTC ping times :) This was 100% reproducible. And every time I did it, the arm would jump down and up several times.

2. Then, after reviewing the WLANOptimzer code, I decided to try it out.
   The program claimed that it would run the optimizer logic while running, and then stop it once exited.

The result was very noticable:

Popping up the Network dialog would no longer cause the arm to jerk around. On the status window, the ping times still jumped up, but they were less than half as log as before.

The only strange thing was that once I stopped the localizer, the "fix" was still in effect. The arm no longer jumped and the ping times were short. So Hmmmm.

The other thing that was significant was that every time I showed the network window, the WLANOptimizer window would add a line or two saying that A wireless LAN had been detected. So I assume to scan the network, Windows is closing the current connection and then re-openning it?

It would be worth while the WPILIB crew have a quick inspection of the WLANOptimzer code.

Extra note: It may also be true that the robot is being a bit too aggressive in disabling itself. It doesn't seem that 40ms Ping times should cause a robot shut down.