- 1 PR2 Problem Timeline (2014- )
- 2 OLD (Pre-2014)
PR2 Problem Timeline (2014- )
- [GATSBII] 9 MAY 2016 - Errors from realtime loop using too much time in the past 30 seconds, halting motors. Bizarre + scary behavior (weird motions, buzzing noises) when running pr2-systemcheck with robot running. realtime loop not immediately helped by power-cycle. Opened ticket with clearpath, but issue resolved after some time, for no apparent reason.
- [GATSBII] 6 MAY 2016 - Errors from realtime loop using too much time in the past 30 seconds, halting motors. Did not respond to robot start, but fixed after power cycle.
- [Monty] 1/4/2016 - The PR2 became unresponsive over wired ethernet, but remained accessible via wireless (MontyLAN), similar to prior incidents. FIXED by running 'sudo service wg-openvpn restart' on Monty (after logging in via wireless).
- [Monty] 11/5/2015 - The PR2 became unresponsive over the wired ethernet connection, but remained responsive over the wireless connection. A full power cycle seems to have fixed the problem. (Megan)
- [Monty] 10/3/2015 - The PR2 became unresponsive over the wired ethernet connection, but remained responsive over the wireless connection. A full power cycle seems to have fixed the problem. (Megan)
- [Monty] 9/25/2015 and 9/28/2015 - The PR2 became unresponsive over the wired ethernet connection, but remained responsive over the wireless connection. A full power cycle seems to have fixed the problem both times (Megan, Phil).
- [Gatsbii] 6/29/2015 - The PR2's internal router is unresponsive again (10.68.0.5 does not respond). A full power cycle was required.
- [Gatsbii] 6/26/2015 - The PR2's internal router appears unresponsive and hence the kinect was unable to publish any data. As suggested by Phil, this required a full power cycle of Gatsbii.
- [Gatsbii] 6/25/2015 - Zackory and Daehyung notice that EtherCat halts motors continuously due to a low frequency. Problem fixed itself after rebooting PR2 and about 20 minutes of waiting.
- [Gatsbii] 5/29/2015 - Megan sees same problem as 5/22 with router not responding and losing kinect data again. Power cycling seems to have fixed it.
- [Gatsbii] 5/22/2015 - Phil sees problem with router not responding and so losing kinect data again.
- [Gatsbii] 3/13/2015 - Kat Graf reported that the WAN port had fallen inside the robot. Phil and Kat repaired, per service instructions originally from Willow on the service wiki. Problems was fixed, no ticket needed to be filed. (Phil)
- [Gatsbii] 2/13/2015 - After installing the Kinect2 backpack from Clearpath, which connects to the robot's internal network via an ethernet cable to the wireless router in the back of the robot, on occasion it seems that the router fails. It can be pinged, but does not present it's web configuration interface @ 10.68.0.5, and does not give an address to the backpack on the internal network, so the kinect cannot be used. Power-cycling seems to have fixed the problem on one occasion. On a second occasion, the situation resolved itself when the robot was moved a short distance with the joystick. Possibly random coincidence, possibly a faulty connection? (Phil, Ari)
- Other Occurrences: 2/27/2015 (GATSBII) - Fixed on power-cycle after ~20 minutes, 3/4/2015 - Power Cycle
- [Monty] 12/11/2014 - One morning we found that a relatively high pitched fan noise was coming from the front vent on the base of our PR2. Higher pitched than usual fan noises. We felt the air coming from the front vent was also warmer than usual. We checked processes on the robot and found 4-5 processes were running using 14-30% cpu each. We ran a robot start to clear those processes. The noise did not subside within 10 minutes so we shut the robot down. Cleaned intake vent. Restarted the robot the next day. No abnormal noises upon start up. (Ari)
- [Monty] 10/1/2014 - Megan reported that Monty dashboard showed motors and breakers greyed out and unresponsive, with a warning in the dashboard under power system stating 'base fan off,' despite the fact that the base fans could be heard running. Phil recorded bag files in this condition, and attempted robot start while recording a second bag file. The did not fix the problem. After power-cycling the robot, the issue appears resolved, and Monty is operating normally. (Phil)
- [Gatsbii] 9/30/2014 - After logging in to Gatsbii, the dashboard showed all breakers and motors greyed out and unresponsive (computers were running normally). Robot start did not correct this. After power-cycling the robot, the items returned to the dashboard, but the motors could not be reset, and were halted due to 'Service Request'. Recorded bag files from robot start, including trying to reset the motors by flipping run-stop, using the dashboard interface, and by calling pr2_etherCAT/halt_motors and pr2_etherCAT/reset_motors services from command line (none of which had any effect). Filed ticket with Clearpath with attached bag file. (Phil)
- [Monty] 8/12/2014 - Unable to connect to Monty via WAN. Unable to ping monty1.hsi.gatech.edu. Was able to connect to Monty wirelessly, shut it down and restart successfully. Afterwards was able to connect to Monty via WAN again. (Megan)
- [Monty] 7/31/2014 - Unable to connect to Monty via WAN. Unable to ping monty1.hsi.gatech.edu. Was able to connect to Monty wirelessly, shut it down and restart successfully. Afterwards was able to connect to Monty via WAN again. (Megan)
- [Monty] 7/9/2014 - All controllers go down. Breakers are greyed out on the dash board and error messages show that the voltage was low. 'robot start' did not fix the problem but a full power cycle seems to have fixed the problem. (Chris)
- [Monty] 5/20/2014 - Right arm camera driver is down. This has happened sporadically over the past week or so, and has previously been fixed by running 'robot start.' This has just failed to solve the problem. Error reported indicates "Driver State: CLOSED; Latest status message: Matching URL name://email@example.com : No cameras matched the URL." Perhaps a physical connectivity issue with the camera? There seem to be lights on on the back of the camera board (matching the working left arm)." pr2-systemcheck returns "Running 'c1/checkwge100.sh'... \\ FAIL (Return code: 1) Unable to find cameras: forearm_r." Problem did resolve with power-cycle, but should keep a close eye on this. (Phil)
- [Monty] 5/20/2014 - Fixed WAN port connection which had come loose from the mounting bracket, per instructions for WAN port repair on WG support site. (Phil)
- [Monty] 3/18/2014 - Computer crash? Motors disengaged, computers unresponsive to ping or ssh via WAN, pingable, but no ssh over wireless. Hard restart (red breaker). Repeated ~10 minutes after start-up (got through calibration successfully). [Phil, Tapo]
- [Monty] Back Right Caster - Dropping Packets, causes RT Loop to stop motors (occurring since ~Dec 2013, frequency increasing.) Contacted Clearpath via Ticket for replacement caster.
- [GATSBII] 3/20/2014 - Per Tucker Hermans and Hai Nguyen, batteries are almost completely shot, Hai was getting ~15-20 minutes of use at last use (~summer 2013?). Phil found the robot powered off in Aware Home, and after starting up, observed approx. 90 seconds of battery life for robot. Dashboard gives no evidence of charging (only some battery modules claim to be charging, after 24 hours plugged in dashboard shows 0% charged, 0 minutes to full charge). This may make it difficult to move the robot for any period when not plugged in. Ticket filed with Clearpath. [Phil]
- [Monty] 3/21/2014 - Robot was used unplugged intentionally to test real-world battery life. Estimated time was 150 minutes at 98% charged when beginning. Under light-moderate load on c1 (very limited physical movement), battery decreased to ~50% in 70 minutes, estimated time showed ~80 minutes. After this point, the base was driven a limited amount (~30s in 2-5s moves over ~2 minutes (adjusting position)), and shortly thereafter the robot powered off, as if the batteries were dead (both computers shut down, lights off, etc.). Robot was plugged back in, restarted (normally), and reported 54% battery life once restarted. Ticked filed with ClearPath. [Phil]
C2 Goes Down
10/19/2011 (Hai) 10/19/2011 (Kelsey x 2)
Computers cannot be contacted by any means, attempts to log in through wireless, service port, and LAN are all unsuccessful. Additionally, the wireless router does not seem to be broadcasting a signal. The robot shows little signs of activity aside from lights and fans. Only fixed through hard boot.
Occurrences: 6/30/11 (Gatsbii) - Kelsey, 7/7/11 (Gatsbii) - Kelsey, 7/7/11 (New PR2) - Kelsey, 8/9/11 Morning (Gatsbii) - HFA, 8/9/11 Evening (Gatsbii) - Kelsey, 9/6/11 (Gatsbii) - Kelsey, 9/6/11 (New PR2) - Tucker, 11/15/11 (New PR2) - Kelsey/Phil, 6/7/12 (Gatsbii)
- Related to networking? Hostnames unresponsive to DNS queries for all computers - coincidence? Also infrequent yet will occur frequently in a small time frame (across both robots in one instance).
Similarly, on a number of occasions, the network connections seem to largely fail, allowing burst of communication every 5-10 seconds. pr2-systemcheck shows 'all/vpnconnectivity/sh...' FAIL (Return code: 1), and the only evident solution appears to be power-cycling the robot. Wireless does still work in this situation, however.
Occurrences: 3-4 times before 7/19/11 (New PR2) - Phil, 7/19/11 (New PR2) - Phil, 7/26/11 (New PR2) - Phil
Occasionally a substantial number of diagnostic messages go idle, and the robot becomes impossible to control through teleop or other control schemes. The computers are still active though and a robot stop and start usually fixes this.
Occurrences: 6/22/11 (New PR2) - Kelsey, 6/27/11 (New PR2) - Kelsey
Motor Breakers Activate on Controller Switch
Sometimes when switching the controllers the breakers will activate, stopping the arms. Reseting the breakers fixes this.
Occurrences: 6/27/11 (New PR2) - Kelsey, 6/28/11 (New PR2) - Kelsey
- If the starting code in the controller loaded takes more time than the realtime loop allows, the motors will stop.
Ethercat / Real-time controllers Down
Logged in and robot was operating slowly and no diagnostic information was being communicated about ethercat devices and the real-time controllers. pr2-systemcheck gave an error when comunicating to the ethercat network. robot stop && robot start did not fix the problem. pr2-shutdown --reboot fixed the issue.
Occurences: 3 July 2012 (Gatsbii) - Tucker