PR2 Problems

From Healthcare Robotics Wiki
Jump to: navigation, search

PR2 Problem Timeline (2014- )

2014

December

  • [Monty] 12/11/2014 - One morning we found that a relatively high pitched fan noise was coming from the front vent on the base of our PR2. Higher pitched than usual fan noises. We felt the air coming from the front vent was also warmer than usual. We checked processes on the robot and found 4-5 processes were running using 14-30% cpu each. We ran a robot start to clear those processes. The noise did not subside within 10 minutes so we shut the robot down. Cleaned intake vent. Restarted the robot the next day. No abnormal noises upon start up. (Ari)

October

  • [Monty] 10/1/2014 - Megan reported that Monty dashboard showed motors and breakers greyed out and unresponsive, with a warning in the dashboard under power system stating 'base fan off,' despite the fact that the base fans could be heard running. Phil recorded bag files in this condition, and attempted robot start while recording a second bag file. The did not fix the problem. After power-cycling the robot, the issue appears resolved, and Monty is operating normally. (Phil)

September

  • [Gatsbii] 9/30/2014 - After logging in to Gatsbii, the dashboard showed all breakers and motors greyed out and unresponsive (computers were running normally). Robot start did not correct this. After power-cycling the robot, the items returned to the dashboard, but the motors could not be reset, and were halted due to 'Service Request'. Recorded bag files from robot start, including trying to reset the motors by flipping run-stop, using the dashboard interface, and by calling pr2_etherCAT/halt_motors and pr2_etherCAT/reset_motors services from command line (none of which had any effect). Filed ticket with Clearpath with attached bag file. (Phil)

August

  • [Monty] 8/12/2014 - Unable to connect to Monty via WAN. Unable to ping monty1.hsi.gatech.edu. Was able to connect to Monty wirelessly, shut it down and restart successfully. Afterwards was able to connect to Monty via WAN again. (Megan)

July

  • [Monty] 7/31/2014 - Unable to connect to Monty via WAN. Unable to ping monty1.hsi.gatech.edu. Was able to connect to Monty wirelessly, shut it down and restart successfully. Afterwards was able to connect to Monty via WAN again. (Megan)
  • [Monty] 7/9/2014 - All controllers go down. Breakers are greyed out on the dash board and error messages show that the voltage was low. 'robot start' did not fix the problem but a full power cycle seems to have fixed the problem. (Chris)

May

  • [Monty] 5/20/2014 - Right arm camera driver is down. This has happened sporadically over the past week or so, and has previously been fixed by running 'robot start.' This has just failed to solve the problem. Error reported indicates "Driver State: CLOSED; Latest status message: Matching URL name://forearm_r#lan1@10.69.0.41 : No cameras matched the URL." Perhaps a physical connectivity issue with the camera? There seem to be lights on on the back of the camera board (matching the working left arm)." pr2-systemcheck returns "Running 'c1/checkwge100.sh'... \\ FAIL (Return code: 1) Unable to find cameras: forearm_r." Problem did resolve with power-cycle, but should keep a close eye on this. (Phil)
  • [Monty] 5/20/2014 - Fixed WAN port connection which had come loose from the mounting bracket, per instructions for WAN port repair on WG support site. (Phil)

March

  • [Monty] 3/18/2014 - Computer crash? Motors disengaged, computers unresponsive to ping or ssh via WAN, pingable, but no ssh over wireless. Hard restart (red breaker). Repeated ~10 minutes after start-up (got through calibration successfully). [Phil, Tapo]
  • [Monty] Back Right Caster - Dropping Packets, causes RT Loop to stop motors (occurring since ~Dec 2013, frequency increasing.) Contacted Clearpath via Ticket for replacement caster.
  • [GATSBII] 3/20/2014 - Per Tucker Hermans and Hai Nguyen, batteries are almost completely shot, Hai was getting ~15-20 minutes of use at last use (~summer 2013?). Phil found the robot powered off in Aware Home, and after starting up, observed approx. 90 seconds of battery life for robot. Dashboard gives no evidence of charging (only some battery modules claim to be charging, after 24 hours plugged in dashboard shows 0% charged, 0 minutes to full charge). This may make it difficult to move the robot for any period when not plugged in. Ticket filed with Clearpath. [Phil]
  • [Monty] 3/21/2014 - Robot was used unplugged intentionally to test real-world battery life. Estimated time was 150 minutes at 98% charged when beginning. Under light-moderate load on c1 (very limited physical movement), battery decreased to ~50% in 70 minutes, estimated time showed ~80 minutes. After this point, the base was driven a limited amount (~30s in 2-5s moves over ~2 minutes (adjusting position)), and shortly thereafter the robot powered off, as if the batteries were dead (both computers shut down, lights off, etc.). Robot was plugged back in, restarted (normally), and reported 54% battery life once restarted. Ticked filed with ClearPath. [Phil]

OLD (Pre-2014)

Robot Unresponsiveness

Major

C2 Goes Down

10/19/2011 (Hai) 10/19/2011 (Kelsey x 2)

Completely Unresponsive

Computers cannot be contacted by any means, attempts to log in through wireless, service port, and LAN are all unsuccessful. Additionally, the wireless router does not seem to be broadcasting a signal. The robot shows little signs of activity aside from lights and fans. Only fixed through hard boot.

Occurrences: 6/30/11 (Gatsbii) - Kelsey, 7/7/11 (Gatsbii) - Kelsey, 7/7/11 (New PR2) - Kelsey, 8/9/11 Morning (Gatsbii) - HFA, 8/9/11 Evening (Gatsbii) - Kelsey, 9/6/11 (Gatsbii) - Kelsey, 9/6/11 (New PR2) - Tucker, 11/15/11 (New PR2) - Kelsey/Phil, 6/7/12 (Gatsbii)

Hypotheses:

  • Related to networking? Hostnames unresponsive to DNS queries for all computers - coincidence? Also infrequent yet will occur frequently in a small time frame (across both robots in one instance).

Network Failure

Similarly, on a number of occasions, the network connections seem to largely fail, allowing burst of communication every 5-10 seconds. pr2-systemcheck shows 'all/vpnconnectivity/sh...' FAIL (Return code: 1), and the only evident solution appears to be power-cycling the robot. Wireless does still work in this situation, however.

Occurrences: 3-4 times before 7/19/11 (New PR2) - Phil, 7/19/11 (New PR2) - Phil, 7/26/11 (New PR2) - Phil


Minor

Occasionally a substantial number of diagnostic messages go idle, and the robot becomes impossible to control through teleop or other control schemes. The computers are still active though and a robot stop and start usually fixes this.

Unresponsive diagnostics


Occurrences: 6/22/11 (New PR2) - Kelsey, 6/27/11 (New PR2) - Kelsey

Motor Breakers Activate on Controller Switch

Sometimes when switching the controllers the breakers will activate, stopping the arms. Reseting the breakers fixes this.

Occurrences: 6/27/11 (New PR2) - Kelsey, 6/28/11 (New PR2) - Kelsey

Hypotheses:

  • If the starting code in the controller loaded takes more time than the realtime loop allows, the motors will stop.

Ethercat / Real-time controllers Down

Logged in and robot was operating slowly and no diagnostic information was being communicated about ethercat devices and the real-time controllers. pr2-systemcheck gave an error when comunicating to the ethercat network. robot stop && robot start did not fix the problem. pr2-shutdown --reboot fixed the issue.

Occurences: 3 July 2012 (Gatsbii) - Tucker