Difference between revisions of "PR2 Problems"

From Healthcare Robotics Wiki
Jump to: navigation, search
(May)
(2014)
Line 1: Line 1:
 
= PR2 Problem Timeline (2014- ) =
 
= PR2 Problem Timeline (2014- ) =
 
==2014==
 
==2014==
 +
=== July ===
 +
*[Monty] 7/9/2014 - All controllers go down. Breakers are greyed out on the dash board and error messages show that the voltage was low. 'robot start' did not fix the problem but a full power cycle seems to have fixed the problem. (Chris)
 
=== May ===
 
=== May ===
 
*[Monty] 5/20/2014 - Right arm camera driver is down.  This has happened sporadically over the past week or so, and has previously been fixed by running 'robot start.'  This has just failed to solve the problem.  Error reported indicates "Driver State: CLOSED; Latest status message: Matching URL name://forearm_r#lan1@10.69.0.41 : No cameras matched the URL."  Perhaps a physical connectivity issue with the camera?  There seem to be lights on on the back of the camera board (matching the working left arm)."  pr2-systemcheck returns "Running 'c1/checkwge100.sh'... \\ FAIL (Return code: 1) Unable to find cameras: forearm_r."  Problem did resolve with power-cycle, but should keep a close eye on this. (Phil)
 
*[Monty] 5/20/2014 - Right arm camera driver is down.  This has happened sporadically over the past week or so, and has previously been fixed by running 'robot start.'  This has just failed to solve the problem.  Error reported indicates "Driver State: CLOSED; Latest status message: Matching URL name://forearm_r#lan1@10.69.0.41 : No cameras matched the URL."  Perhaps a physical connectivity issue with the camera?  There seem to be lights on on the back of the camera board (matching the working left arm)."  pr2-systemcheck returns "Running 'c1/checkwge100.sh'... \\ FAIL (Return code: 1) Unable to find cameras: forearm_r."  Problem did resolve with power-cycle, but should keep a close eye on this. (Phil)

Revision as of 13:07, 9 July 2014

PR2 Problem Timeline (2014- )

2014

July

  • [Monty] 7/9/2014 - All controllers go down. Breakers are greyed out on the dash board and error messages show that the voltage was low. 'robot start' did not fix the problem but a full power cycle seems to have fixed the problem. (Chris)

May

  • [Monty] 5/20/2014 - Right arm camera driver is down. This has happened sporadically over the past week or so, and has previously been fixed by running 'robot start.' This has just failed to solve the problem. Error reported indicates "Driver State: CLOSED; Latest status message: Matching URL name://forearm_r#lan1@10.69.0.41 : No cameras matched the URL." Perhaps a physical connectivity issue with the camera? There seem to be lights on on the back of the camera board (matching the working left arm)." pr2-systemcheck returns "Running 'c1/checkwge100.sh'... \\ FAIL (Return code: 1) Unable to find cameras: forearm_r." Problem did resolve with power-cycle, but should keep a close eye on this. (Phil)
  • [Monty] 5/20/2014 - Fixed WAN port connection which had come loose from the mounting bracket, per instructions for WAN port repair on WG support site. (Phil)

March

  • [Monty] 3/18/2014 - Computer crash? Motors disengaged, computers unresponsive to ping or ssh via WAN, pingable, but no ssh over wireless. Hard restart (red breaker). Repeated ~10 minutes after start-up (got through calibration successfully). [Phil, Tapo]
  • [Monty] Back Right Caster - Dropping Packets, causes RT Loop to stop motors (occurring since ~Dec 2013, frequency increasing.) Contacted Clearpath via Ticket for replacement caster.
  • [GATSBII] 3/20/2014 - Per Tucker Hermans and Hai Nguyen, batteries are almost completely shot, Hai was getting ~15-20 minutes of use at last use (~summer 2013?). Phil found the robot powered off in Aware Home, and after starting up, observed approx. 90 seconds of battery life for robot. Dashboard gives no evidence of charging (only some battery modules claim to be charging, after 24 hours plugged in dashboard shows 0% charged, 0 minutes to full charge). This may make it difficult to move the robot for any period when not plugged in. Ticket filed with Clearpath. [Phil]
  • [Monty] 3/21/2014 - Robot was used unplugged intentionally to test real-world battery life. Estimated time was 150 minutes at 98% charged when beginning. Under light-moderate load on c1 (very limited physical movement), battery decreased to ~50% in 70 minutes, estimated time showed ~80 minutes. After this point, the base was driven a limited amount (~30s in 2-5s moves over ~2 minutes (adjusting position)), and shortly thereafter the robot powered off, as if the batteries were dead (both computers shut down, lights off, etc.). Robot was plugged back in, restarted (normally), and reported 54% battery life once restarted. Ticked filed with ClearPath. [Phil]

OLD (Pre-2014)

Robot Unresponsiveness

Major

C2 Goes Down

10/19/2011 (Hai) 10/19/2011 (Kelsey x 2)

Completely Unresponsive

Computers cannot be contacted by any means, attempts to log in through wireless, service port, and LAN are all unsuccessful. Additionally, the wireless router does not seem to be broadcasting a signal. The robot shows little signs of activity aside from lights and fans. Only fixed through hard boot.

Occurrences: 6/30/11 (Gatsbii) - Kelsey, 7/7/11 (Gatsbii) - Kelsey, 7/7/11 (New PR2) - Kelsey, 8/9/11 Morning (Gatsbii) - HFA, 8/9/11 Evening (Gatsbii) - Kelsey, 9/6/11 (Gatsbii) - Kelsey, 9/6/11 (New PR2) - Tucker, 11/15/11 (New PR2) - Kelsey/Phil, 6/7/12 (Gatsbii)

Hypotheses:

  • Related to networking? Hostnames unresponsive to DNS queries for all computers - coincidence? Also infrequent yet will occur frequently in a small time frame (across both robots in one instance).

Network Failure

Similarly, on a number of occasions, the network connections seem to largely fail, allowing burst of communication every 5-10 seconds. pr2-systemcheck shows 'all/vpnconnectivity/sh...' FAIL (Return code: 1), and the only evident solution appears to be power-cycling the robot. Wireless does still work in this situation, however.

Occurrences: 3-4 times before 7/19/11 (New PR2) - Phil, 7/19/11 (New PR2) - Phil, 7/26/11 (New PR2) - Phil


Minor

Occasionally a substantial number of diagnostic messages go idle, and the robot becomes impossible to control through teleop or other control schemes. The computers are still active though and a robot stop and start usually fixes this.

Unresponsive diagnostics


Occurrences: 6/22/11 (New PR2) - Kelsey, 6/27/11 (New PR2) - Kelsey

Motor Breakers Activate on Controller Switch

Sometimes when switching the controllers the breakers will activate, stopping the arms. Reseting the breakers fixes this.

Occurrences: 6/27/11 (New PR2) - Kelsey, 6/28/11 (New PR2) - Kelsey

Hypotheses:

  • If the starting code in the controller loaded takes more time than the realtime loop allows, the motors will stop.

Ethercat / Real-time controllers Down

Logged in and robot was operating slowly and no diagnostic information was being communicated about ethercat devices and the real-time controllers. pr2-systemcheck gave an error when comunicating to the ethercat network. robot stop && robot start did not fix the problem. pr2-shutdown --reboot fixed the issue.

Occurences: 3 July 2012 (Gatsbii) - Tucker