Subscribe via Email

Tuesday, July 26, 2016

Managing Low Sleep and High Stress With Some Robot Magic

 Post #19 of "Today In The AA" series. (~10 minute read)
Technical: Directed to those in CS/Robotics. General reader may still keep up..

This post follows an experience I had working at NTNU on the THORMANG3 robot, meeting tight deadlines under high stress with low energy.

I believe sharing the events may serve as a pearl for other CS students, and undergraduate researchers.

The Situation

    It's Saturday, July 16th. I'm now settled into the lab working on the THORMANG3 designing motions for a magic trick performance.
Jacky (Dr.Baltes) indicated to me he'd like a demo of the robot composed, showing our progress to a group of NTNU advisers who largely assist in funding the lab.

If you're curious
    The magic trick would involve 4 bags in front of the robot, one of which contains a nail that would hurt the robots hand. The magician would shift bags around, and a volunteer would then tell the robot which one to crush first. The robot will crush each bag until 1 remains, and this would be the one with the nail. The whole magic, is that the robot never hits the nail. There are many simple approaches to this, mainly having fake nails in each bag, that can be crushed, only to reveal the last nail is in tact, when really it is just paper as well.


        The THORMANG3 is a recent development from Robotis: the third generation of Thormang, a full-size humanoid robot.
The lab I'm working in has the upper half for the time being, so torso and above, this eliminates the complexity around walk tuning.
        3 separate PC's are used for perception, manipulation, and operation. Each comes with corresponding ros packages that provide demos and interfacing capabilities.
THORMANG3 Specification for weight, height, controller, sensors, actuator, DOF, battery, control environment from Robotis.

Robotis THORMANG3 Joint name and ID Map
THORMANG3 Joint Name and ID map

Crunch Time

    This is a newer experience for me, and there are no chances to mess up. Actual funding is at stake here, with the expensive equipment I'm working on alone.
Put someone like me in a situation like this, and naturally, things get interesting...

    The demo is July 19th at 9am. I'll try to sleep tonight, as with recent jetlag arriving in Taiwan, I've gotten about 12 hours over the last 3 days.

2 Days Until the Demo

        Welp, so much for sleep. Pulled in about 3 hours and my circadian rhythm jolted me up.
I head into the lab early and began working on the motions. The first 6 hours were spent exploring documentation, and figuring out how to use the available THORMANG3 tools.
        I'm working on the OPC (operating PC) using the thormang3_demo module to create motion frames.
National Taiwan Normal University Autonomous Agents Lab working on thormang3 rviz qt gui        Forward and inverse kinematics are provided through this QT GUI, and with my limited knowledge and time it will have to suffice. I begin by moving the left arms effector towards a location for the first frame, then I run IK and record the pose. The issue I was finding, however, is that upon playing back the pose I recorded, the Thormangs' effector destination was slightly off. I understand that by mapping an effectors position in Cartesian space (R3), to a quaternion joint space, there may be infinitely many solutions, or perhaps none.
       Maybe if iterative approximation is being used, some nearest solution was found, but that solution didn't put the effector in the exact location I want?
 I moved the arm to where it should go... I'm physically looking at it right now, and it's there! The joints are lined up, and this exists, in the real world. Whatever state these joints are in, form the solution I'm looking for... yet when forward kinematics are used, the end effector still ends up somewhere else...
I get some supper, come back, and reboot the robot.
Running the same motion I had before, the effector now moves even further away than it did previously.


Welcome to lesson #1 for CS students who work on Robotics:
The real world isn't a simple program. A robot that physically interacts with it's environment, is not 'as reliable' as a bullet proof insertion sort you coded up for intro to algorithms.

I played the motion again. The effector moved somewhere further off target this time... I internally scolded the rebellious machine.
I typed the code, you're supposed to operate perfectly damn it! That's how it works right?
 I tried designing another motion... this time, it wouldn't play.
What's going on? There is at least some solution for this system... Now you won't even move?

CS Insight kicking in, I tried the ol " turning it off and on again"... and then tried running the manipulation module.
       ERROR [ID -3002] 
bleeds on the screen.
        Servo ID 8 (left arm elbow) couldn't be read!!!

I stare in confusion. Did I over torque it?
The next few hours were spend trying to diagnose the issue.

It's 8:30pm, I go get a Bubble tea and let the machine rest a bit.

        *restart program*
ERROR [ID -3002]       W
ERROR [ID -3002]         H
ERROR [ID -3002]           Y
ERROR [ID -3002]              T
ERROR [ID -3002]                 H
ERROR [ID -3002]                    E
ERROR [ID -3002]                         FUUUUUUUUUU
Thormang3 robotis servo could not be read
Actual photo of the headache in action
Countless more errors welcome me. None of the servos in the left arm are working. The right arm elbow has bailed on me as well.

I'm tired. I've got no clue what's going on.
It's best to message my professor...

I head back to the dorm for a few hours of sleep and await a response.

24 Hours Remain

 9am the next morning, I arrive at the lab. I woke up again in the night, too stressed to sleep well.
2 Hours will have to do. I have to fix things before it's too late. The 3km walk to University felt more like 10km today, but there's no time to complain.

I open up facebook and Jacky has left his feedback.
               "Check the battery voltage! Charge the battery! You're going to damage the servos!"
The battery is a 22.2V MaxAmps Lipo battery.

        I run a multimeter on it: 8.2V

Max amps Lipo 22000 22.2v 40c battery for thormang3

IMAXB6AC balanced battery charger
Charging the battery
I spend 3 hours figuring out how to charge the battery correctly. It's not as easy as it seems. One wrong move and you could blow something up. I don't have the chance to make that type of mistake, not right now. Balanced charge 1 amp seemed to do the trick using an IMAX B6AC balance charger.

The remaining chunk of the day was spent waiting for the battery to charge (approximately 7 hours) while I read the documentation and studied inverse kinematics for the motions.

By 4pm I booted up the machine again.
        [ERROR ID -3002]
for servo 8, the same joint as before, as well as servo 13 now, which was the one that crashed on the right arm originally!

I'm boggled here. The battery is fully charged, it was all working, now it isn't.

The night approaches. Midnight hits and I'm at the lab still diagnosing the issue. Some of the NTNU guys brought me chicken wings, a good choice of rations.

I had opened up the arm completely, checking connections of all the servos, doing voltage checks, nothing was working.
        Everyone is asleep.
            This demo is in less than 7 hours.
                It's now 2 am.
The next 4 hours were spent with me continuing to diagnose the issue, with intermediate 15-minute naps on the floor. The benefit of being extremely tired, is I was able to focus in only on the goal, I had no energy for anything else, hell I feel too tired to sleep.

I message several people from the lab back in Canada for advice, and try to connect with some of the NTNU students. Naturally people are asleep, they should be.
At this point, you have to decide where to invest your time. Is there a point in throwing useless attempts at fixing the arms, when the demo is in hours? I clearly couldn't figure this out. Let's just put this in perspective again, really. I've slept 5 hours since Friday morning. It's now approaching Tuesday morning, and I've been sitting in this very spot for nearly 26 hours straight excluding the total of ~20 minutes to use a washroom. It's time to change my focus, I need SOMETHING to show by the morning.

I'm fretting.
This isn't doing me any good. You have to get something working, so stop freaking out about failing. You won't fail, because you're better than that.

I opened up the MPC's manipulation module and manually disabled the left arm elbow and right arm wrist servos (8 and 13 respectively). This allowed for the thormang3_manipulation_manager node to be executed without that blasted -3002 error.

Inverse kinematics were chaotic as 2 crucial joints were not functioning. I designed motions to reach outward and do waving gestures which didn't require an elbow bend. The Lidar sensor also worked, so I quickly had a 3D point cloud generated and rendered in RViZ for a visual demo.
Rviz ros point cloud 3D using thormang3 lidar sensor
RVIZ 3D Point Cloud generation from Lidar
I'd like to say the morning came too fast; but really it arrived just on schedule, following physical laws and all that jazz.
Jacky rushed in by 9am, 30 minutes before the demo would begin. He has his own things to focus on so of course he wouldn't have the time to throw a night away fixing my problems.
We spent the next 30 minutes with me sprinting to and from his lab in the building next door.

He requested a USB2Dynamixel device, and to open up servo 8 and 13 in RoboPlus dynamixel wizard.
Usb2dynamixel connected to robotis servos

We did so, and the issue revealed itself....

Something flashed the firmware. The model ID for servo 8 was off, along with many low-level configurations...

Jacky exclaimed this was due to the fact that I let the battery run too low.
I had no damn clue something like voltage could flash the firmware.

        FACT: Software runs on hardware. Hardware needs attention too.

Not having enough time, we reset the servos and I loaded my motions as the administrators walked in. The demo overall was a close call; evading the motion demonstrations all so strategically. It was in an honest effort, we did have it working... yesterday.

The fact a 2nd-year undergraduate goofed up the kinematics has nothing to do with the fact that the NTNU lab is doing valuable research with the funding.
 All in all the gentlemen were satisfied with our demo. The Lidar point cloud scan worked just fine, which was fantastic considering I was creating a demo in real-time.
 I guess you could say, this was just a bit of Robot Magic at hand.

The whole demonstration was an elaborate performance really, despite the actual magic trick not yet working. We were honest about that, the motion wasn't complete; but everything else went smoothly after I disabled 8 and 13.

roboplus dynamixel wizard on thormang3 servos flashing firmware
Fixing the firmware on servos
Immediately after the demo me and Jacky spent the next 6 hours diagnosing the issue, which was indeed the firmware. Inevitably, we had things up and running again after a nearly complete disassembly of the left arm.

I've been writing a lot....


  Check your batteries.
 Mathematics holds true. 
Software holds to invariants... 
but Hardware couldn't care less.
When you're in a lab, in charge of a machine, you're in charge of ALL of it. Keep an eye out for hardware signs such as this instead of focusing only on the software paradigm, it could save you many hours.

This whole issue could have also been avoided, if I realized earlier on what was happening, in that on each reboot performance was decreasing.

Does that not appear to be a battery issue? It does, doesn't it?

I was too frantic with the deadline to realize the root problem, which I had to first fix to meet said deadline. This leads to the second important lesson....
Use deadlines as a guidance, but avoid letting them become a  hindrance and a stress inducer. If you can't meet a deadline, fretting about it wouldn't help before, during or after.
You have to remain fully calm in situations of high stress, and look at things from first principles.

What is happening RIGHT NOW, can you model it? Is there some noticeable reoccurring event or trend over time? What can you do to improve it?


This whole post is quite lengthy and jumbled, my apologies. I hope anyone involved as an undergraduate researcher can benefit from this and hopefully not repeat the mistakes I've made.

This isn't as specific as to pertain only to robotics... these are life lessons.
Manage your stress, and identify how you view deadlines.

Facing stress, deadlines, and unexpected events efficiently is a key component to making it further in an academic career.

No comments:

Post a Comment

Please feel free to give feedback/criticism, or other suggestions! (be gentle)

Subscribe to Updates via Email