Franka

We assume that you have a NUC running with a real-time kernel as the Franka controller, and a separate workstation for policy inference, on the same network.

Gripper hardware setup

Follow this guide to assemble the gripper: Assembly Guide

Have the following materials ready:

• Download and print the gripper mount

• 4 M6 x 10mm Screws

• 4 M2.5 x 10mm Screws

• The Dynamixel Starter Set, which includes parts 2-6 in the photo below:

1. Connect the Dynamixel Motor to the U2D2 Power Hub. You will hear a clicking sound when connected:

1. Connect the Power Hub to the U2D2 Board

1. Connect the U2D2 and your computer with the USB cable. A red light will turn on.

1. Plug the 12V 5A power supply into the U2D2 Power Hub board and switch on the power. A red LED should turn on.

Go to the robot-server directory and run this script to initialize the gripper:

cd robot-server
python gripper_init.py

1. Attach gripper to the mount with M2.5 x 10mm screws

1. Mount your 3D printed franka mount with M6 x 10mm screws

NUC Set-Up

1. SSH into the NUC hosting your Franka.

ssh -o LogLevel=ERROR -D 1337 user@nuc_IP

Host nuc
    HostName (nuc_IP)
    User ???
    LogLevel ERROR
    DynamicForward 1337

1. Setup Deoxys, select 0.13.3 for libfranka when prompted

git clone [e-mail]:NYU-robot-learning/deoxys_control.git
cd deoxys_control/deoxys

# Instructions from deoxys repo (this takes a while to build everything)
./InstallPackage
make -j build_deoxys=1

1. Input necessary IPs into the following config

./deoxys_control/deoxys/config/franka_config.yml

PC:
  NAME: "lambda"
  IP: ??? ## <<<<<<

NUC:
  NAME: "nuc"
  IP: ??? ## <<<<<<
  PUB_PORT: 5566
  SUB_PORT: 5565
  GRIPPER_PUB_PORT: 5568
  GRIPPER_SUB_PORT: 5567

ROBOT:
  IP: ??? ## <<<<<<
...

1. Run the following from the deoxys directory to start deoxys server.

• Errors will repeat until the next section is complete

./auto_scripts/auto_arm.sh config/franka_config.yml

Franka Setup

1. Setup and activate a proxy in your browser (e.g., FoxyProxy on Firefox).

   • Example FoxyProxy config:

1. Connect to Franka Desk using the Franka IP inputted into the config.

• Where the URL is, you'll see the input is an IP:

1. Unlock the joints in Franka Desk (open lock icon under the Joints tab).

1. Set Execution mode (bottom-right in the UI).

1. Activate FCI mode:

• Click the IP (top-right) → Activate FCI Mode. The arm indicator should turn green (Not guaranteed for all versions of Franka):

1. Review NUC SSH terminal that errors have stopped.

2. Download the Franka gripper profile below

468B

Nrum.endeffector-profile.zip

archive

Open

1. Head to Settings in Franka Desk

1. Click on End Effector

1. Import the JSON from the downloaded zip

1. Activate the Gripper Profile

Deploying a Trained Policy

1. Set up the iPhone on the robot:

   • Slide the iPhone into the gripper mount until the camera block hits the mount, then tighten the knob (snug, not over-tight).

• Connect the iPhone to the robot via USB.

• In Record3D app settings: set RGBD Streaming mode to USB and FPS to 30.

• Press Record to enter “stream-ready” mode.

Unless otherwise specified, the following steps assume that we are in the min-stretch repo root directory. 2. Set up Deoxys

git clone [e-mail]:NYU-robot-learning/deoxys_control.git
conda activate home_robot
cd deoxys_control/deoxys

# Instructions from deoxys repo (this takes a while to build everything)
./InstallPackage
make -j build_deoxys=1
pip install -U -r requirements.txt
pip install -e .

1. Edit the following config and input necessary IPs

./min-stretch/robot-server/robot/franka/deoxys_utils/configs/franka_config.yml

PC:
  NAME: "lambda"
  IP: ??? ## <<<<<<

NUC:
  NAME: "nuc"
  IP: ??? ## <<<<<<
  PUB_PORT: 5566
  SUB_PORT: 5565
  GRIPPER_PUB_PORT: 5568
  GRIPPER_SUB_PORT: 5567

ROBOT:
  IP: ??? ## <<<<<<
...

1. Calibrate Gripper:

   cd robot-server
   conda activate home_robot
   python auto_calibrate_gripper.py

2. Start the robot server:

   python3 start_server.py controller=franka

   This starts streaming images from the iPhone and the robot begins listening for action commands. If you have missing imports, try pip3 install hydra-core record3d (and any other missing packages).

3. Run the policy in another terminal (requires display):

   cd imitation-in-homes
   conda activate home_robot
   python run.py

4. Head to the Playbook

---