Workstation¶

Interactive development, light experiments, dataset prep. Push heavy training to Torch.

Machines¶

DHCP IP address: 10.21.35.152 (requires NYU VPN)
Connecting with Tailscale (Recommended): download Tailscale and find the “dl” machine under name "Yanlai Yang" in the devices list. Works without NYU VPN and when DHCP IP address changes.
Physical location: Desk 428C
GPU: 4x NVIDIA RTX A4000 GPUs (16 GB VRAM each)
CPU: 64x AMD Ryzen Threadripper PRO 5975WX
Memory: 252 GB of RAM
OS: Ubuntu 20.04 LTS (GNU/Linux 5.15.0-107-generic x86_64)

Admin (PI / lab admin) SSHes in as bizon:
```
ssh bizon@10.21.35.152
```
Password lives in the lab password manager.
Admin creates the new account:
```
sudo adduser <username>
```
New user logs in:
```
ssh <username>@10.21.35.152
```

Set up ssh-key auth so you don't type the password every time. On your local machine:

ssh-keygen          # skip if you already have a key
ssh-copy-id <username>@10.21.35.152

Add CUDA to your PATH. Append to ~/.bashrc on the workstation, then exit and re-login:

export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Verify GPUs and CUDA version:

nvidia-smi   # should list the GPUs
nvcc -V      # CUDA version (≥ 11.4 per the 2022 doc; likely higher now)

Workstations are for debugging; push training to Torch. Brief heavy use during deadlines is OK.

No experiments as admin/root (Docker is fine).
For heavy use during deadlines, coordinate with other users about resource allocation and stay within your GPU / CPU / RAM share. Borrow by asking first.
Limit CPU cores with taskset:
```
taskset -c 0-16 python train.py
```

Limit GPUs with CUDA_VISIBLE_DEVICES:

CUDA_VISIBLE_DEVICES=1,2 python train.py

Combined:

CUDA_VISIBLE_DEVICES=1,2 taskset -c 0-16 python train.py

Compute-intensive programs running as admin/root can be killed by anyone without notice.
If someone violates their resource share, you may kill their program with 1 hour's notice.
Do not use root permissions on other users' jobs.
Avoid the admin account in general; it's for setup, not work.
Careful with shared settings (NVIDIA driver, kernel modules, Docker); they break others' code.
Always ask before rebooting.