Skip to main content
Parallelize your sweep agents on a multi-core or multi-GPU machine to run multiple sweep runs at the same time. Parallelization reduces the time needed to explore your hyperparameter space. This page shows you how to launch parallel agents on multi-CPU and multi-GPU machines so you can use all available compute on a single host. Before you start, you must initialize your sweep. For more information, see Initialize sweeps.

​
Parallelize on a multi-CPU machine

The following tabs describe how to run multiple sweep agents in parallel on the same machine. Parallel agents are most useful on a machine with multiple CPU cores, where each agent can run on its own core. Choose the tab for the CLI or a Jupyter Notebook based on your workflow.
Use the wandb agent command to parallelize your sweep agent across multiple CPUs with the terminal. Provide the sweep ID that W&B returned when you initialized the sweep.
  1. Open more than one terminal window on your local machine.
  2. Copy and paste the following code snippet and replace [SWEEP-ID] with your sweep ID:

​
Parallelize on a multi-GPU machine

Run sweep agents in parallel across multiple GPUs on the same machine by using CUDA_VISIBLE_DEVICES to assign each wandb agent to a different GPU, so the agents run in parallel without competing for the same device. To follow this procedure, you need:
  • A machine with more than one GPU.
  • The NVIDIA CUDA Toolkit installed.
  • A terminal.
For example, suppose you have two NVIDIA GPUs on your local machine. Replace [SWEEP-ID] in each command with the sweep ID that W&B returns when you initialize a sweep:
  1. In one terminal, set CUDA_VISIBLE_DEVICES to 0 and start an agent:
  2. In a separate terminal, set CUDA_VISIBLE_DEVICES to 1 and start another agent with the same sweep ID: