parallel
Dask helper functions.
configure_dask_scheduler(scheduler_address, name, nproc=1)
Configure the Dask scheduler by reusing existing or creating a new cluster.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scheduler_address
|
str | Cluster | None
|
Address of the Dask scheduler to connect to, or None for local cluster. |
required |
name
|
str
|
Name for the Dask cluster. |
required |
nproc
|
int
|
Number of processes to use per worker for CPU support. |
1
|
Returns:
| Type | Description |
|---|---|
str | Cluster
|
A Dask Cluster instance or a string address for the scheduler. |
nr_cpus()
Determine the number of CPU cores to use.
If the environment variables SLURM_CPUS_PER_TASK or OMP_NUM_THREADS are set, their value is used. Otherwise, the number of physical CPU cores is returned.
Returns:
| Type | Description |
|---|---|
int
|
The number of CPU cores to use. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the number of physical CPU cores cannot be determined. |