haddock.libs.libgrid module

class haddock.libs.libgrid.CompositeGridJob(input: list[str], toppar_path: Path, module_path: Path)

Bases: GridInterface

create_job_script() → None

Create the job script that will be executed in the grid.

parse_input(input: Path | str | list[str]) → None

Read a concatenated list of .inp files and split them into separate recipes.

process_input_f() → None

Process each recipe section separately.

class haddock.libs.libgrid.GRIDScheduler(tasks: list[CNSJob], params: dict, probing: float = 0.05)

Bases: object

Scheduler to manage and run jobs on the GRID via DIRAC.

static calculate_optimal_batch_size(N: int, W: float, R: float, T: float) → int

Calculate the optimal batch size to achieve target efficiency.

create_batches() → None

Create batches of jobs to be submitted together.

probe_grid_efficiency() → None

Submit a small number of jobs to probe the efficiency of the GRID.

static process_job(job: GridJob) → None

Process a single job: update status, retrieve output if done, handle retries if failed.

NOTE: This function is parallelized, that is why things like cleaning, download, retry

are here. If you are adding new functionality, consider if it should be here or in the sequential part of the code.

run() → None

Execute the tasks.

submit_jobs(tag: Tag = Tag.DEFAULT) → None

Submit jobs to the GRID in parallel.

wait_for_completion() → None

Wait for jobs with status WAITING or RUNNING to complete.

class haddock.libs.libgrid.GridInterface(input: Path | str | list[str], toppar_path: Path, module_path: Path)

Bases: ABC

clean() → None

Clean up the temporary directory where the job lives.

clean_timings() → None

Clean the timings dictionary.

create_jdl() → None

Create the JDL file that describes the job to DIRAC.

abstract create_job_script() → None

Create the job script that will be executed in the grid.

package() → None

abstract parse_input(input: Path | str | list[str]) → None

placeholder

static parse_output(output_str: str) → dict[str, str]

Parse the output string from DIRAC commands into a dictionary.

prepare_payload(cns_script_path: Path, toppar_path: Path) → None

Prepare the payload.zip file containing all necessary files.

abstract process_input_f() → None

Process the input file to adjust paths and identify outputs.

retrieve_output() → None

Retrieve the output files from DIRAC.

The dirac-wms-job-get-output command downloads the output sandbox, this means that

anything that was specified in the OutputSandbox section of the JDL file will be put in the current working directory following the pattern: working_dir/job_id/

submit() → None

Interface to submit the job to DIRAC.

update_status() → None

Update the status of the job by querying DIRAC.

class haddock.libs.libgrid.GridJob(input: Path | str, toppar_path: Path, module_path: Path)

Bases: GridInterface

GridJob is a class tha represents a job to be run on the GRID via DIRAC.

create_job_script() → None

Create the job script that will be executed in the grid.

parse_input(input: Path | str | list[str]) → None

placeholder

process_input_f() → None

Process the input file to adjust paths and identify outputs.

class haddock.libs.libgrid.JobStatus(value)

Bases: Enum

An enumeration.

COMPLETING = 'Completing'

DONE = 'Done'

FAILED = 'Failed'

MATCHED = 'Matched'

RUNNING = 'Running'

STAGED = 'Staged'

UNKNOWN = 'Unknown'

WAITING = 'Waiting'

classmethod from_string(value)

Convert string to JobStatus enum.

class haddock.libs.libgrid.Tag(value)

Bases: Enum

An enumeration.

DEFAULT = 'Default'

PROBING = 'Probing'

haddock.libs.libgrid.ping_dirac() → bool

Ping the Dirac server to check if it’s reachable.

haddock.libs.libgrid.validate_dirac() → bool

Check if the DIRAC client is valid and configured.