9. GridPath Architecture

This section describes GridPath’s modular architecture, i.e. how the modules are linked and work together (as opposed to their formulation or the functionality they add).

9.1. GridPath Modules

9.1.1. gridpath.auxiliary.module_list

This module contains:

1. the list of all GridPath modules;

2. the modules included in each optional feature;

3. the ‘cross-feature’ modules;

4. the method for determining the user-requested features for the scenarios;

5. the method for loading modules.

all_modules_list()

Returns:

list of all GridPath modules in order they are loaded

This is the list of all GridPath modules in the order they would be loaded if all optional features were selected.

cross_feature_modules_list()

Returns:

dictionary with a tuple of features as keys and a list of modules to be included if all those features are selected as values

Some modules depend on more than one feature, i.e. they are included only if multiple features are selected. These relationships are described in the ‘cross_modules’ dictionary here.

determine_modules(features=None, scenario_directory=None, multi_stage=None)

Parameters:

• features – List of requested features. Optional input; if not specified, function will try to load ‘features.csv’ file to determine the requested features.

• scenario_directory – the scenario directory, where we will look for the list of requested features. Optional input; if not specified, function will look for the ‘features’ input parameter

• multi_stage – Boolean. Optional input that determines whether the modules that fix variables are used (yes if True, no if False); if not specified, this function will check the scenario_directory to determine whether there are stage subdirectories (if there are not, the ‘fix variables’ modules are removed).

Returns:

the list of modules – a subset of all GridPath modules – needed for a scenario. These are the module names, not the actual modules.

This method determines which modules are needed for a scenario based on the features specified for the scenario. The features can be either directly specified as a list or by providing the directory where a ‘features.csv’ file lists the requested features.

We start with the list of all GridPath modules from all_modules_list() as the list of modules to use in the scenario. We then iterate over all optional features, which we get from the keys of the optional_modules_list() method above; if the feature is in the list of user-requested features, we do nothing; if it is not, we remove all of the feature’s modules from the list of modules to use. Similarly, for the cross feature modules, which we get from the cross_feature_module_list() method, we check if all features they depend on are included and, if not, remove those modules from the list of modules to use.

load_modules(modules_to_use)

Parameters:

modules_to_use – a list of the names of the modules to use

Returns:

list of imported modules (Python <class ‘module’> objects)

Load the requested modules and return them as a list of Python module objects.

optional_modules_list()

Returns:

dictionary with the optional feature names as keys and a list of the modules included in each feature as values

These are all of GridPath’s optional modules grouped by features (features as the dictionary keys). Each of these modules belongs to only one feature.

9.1.2. gridpath.auxiliary.dynamic_components

This module creates the DynamicComponents class, which contains the lists and dictionaries of the names of dynamic optimization components. These are components that are populated by GridPath modules based on the selected features and the scenario input data.

class DynamicComponents

Here we initialize the class object and its components that will contain the dynamic model components, i.e. lists and dictionary with the names of the optimization components that are populated based on whether features are selected (i.e. certain modules are called) and based on the scenario input data.

9.2. Running GridPath

9.2.1. Running a Scenario

gridpath.run_scenario

This script runs a GridPath scenario. It assumes that scenario inputs have already been written.

The main() function of this script can also be called with the gridpath_run command when GridPath is installed.

create_abstract_model(model, dynamic_components, loaded_modules, scenario_directory, weather_iteration, hydro_iteration, availability_iteration, subproblem, stage)

Parameters:

• model – the Pyomo AbstractModel object

• dynamic_components – the populated dynamic model components class

• loaded_modules – list of the required modules as Python objects

• scenario_directory –

• subproblem –

• stage –

To create the abstract model, we iterate over all required modules and call their add_model_components method to add components to the Pyomo AbstractModel. Some modules’ add_model_components method also require the dynamic component class as an argument for any dynamic components to be added to the model.

create_problem(scenario_directory, weather_iteration, hydro_iteration, availability_iteration, subproblem, stage, multi_stage, parsed_arguments, timing_summary_file_path=None)

Parameters:

• scenario_directory – the main scenario directory

• subproblem – the horizon subproblem name

• stage – the stage subproblem name

• parsed_arguments – the user-defined script arguments

• timing_summary_file_path – the timing summary file path (None if not logging)

Returns:

modules_to_use (list of module names used in scenario), loaded_modules (Python objects), dynamic_inputs (the populated dynamic components class), instance (the problem instance), results (the optimization results)

This method creates the problem instance.

To create the problem, we use a Pyomo AbstractModel() class. We will add Pyomo optimization components to this class, will load data into the components, and will then compile the problem.

We first need to determine which GridPath modules we need to use. See determine_modules method (imported from gridpath.auxiilary.module_list) and import those modules (via the load_modules method imported from gridpath.auxiliary.module_list).

We then determine the dynamic model components based on the selected modules and input data. See populate_dynamic_components method.

The next step is to create the abstract model (see create_abstract_model method) and load the input data into its components (see load_scenario_data).

Finally, we compile the problem (see create_problem_instance method). If any variables need to be fixed, this is done as the last step here (see the fix_variables method).

create_problem_instance(model, loaded_data)

Parameters:

• model – the AbstractModel Pyomo object with components added

• loaded_data – the DataPortal object with the data loaded in and linked to the relevant model components

Returns:

the compiled problem instance

Compile the problem based on the abstract model formulation and the data loaded into the model components.

fix_variables(instance, dynamic_components, scenario_directory, weather_iteration, hydro_iteration, availability_iteration, subproblem, stage, loaded_modules)

Parameters:

• instance – the compiled problem instance

• dynamic_components – the dynamic component class

• scenario_directory – str

• subproblem – str

• stage – str

• loaded_modules – list of imported GridPath modules as Python objects

Returns:

the problem instance with the relevant variables fixed

Iterate over the required GridPath modules and fix variables by calling the modules’ fix_variables, if applicable. Return the modified problem instance with the relevant variables fixed.

load_scenario_data(model, dynamic_components, loaded_modules, scenario_directory, weather_iteration, hydro_iteration, availability_iteration, subproblem, stage)

Parameters:

• model – the Pyomo abstract model object with components added

• dynamic_components – the dynamic components class

• loaded_modules – list of the imported GridPath modules as Python objects

• scenario_directory – the main scenario directory

• subproblem – the horizon subproblem

• stage – the stage subproblem

Returns:

the DataPortal object populated with the input data

Iterate over all required GridPath modules and call their load_model_data method in order to load input data into the relevant model components. Return the resulting DataPortal object with the data loaded in.

main(args=None)

This is the ‘main’ method that runs a scenario. It takes in and parses the script arguments, determines the scenario structure (i.e. whether it is a single optimization or has subproblems), and runs the scenario. This method also returns the objective function value(s).

parse_arguments(args)

Parameters:

args – the script arguments specified by the user

Returns:

the parsed known argument values (<class ‘argparse.Namespace’> Python object)

Parse the known arguments.

run_optimization_for_subproblem(scenario_directory, weather_iteration_directory, hydro_iteration_directory, availability_iteration_directory, subproblem_directory, stage_directories, multi_stage, parsed_arguments, objective_values)

Check if there are stages in the subproblem; if not solve subproblem; if, yes, solve each stage sequentially

run_scenario(scenario_directory, scenario_structure, parsed_arguments)

Check the scenario structure, iterate over all subproblems if they exist, and run the subproblem optimization.

The objective function is returned, but it’s only really used if we are in ‘testing’ mode.

Parameters:

• scenario_directory – scenario directory path

• scenario_structure – the subproblem structure object

• parsed_arguments –

Returns:

the objective function value (NPV); only used in ‘testing’ mode.

solve(instance, parsed_arguments)

Parameters:

• instance – the compiled problem instance

• parsed_arguments – the user-defined arguments (parsed)

Returns:

the problem results

Send the compiled problem instance to the solver and solve.

9.2.2. Database Access

gridpath.get_scenario_inputs

This script iterates over all modules required for a GridPath scenario and calls their write_model_inputs() method, which queries the GridPath database and writes the .tab input files to the scenario directory.

The main() function of this script can also be called with the gridpath_get_inputs command when GridPath is installed.

Two mechanisms can restrict the run to part of the scenario’s structure:

• --temporal_structure_csv_overwrite with --temporal_structure_csv_path REPLACES the database-derived scenario structure with the CSV’s (columns: weather_iteration, hydro_iteration, availability_iteration, subproblem, stage; one row per subproblem/stage). Fully general – any subset of cells, down to individual subproblems/stages. Every listed combination is validated to exist in the scenario’s database-derived structure (combinations outside it have no input data and would otherwise fail late); the directory-layout flags are derived from the CSV’s values (see get_scenario_structure_from_csv).

• The --single_draw WEATHER HYDRO AVAILABILITY option (0 for iteration levels the scenario doesn’t use) SLICES the resolved structure to a single iteration draw. Use, for example, for re-materializing one draw of a scenario directory that was cleaned after import. The draw is checked to exist, and the scenario’s directory-layout flags are preserved. The slice applies to whatever structure was resolved.

With either mechanism, the post-import cleanup marker (if present) is left in place, since the rest of the tree is still in its cleaned state; solve the regenerated part with run_scenario’s --ignore_cleanup_marker. Only a full regeneration of the database-derived structure removes the marker.

gridpath.import_scenario_results

This script iterates over all modules required for a GridPath scenario and calls their import_results_into_database() method, which loads the scenario results files into their respective database table.

The main()_ function of this script can also be called with the gridpath_import_results command when GridPath is installed.

This script always deletes ALL of the scenario’s prior database results before importing. Note the implication for --temporal_structure_csv_overwrite: only the CSV’s iteration/subproblem/stage cells are then imported, so on a scenario that already has results, the import replaces them with the CSV’s subset only (intended for scenarios with no prior results for other cells, e.g. gridpath_iterate’s freshly cloned conditions-only re-runs).

There is deliberately no single-draw mode here: gridpath_run_e2e --per_draw_lifecycle --single_draw instead runs the per-draw machinery (run_end_to_end_per_draw), whose importer deletes and re-imports ONLY the requested draw’s results (via delete_scenario_results_for_draw) and never calls this script’s delete-all step. Passing –single_draw to this script is refused rather than silently ignored, since the user would get the delete-all behavior they were specifically trying to avoid.

The import assigns each (weather iteration, hydro iteration, availability iteration, subproblem, stage) an import status – see the IMPORT_STATUS_* constants – and main() returns the statuses as a dictionary keyed by that tuple. If results were imported for none or only some of the subproblems/stages, a warning is printed regardless of the –quiet setting: skipped subproblems are otherwise silent, and, since all prior results for the scenario are deleted at the start of the import step, an import that skips everything leaves the scenario with no results in the database while appearing to have succeeded. Subproblems/stages skipped because the solver status was not ‘ok’ are reported by their solver termination condition (e.g. ‘infeasible’), so the warning says WHY there are no results for them.

gridpath.process_results

This script iterates over all modules required for a GridPath scenario and calls their process_results() method, which makes updates to database tables.

The main() function of this script can also be called with the gridpath_process_results command when GridPath is installed.

9.2.3. Running End-to-End

gridpath.run_end_to_end

This script calls the __main__ functions of get_scenario_inputs.py, run scenario.py, import_scenario_results.py, and process_results.py to run a scenario end-to-end, i.e. get the scenario inputs from the database, solve the scenario problem, import the results the database and perform any necessary results-processing.

The main() function of this script can also be called with the gridpath_process_results command when GridPath is installed.

gridpath.run_end_to_end_per_draw

Per-draw end-to-end mode (gridpath_run_e2e --per_draw_lifecycle) for Monte Carlo scenarios with many iteration draws.

Instead of writing all inputs, solving everything, and then importing everything – which materializes the entire scenario directory (millions of small files for large Monte Carlo cases) before any of it can be reclaimed – this mode pipelines the run one iteration draw (weather iteration, hydro iteration, availability iteration) at a time:

• The main loop writes a batch of draws’ inputs (--n_draws_per_solve_batch draws per batch, default 1), solves the batch with one run_scenario call – whose --n_parallel_solve pool parallelizes over the batch’s draws x subproblems – and hands the solved batch to the importer queue.

• A single importer thread – the only database writer – imports queued batches while other draws are still solving, and, with --cleanup_after_import/--archive_after_import, cleans each draw’s directory as soon as its batch’s import succeeds. The queue is bounded (--max_draws_pending_import batches), so if importing falls behind, solving pauses and the on-disk footprint stays bounded.

• The database is switched to WAL journal mode for the duration of the run so the main loop’s input-writing reads can proceed alongside the importer’s writes; the prior journal mode is restored at the end.

Choosing the parallelization settings (peak on-disk footprint is about (1 + –max_draws_pending_import) x –n_draws_per_solve_batch draws):

• --n_parallel_solve is the CPU (and memory) knob: each in-flight subproblem occupies roughly one core – assuming single-threaded solver settings; if the solver is configured to use multiple threads, budget cores ~= –n_parallel_solve x solver threads instead – plus the memory for its model and the solver’s workspace. Start at about the machine’s core count minus one (the importer thread and the main loop overlap with solving), and lower it if memory binds first: in-flight subproblems x per-subproblem peak memory must fit in RAM.

• --n_draws_per_solve_batch is NOT a CPU knob – it only determines how much work the pool can see at once. Actual concurrency is min(–n_parallel_solve, batch size x subproblems per draw), so make the batch just large enough to feed the pool:

  • Many subproblems per draw (e.g. weekly subproblems over a year): the default batch of 1 already offers a full pool of tasks; leave it.

  • One subproblem per draw: set the batch to –n_parallel_solve (e.g. on a 10-core budget, both 10) – with the default batch of 1, the draws solve sequentially no matter how many cores are available.

  • In-between shapes: the smallest batch with batch size x subproblems per draw >= –n_parallel_solve.

• Batches larger than needed add no speed – the pool caps concurrency – and only raise the disk footprint and delay each batch’s import/cleanup (a batch is imported only once it has fully solved). One exception: if solve times vary a lot across a batch, workers idle while the last tasks finish, so a batch of 2-3x the pool size amortizes that end-of-batch tail at proportionally higher footprint.

Each draw’s import is idempotent: the importer first deletes the draw’s prior database rows, so a crashed or killed run can simply be re-run. Completed draws are recognized on re-run by their rows in the cleanup marker file (with cleanup/archiving on) and are skipped entirely; all other draws are re-solved by default, exactly like the classic whole-scenario mode – pass --incomplete_only to skip re-solving subproblems whose results are already on disk.

Some more notes:

Linked-subproblem scenarios are refused: subproblems then depend on each other’s inputs and the draws cannot be processed independently.

--temporal_structure_csv_overwrite works with this mode: the draws are then iterated from the CSV’s structure instead of the database’s, so a per-draw run can be restricted to a subset of the scenario’s draws. The CSV must list WHOLE draws here (each processed draw’s database results are deleted in full before its re-import, so a partial draw would lose its unlisted subproblems’ results – refused with a clear error); sub-draw subsets belong in the classic pipeline.

gridpath_run_e2e --per_draw_lifecycle --single_draw WEATHER HYDRO AVAILABILITY (0 for iteration levels the scenario doesn’t use; the two flags are required together, –single_draw being a selector for this mode) runs this same machinery for one requested draw: its inputs are (re)written, it is solved and imported – deleting only THIS draw’s prior database rows, so the scenario’s other results are untouched – and it is cleaned/archived if those options are set. An explicitly requested draw is never skipped as already-completed, and a scenario directory cleaned after import needs no special handling (the draw is simply re-materialized). This is the one-command way to re-run or debug a single draw of a large Monte Carlo case.

gridpath.scenario_directory_cleanup

Optional lifecycle management for the on-disk scenario directory after results have been imported into the database.

For database-driven workflows the scenario directory is a regenerable intermediate: inputs are written from the database and results are read back into it. Monte Carlo cases with many iterations accumulate millions of small files, so gridpath_run_e2e offers --cleanup_after_import (delete) and --archive_after_import (one tarball per iteration “draw”, then delete) to reclaim the directory once its contents are safely in the database.

The unit of cleanup is one iteration draw – a (weather iteration, hydro iteration, availability iteration) directory path, or the scenario directory’s own contents when the scenario has no iteration levels. A draw is only cleaned if EVERY one of its subproblems/stages has import status “imported” (see import_scenario_results); by default, draws with any skipped or failed subproblem are left fully intact. With --cleanup_granularity subproblem, the imported subproblems WITHIN such partially imported draws are cleaned too, retaining only the not-imported subproblems (useful when a single stuck subproblem would otherwise strand a large draw on disk); fully imported draws are still cleaned as whole draws, so re-run resume bookkeeping is the same at both granularities.

Retained in all cases: the scenario-level files (scenario_description.csv, features.csv, solver_options.csv, units.csv, multi_stage_flag.txt, linked_subproblems_map.csv) and the scenario-level logs directory (small, and the only non-regenerable content).

Cleanup writes a marker file (scenario_directory_cleaned.csv, one row per cleaned draw) to the scenario directory. Each entry point has a deliberate, different relationship with the marker:

• import_scenario_results refuses to run on a marked directory, with NO override: it deletes all of the scenario’s database results before importing, so importing from a cleaned directory would wipe the results and find nothing to re-import. (gridpath_run_e2e --per_draw_lifecycle --single_draw is the sanctioned way to re-import one draw: it deletes only that draw’s rows.)

• run_scenario refuses unless passed --ignore_cleanup_marker: the scenario structure is inferred from the directory tree, so a partially cleaned tree yields a silently wrong structure – but deliberately solving just what is on disk (a re-materialized subset) is a legitimate, explicit choice.

• get_scenario_inputs is never blocked – it is the recovery path: a full regeneration of the database-derived structure removes the marker, while a partial regeneration (--single_draw or a temporal-structure CSV) leaves it in place, since the rest of the tree is still cleaned.

9.2.4. Input Validation

gridpath.validate_inputs

This script iterates over all modules required for a GridPath scenario and calls their validate_inputs() method, which performs various validations of the input data and scenario setup.