Solving the entire dataset

To solve the entire dataset, run:

Copy

Copy

python -m alpha_codium.solve_dataset \
--dataset_name /path/to/dataset \
--split_name test
--database_solution_path /path/to/output/dir/dataset_output.json

• Parameters:

  • split_name: Could be either valid or test.

  • database_solution_path: Path to the directory where solutions will be saved

  • The dataset section in the configuration file contains the configuration for the running and evaluation of a dataset.

dataset.num_iterations defines the number of iterations for each problem (pass@K). For a large number of iterations, it is recommended to introduce some randomness and different options for each iteration to achieve top results.

Important Note: Solving the entire dataset is a long process, and it may take a few days to complete with large models (e.g. GPT-4) and several iterations per problem.