imputegap.recovery.benchmark package¶

Module contents¶

class imputegap.recovery.benchmark.Benchmark[source]¶

Bases: object

A class to evaluate the performance of imputation algorithms through benchmarking across datasets and patterns.

Methods¶

average_runs_by_names(self, data):

Average the results of all runs depending on the dataset.

avg_results():

Calculate average metrics (e.g., RMSE) across multiple datasets and algorithm runs.

generate_heatmap():

Generate and save a heatmap visualization of RMSE scores for datasets and algorithms.

generate_reports_txt():

Create detailed text-based reports summarizing metrics and timing results for all evaluations.

generate_reports_excel():

Create detailed excel-based reports summarizing metrics and timing results for all evaluations.

generate_plots():

Visualize metrics (e.g., RMSE, MAE) and timing (e.g., imputation, optimization) across patterns and datasets.

eval():

Perform a complete benchmarking pipeline, including contamination, imputation, evaluation, and reporting.

Example¶

output : {‘eegalcohol’: {‘mcar’: {‘MeanImpute’: {‘default_params’: {‘0.05’: {‘scores’: {‘RMSE’: 1.107394798606378, ‘MAE’: 0.9036474830477748, ‘CORRELATION’: nan, ‘RUNTIME’: 10.07390022277832, ‘RUNTIME_LOG’: 1.00319764506136}}, ‘0.1’: {‘scores’: {‘RMSE’: 0.8569349076796438, ‘MAE’: 0.6416542359734557, ‘CORRELATION’: nan, ‘RUNTIME’: 1.0, ‘RUNTIME_LOG’: 0.0}}, ‘0.2’: {‘scores’: {‘RMSE’: 0.9924113085421721, ‘MAE’: 0.7939689811173046, ‘CORRELATION’: nan, ‘RUNTIME’: 1.0, ‘RUNTIME_LOG’: 0.0}}, ‘0.4’: {‘scores’: {‘RMSE’: 1.0058063455061463, ‘MAE’: 0.8076546785476064, ‘CORRELATION’: nan, ‘RUNTIME’: 1.0, ‘RUNTIME_LOG’: 0.0}}, ‘0.6’: {‘scores’: {‘RMSE’: 0.9891809506243663, ‘MAE’: 0.7914550709031675, ‘CORRELATION’: nan, ‘RUNTIME’: 1.0, ‘RUNTIME_LOG’: 0.0}}, ‘0.8’: {‘scores’: {‘RMSE’: 0.9927953862507292, ‘MAE’: 0.7925635744718286, ‘CORRELATION’: nan, ‘RUNTIME’: 1.0, ‘RUNTIME_LOG’: 0.0}}}}, ‘SoftImpute’: {‘default_params’: {‘0.05’: {‘scores’: {‘RMSE’: 0.4359915238078244, ‘MAE’: 0.3725965559420608, ‘CORRELATION’: 0.9530448037164908, ‘RUNTIME’: 199.30577278137207, ‘RUNTIME_LOG’: 2.2995198779819055}}, ‘0.1’: {‘scores’: {‘RMSE’: 0.3665001858394363, ‘MAE’: 0.2989983612840734, ‘CORRELATION’: 0.9049909722894052, ‘RUNTIME’: 117.54822731018066, ‘RUNTIME_LOG’: 2.0702160841184516}}, ‘0.2’: {‘scores’: {‘RMSE’: 0.39833006221984, ‘MAE’: 0.30824644022807457, ‘CORRELATION’: 0.9161465703422209, ‘RUNTIME’: 317.5652027130127, ‘RUNTIME_LOG’: 2.5018329084349737}}, ‘0.4’: {‘scores’: {‘RMSE’: 0.435591016228979, ‘MAE’: 0.3335144215651955, ‘CORRELATION’: 0.9021032587324183, ‘RUNTIME’: 302.2916316986084, ‘RUNTIME_LOG’: 2.4804261248244566}}, ‘0.6’: {‘scores’: {‘RMSE’: 0.4500113661547204, ‘MAE’: 0.338085865703361, ‘CORRELATION’: 0.8893263437029546, ‘RUNTIME’: 314.93282318115234, ‘RUNTIME_LOG’: 2.498217926383076}}, ‘0.8’: {‘scores’: {‘RMSE’: 0.46554422402146944, ‘MAE’: 0.3508926604243284, ‘CORRELATION’: 0.8791443563129441, ‘RUNTIME’: 311.9697570800781, ‘RUNTIME_LOG’: 2.4941124947560986}}}}}}}

average_runs_by_names(data)[source]¶

Average the results of all runs depending on the dataset

Parameters¶

datalist

list of dictionary containing the results of the benchmark runs.

Returns¶

list

list of dictionary containing the results of the benchmark runs averaged by datasets.

avg_results(*datasets, metric='RMSE')[source]¶

Calculate the average of all metrics and times across multiple datasets.

Parameters¶

datasetsdict

Multiple dataset dictionaries to be averaged.

metricstr

Metric to group.

Returns¶

List

Matrix with averaged scores and times for all levels, list of algorithms, list of datasets

eval(algorithms=['cdrec'], datasets=['eeg-alcohol'], patterns=['mcar'], x_axis=[0.05, 0.1, 0.2, 0.4, 0.6, 0.8], optimizers=['default_params'], metrics=['*'], save_dir='./imputegap_assets/benchmark', runs=1, normalizer='z_score', nbr_series=2500, nbr_vals=2500, dl_ratio=0.9, verbose=False)[source]¶

Execute a comprehensive evaluation of imputation algorithms over multiple datasets and patterns.

Parameters¶

algorithmslist of str

List of imputation algorithms to test.

datasetslist of str

List of dataset names to evaluate.

patternslist of str

List of contamination patterns to apply.

x_axislist of float

List of missing rates for contamination.

optimizerslist

List of optimizers with their configurations.

metricslist of str

List of metrics for evaluation.

save_dirstr, optional

Directory to save reports and plots (default is “./reports”).

runsint, optional

Number of executions with a view to averaging them

normalizerstr, optional

Normalizer to pre-process the data (default is “z_score”).

nbr_seriesint, optional

Number of series to take inside the dataset (default is 2500 (as the max values)).

nbr_valsint, optional

Number of values to take inside the series (default is 2500 (as the max values)).

dl_ratiofloat, optional

Training ratio for Deep Learning techniques (default is 0.8)

verbosebool, optional

Whether to display the contamination information (default is False).

Returns¶

List

List of all runs results, matrix with averaged scores and times for all levels

Notes¶

Runs contamination, imputation, and evaluation, then generates plots and a summary reports.

generate_heatmap(scores_list, algos, sets, metric='RMSE', save_dir='./reports', display=True)[source]¶

Generate and save RMSE matrix in HD quality.

Parameters¶

scores_listnp.ndarray

2D numpy array containing RMSE values.

algoslist of str

List of algorithm names (columns of the heatmap).

setslist of str

List of dataset names (rows of the heatmap).

metricstr, optional

metric to extract

save_dirstr, optional

Directory to save the generated plot (default is “./reports”).

displaybool, optional

Display or not the plot

Returns¶

Bool

True if the matrix has been generated

generate_plots(runs_plots_scores, ticks, metrics=None, subplot=False, y_size=4, title=None, save_dir='./reports', display=False, verbose=True)[source]¶

Generate and save plots for each metric and pattern based on provided scores.

Parameters¶

runs_plots_scoresdict

Dictionary containing scores and timing information for each dataset, pattern, and algorithm.

tickslist of float

List of missing rates for contamination.

metricslist of string

List of metrics used

subplotbool, optional

If True, generates a single figure with subplots for all metrics (default is False).

y_sizeint, optional

Default size of the graph (default is 4).

titlestr, optional

Title of the graph (default is “imputegap benchmark”).

save_dirstr, optional

Directory to save generated plots (default is “./reports”).

displaybool, optional

Display or not the plots (default is False).

verbosebool, optional

Whether to display the contamination information (default is True).

Returns¶

None

Notes¶

Saves generated plots in save_dir, categorized by dataset, pattern, and metric.

generate_reports_excel(runs_plots_scores, save_dir='./reports', dataset='', run=-1, verbose=True)[source]¶

Generate and save an Excel-like text report of metrics and timing for each dataset, algorithm, and pattern.

Parameters¶

runs_plots_scoresdict

Dictionary containing scores and timing information for each dataset, pattern, and algorithm.

save_dirstr, optional

Directory to save the Excel-like file (default is “./reports”).

datasetstr, optional

Name of the data for the Excel-like file name.

runint, optional

Number of the run

verbosebool, optional

Whether to display the contamination information (default is True).

Returns¶

None

generate_reports_txt(runs_plots_scores, save_dir='./reports', dataset='', metrics=['RMSE'], run=-1, rt=0, verbose=True)[source]¶

Generate and save a text report of metrics and timing for each dataset, algorithm, and pattern.

Parameters¶

runs_plots_scoresdict

Dictionary containing scores and timing information for each dataset, pattern, and algorithm.

save_dirstr, optional

Directory to save the reports file (default is “./reports”).

datasetstr, optional

Name of the data for the report name.

metricsstr, optional

List of metrics asked for in the report.

runint, optional

Number of the run.

rtfloat, optional

Total time of the run.

verbosebool, optional

Whether to display the contamination information (default is True).

Returns¶

None

Notes¶

The report is saved in a “report.txt” file in save_dir, organized in sections with headers and results.