Make "compare" task derive METRIC_FILES dynamically from recipe datasets (multi-run safe)

[mo-nikosbaltas]

Description
The current compare task relies on hard-coded METRIC_FILES in
app/compare/opt/rose-app-<recipe>.conf, e.g.:

HadGEM3-GC31-LL_metrics.csv
UKESM1-0-LL_metrics.csv

This approach breaks as soon as:
• the evaluation model changes,
• additional runs are introduced (e.g. eval2, eval3),
• or dataset names are modified via the recipe.

Observed Errors
During recent test runs, the compare task failed with:
... UKESM1-0-LL_metrics.csv does not exist
Even though:
• the file did exist for the actual dataset used, and
• other metric files in the same loop expanded correctly.
Root cause:
• METRIC_FILES is model-name hard-coded and no longer matches the
datasets produced by run_recipe.
• Shell glob expansion masked the issue for the first file but failed for subsequent entries.
Root Cause Summary
• compare.sh assumes exact model identifiers in filenames.
• These identifiers are duplicated in Rose config instead of being derived from the recipe.
• This creates technical debt and blocks multi-run scalability.

---

Proposed Solution
Derive METRIC_FILES dynamically from the ESMValTool recipe at runtime.
Implementation outline:

1. Parse the datasets list from RECIPE_PATH in compare.sh (or a helper script).
2. Extract dataset names actually used by the recipe.
3. Construct expected metric and plot filenames programmatically.
4. Validate existence of generated paths instead of static strings.

Benefits:
• Fully multi-run compatible (ref, eval, eval2, …).
• Eliminates hard-coded model names in Rose configuration.
• Ensures compare logic stays consistent with the recipe used to generate outputs.
• Reduces maintenance and configuration duplication.