Symbolic overhaul: implementation provenance#

This page preserves the original design, decisions, implementation notes, and remaining validation work for the symbolic-backend overhaul. The status below is a project record, not a claim that every roadmap item or scientific comparison is complete.

Objective#

Integrate SYREN-NEW as the default symbolic backend in CosmicJellyfish, add CLASS+EuclidEmu2 (EE2) as a composable nonlinear boost, and validate both through CLASS-comparison forecasts (GCsp and WL+GCph Fisher matrices).

Design decisions#

  • Total matter first: CDM+baryon (cb) requests are rejected explicitly, not silently approximated.

  • Validation proceeds through spectra → derivatives → Fisher forecasts.

  • New symbolic is the default; an explicit legacy mode is kept. Wider-SYREN and baryonic corrections are later work (see §7).

  • euclidemu2 is an optional emulators extra. Its native grid spans approximately 8.73e-3 - 9.41 h/Mpc; each call accepts at most 100 redshifts.

  • CLASS+EE2 uses P_nl = P_lin_CLASS x B_EE2, reusing the same total-matter boost for cb as an explicit approximation. The top-level NONLINEAR.model YAML selector defaults to class_native and is never passed to CLASS itself.

  • EE2 k is in h/Mpc (CLASS k in 1/Mpc, divided by h). Below the native grid the boost is unity; above it the adapter raises ValueError. CLASS’s padded redshift grid must fit EE2’s z = 0-10 range.

  • class/syren_new_minimal_matched.yaml requires output: mPk,mTk; changebasis_class() has an opt-in three_degenerate neutrino translation via PARAMETER_TRANSLATION.

  • build_analysis_context is the spectrum-only configuration API; FisherMatrix is for derivatives/Fisher computations. Call likelihood.loglike(param_dict={}) by keyword.

Status at implementation time#

Completed

  • SyrenNewProvider implemented in cosmicjellyfish/cosmology/symbolic_new.py; wired into cosmology.py via input_type='symbolic' dispatch; uses Astropy backgrounds, physical-unit Pmm, total-matter only.

  • SYREN-NEW GCsp and WL Fisher matrices produced and matched CLASS/symbolic YAML profiles, the three_degenerate CLASS neutrino scheme, and aggregate derivative plotting put in place.

  • 3-point GCsp derivatives run through the derivatives.py engine for all 4 matched configs (CLASS LCDM/w0wa, SYREN LCDM/w0wa); all derivatives finite.

  • scripts/likelihood/diagnostics/run_syren_new_likelihoods.py created; fiducial loglike(param_dict={}) verified (GCsp and WL+GCph both finite); parameter responsiveness confirmed.

  • CLASS+EE2 implemented in cosmicjellyfish/cosmology/ee2.py and cosmology.py, with a new class/ee2_boost.yaml profile. CLASS linear spectra are multiplied by the emulator boost when nonlinear is enabled; native CLASS remains the default.

  • Focused adapter and CLASS+EE2 integration tests passed for both A_s and sigma8 normalization.

Active / next

  1. Review the CLASS+EE2 integration in representative spectra/derivatives/forecasts as the broader comparison work proceeds.

  2. Select class_config_yaml=cosmicjellyfish/configs/default_boltzmann_yaml_files/class/ee2_boost.yaml with code=class and nonlinear=True for a CLASS+EE2 comparison.

  3. Compare spectra, derivatives and forecasts against the matched CLASS HMcode and SYREN-NEW configurations.

Relevant files#

  • cosmicjellyfish/cosmology/ee2.py: CLASS to EE2 parameter mapping and boost grid adapter.

  • cosmicjellyfish/cosmology/cosmology.py: class_setparams() selector and class_results() multiplication.

  • cosmicjellyfish/configs/default_boltzmann_yaml_files/class/ee2_boost.yaml: opt-in CLASS+EE2 profile.

  • tests/cosmology_test.py: unit and integration checks.

The plan was to make SYREN-NEW the default symbolic backend, separate nonlinear modelling from the underlying cosmology calculation, and validate the result against CLASS-based references through representative forecasts, per the design decisions above. The sections below describe the wider roadmap that followed from that plan.

1. Design: cosmology provider + nonlinear prescription#

Keep the public cosmology interface—Pmm, Hubble, distances, growth, and so on—but separate its internal responsibilities:

Cosmological parameters
        │
        ▼
Background + linear-spectrum provider
        │
        ▼
Nonlinear prescription
  ├─ native calculation
  ├─ direct nonlinear emulator
  └─ linear power × nonlinear boost
        │
        ▼
Optional baryonic correction
        │
        ▼
Existing CosmicJellyfish observables and forecasts

This accommodates the requested combinations:

Configuration

Linear power

Nonlinear power

New symbolic default

linear_new

syren_new

CLASS + EuclidEmu2

CLASS

CLASS linear × EE2 boost

Legacy symbolic

Existing implementation

Existing legacy choices

Later: wider symbolic

wider_syren.linear

wider_syren.halofit

Later: baryonic symbolic

linear_new

SYREN-NEW × baryonic correction

Future: MG combination

MGCLASS

Compatible MG linear power × MG boost

Each boost must declare what it multiplies. A nonlinear-to-linear boost and an MG-to-GR nonlinear ratio are different objects. Recording that distinction now enables future MG combinations without assuming every emulator can multiply every spectrum.

SYREN-NEW remains a direct nonlinear-spectrum provider: its implementation evaluates its own linear model internally. We should not pretend it accepts arbitrary CLASS linear power.

2. Replace the symbolic calculation completely#

The main integration point is cosmicjellyfish/cosmology/cosmology.py, especially changebasis_symb, symbolic_setparams, and symbolic_results.

Parameter handling#

Resolve a consistent parameter set:

[ {10^9A_s,\Omega_m,\Omega_b,h,n_s,\Sigma m_\nu,w_0,w_a}. ]

Changes include:

  • Support flat ΛCDM and (w_0w_a)CDM within the emulator’s supported domain.

  • Include neutrinos when converting between total matter and CDM densities.

  • Use the new amplitude conversions, including their (m_\nu,w_0,w_a) dependence.

  • Preserve a supplied (A_s); avoid the current unnecessary (A_s\to\sigma_8\to A_s) round trip.

  • Define how conflicting amplitude inputs are handled.

  • Reject unsupported cosmological freedom rather than silently ignoring it.

Spectra and units#

Evaluate the new linear spectrum at each redshift, rather than scaling a (z=0) spectrum with the old scale-independent Colossus growth.

Preserve CosmicJellyfish’s physical-unit interface:

[ k_{\rm symbolic}=k_{\rm CFP}/h,\qquad P_{\rm CFP}=P_{\rm symbolic}/h^3. ]

Internally, use explicit (n_z, n_k) grids and retain the existing interpolation interface.

Background, growth, and normalization#

My recommendation is:

  • Astropy backgrounds, already a dependency, for (H(z)), distances, and density evolution, tabulated once per cosmology.

  • Configure radiation and massive neutrinos explicitly; account for Astropy’s distinction between matter and neutrino densities.

  • Derive growth from the complete new linear spectrum: [ D(k,z)=\sqrt{\frac{P_{\rm lin}(k,z)}{P_{\rm lin}(k,0)}}, \qquad f(k,z)=-(1+z)\frac{\partial\ln D(k,z)}{\partial z}. ]

  • Compute (\sigma_8(z)) from the linear spectrum, with controlled integration limits and normalization checks.

This keeps growth and normalization consistent with the emulator’s scale dependence. Its internal approximate growth factor alone is insufficient because the final spectrum includes additional redshift-dependent corrections.

Colossus compatibility#

The new path will stop creating Colossus cosmologies or using its background/growth routines.

One source detail matters: linear_new.py imports a function from linear.py, and that module imports Colossus at module level. Upstream also declares Colossus as a dependency. Therefore, inactive but still installed is the appropriate first implementation, matching your compatibility preference.

3. Add EuclidEmu2 as a reusable nonlinear boost#

Implement a small adapter around:

ee2.get_boost(parameters, redshifts)

Then construct:

[ P_{\rm nl}(k,z)=B_{\rm EE2}(k,z),P_{\rm lin}(k,z). ]

For CLASS+EE2:

  • Use CosmicJellyfish’s existing CLASS linear calculation.

  • Avoid computing native CLASS nonlinear power unnecessarily.

  • Keep CLASS backgrounds and linear growth.

  • Obtain (A_s), densities, and neutrino parameters from the same resolved cosmology.

  • Expose total-matter nonlinear results consistently; do not leave native CLASS cb nonlinear results attached to an EE2-labelled calculation.

  • Import EE2 lazily and provide it as an optional dependency.

The installed EE2 wrapper warrants a little care: it extrapolates custom (k) inputs and has fixed-size native redshift storage. I recommend requesting its native (k) grid, handling interpolation and range policies in our adapter, and chunking redshifts within its capacity.

All provider choices must travel with the existing immutable run configuration, including metadata and any cache identity.

4. Set the numerical domain before running forecasts#

This is the main scientific constraint to resolve early.

The SYREN-NEW notebook exercises:

  • (0\le z\le3);

  • (0.009\le k\le9,h,{\rm Mpc}^{-1});

  • bounded cosmological parameters, including (0\le\Sigma m_\nu\le0.15) eV.

The installed EE2 boost covers approximately (0.00873)–(9.41,h,{\rm Mpc}^{-1}), while some existing photometric configurations request substantially higher (k).

The implementation should therefore:

  1. Distinguish documented emulator validity from the notebook’s sampled test range.

  2. Check actual requested scales—including growth defaults, smoothing grids, and (\sigma_8) integration.

  3. Define explicit low-/high-(k) behaviour.

  4. Validate the whole derivative stencil, not just the fiducial cosmology.

  5. Use identical physical scale cuts for both sides of each forecast comparison.

For initial forecast acceptance, I recommend a clearly labelled common-domain survey configuration. Full survey settings extending outside that domain require a separately justified extension policy; spline extrapolation must not silently supply it.

5. CLASS-only validation, in successive stages#

Stage A — Reproduce the notebook reference#

Start with a fiducial and a small deterministic set of cosmologies, then extend coverage.

The CLASS settings need a dedicated profile: the notebook uses three degenerate massive neutrinos and a density convention that differs from CosmicJellyfish’s current CLASS mapping. Match those conventions deliberately before attributing residuals to the emulator.

Stage B — Validate cosmology outputs#

Compare:

  • linear (P(k,z));

  • (H(z)), comoving and angular-diameter distances;

  • (\sigma_8(z));

  • (D(k,z)) and (f(k,z)).

Separate raw emulator discrepancies from interpolation, integration, and parameter-conversion errors.

Stage C — Validate nonlinear power#

Use the notebook’s reference:

[ \boxed{P_{\rm nl}^{\rm reference} =P_{\rm lin}^{\rm CLASS},B_{\rm EE2}} ]

Native CLASS HMcode/Halofit can be a separately labelled comparison if useful, but it should not be the sole pass/fail reference for an emulator targeting EE2.

Stage D — Validate derivatives#

Check parameter responses and finite-difference stability, especially for (m_\nu,w_0,w_a) and amplitude normalization.

Use absolute or scaled residuals where derivatives approach zero. Agreement in spectra does not guarantee agreement in their derivatives.

Stage E — Validate representative forecasts#

Run matched, total-matter cases for:

  • photometric observables and Fisher matrices;

  • spectroscopic observables and Fisher matrices.

Reuse scripts/validation/run_fisher_compare_backends.py and the current reporting pipeline (see scripts reference). Compare observable derivatives before interpreting differences in marginalized errors.

Define acceptance thresholds separately for each stage before the formal campaign. Any slightly different CLASS settings and survey cuts should be recorded alongside the results.

The existing CAMB–CLASS validation remains supporting evidence. The new campaign establishes consistency with the chosen CLASS-based reference; it does not automatically establish agreement for every earlier nonlinear prescription or parameter setting.

6. Dashboard, tests, and dependency integration#

Extend the current validation machinery to record:

  • background and linear provider;

  • nonlinear prescription and optional correction;

  • tracer, amplitude basis, neutrino conventions, and domain policy;

  • package revisions and resolved numerical settings;

  • spectrum, derivative, observable, and Fisher metrics.

Relevant files include:

  • cosmicjellyfish/configs/config.py and configs/context.py;

  • configs/default_boltzmann_yaml_files/symbolic/;

  • scripts/validation/run_fisher_compare_backends.py;

  • scripts/validation/render_validation_dashboard.py;

  • scripts/validation/configs/;

  • focused cosmology and runtime-context tests.

Software tests should cover units, shape/order handling, nonlinear toggles, unsupported cb requests, amplitude conversion, range handling, and configuration isolation.

A useful finding: the current project pin already matches the local upstream commit containing these new modules. We should verify packaging and align dependency declarations rather than assume an upstream version bump is necessary.

7. Final optional additions#

Wider SYREN#

Add it as an explicit ΛCDM-only option, with its own parameter/domain metadata, linear spectrum, and Halofit calculation. Its wider priors do not extend SYREN-NEW’s massive-neutrino or dynamical-dark-energy support.

Baryonic corrections#

Add a final correction stage:

[ P_{\rm nl,baryonic}=S_{\rm baryon},P_{\rm nl,DMO}. ]

Expose the hydro-model choice and feedback parameters. Keep the supplied scatter estimate separate from the deterministic spectrum; incorporating it into forecast covariance would be a further feature.

As requested, these final options receive integration checks, not another scientific-validation campaign.

Implementation order#

  1. Parameter, unit, capability, and domain contracts.

  2. New symbolic spectra plus Colossus-free background/growth calculation.

  3. Composable nonlinear interface and CLASS+EE2 adapter.

  4. CLASS validation: cosmology outputs → derivatives → forecasts.

  5. Dashboard integration and default/legacy documentation.

  6. Wider SYREN, then baryonic corrections.

The most important early milestone is a complete new symbolic cosmology object—not merely matching (P(k))—with consistent backgrounds, growth, normalization, and explicit numerical coverage. Once that is established, EE2 and the later corrections fit naturally into the same design.