SILVA Fixed-Point Diffusion Models¶
This lab derives a timestep-conditioned implicit denoiser, verifies variable compute and equilibrium reuse, exercises stochastic Jacobian-free training, and distinguishes per-timestep fixed points from a joint diffusion-restoration trajectory. The first mechanism follows Fixed-Point Diffusion Models [74]; the joint trajectory route follows DeqIR [49].
Numbered literature: [1], [4], [49], [74]. Each number opens the complete citation and its primary external source.
from pathlib import Path
import importlib.util
import subprocess
import sys
REPO_URL = "https://github.com/jseluis/silva-networks.git"
def find_local_silva_root():
candidates = [Path.cwd(), Path("/content/silva-networks")]
root = Path.cwd()
while root != root.parent:
candidates.append(root)
root = root.parent
for candidate in candidates:
if (candidate / "src" / "silva_networks").exists():
return candidate
return None
root = find_local_silva_root()
if root is not None:
sys.path.insert(0, str(root / "src"))
elif importlib.util.find_spec("silva_networks") is None:
subprocess.check_call([sys.executable, "-m", "pip", "install", f"git+{REPO_URL}"])
root = Path.cwd()
else:
root = Path.cwd()
import matplotlib.pyplot as plt
import torch
plt.rcParams.update({"figure.dpi": 300, "savefig.dpi": 300})
torch.manual_seed(33)
from torch import nn
from silva_networks import (
SILVADiffusionEquilibrium,
SILVAFixedPointDenoiser,
SILVAFixedPointDiffusionModel,
SILVATimestepFixedPointBlock,
SolverConfig,
make_fixed_point_diffusion_dataset,
)
1. One Fixed Point at Every Diffusion Time¶
An explicit diffusion block first computes input features $h_t=f_{pre}(x_t)$ and an injection $p_t=P(h_t)$. The implicit block solves
$$ z_t^\star=F_\theta(z_t^\star,p_t,e(t),c), \qquad \widehat\epsilon_t=f_{post}(z_t^\star), $$
where $e(t)$ is the timestep embedding and $c$ is optional conditioning. A reverse update then maps $(x_t,\widehat\epsilon_t)$ to $x_s$ for $s<t$.
The architecture has four independently replaceable regions:
$$ x_t\xrightarrow{f_{pre}}h_t\xrightarrow{P}p_t \xrightarrow{\operatorname{root}(F_\theta)}z_t^\star \xrightarrow{f_{post}}\widehat\epsilon_t. $$
SILVA does not require the internal transition to be convolutional. It may be a residual block, U-Net, attention block, transformer, Fourier operator, or any shape-preserving module with the declared four-argument transition contract.
2. Exact Contractive Denoiser¶
For a transparent verification problem, define
$$ F(z,p,t)=\rho z+(1-\rho)(0.5p+0.1t),\qquad 0<\rho<1. $$
The exact fixed point is $z_t^\star=0.5p+0.1t$. This lets us measure how the iteration allocation changes numerical error without requiring a pretrained image model.
class ContractiveTimestepTransition(nn.Module):
def __init__(self, gain=0.35):
super().__init__()
self.gain = nn.Parameter(torch.tensor(gain))
def forward(self, state, injection, time, condition=None):
del condition
time_field = time.reshape(-1, 1, 1, 1).to(state)
target = 0.5 * injection + 0.1 * time_field
return self.gain * state + (1.0 - self.gain) * target
transition = ContractiveTimestepTransition()
denoiser = SILVAFixedPointDenoiser(
channels=1,
preprocessor=nn.Identity(),
projection=nn.Identity(),
transition=transition,
postprocessor=nn.Identity(),
config=SolverConfig(
solver="picard",
max_iter=20,
tol=1e-8,
backward_mode="unrolled",
anderson_batch_dims=1,
return_best=False,
),
)
compact_data = make_fixed_point_diffusion_dataset(
samples=3, channels=1, size=18, seed=74
)
noise = compact_data.noise
time = compact_data.times
exact = compact_data.target
result = denoiser(noise, time, return_result=True)
assert torch.allclose(result.output, exact, atol=2e-6)
print("equilibrium shape:", tuple(result.equilibrium.shape))
print("iterations/residual:", result.solver_result.iterations, result.solver_result.residual)
equilibrium shape: (3, 1, 18, 18) iterations/residual: 19 2.8799560514158884e-09
budgets = (1, 2, 4, 8, 12)
errors = []
for budget in budgets:
estimate = denoiser(noise, time, iterations=budget)
errors.append(float((estimate - exact).square().mean().sqrt()))
figure, axis = plt.subplots(figsize=(4.8, 2.8))
axis.semilogy(budgets, errors, marker="o")
axis.set(xlabel="fixed-point block evaluations", ylabel="RMSE to exact fixed point")
axis.grid(alpha=0.25)
figure.tight_layout()
plt.show()
3. Reverse Schedule, Variable Compute, and State Reuse¶
For reverse times $t_0>t_1>\cdots>t_K$, the computation is a sequence of related roots:
$$ z_{t_k}^\star=F_\theta(z_{t_k}^\star,P(x_{t_k}),t_k,c), \qquad x_{t_{k+1}}=R(x_{t_k},Q(z_{t_k}^\star),t_k,t_{k+1},\xi_k). $$
The previous equilibrium is a warm start for the next time. An allocation $(m_0,\ldots,m_{K-1})$ controls block evaluations independently at each reverse step. The reverse operator remains replaceable, so the same wrapper can host a declared DDIM, DDPM, ODE, or task-specific schedule.
def reverse_step(sample, prediction, timestep, next_timestep, condition, step_noise):
del timestep, next_timestep, condition
return 0.45 * sample + 0.55 * prediction + step_noise
process = SILVAFixedPointDiffusionModel(
denoiser,
timesteps=(8, 5, 3, 1, 0),
allocations=(2, 3, 5, 8),
step_operator=reverse_step,
reuse_equilibria=True,
)
process_result = process(noise, return_result=True)
assert process_result.allocations == (2, 3, 5, 8)
print("reverse samples:", len(process_result.samples))
print("solver residuals:", [item.residual for item in process_result.solver_results])
figure, axes = plt.subplots(1, len(process_result.samples), figsize=(10.0, 2.1))
for index, (axis, sample) in enumerate(zip(axes, process_result.samples)):
image = axis.imshow(sample[0, 0].detach(), cmap="coolwarm")
axis.set_title(f"state {index}")
axis.set_xticks([])
axis.set_yticks([])
figure.colorbar(image, ax=axes, fraction=0.018, pad=0.02)
plt.show()
reverse samples: 5 solver residuals: [2.1534578800201416, 0.2675005793571472, 0.019896134734153748, 0.0006140271434560418]
4. Stochastic Jacobian-Free Training¶
A memory-limited training step can sample $n$ no-gradient transitions followed by $m$ differentiable transitions:
$$ z_n=F_\theta^{\,n}(z_0,p,t),\quad n\sim\mathcal U\{0,\ldots,N\}, $$
$$ \widetilde z=F_\theta^{\,m}(\operatorname{stopgrad}(z_n),p,t), \quad m\sim\mathcal U\{1,\ldots,M\}. $$
This does not store the first $n$ states. The following deterministic choice
checks the same gradient route; omitting no_grad_steps and grad_steps samples
them from the configured ranges.
transition.gain.grad = None
prediction = denoiser.stochastic_jfb(
noise,
time,
no_grad_steps=3,
grad_steps=2,
max_no_grad=8,
max_grad=4,
)
training_loss = prediction.square().mean()
training_loss.backward()
assert transition.gain.grad is not None
print("training loss:", float(training_loss.detach()))
print("transition gradient:", float(transition.gain.grad.detach()))
training loss: 0.2486223727464676 transition gradient: 0.014318034052848816
5. Install a Full Spatial Transition¶
The built-in transition concatenates the current state, input injection, and a scalar timestep field, applies a shape-preserving spatial network, and bounds its correction. It is a usable default and a reference implementation of the contract. Full source reproduction can replace it with the reported transformer blocks while retaining the outer denoiser, solver, allocation, and diagnostic interfaces.
spatial_transition = SILVATimestepFixedPointBlock(
channels=4,
hidden_channels=64,
scale=0.15,
)
spatial_denoiser = SILVAFixedPointDenoiser(
channels=4,
preprocessor=nn.Conv2d(4, 4, 3, padding=1),
projection=nn.Conv2d(4, 4, 1),
transition=spatial_transition,
postprocessor=nn.Conv2d(4, 4, 3, padding=1),
config=SolverConfig(
solver="anderson",
max_iter=32,
tol=1e-5,
backward_mode="implicit",
backward_solver="gmres",
anderson_batch_dims=1,
return_best=True,
),
)
print(spatial_denoiser)
print("trainable parameters:", sum(p.numel() for p in spatial_denoiser.parameters()))
SILVAFixedPointDenoiser(
(preprocessor): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(projection): Conv2d(4, 4, kernel_size=(1, 1), stride=(1, 1))
(transition_module): SILVATimestepFixedPointBlock(
(network): Sequential(
(0): Conv2d(9, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): SiLU()
(2): Conv2d(64, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
)
)
(postprocessor): Conv2d(4, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
)
trainable parameters: 7872
6. Distinguish the Joint DeqIR Route¶
The model above solves one implicit feature state at each diffusion timestep
[74]. SILVADiffusionEquilibrium instead places the complete selected reverse
trajectory in one triangular fixed point. Its step_operator replaces the
reverse rule and data_consistency projects every candidate against an
observation, which is the SILVA route for joint diffusion restoration [49].
These are related but different abstractions; both remain available.
class JointRestorationStep(nn.Module):
def forward(self, state, timestep, next_timestep, condition, step_noise):
del timestep, next_timestep, condition
return 0.7 * state + step_noise
class MeasurementProjection(nn.Module):
def forward(self, candidate, observation, next_timestep):
del next_timestep
return 0.8 * candidate + 0.2 * observation
joint = SILVADiffusionEquilibrium(
denoiser=None,
alphas_cumprod=torch.linspace(1.0, 0.2, 6),
timesteps=(5, 3, 1, -1),
step_operator=JointRestorationStep(),
data_consistency=MeasurementProjection(),
config=SolverConfig(
solver="picard", max_iter=8, tol=1e-7,
anderson_batch_dims=0, return_best=True
),
)
observation = torch.zeros_like(noise)
joint_result = joint(noise, observation=observation, return_result=True)
assert joint_result.trajectory.shape == (4, *noise.shape)
print("joint trajectory:", tuple(joint_result.trajectory.shape))
print("joint residual:", joint_result.solver_result.residual)
joint trajectory: (4, 3, 1, 18, 18) joint residual: 0.0
7. Source-Scale Reproduction Checklist¶
To reproduce the fixed-point diffusion study [74], retain the published image preprocessing, latent encoder, diffusion schedule, timestep conditioning, architecture widths, iteration-allocation policy, stochastic backward ranges, optimizer, precision, checkpoint selection, and FID-50K protocol. Report both quality and the number of transformer-block evaluations, wall time, peak memory, and equilibrium residuals.
To reproduce a DeqIR restoration result [49], additionally retain the exact pretrained denoiser, degradation/SVD operator, observation noise, initialization, reverse schedule, data-consistency rule, and restoration metrics. Compact results on this page verify the public mechanism contracts; they are not substitutes for either full source experiment.
Source Data and Full Experiment Preflight¶
The source image experiments [74] require their licensed or registered datasets, latent encoder, diffusion schedule, full architecture, training checkpoints, and FID-50K budget. Begin with 128-512 encoded samples and a short timestep schedule before restoring the complete allocation and generation protocol.
The executable record below distinguishes public, generated, and licensed inputs and keeps storage and launch steps next to the model contract. Compact results validate the implementation route; the cited benchmark additionally requires every recorded source-scale step.
from silva_networks import silva_reproduction_spec
source_plan = silva_reproduction_spec('silva_fixed_point_diffusion')
print("data sources:")
for source in source_plan.data_sources:
print(" -", source)
print("access:")
for item in source_plan.data_access:
print(" -", item)
print("storage:")
for item in source_plan.storage_plan:
print(" -", item)
print("source-scale steps:")
for index, item in enumerate(source_plan.source_scale_steps, start=1):
print(f" {index}. {item}")
data sources: - https://arxiv.org/abs/2401.08741 - https://openaccess.thecvf.com/content/CVPR2024/html/Bai_Fixed-Point_Diffusion_Models_CVPR_2024_paper.html access: - ImageNet requires registration and its stated access terms; face and scene datasets each retain their own licenses and acquisition routes. - Store dataset checksums separately from latent-encoder, diffusion-schedule, and checkpoint revisions so a source-scale claim is auditable. storage: - Budget raw images, encoded latents, checkpoints, optimizer state, and generated evaluation samples separately. - A standard FID-50K evaluation alone stores 50,000 decoded samples; latent trajectory caches scale again with timesteps and retained fixed-point states. source-scale steps: 1. Acquire one declared image task and reproduce its resize/crop, latent encoder, diffusion schedule, split, and evaluation preprocessing. 2. Configure explicit pre/projection/post blocks around the timestep-conditioned fixed point, then reproduce the source per-timestep iteration allocation and state reuse. 3. Train with the declared stochastic Jacobian-free schedule and compare FID-50K, block evaluations, latency, memory, and residuals at equal sampling budgets.
From 35 Silva Fixed Point Diffusion to a Custom SILVA Family¶
The construction in this notebook can be separated into the universal conditioned-equilibrium contract
$$ z_0=I_\eta(x),\qquad z^\star=T_\theta(z^\star,x),\qquad \widehat y=Q_\psi(z^\star). $$
For this topic:
| Part | Concrete interpretation |
|---|---|
| Equilibrium state | the latent vector or tensor z |
| Condition | the injected observation x |
| Repeated computation | the tied map f_theta(z, x) |
| Required invariants | state shape and a decreasing or bounded residual |
| Replaceable components | transition, damping, stopping rule, backward solver, and readout |
The initializer and source path are evaluated outside or alongside the root solve. Only the state-preserving transition is repeated. Replacing an internal architecture does not change this equation, provided the transition still maps the same state space into itself.
Replace Every Denoising Stage and the Reverse Process¶
denoiser = SILVAFixedPointDenoiser(
channels=latent_channels,
preprocessor=my_preprocessor,
projection=my_condition_projection,
transition=my_timestep_conditioned_transition,
postprocessor=my_noise_head,
config=solver_config,
)
model = SILVAFixedPointDiffusionModel(
denoiser,
timesteps=reverse_timesteps,
allocations=iterations_per_timestep,
step_operator=my_reverse_step,
reuse_equilibria=True,
)
The transition preserves latent shape and receives the projected noisy input, timestep embedding, and optional class condition. Full runs should preserve the source noise schedule, loss weighting, variable-compute policy, equilibrium reuse, sampling seeds, checkpoints, and generation metrics.
import torch as silva_extension_torch
from torch import nn as silva_extension_nn
from silva_networks import (
SILVAConditionedEquilibrium,
SILVAZeroInitializer,
SolverConfig,
validate_silva_transition,
)
class NotebookExtensionTransition(silva_extension_nn.Module):
def __init__(self, condition_dim=2, state_dim=3):
super().__init__()
self.source = silva_extension_nn.Linear(condition_dim, state_dim)
self.state_field = silva_extension_nn.Sequential(
silva_extension_nn.Linear(state_dim, 2 * state_dim),
silva_extension_nn.Tanh(),
silva_extension_nn.Linear(2 * state_dim, state_dim),
)
def forward(self, state, condition):
return silva_extension_torch.tanh(
self.source(condition) + 0.15 * self.state_field(state)
)
silva_extension_torch.manual_seed(610)
notebook_condition = silva_extension_torch.linspace(-1.0, 1.0, 8).reshape(4, 2)
notebook_state0 = silva_extension_torch.zeros(4, 3)
notebook_transition = NotebookExtensionTransition()
notebook_report = validate_silva_transition(
notebook_transition,
notebook_state0,
notebook_condition,
)
assert notebook_report.valid
with silva_extension_torch.no_grad():
notebook_reference_step = silva_extension_torch.tanh(
notebook_transition.source(notebook_condition)
+ 0.15 * notebook_transition.state_field(notebook_state0)
)
silva_extension_torch.testing.assert_close(
notebook_transition(notebook_state0, notebook_condition),
notebook_reference_step,
)
notebook_custom_model = SILVAConditionedEquilibrium(
notebook_transition,
SILVAZeroInitializer(3),
readout=silva_extension_nn.Linear(3, 1),
config=SolverConfig(
solver="picard",
max_iter=40,
tol=1e-7,
backward_mode="implicit",
backward_solver="gmres",
anderson_batch_dims=1,
),
)
notebook_custom_result = notebook_custom_model(
notebook_condition,
return_result=True,
)
assert notebook_custom_result.output.shape == (4, 1)
assert notebook_custom_result.solver_result.residual < 1e-5
notebook_custom_result.output.square().mean().backward()
assert all(
parameter.grad is not None and silva_extension_torch.isfinite(parameter.grad).all()
for parameter in notebook_custom_model.parameters()
)
print("custom transition:", notebook_report)
print("equilibrium residual:", notebook_custom_result.solver_result.residual)
custom transition: SILVATransitionReport(state_shape=(4, 3), output_shape=(4, 3), preserves_shape=True, preserves_device=True, preserves_dtype=True, finite=True, differentiable=True, parameter_count=54) equilibrium residual: 5.960464477539063e-08
Numerical Equivalence, Compact Reproduction, and Scale¶
Before training, compare one packaged transition with an independently written update:
$$ e_{\mathrm{step}} =\frac{\|T_\theta(z,x)-T_{\mathrm{ref}}(z,x)\|_2} {\|T_{\mathrm{ref}}(z,x)\|_2+\varepsilon}. $$
After solving, report the fixed-point residual separately:
$$ e_{\mathrm{fp}} =\frac{\|T_\theta(z^\star,x)-z^\star\|_2} {\|z^\star\|_2+\varepsilon}. $$
For this notebook, a compact reproduction must declare and assert distance to an analytic fixed point and final relative residual. A full experiment must additionally record the source dataset version and split, preprocessing, architecture widths, solver and optimizer schedules, random seeds, baseline configuration, checkpoints, and every deviation from the cited protocol.
The principal scaling axes are latent width, solver tolerance, and iteration budget. Increase one axis at a time, retain the compact deterministic case as a regression test, and record task error, domain-specific residual, forward residual, backward linear residual, memory use, and runtime independently.
Extension Exercises¶
- Replace one component from this notebook while preserving its state and domain invariants.
- Write the replacement first as an independent reference function, then as a module, and assert one-step equivalence.
- Compare two solver configurations on the identical trained transition.
- Add a compact baseline and a predeclared metric threshold.
- Create a full-scale configuration without weakening the compact tests.
The complete authoring protocol is documented in Extending SILVA.
notebook_reproduction_record = {
"notebook": '35_silva_fixed_point_diffusion.ipynb',
"state": 'the latent vector or tensor z',
"condition": 'the injected observation x',
"transition": 'the tied map f_theta(z, x)',
"invariants": 'state shape and a decreasing or bounded residual',
"compact_metric": 'distance to an analytic fixed point and final relative residual',
"scale_axis": 'latent width, solver tolerance, and iteration budget',
}
assert all(notebook_reproduction_record.values())
notebook_reproduction_record
{'notebook': '35_silva_fixed_point_diffusion.ipynb',
'state': 'the latent vector or tensor z',
'condition': 'the injected observation x',
'transition': 'the tied map f_theta(z, x)',
'invariants': 'state shape and a decreasing or bounded residual',
'compact_metric': 'distance to an analytic fixed point and final relative residual',
'scale_axis': 'latent width, solver tolerance, and iteration budget'}
Worked Convergence and Sensitivity Study¶
The preceding example demonstrates one configured solve. This additional study changes the refinement feedback factor while keeping the source fixed, so solver effort and implicit sensitivity can be read separately from task behavior. Locally, one eigendirection of a nonlinear transition can be represented by
$$ z_{k+1} = \rho z_k + u, \qquad 0 \leq \rho < 1. $$
Its equilibrium is
$$ z^\star = \frac{u}{1-\rho}. $$
Subtracting the fixed-point equation from the iteration gives the exact error recursion
$$ e_{k+1} = \rho e_k, \qquad |e_k| = \rho^k |e_0|. $$
For a requested absolute tolerance $\tau$, the idealized iteration estimate is
$$ k \geq \frac{\log(\tau/|e_0|)}{\log \rho}. $$
The same factor controls sensitivity. Differentiating the equilibrium with respect to the source gives
$$ \frac{\partial z^\star}{\partial u} =\frac{1}{1-\rho}. $$
Thus a transition can remain contractive while becoming expensive and highly sensitive as $\rho$ approaches one. The table and figure below measure this effect rather than merely stating it. They provide a reference envelope for the notebook's actual state, the latent vector or tensor z, and its repeated map, the tied map f_theta(z, x). The scalar study does not replace the domain model; it supplies a result whose convergence rate and derivative are known exactly, so the same reporting code can be trusted before it is applied to the larger transition.
import math as silva_deepening_math
import torch as silva_deepening_torch
silva_deepening_rates = (0.20, 0.45, 0.70, 0.85)
silva_deepening_source = 0.35
silva_deepening_tolerance = 1e-8
silva_deepening_histories = {}
silva_deepening_rows = []
for silva_deepening_rho in silva_deepening_rates:
silva_deepening_state = silva_deepening_torch.tensor(0.0)
silva_deepening_exact = silva_deepening_source / (1.0 - silva_deepening_rho)
silva_deepening_history = []
for silva_deepening_iteration in range(1, 241):
silva_deepening_next = (
silva_deepening_rho * silva_deepening_state + silva_deepening_source
)
silva_deepening_residual = abs(
float(silva_deepening_next - silva_deepening_state)
)
silva_deepening_history.append(silva_deepening_residual)
silva_deepening_state = silva_deepening_next
if silva_deepening_residual < silva_deepening_tolerance:
break
silva_deepening_u = silva_deepening_torch.tensor(
silva_deepening_source, requires_grad=True
)
silva_deepening_solution = silva_deepening_u / (1.0 - silva_deepening_rho)
silva_deepening_solution.backward()
silva_deepening_expected_sensitivity = 1.0 / (1.0 - silva_deepening_rho)
silva_deepening_gradient_error = abs(
float(silva_deepening_u.grad) - silva_deepening_expected_sensitivity
)
silva_deepening_histories[silva_deepening_rho] = silva_deepening_history
silva_deepening_rows.append(
(
silva_deepening_rho,
silva_deepening_iteration,
silva_deepening_history[-1],
abs(float(silva_deepening_state) - silva_deepening_exact),
float(silva_deepening_u.grad),
silva_deepening_gradient_error,
)
)
print('refinement feedback factor')
print("rho | iterations | final residual | exact-state error | sensitivity | gradient error")
for silva_deepening_row in silva_deepening_rows:
print(
f"{silva_deepening_row[0]:.2f} | {silva_deepening_row[1]:3d} | "
f"{silva_deepening_row[2]:.3e} | {silva_deepening_row[3]:.3e} | "
f"{silva_deepening_row[4]:.4f} | {silva_deepening_row[5]:.3e}"
)
assert all(row[2] < silva_deepening_tolerance for row in silva_deepening_rows)
assert all(row[3] < 1e-6 for row in silva_deepening_rows)
assert all(row[5] < 1e-6 for row in silva_deepening_rows)
refinement feedback factor rho | iterations | final residual | exact-state error | sensitivity | gradient error 0.20 | 12 | 0.000e+00 | 5.551e-17 | 1.2500 | 0.000e+00 0.45 | 23 | 0.000e+00 | 1.084e-08 | 1.8182 | 6.502e-08 0.70 | 45 | 0.000e+00 | 1.589e-07 | 3.3333 | 7.947e-08 0.85 | 93 | 0.000e+00 | 5.563e-07 | 6.6667 | 1.589e-07
import matplotlib.pyplot as silva_deepening_plt
silva_deepening_plt.rcParams.update({"figure.dpi": 300, "savefig.dpi": 300})
silva_deepening_figure, silva_deepening_axes = silva_deepening_plt.subplots(
1, 2, figsize=(8.6, 3.2)
)
for silva_deepening_rho, silva_deepening_history in silva_deepening_histories.items():
silva_deepening_axes[0].semilogy(
range(1, len(silva_deepening_history) + 1),
silva_deepening_history,
marker="o",
markersize=2,
linewidth=1.2,
label=f"rho={silva_deepening_rho:.2f}",
)
silva_deepening_axes[0].axhline(
silva_deepening_tolerance, color="black", linestyle="--", linewidth=0.9
)
silva_deepening_axes[0].set_xlabel("iteration")
silva_deepening_axes[0].set_ylabel("absolute residual")
silva_deepening_axes[0].set_title("Residual trajectories")
silva_deepening_axes[0].legend(fontsize=7)
silva_deepening_axes[1].plot(
[row[0] for row in silva_deepening_rows],
[row[1] for row in silva_deepening_rows],
marker="o",
label="iterations",
)
silva_deepening_sensitivity_axis = silva_deepening_axes[1].twinx()
silva_deepening_sensitivity_axis.plot(
[row[0] for row in silva_deepening_rows],
[row[4] for row in silva_deepening_rows],
color="tab:red",
marker="s",
label="sensitivity",
)
silva_deepening_axes[1].set_xlabel('refinement feedback factor')
silva_deepening_axes[1].set_ylabel("iterations")
silva_deepening_sensitivity_axis.set_ylabel("implicit sensitivity", color="tab:red")
silva_deepening_axes[1].set_title("Cost and sensitivity")
silva_deepening_figure.tight_layout()
silva_deepening_plt.show()
Reading and Extending the Result¶
The measured residual curves become flatter as the refinement feedback factor increases. The iteration count and the exact sensitivity rise together, but they answer different questions: iterations measure numerical work, while sensitivity describes how strongly the equilibrium reacts to the source. The gradient-error column verifies the differentiation path against the analytic derivative.
Apply the same separation to this notebook's full model:
| Report | Notebook-specific interpretation |
|---|---|
| Task evidence | distance to an analytic fixed point and final relative residual |
| Forward residual | Re-evaluate the complete transition at the returned state |
| Empirical rate | Compare consecutive residuals only after the transient regime |
| Backward residual | Record the linear-adjoint stopping value independently |
| Sensitivity | Perturb one declared source field while preserving all other inputs |
| Structural checks | state shape and a decreasing or bounded residual |
| Scale sweep | Change one of latent width, solver tolerance, and iteration budget at a time |
A richer experiment should now repeat the sweep with at least two forward solvers, two tolerances, and multiple seeds. Keep model parameters and data identical when comparing solvers. Then change one architecture or data-scale axis, retain the compact analytic study as a regression test, and report task quality, residuals, iterations, runtime, memory, gradient norms, and failed convergence cases together.