SILVA Physics-Informed Equilibrium¶
This lab derives a physics-informed equilibrium for an ODE initial-value problem, computes time derivatives with the implicit function theorem, and trains a tiny linear-decay task with boundary, equation, and Jacobian terms. The construction follows Physics-Informed Deep Equilibrium Models [51].
Numbered literature: [1], [4], [6], [14], [51]. Each number opens the complete citation and its primary external source.
from pathlib import Path
import importlib.util
import subprocess
import sys
REPO_URL = "https://github.com/jseluis/silva-networks.git"
def find_local_silva_root():
candidates = [Path.cwd(), Path("/content/silva-networks")]
root = Path.cwd()
while root != root.parent:
candidates.append(root)
root = root.parent
for candidate in candidates:
if (candidate / "src" / "silva_networks").exists():
return candidate
return None
root = find_local_silva_root()
if root is not None:
sys.path.insert(0, str(root / "src"))
elif importlib.util.find_spec("silva_networks") is None:
subprocess.check_call([sys.executable, "-m", "pip", "install", f"git+{REPO_URL}"])
root = Path.cwd()
else:
root = Path.cwd()
import torch
import matplotlib.pyplot as plt
plt.rcParams.update({"figure.dpi": 300, "savefig.dpi": 300})
from silva_networks import (
SILVAPhysicsInformedEquilibrium,
SolverConfig,
make_linear_ivp_dataset,
)
torch.manual_seed(240)
<torch._C.Generator at 0x117d6ceb0>
1. Initial-Value Problem¶
Consider
$$ \frac{dy}{dt}=N(t,y(t)), \qquad y(t_0)=y_0. $$
A physics-informed model is trained at collocation times without requiring a target state at every point. The package's analytic batch supplies targets only so this notebook can measure error after training.
data = make_linear_ivp_dataset(points=9, final_time=1.5, rate=-0.5)
assert data.equation_residual().abs().max() == 0
assert torch.allclose(data.target[:1], data.initial_state)
print("collocation points:", data.times.shape[0])
print("exact final state:", float(data.target[-1]))
collocation points: 9 exact final state: 0.2361832708120346
2. State Is Defined Implicitly¶
Instead of evaluating a finite stack, define
$$ z^\star(t)=f_\theta(z^\star(t),t), \qquad \widehat y(t)=Q_\psi(z^\star(t)). $$
In SILVA, time enters through the source branch and the latent state enters through the self-interaction branch. A standard output loss can use the package's implicit adjoint, avoiding storage of every forward solver iterate.
3. Time Derivative from the Implicit Function Theorem¶
Differentiate the fixed-point equation:
$$ \frac{dz^\star}{dt} =J_zf_\theta\frac{dz^\star}{dt}+J_tf_\theta. $$
Therefore
$$ \frac{dz^\star}{dt} =(I-J_zf_\theta)^{-1}J_tf_\theta, \qquad \frac{d\widehat y}{dt}=J_Q\frac{dz^\star}{dt}. $$
implicit_time_derivative solves this system with either a dense latent
Jacobian or matrix-free JVPs and GMRES. auto uses the dense path only below
the configured latent-dimension threshold. Both paths differentiate the same
implicit equation.
model = SILVAPhysicsInformedEquilibrium(
state_dim=4,
output_dim=1,
state_scale=0.15,
config=SolverConfig(
solver="picard",
max_iter=15,
tol=1e-6,
backward_mode="implicit",
backward_solver="gmres",
anderson_batch_dims=1,
),
)
initial = model(data.times, return_result=True)
derivative = model.implicit_time_derivative(data.times, initial.state, mode="dense")
matrix_free_derivative = model.implicit_time_derivative(
data.times,
initial.state,
mode="matrix_free",
)
assert derivative.shape == initial.output.shape == data.target.shape
torch.testing.assert_close(matrix_free_derivative, derivative, atol=1e-5, rtol=1e-5)
print("equilibrium residual:", initial.solver_result.residual)
equilibrium residual: 5.5275074828387e-07
4. Three-Term Physics-Informed Objective¶
The decomposed objective is
$$ \mathcal J_b=\|\widehat y(t_0)-y_0\|_2^2, $$
$$ \mathcal J_N =\frac1M\sum_{i=1}^M \left\|\frac{d\widehat y(t_i)}{dt} -N(t_i,\widehat y(t_i))\right\|_2^2, $$
$$ \mathcal J =\mathcal J_b+\lambda\mathcal J_N +\kappa\|J_zf_\theta\|_F^2. $$
The Jacobian term is estimated with Rademacher probes [6, 14, 51]. It is a solver-conditioning term, not the differential-equation residual.
loss = model.physics_loss(
data.times,
data.dynamics,
initial_time=data.times[:1],
initial_state=data.initial_state,
physics_weight=1.0,
jacobian_weight=1e-3,
jacobian_samples=1,
)
print("boundary:", float(loss.initial))
print("ODE residual:", float(loss.residual))
print("Jacobian estimate:", float(loss.jacobian))
boundary: 0.4718993604183197 ODE residual: 0.23153440654277802 Jacobian estimate: 0.08972296863794327
5. Tiny Physics-Only Training Run¶
No trajectory targets appear in the optimization objective below. They are used afterward only to calculate an error curve. This separation is essential when describing a physics-informed experiment.
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
history = []
for _ in range(8):
optimizer.zero_grad()
terms = model.physics_loss(
data.times,
data.dynamics,
initial_time=data.times[:1],
initial_state=data.initial_state,
jacobian_weight=1e-3,
)
terms.total.backward()
optimizer.step()
history.append((float(terms.initial.detach()), float(terms.residual.detach())))
trained = model(data.times, return_result=True)
trajectory_mse = torch.nn.functional.mse_loss(trained.output, data.target)
print("trajectory MSE used for evaluation:", float(trajectory_mse))
trajectory MSE used for evaluation: 0.0450901985168457
figure, axes = plt.subplots(1, 2, figsize=(7.0, 2.6))
axes[0].plot(data.times[:, 0], data.target[:, 0], label="exact", linewidth=2)
axes[0].plot(data.times[:, 0], trained.output.detach()[:, 0], "--", label="SILVA")
axes[0].set(xlabel="time", ylabel="state")
axes[0].legend()
axes[1].semilogy([item[0] for item in history], label="boundary")
axes[1].semilogy([item[1] for item in history], label="ODE residual")
axes[1].set(xlabel="optimization step", ylabel="loss")
axes[1].legend()
figure.tight_layout()
plt.show()
6. Stiff Systems and Scaling¶
An equilibrium layer does not automatically solve stiffness. Stability depends on the transition, root solver, Jacobian spectrum, collocation distribution, and loss balancing. For larger states, use matrix-free implicit products, report forward and backward tolerances separately, and compare against a trusted numerical integrator on the same time interval.
From 24 Silva Physics Informed Equilibrium to a Custom SILVA Family¶
The construction in this notebook can be separated into the universal conditioned-equilibrium contract
$$ z_0=I_\eta(x),\qquad z^\star=T_\theta(z^\star,x),\qquad \widehat y=Q_\psi(z^\star). $$
For this topic:
| Part | Concrete interpretation |
|---|---|
| Equilibrium state | an implicit latent state or coupled differential/algebraic stage state |
| Condition | time, initial/boundary values, dynamics, and algebraic constraints |
| Repeated computation | a time-conditioned fixed point or implicit Runge-Kutta root map |
| Required invariants | initial/boundary conditions, equation shape, and constraint consistency |
| Replaceable components | time/source lift, transition, readout, dynamics, constraints, losses, and solvers |
The initializer and source path are evaluated outside or alongside the root solve. Only the state-preserving transition is repeated. Replacing an internal architecture does not change this equation, provided the transition still maps the same state space into itself.
Build the Physics-Informed Transition Explicitly¶
class MyPhysicsTransition(nn.Module):
time_dim = 1
state_dim = 64
def __init__(self):
super().__init__()
self.time_source = MyTimeEncoder()
self.state_operator = MyResidualOrOperatorNetwork()
def forward(self, state, times):
source = self.time_source(times)
return torch.tanh(source + 0.2 * self.state_operator(state))
model = SILVAPhysicsInformedEquilibrium(
state_dim=64,
output_dim=physical_dimension,
transition=MyPhysicsTransition(),
readout=my_physical_readout,
derivative_mode="matrix_free",
derivative_max_iter=100,
derivative_tol=1e-7,
config=solver_config,
)
physics_loss then accepts the user-supplied dynamics. The transition defines
the implicit representation; the dynamics define the differential-equation
residual. They are related by the implicit time derivative but are not the same
module.
import torch as silva_extension_torch
from torch import nn as silva_extension_nn
from silva_networks import (
SILVAConditionedEquilibrium,
SILVAZeroInitializer,
SolverConfig,
validate_silva_transition,
)
class NotebookExtensionTransition(silva_extension_nn.Module):
def __init__(self, condition_dim=2, state_dim=3):
super().__init__()
self.source = silva_extension_nn.Linear(condition_dim, state_dim)
self.state_field = silva_extension_nn.Sequential(
silva_extension_nn.Linear(state_dim, 2 * state_dim),
silva_extension_nn.Tanh(),
silva_extension_nn.Linear(2 * state_dim, state_dim),
)
def forward(self, state, condition):
return silva_extension_torch.tanh(
self.source(condition) + 0.15 * self.state_field(state)
)
silva_extension_torch.manual_seed(610)
notebook_condition = silva_extension_torch.linspace(-1.0, 1.0, 8).reshape(4, 2)
notebook_state0 = silva_extension_torch.zeros(4, 3)
notebook_transition = NotebookExtensionTransition()
notebook_report = validate_silva_transition(
notebook_transition,
notebook_state0,
notebook_condition,
)
assert notebook_report.valid
with silva_extension_torch.no_grad():
notebook_reference_step = silva_extension_torch.tanh(
notebook_transition.source(notebook_condition)
+ 0.15 * notebook_transition.state_field(notebook_state0)
)
silva_extension_torch.testing.assert_close(
notebook_transition(notebook_state0, notebook_condition),
notebook_reference_step,
)
notebook_custom_model = SILVAConditionedEquilibrium(
notebook_transition,
SILVAZeroInitializer(3),
readout=silva_extension_nn.Linear(3, 1),
config=SolverConfig(
solver="picard",
max_iter=40,
tol=1e-7,
backward_mode="implicit",
backward_solver="gmres",
anderson_batch_dims=1,
),
)
notebook_custom_result = notebook_custom_model(
notebook_condition,
return_result=True,
)
assert notebook_custom_result.output.shape == (4, 1)
assert notebook_custom_result.solver_result.residual < 1e-5
notebook_custom_result.output.square().mean().backward()
assert all(
parameter.grad is not None and silva_extension_torch.isfinite(parameter.grad).all()
for parameter in notebook_custom_model.parameters()
)
print("custom transition:", notebook_report)
print("equilibrium residual:", notebook_custom_result.solver_result.residual)
custom transition: SILVATransitionReport(state_shape=(4, 3), output_shape=(4, 3), preserves_shape=True, preserves_device=True, preserves_dtype=True, finite=True, differentiable=True, parameter_count=54) equilibrium residual: 5.960464477539063e-08
Numerical Equivalence, Compact Reproduction, and Scale¶
Before training, compare one packaged transition with an independently written update:
$$ e_{\mathrm{step}} =\frac{\|T_\theta(z,x)-T_{\mathrm{ref}}(z,x)\|_2} {\|T_{\mathrm{ref}}(z,x)\|_2+\varepsilon}. $$
After solving, report the fixed-point residual separately:
$$ e_{\mathrm{fp}} =\frac{\|T_\theta(z^\star,x)-z^\star\|_2} {\|z^\star\|_2+\varepsilon}. $$
For this notebook, a compact reproduction must declare and assert equation residual, boundary error, trajectory error, and solver residual. A full experiment must additionally record the source dataset version and split, preprocessing, architecture widths, solver and optimizer schedules, random seeds, baseline configuration, checkpoints, and every deviation from the cited protocol.
The principal scaling axes are collocation count, latent dimension, stages, stiffness, and time horizon. Increase one axis at a time, retain the compact deterministic case as a regression test, and record task error, domain-specific residual, forward residual, backward linear residual, memory use, and runtime independently.
Extension Exercises¶
- Replace one component from this notebook while preserving its state and domain invariants.
- Write the replacement first as an independent reference function, then as a module, and assert one-step equivalence.
- Compare two solver configurations on the identical trained transition.
- Add a compact baseline and a predeclared metric threshold.
- Create a full-scale configuration without weakening the compact tests.
The complete authoring protocol is documented in Extending SILVA.
notebook_reproduction_record = {
"notebook": '24_silva_physics_informed_equilibrium.ipynb',
"state": 'an implicit latent state or coupled differential/algebraic stage state',
"condition": 'time, initial/boundary values, dynamics, and algebraic constraints',
"transition": 'a time-conditioned fixed point or implicit Runge-Kutta root map',
"invariants": 'initial/boundary conditions, equation shape, and constraint consistency',
"compact_metric": 'equation residual, boundary error, trajectory error, and solver residual',
"scale_axis": 'collocation count, latent dimension, stages, stiffness, and time horizon',
}
assert all(notebook_reproduction_record.values())
notebook_reproduction_record
{'notebook': '24_silva_physics_informed_equilibrium.ipynb',
'state': 'an implicit latent state or coupled differential/algebraic stage state',
'condition': 'time, initial/boundary values, dynamics, and algebraic constraints',
'transition': 'a time-conditioned fixed point or implicit Runge-Kutta root map',
'invariants': 'initial/boundary conditions, equation shape, and constraint consistency',
'compact_metric': 'equation residual, boundary error, trajectory error, and solver residual',
'scale_axis': 'collocation count, latent dimension, stages, stiffness, and time horizon'}
Worked Convergence and Sensitivity Study¶
The preceding example demonstrates one configured solve. This additional study changes the implicit dynamics factor while keeping the source fixed, so solver effort and implicit sensitivity can be read separately from task behavior. Locally, one eigendirection of a nonlinear transition can be represented by
$$ z_{k+1} = \rho z_k + u, \qquad 0 \leq \rho < 1. $$
Its equilibrium is
$$ z^\star = \frac{u}{1-\rho}. $$
Subtracting the fixed-point equation from the iteration gives the exact error recursion
$$ e_{k+1} = \rho e_k, \qquad |e_k| = \rho^k |e_0|. $$
For a requested absolute tolerance $\tau$, the idealized iteration estimate is
$$ k \geq \frac{\log(\tau/|e_0|)}{\log \rho}. $$
The same factor controls sensitivity. Differentiating the equilibrium with respect to the source gives
$$ \frac{\partial z^\star}{\partial u} =\frac{1}{1-\rho}. $$
Thus a transition can remain contractive while becoming expensive and highly sensitive as $\rho$ approaches one. The table and figure below measure this effect rather than merely stating it. They provide a reference envelope for the notebook's actual state, an implicit latent state or coupled differential/algebraic stage state, and its repeated map, a time-conditioned fixed point or implicit Runge-Kutta root map. The scalar study does not replace the domain model; it supplies a result whose convergence rate and derivative are known exactly, so the same reporting code can be trusted before it is applied to the larger transition.
import math as silva_deepening_math
import torch as silva_deepening_torch
silva_deepening_rates = (0.20, 0.45, 0.70, 0.85)
silva_deepening_source = 0.35
silva_deepening_tolerance = 1e-8
silva_deepening_histories = {}
silva_deepening_rows = []
for silva_deepening_rho in silva_deepening_rates:
silva_deepening_state = silva_deepening_torch.tensor(0.0)
silva_deepening_exact = silva_deepening_source / (1.0 - silva_deepening_rho)
silva_deepening_history = []
for silva_deepening_iteration in range(1, 241):
silva_deepening_next = (
silva_deepening_rho * silva_deepening_state + silva_deepening_source
)
silva_deepening_residual = abs(
float(silva_deepening_next - silva_deepening_state)
)
silva_deepening_history.append(silva_deepening_residual)
silva_deepening_state = silva_deepening_next
if silva_deepening_residual < silva_deepening_tolerance:
break
silva_deepening_u = silva_deepening_torch.tensor(
silva_deepening_source, requires_grad=True
)
silva_deepening_solution = silva_deepening_u / (1.0 - silva_deepening_rho)
silva_deepening_solution.backward()
silva_deepening_expected_sensitivity = 1.0 / (1.0 - silva_deepening_rho)
silva_deepening_gradient_error = abs(
float(silva_deepening_u.grad) - silva_deepening_expected_sensitivity
)
silva_deepening_histories[silva_deepening_rho] = silva_deepening_history
silva_deepening_rows.append(
(
silva_deepening_rho,
silva_deepening_iteration,
silva_deepening_history[-1],
abs(float(silva_deepening_state) - silva_deepening_exact),
float(silva_deepening_u.grad),
silva_deepening_gradient_error,
)
)
print('implicit dynamics factor')
print("rho | iterations | final residual | exact-state error | sensitivity | gradient error")
for silva_deepening_row in silva_deepening_rows:
print(
f"{silva_deepening_row[0]:.2f} | {silva_deepening_row[1]:3d} | "
f"{silva_deepening_row[2]:.3e} | {silva_deepening_row[3]:.3e} | "
f"{silva_deepening_row[4]:.4f} | {silva_deepening_row[5]:.3e}"
)
assert all(row[2] < silva_deepening_tolerance for row in silva_deepening_rows)
assert all(row[3] < 1e-6 for row in silva_deepening_rows)
assert all(row[5] < 1e-6 for row in silva_deepening_rows)
implicit dynamics factor rho | iterations | final residual | exact-state error | sensitivity | gradient error 0.20 | 12 | 0.000e+00 | 5.551e-17 | 1.2500 | 0.000e+00 0.45 | 23 | 0.000e+00 | 1.084e-08 | 1.8182 | 6.502e-08 0.70 | 45 | 0.000e+00 | 1.589e-07 | 3.3333 | 7.947e-08 0.85 | 93 | 0.000e+00 | 5.563e-07 | 6.6667 | 1.589e-07
import matplotlib.pyplot as silva_deepening_plt
silva_deepening_plt.rcParams.update({"figure.dpi": 300, "savefig.dpi": 300})
silva_deepening_figure, silva_deepening_axes = silva_deepening_plt.subplots(
1, 2, figsize=(8.6, 3.2)
)
for silva_deepening_rho, silva_deepening_history in silva_deepening_histories.items():
silva_deepening_axes[0].semilogy(
range(1, len(silva_deepening_history) + 1),
silva_deepening_history,
marker="o",
markersize=2,
linewidth=1.2,
label=f"rho={silva_deepening_rho:.2f}",
)
silva_deepening_axes[0].axhline(
silva_deepening_tolerance, color="black", linestyle="--", linewidth=0.9
)
silva_deepening_axes[0].set_xlabel("iteration")
silva_deepening_axes[0].set_ylabel("absolute residual")
silva_deepening_axes[0].set_title("Residual trajectories")
silva_deepening_axes[0].legend(fontsize=7)
silva_deepening_axes[1].plot(
[row[0] for row in silva_deepening_rows],
[row[1] for row in silva_deepening_rows],
marker="o",
label="iterations",
)
silva_deepening_sensitivity_axis = silva_deepening_axes[1].twinx()
silva_deepening_sensitivity_axis.plot(
[row[0] for row in silva_deepening_rows],
[row[4] for row in silva_deepening_rows],
color="tab:red",
marker="s",
label="sensitivity",
)
silva_deepening_axes[1].set_xlabel('implicit dynamics factor')
silva_deepening_axes[1].set_ylabel("iterations")
silva_deepening_sensitivity_axis.set_ylabel("implicit sensitivity", color="tab:red")
silva_deepening_axes[1].set_title("Cost and sensitivity")
silva_deepening_figure.tight_layout()
silva_deepening_plt.show()
Reading and Extending the Result¶
The measured residual curves become flatter as the implicit dynamics factor increases. The iteration count and the exact sensitivity rise together, but they answer different questions: iterations measure numerical work, while sensitivity describes how strongly the equilibrium reacts to the source. The gradient-error column verifies the differentiation path against the analytic derivative.
Apply the same separation to this notebook's full model:
| Report | Notebook-specific interpretation |
|---|---|
| Task evidence | equation residual, boundary error, trajectory error, and solver residual |
| Forward residual | Re-evaluate the complete transition at the returned state |
| Empirical rate | Compare consecutive residuals only after the transient regime |
| Backward residual | Record the linear-adjoint stopping value independently |
| Sensitivity | Perturb one declared source field while preserving all other inputs |
| Structural checks | initial/boundary conditions, equation shape, and constraint consistency |
| Scale sweep | Change one of collocation count, latent dimension, stages, stiffness, and time horizon at a time |
A richer experiment should now repeat the sweep with at least two forward solvers, two tolerances, and multiple seeds. Keep model parameters and data identical when comparing solvers. Then change one architecture or data-scale axis, retain the compact analytic study as a regression test, and report task quality, residuals, iterations, runtime, memory, gradient norms, and failed convergence cases together.