算法 on Detail's Blog

拉格朗日乘子法

Wed, 18 Mar 2026 00:00:00 +0000

基本概念

拉格朗日乘子法（Lagrange Multipliers）是一种寻找多元函数在一组约束下的极值的方法。通过引入拉格朗日乘子，可以将有$d$个变量与$k$个约束条件的最优化问题转化为具有$d+k$个变量的无约束优化问题。¹

问题求解

等式约束

$$ \min_\mathbf x f(\mathbf x)\qquad\text{s.t.}\quad g(\mathbf x)=0, $$

约束曲面与极值曲面相切的点为极值点$\mathbf x^\ast$。

对于约束曲面上的任意点$\mathbf x$，该点的梯度$\nabla g(\mathbf x)$正交于约束曲面。
在最优点$\mathbf x^\ast$，目标函数在该点的梯度$\nabla f(\mathbf x^\ast)$正交于约束曲面。

在最优点$\mathbf x^\ast$梯度$\nabla g(\mathbf x)$和$\nabla f(\mathbf x)$的方向必相同或相反，即存在$\lambda\ne 0$，使得$\nabla f(\mathbf x^\ast)+\lambda\nabla g(\mathbf x^\ast)=0$ 构造拉格朗日函数$L(\mathbf x,\lambda)=f(\mathbf x)+\lambda g(\mathbf x)$

$\nabla_\mathbf xL(\mathbf x,\lambda)=0 \Rightarrow \nabla f(\mathbf x^\ast)+\lambda\nabla g(\mathbf x^\ast)=0$
$\nabla_\lambda L(\mathbf x,\lambda)=0 \Rightarrow g(\mathbf x)=0$

不等式约束

$$ \min_\mathbf x f(\mathbf x)\qquad\text{s.t.}\quad g(\mathbf x)\le0, $$

最优点在不等式约束的区域内时，等价于无约束问题。最优点在不等式约束边界上时，等价于等式约束问题。

对于情况一，最优点在$g(\mathbf x)<0$区域内，令$\nabla f(\mathbf x)=0$求解。即$\nabla_\mathbf xL(\mathbf x, 0)=0$。
对于情况二，最优解在$g(\mathbf x)=0$上，即存在$\nabla f(\mathbf x^\ast)+\lambda\nabla g(\mathbf x^\ast)=0$。注意到若两个梯度方向相同，则还可再优化，$\mathbf x$不是最优解。因此，需要取两个梯度方向相反，即$\lambda > 0$。

结合上述两种情况，二者都满足$\lambda g(\mathbf x)=0$。因此，原优化问题可以转化成在KKT（Karush-Kuhn-Tucker）条件下的最小化拉格朗日函数

$$ L(\mathbf x,\lambda)=f(\mathbf x)+\lambda g(\mathbf x) \qquad\text{s.t.}\quad \left\{\begin{align*} g(\mathbf x)\le0\\ \lambda\ge0\\ \lambda g(\mathbf x)=0 \end{align*}\right.. $$

马轶荀. 优化-拉格朗日乘子法[EB/OL], (2020-07-03). https://zhuanlan.zhihu.com/p/154517678 ↩︎

Nesterov 牛顿动量

Fri, 06 Mar 2026 00:00:00 +0000

基本理论

动量法（Momentum）

普通梯度下降法$\theta\leftarrow\theta-\eta\nabla_\theta J(\theta)$在接近最优值时收敛速度会变慢，有事甚至陷入局部最优。这时如果考虑历史梯度，将会引导参数向最优值更快收敛，这就是动量算法基本思想。

$$ \begin{align*} m&\leftarrow\gamma m+\eta\nabla_\theta J(\theta) \\ \theta&\leftarrow\theta -m \end{align*}. $$

牛顿动量（Nesterov）

尽管动量法能最终找到最优解，但是在不同维度梯度差较大时容易震荡。牛顿动量法在超前一个动量单位处进行梯度更新，公式修正为

$$ \begin{align*} m&\leftarrow\gamma m+\eta\nabla_\theta J(\theta+\beta m) \\ \theta&\leftarrow\theta -m \end{align*}. $$

仿真

对二元函数$z=x^2-4xy+10y^2$分别用梯度下降法、动量法和牛顿动量法求最小值。$\eta=0.012, \gamma=0.8$。梯度下降方法速度慢，动量法震荡明显。

结果

迭代结果和迭代次数

1
2
3

Gradient 最优点: (0.0000, 0.0000), 迭代次数: 816
Momentum 最优点: (0.0000, -0.0000), 迭代次数: 176
Nesterov 最优点: (0.0000, 0.0000), 迭代次数: 137

优化路径

梯度下降

动量法

牛顿动量

代码

global A;   %#ok
% 此处仅作演示，实际编程过程中应避免使用全局变量。
A = [...
    1, -2; ...
    -2, 10 ...
];
x_0 = [150, 100].';
eta = 1.2e-2;
gamma = 0.8;
max_iter = 1000;

dim1_space = linspace(-200, 200, 1e2);
dim2_space = linspace(-200, 200, 1e2);
[X, Y] = meshgrid(dim1_space, dim2_space);
Z = arrayfun(@(x, y) dst_function([x; y]), X, Y);

% 三维曲面图
figure;
surf(X, Y, Z, 'EdgeColor', 'none');
colormap(jet);
colorbar;
xlabel('x_1');
ylabel('x_2');

[x_opt, x_history] = gradient_descent(x_0, eta, max_iter);
% 绘制优化路径
figure;
contour(X, Y, Z, 50);
colormap(jet);
colorbar;
hold on;
plot(x_history(1, :), x_history(2, :), 'ro-', 'LineWidth', 2, 'MarkerSize', 5);
xlabel('x_1');
ylabel('x_2');
% 绘制最优点
hold on;
plot(x_opt(1), x_opt(2), 'kx', 'LineWidth', 2, 'MarkerSize', 10);
legend('等高线', '优化路径', '最优点');
fprintf('Gradient 最优点: (%.4f, %.4f), 迭代次数: %d\n', x_opt(1), x_opt(2), size(x_history, 2) - 1);

[x_opt, x_history] = momentum_method(x_0, eta, gamma, max_iter);
% 绘制优化路径
figure;
contour(X, Y, Z, 50);
colormap(jet);
colorbar;
hold on;
plot(x_history(1, :), x_history(2, :), 'ro-', 'LineWidth', 2, 'MarkerSize', 5);
xlabel('x_1');
ylabel('x_2');
% 绘制最优点
hold on;
plot(x_opt(1), x_opt(2), 'kx', 'LineWidth', 2, 'MarkerSize', 10);
legend('等高线', '优化路径', '最优点');
fprintf('Momentum 最优点: (%.4f, %.4f), 迭代次数: %d\n', x_opt(1), x_opt(2), size(x_history, 2) - 1);

[x_opt, x_history] = nesterov_method(x_0, eta, gamma, max_iter);
% 绘制优化路径
figure;
contour(X, Y, Z, 50);
colormap(jet);
colorbar;
hold on;
plot(x_history(1, :), x_history(2, :), 'ro-', 'LineWidth', 2, 'MarkerSize', 5);
xlabel('x_1');
ylabel('x_2');
% 绘制最优点
hold on;
plot(x_opt(1), x_opt(2), 'kx', 'LineWidth', 2, 'MarkerSize', 10);
legend('等高线', '优化路径', '最优点');
fprintf('Nesterov 最优点: (%.4f, %.4f), 迭代次数: %d\n', x_opt(1), x_opt(2), size(x_history, 2) - 1);

function [x_opt, x_history] = gradient_descent(x_0, eta, max_iter)
    x_history = zeros(length(x_0), max_iter+1);
    x_history(:, 1) = x_0;
    x = x_0;
    for i = 1:max_iter
        grad = dst_function_gradient(x);
        x = x - eta * grad;
        x_history(:, i+1) = x;
        if norm(grad) < 1e-6  % 梯度足够小，认为收敛
            x_history = x_history(:, 1:i+1);  % 截断历史记录
            break;
        end
    end
    x_opt = x;
end

function [x_opt, x_history] = momentum_method(x_0, eta, gamma, max_iter)
    x_history = zeros(length(x_0), max_iter+1);
    x_history(:, 1) = x_0;
    x = x_0;
    m = zeros(size(x_0));
    for i = 1:max_iter
        grad = dst_function_gradient(x);
        m = gamma * m + eta * grad;
        x = x - m;
        x_history(:, i+1) = x;
        if norm(grad) < 1e-6  % 梯度足够小，认为收敛
            x_history = x_history(:, 1:i+1);  % 截断历史记录
            break;
        end
    end
    x_opt = x;
end

function [x_opt, x_history] = nesterov_method(x_0, eta, gamma, max_iter)
    x_history = zeros(length(x_0), max_iter+1);
    x_history(:, 1) = x_0;
    x = x_0;
    m = zeros(size(x_0));
    for i = 1:max_iter
        y = x - gamma * m;  % 计算预测点
        grad = dst_function_gradient(y);  % 在预测点计算梯度
        m = gamma * m + eta * grad;  % 更新动量
        x = x - m;  % 更新参数
        x_history(:, i+1) = x;
        if norm(grad) < 1e-6  % 梯度足够小，认为收敛
            x_history = x_history(:, 1:i+1);  % 截断历史记录
            break;
        end
    end
    x_opt = x;
end

%% 目标函数
function y = dst_function(x)
    global A;   %#ok
    y = x.' * A * x;
end

function grad = dst_function_gradient(x)
    global A;   %#ok
    grad = 2 * A * x;
end

IST

Wed, 04 Mar 2026 00:00:00 +0000

基本原理

迭代软阈值（Iterative Soft Thresholding, IST）利用LASSO回归模型将问题表述为

$$ \arg\min_\mathbf x \lVert{\mathbf y-\Phi\mathbf x}\rVert_2^2+\lambda\lVert \mathbf x\rVert_1, $$

可以得到其迭代方法

初始化：$\mathbf x^{(0)}=0$或$\mathbf x^{(0)}=\Phi ^\text T \mathbf y$
计算梯度下降：$\mathbf r^{(k)}=\mathbf x^{(k)}-t\nabla\lVert\mathbf y-\Phi \mathbf x^{(k)}\rVert_2^2$
迭代优化直至收敛：$\mathbf x^{(k+1)}=S_{\lambda t}(\mathbf r^{(k)})$ 其中$t$为步长，$S_\omega(x)$表示软阈值函数，即 $$ [S_\omega(x)]_i = \left\{ \begin{align*} &x_i-\omega, \quad x_i>\omega\\ &0, \quad \lvert x_i\rvert\le\omega\\ &x_i+\omega, \quad x_i<-\omega \end{align*}\right. $$

推导过程

IST的推导需要使用近端梯度下降（PGD）方法，可以将原问题修改为

$$ \mathbf x = \arg\min_\mathbf x {\frac12\lVert A\mathbf x-\mathbf b\rVert_2^2+\lambda\lVert\mathbf x\rVert_1}, $$

令$f(\mathbf x)=\frac12\lVert A\mathbf x-\mathbf b\rVert_2^2+\lambda\lVert\mathbf x\rVert_1$，对其求导

$$ f^\prime(\mathbf x) = A^\text T (A\mathbf x-\mathbf b)+\lambda\text{sgn}(\mathbf x), $$

求其最小值即令$f^\prime(\mathbf x)=0$。

特殊情况$A=I$

特别的当$A=I$时，$f^\prime(\mathbf x) = (\mathbf x-\mathbf b)+\lambda\text{sgn}(\mathbf x)=0$，其解为软阈值函数

$$ \mathbf x = S_\lambda(\mathbf b) = \left\{ \begin{align*} &b+\lambda, \quad b< -\lambda \\ &0, \quad b\le\lvert\lambda\rvert \\ &b-\lambda, \quad b>\lambda \end{align*}\right.. $$

一般情况

回到一般情况，令$f(\mathbf x) = \lVert A\mathbf x-\mathbf b\rVert^2_2$。对$f(\mathbf x)$在$\mathbf x^{(k)}$处做二阶泰勒展开，令$L=\nabla^2f(\mathbf x)=\frac1t$

$$ \begin{align*} \arg\min_\mathbf x f(\mathbf x) &= \arg\min_\mathbf x{f(\mathbf x^{(k)})+\nabla f(\mathbf x^{(k)})^\text T (\mathbf x -\mathbf x^{(k)})+\frac12(\mathbf x-\mathbf x^{(k)})^\text T\nabla^2f(\mathbf x^{(k)})(\mathbf x-\mathbf x^{(k)})} \\ &= \arg\min_\mathbf x\frac1{2t}\sum_i{t^2[\nabla f(\mathbf x^{(k)})]_i^2+2t[\nabla f(\mathbf x^{(k)})]_i (x_i -x_i^{(k)})+(x_i-x_i^{(k)})^2} \\ &= \arg\min_\mathbf x\frac1{2t}{\lVert\mathbf x-[\mathbf x^{(k)}-t\nabla f(\mathbf x^{(k)})]\rVert^2_2} \\ \end{align*}. $$

此时，令$\mathbf z^{(k)}=\mathbf x^{(k)}-t\nabla f(\mathbf x^{(k)})$，原凸优化问题转换为$A=I$的形式（注意这里是$\mathbf x^{(k+1)}=\text{prox}_{h,t}(\mathbf z^{(k)})$），于是有

$$ \begin{align*} \mathbf x^{(k+1)} &= \arg\min_\mathbf x \frac1{2t}\lVert\mathbf x-\mathbf z\rVert^2_2+\lambda\lVert\mathbf x\rVert_1 \\ &= \arg\min_\mathbf x \frac1{2}\lVert\mathbf x-\mathbf z\rVert^2_2+\lambda t\lVert\mathbf x\rVert_1 \\ &= S_{\lambda t}(\mathbf z^{(k)}) \end{align*}. $$

步长选取

固定步长

假设函数$f(\mathbf x) = \lVert A\mathbf x-\mathbf b\rVert^2_2$满足梯度上的Lipschitz连续。当步长$t\le\frac1L$时，能保证多次迭代后最终收敛至梯度为0的点，$L$表示$\nabla f$的Lipschitz常数。

$$ L = \sup\frac{\lVert\nabla f(a)-\nabla f(b)\rVert_2}{\lVert a-b\rVert_2} $$

令$a=\mathbf x+\mathbf x_0,b=\mathbf x$则

$$ \begin{align*} L &= \sup\frac{\lVert\nabla f(\mathbf x+\mathbf x_0)-\nabla f(\mathbf x)\rVert_2}{\lVert\mathbf x_0\rVert_2} \\ &= \sup\frac{\lVert 2A^\text T [A(\mathbf x+\mathbf x_0)-\mathbf b]-2A^\text T (A\mathbf x-\mathbf b)\rVert_2}{\lVert\mathbf x_0\rVert_2} \\ &= \sup\frac{2\lVert A^\text TA\mathbf x_0\rVert_2}{\lVert\mathbf x_0\rVert_2} \\ &\le\frac{2\lVert\lambda_\text{max}(A^\text TA)\mathbf x_0\rVert_2}{\lVert\mathbf x_0\rVert_2} \\ &=2\lambda_\text{max}(A^\text TA) \end{align*} $$

其中$\lambda_\text{max}(\cdot)$表示取最大特征值，令$t=\frac1L=\frac1{2\lambda_\text{max}(A^\text TA)}$

回溯型步长

当矩阵$A$很大时，不容易计算$A^\text TA$的特征值，或者不知道矩阵$A$时，可以通过回溯搜索的方式估计Lipschitz常数。将$f(\mathbf x)$泰勒展开并放缩得到上界函数

$$ \begin{align*} f(\mathbf x) &={f(\mathbf x_0)+\nabla f(\mathbf x_0)^\text T (\mathbf x -\mathbf x_0)+\frac12(\mathbf x-\mathbf x_0)^\text T\nabla^2f(\mathbf x_0)(\mathbf x-\mathbf x_0)}+o(x^3) \\ &\le{f(\mathbf x_0)+\nabla f(\mathbf x_0)^\text T (\mathbf x -\mathbf x_0)+\frac L2(\mathbf x-\mathbf x_0)^\text T(\mathbf x-\mathbf x_0)} \\ &=Q(\mathbf x,\mathbf x_0;L) \end{align*}, $$

令$\mathbf x=\mathbf x^{(k+1)},\mathbf x_0=x^{(k)}$，不断增大$\hat L$，直到$f(\mathbf x^{(k+1)})\le Q(\mathbf x^{(k+1)},\mathbf x^{(k)};\hat L)$满足。

\begin{algorithm}
\caption{BackTracking}
\begin{algorithmic}
	\STATE 初始化$L_0>0, \eta>1, \mathbf x_0\in\mathbb R^n$
	\REPEAT
		\STATE $\hat L = \eta L_\text{before}$
	\UNTIL{$f(\mathbf x^{(k+1)})\le Q(\mathbf x^{(k+1)},\mathbf x^{(k)};\hat L)$}
\end{algorithmic}
\end{algorithm}

PGD

Wed, 04 Mar 2026 00:00:00 +0000

基本原理

近端梯度下降（Proximal Gradient Descent）是梯度下降方法的一种，其适用于目标函数存在不可微部分的凸优化问题。问题描述为

$$ \mathbf x = \arg\min_\mathbf x {g(\mathbf x)+h(\mathbf x)}, $$

其中，$g(\mathbf x)$为可微凸函数、$h(\mathbf x)$为不可微凸函数。

求解方法

求解该类型凸优化问题可以通过近端映射（Proximal Mapping）和梯度迭代的手段实现。近端映射函数定义为

$$ \text{prox}_{h,t}(\mathbf x) = \arg\min_\mathbf z{\frac1{2t}\lVert\mathbf x-\mathbf z\rVert^2_2+h(\mathbf z)}, $$

梯度迭代方法为

$$ \begin{align*} &\mathbf z^{(k)}=\mathbf x^{(k)}-t_k\nabla g(\mathbf x^{(k)}) \\ &\mathbf x^{(k+1)}=\text{prox}_{h,t_k}(\mathbf z^{(k)}) \end{align*} $$

证明

对$g(\mathbf x)$在$\mathbf x^{(k)}$处做二阶泰勒展开，令$L=\nabla^2g(\mathbf x)=\frac1t$

$$ \begin{align*} \arg\min_\mathbf x g(\mathbf x)+h(\mathbf x) &\approx \arg\min_\mathbf x{g(\mathbf x^{(k)})+\nabla g(\mathbf x^{(k)})^\text T (\mathbf x -\mathbf x^{(k)})+\frac12(\mathbf x-\mathbf x^{(k)})^\text T\nabla^2g(\mathbf x^{(k)})(\mathbf x-\mathbf x^{(k)})}+h(\mathbf x) \\ \mathbf{\hat x}&= \arg\min_\mathbf x{\nabla g(\mathbf x^{(k)})^\text T (\mathbf x -\mathbf x^{(k)})+\frac1{2t}(\mathbf x-\mathbf x^{(k)})^\text T(\mathbf x-\mathbf x^{(k)})}+h(\mathbf x) \\ &= \arg\min_\mathbf x\sum_i{[\nabla g(\mathbf x^{(k)})]_i (x_i -x_i^{(k)})+\frac1{2t}(x_i-x_i^{(k)})^2}+h(\mathbf x) \\ &= \arg\min_\mathbf x\frac1{2t}\sum_i{2t[\nabla g(\mathbf x^{(k)})]_i (x_i -x_i^{(k)})+(x_i-x_i^{(k)})^2}+h(\mathbf x) \\ &= \arg\min_\mathbf x\frac1{2t}\sum_i{t^2[\nabla g(\mathbf x^{(k)})]_i^2+2t[\nabla g(\mathbf x^{(k)})]_i (x_i -x_i^{(k)})+(x_i-x_i^{(k)})^2}+h(\mathbf x) \\ &= \arg\min_\mathbf x\frac1{2t}\sum_i{[x_i-x_i^{(k)}+t[\nabla g(\mathbf x^{(k)})]_i]^2}+h(\mathbf x) \\ &= \arg\min_\mathbf x\frac1{2t}{\lVert\mathbf x-[\mathbf x^{(k)}-t\nabla g(\mathbf x^{(k)})]\rVert^2_2}+h(\mathbf x) \\ \end{align*}. $$

令$\mathbf z^{(k)}=\mathbf x^{(k)}-t\nabla g(\mathbf x^{(k)})$，原问题即可转换为近端映射

$$ \begin{align*} \mathbf{\hat x} &= \arg\min_\mathbf x\frac1{2t}{\lVert\mathbf x-\mathbf z\rVert^2_2}+h(\mathbf x) \\ &=\text{prox}_{h,t}(\mathbf z) \end{align*}. $$

设定合适的参数，即可使用迭代方法求得$\arg\min_\mathbf x g(\mathbf x)+h(\mathbf x)$的结果。

Lipschitz常数

假定目标凸函数满足梯度上的Lipschitz连续。当步长$t\le\frac1L$时，保证多次迭代后能收敛到梯度为0的点。梯度Lipschitz连续表示为

$$ \frac{\lVert\nabla f(a)-\nabla f(b)\rVert_2}{\lVert a-b\rVert_2} \le L(f) \quad \forall a,b\in\text{dom}f, $$

函数$\nabla f$的Lipschitz常数为

$$ L = \sup\frac{\lVert\nabla f(a)-\nabla f(b)\rVert_2}{\lVert a-b\rVert_2} $$

IHT

Tue, 10 Feb 2026 00:00:00 +0000

基本原理

迭代硬阈值算法（Iterative Hard Thresholding, IHT），在提出时称M-Sparse Algorithm。$M$稀疏问题可以表示为

$$ \min_\mathbf y\lVert \mathbf x - \Phi\mathbf y \rVert^2_2\quad \text{s.t.}\quad \lVert\mathbf y\rVert_0\le M, $$

可以得到其迭代算法

$$ \mathbf y^{n+1}=H_M(\mathbf y^n+\Phi^\rm H(\mathbf x-\Phi\mathbf y^n)), $$

其中，$H_M$为非线性算子，其返回$M$维最大幅值

$$ H_M(y_i)= \left\{ \begin{align} 0\quad \lvert y_i\rvert\lt\lambda^{0.5}_M(\mathbf y)\\ y_i\quad \lvert y_i\rvert\ge\lambda^{0.5}_M(\mathbf y) \end{align} \right.. $$

推导过程

替代目标函数（Surrogate Objective Function）

$$ C_M^S(\mathbf{y,z})=\lVert \mathbf{x-\Phi y} \rVert_2^2 - \lVert \mathbf{\Phi y-\Phi z} \rVert_2^2 + \lVert \mathbf{y-z} \rVert_2^2 \qquad \lVert\mathbf\Phi\rVert_2\lt1, $$

当$\mathbf{y=z}$时，该函数即为原目标函数，其余情况下均大于目标函数。$\lVert\mathbf\Phi\rVert_2\lt1$由$0\lt eig(\mathbf I-\mathbf \Phi^\rm{H} \mathbf\Phi)\lt 1$推导出。

替代目标函数变形

$$ C_M^S(\mathbf{y,z})=\sum_i[y_i^2-2y_i(z_i+\phi_i^\rm{T}\mathbf x-\phi_i^\rm{T}\mathbf{\Phi z})]+\lVert\mathbf x\rVert_2^2+\lVert\mathbf z\rVert_2^2+\lVert\mathbf{\Phi z}\rVert_2^2, $$

由于优化目标为$\mathbf y$，而后面三项与其无关。因此，对其优化时忽略这三项不影响结果。

$$ C_M^S(\mathbf{y,z})\propto\sum_i[y_i^2-2y_i(z_i+\phi_i^\rm{T}\mathbf x-\phi_i^\rm{T}\mathbf{\Phi z})]. $$

极值点获取

$$ y_i^*=z_i+\phi_i^\rm H\mathbf x-\phi_i^\rm H\mathbf{\Phi z}, $$

极值点通过简单的配方法即可获得。这时，取得最小值$\sum_i{-(y_i^*)^2}$。

迭代公式获取

考虑到$\lVert\mathbf y\rVert_0\le M$的约束，需要保留$\lVert\mathbf y^*\rVert$的最大$M$项，即使用硬阈值函数。

代码

简单

function [s_est, info] = iht(g, Phi, max_iter, K)
    [~, N] = size(Phi);
    s_est = zeros(N, 1);
    g = g(:);

    g_norm = norm(g);
    L = compute_lipschitz_constant(Phi);
    t = 1 / L;  % 计算步长
    
    residual_tol = 1e-2;
    plateau_tol = 1e-6;
    plateau_patience = 20;
    plateau_count = 0;

    residual = Phi * s_est - g;  % 初始残差
    residual_norm = norm(residual);
    stop_reason = "max_iter";
    iter_done = 0;

    for iter = 1:max_iter
        z = s_est - Phi' * residual * t;    % 计算梯度步长更新
        s_est = hard_thresholding(z, K); % 应用硬阈值处理

        prev_residual_norm = residual_norm;
        residual = Phi * s_est - g;  % 更新残差
        residual_norm = norm(residual);
        iter_done = iter;

        if residual_norm < residual_tol * g_norm
            stop_reason = "residual_tol";
            break;
        end

        improvement = (prev_residual_norm - residual_norm) / max(prev_residual_norm, eps);
        if improvement < plateau_tol
            plateau_count = plateau_count + 1;
        else
            plateau_count = 0;
        end

        if plateau_count >= plateau_patience
            stop_reason = "plateau";
            break;
        end
    end

    if nargout > 1
        info = struct();
        info.iter = iter_done;
        info.residual_norm = residual_norm;
        info.relative_residual = residual_norm / max(g_norm, eps);
        info.stop_reason = stop_reason;
        info.step_size = t;
        info.L = L;
        info.plateau_count = plateau_count;
    end
end

function r = hard_thresholding(x, K)
    % 对向量x进行硬阈值处理，保留最大的K个元素
    r = zeros(size(x));
    [~, idx] = sort(abs(x), 'descend');
    r(idx(1:K)) = x(idx(1:K));
end

function L = compute_lipschitz_constant(Phi)
    ATA = Phi' * Phi;
    L = max(eig(ATA));
end

原文

function [s, err_mse, iter_time]=hard_l0_Mterm(x,A,m,M,varargin)
% hard_l0_Mterm: Hard thresholding algorithm that keeps exactly M elements 
% in each iteration. 
%
% This algorithm has certain performance guarantees as described in [1],
% [2] and [3].
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Usage
%
%   [s, err_mse, iter_time]=hard_l0_Mterm(x,P,m,M,'option_name','option_value')
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% Input
%
%   Mandatory:
%               x   Observation vector to be decomposed
%               P   Either:
%                       1) An nxm matrix (n must be dimension of x)
%                       2) A function handle (type "help function_format" 
%                          for more information)
%                          Also requires specification of P_trans option.
%                       3) An object handle (type "help object_format" for 
%                          more information)
%               m   length of s 
%               M   non-zero elements to keep in each iteration
%
%   Possible additional options:
%   (specify as many as you want using 'option_name','option_value' pairs)
%   See below for explanation of options:
%__________________________________________________________________________
%   option_name    |     available option_values                | default
%--------------------------------------------------------------------------
%   stopTol        | number (see below)                         | 1e-16
%   P_trans        | function_handle (see below)                | 
%   maxIter        | positive integer (see below)               | n^2
%   verbose        | true, false                                | false
%   start_val      | vector of length m                         | zeros
%   step_size      | number                                     | 0 (auto)
%
%   stopping criteria used : (OldRMS-NewRMS)/RMS(x) < stopTol
%
%   stopTol: Value for stopping criterion.
%
%   P_trans: If P is a function handle, then P_trans has to be specified and 
%            must be a function handle. 
%
%   maxIter: Maximum number of allowed iterations.
%
%   verbose: Logical value to allow algorithm progress to be displayed.
%
%   start_val: Allows algorithms to start from partial solution.
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% Outputs
%
%    s              Solution vector 
%    err_mse        Vector containing mse of approximation error for each 
%                   iteration
%    iter_time      Vector containing computation times for each iteration
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% Description
%
%   Implements the M-sparse algorithm described in [1], [2] and [3].
%   This algorithm takes a gradient step and then thresholds to only retain
%   M non-zero elements. It allows the step-size to be calculated
%   automatically as described in [3] and is therefore now independent from 
%   a rescaling of P.
%   
%   
% References
%   [1]  T. Blumensath and M.E. Davies, "Iterative Thresholding for Sparse 
%        Approximations", submitted, 2007
%   [2]  T. Blumensath and M. Davies; "Iterative Hard Thresholding for 
%        Compressed Sensing" to appear Applied and Computational Harmonic 
%        Analysis 
%   [3] T. Blumensath and M. Davies; "A modified Iterative Hard 
%        Thresholding algorithm with guaranteed performance and stability" 
%        in preparation (title may change) 
% See Also
%   hard_l0_reg
%
% Copyright (c) 2007 Thomas Blumensath
%
% The University of Edinburgh
% Email: thomas.blumensath@ed.ac.uk
% Comments and bug reports welcome
%
% This file is part of sparsity Version 0.4
% Created: April 2007
% Modified January 2009
%
% Part of this toolbox was developed with the support of EPSRC Grant
% D000246/1
%
% Please read COPYRIGHT.m for terms and conditions.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                    Default values and initialisation
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



[n1 n2]=size(x);
if n2 == 1
    n=n1;
elseif n1 == 1
    x=x';
    n=n2;
else
   error('x must be a vector.');
end
    
sigsize     = x'*x/n;
oldERR      = sigsize;
err_mse     = [];
iter_time   = [];
STOPTOL     = 1e-16;
MAXITER     = n^2;
verbose     = false;
initial_given=0;
s_initial   = zeros(m,1);
MU          = 0;

if verbose
   display('Initialising...') 
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                           Output variables
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

switch nargout 
    case 3
        comp_err=true;
        comp_time=true;
    case 2 
        comp_err=true;
        comp_time=false;
    case 1
        comp_err=false;
        comp_time=false;
    case 0
        error('Please assign output variable.')        
    otherwise
        error('Too many output arguments specified')
end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                       Look through options
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Put option into nice format
Options={};
OS=nargin-4;
c=1;
for i=1:OS
    if isa(varargin{i},'cell')
        CellSize=length(varargin{i});
        ThisCell=varargin{i};
        for j=1:CellSize
            Options{c}=ThisCell{j};
            c=c+1;
        end
    else
        Options{c}=varargin{i};
        c=c+1;
    end
end
OS=length(Options);
if rem(OS,2)
   error('Something is wrong with argument name and argument value pairs.') 
end
for i=1:2:OS
   switch Options{i}
        case {'stopTol'}
            if isa(Options{i+1},'numeric') ; STOPTOL     = Options{i+1};   
            else error('stopTol must be number. Exiting.'); end
        case {'P_trans'} 
            if isa(Options{i+1},'function_handle'); Pt = Options{i+1};   
            else error('P_trans must be function _handle. Exiting.'); end
        case {'maxIter'}
            if isa(Options{i+1},'numeric'); MAXITER     = Options{i+1};             
            else error('maxIter must be a number. Exiting.'); end
        case {'verbose'}
            if isa(Options{i+1},'logical'); verbose     = Options{i+1};   
            else error('verbose must be a logical. Exiting.'); end 
        case {'start_val'}
            if isa(Options{i+1},'numeric') && length(Options{i+1}) == m ;
                s_initial     = Options{i+1};  
                initial_given=1;
            else error('start_val must be a vector of length m. Exiting.'); end
        case {'step_size'}
            if isa(Options{i+1},'numeric') && (Options{i+1}) > 0 ;
                MU     = Options{i+1};   
            else error('Stepsize must be between a positive number. Exiting.'); end
        otherwise
            error('Unrecognised option. Exiting.') 
   end
end

if nargout >=2
    err_mse = zeros(MAXITER,1);
end
if nargout ==3
    iter_time = zeros(MAXITER,1);
end


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                        Make P and Pt functions
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

if          isa(A,'float')      P =@(z) A*z;  Pt =@(z) A'*z;
elseif      isobject(A)         P =@(z) A*z;  Pt =@(z) A'*z;
elseif      isa(A,'function_handle') 
    try
        if          isa(Pt,'function_handle'); P=A;
        else        error('If P is a function handle, Pt also needs to be a function handle. Exiting.'); end
    catch error('If P is a function handle, Pt needs to be specified. Exiting.'); end
else        error('P is of unsupported type. Use matrix, function_handle or object. Exiting.'); end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                        Do we start from zero or not?
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



if initial_given ==1;
    
    if length(find(s_initial)) > M
        display('Initial vector has more than M non-zero elements. Keeping only M largest.')
    
    end
    s                   =   s_initial;
    [ssort sortind]     =   sort(abs(s),'descend');
    s(sortind(M+1:end)) =   0;
    Ps                  =   P(s);
    Residual            =   x-Ps;
    oldERR      = Residual'*Residual/n;
else
    s_initial   = zeros(m,1);
    Residual    = x;
    s           = s_initial;
    Ps          = zeros(n,1);
    oldERR      = sigsize;
end


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                 Random Check to see if dictionary norm is below 1 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

        
        x_test=randn(m,1);
        x_test=x_test/norm(x_test);
        nP=norm(P(x_test));
        if abs(MU*nP)>1;
            display('WARNING! Algorithm likely to become unstable.')
            display('Use smaller step-size or || P ||_2 < 1.')
        end


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                        Main algorithm
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
if verbose
   display('Main iterations...') 
end
tic
t=0;
done = 0;
iter=1;

while ~done
    
    if MU == 0

        %Calculate optimal step size and do line search
        olds                =   s;
        oldPs               =   Ps;
        IND                 =   s~=0;
        d                   =   Pt(Residual);
        % If the current vector is zero, we take the largest elements in d
        if sum(IND)==0
            [dsort sortdind]    =   sort(abs(d),'descend');
            IND(sortdind(1:M))  =   1;    
         end  

        id                  =   (IND.*d);
        Pd                  =   P(id);
        mu                  =   id'*id/(Pd'*Pd);
        s                   =   olds + mu * d;
        [ssort sortind]     =   sort(abs(s),'descend');
        s(sortind(M+1:end)) =   0;
        Ps                  =   P(s);
        
        % Calculate step-size requirement 
        omega               =   (norm(s-olds)/norm(Ps-oldPs))^2;

        % As long as the support changes and mu > omega, we decrease mu
        while mu > (0.99)*omega && sum(xor(IND,s~=0))~=0 && sum(IND)~=0
%             display(['decreasing mu'])
                    
                    % We use a simple line search, halving mu in each step
                    mu                  =   mu/2;
                    s                   =   olds + mu * d;
                    [ssort sortind]     =   sort(abs(s),'descend');
                    s(sortind(M+1:end)) =   0;
                    Ps                  =   P(s);
                    % Calculate step-size requirement 
                    omega               =   (norm(s-olds)/norm(Ps-oldPs))^2;
        end
        
    else
        % Use fixed step size
        s                   =   s + MU * Pt(Residual);
        [ssort sortind]     =   sort(abs(s),'descend');
        s(sortind(M+1:end)) =   0;
        Ps                  =   P(s);
        
    end
        Residual            =   x-Ps;

        
     ERR=Residual'*Residual/n;
     if comp_err
         err_mse(iter)=ERR;
     end
     
     if comp_time
         iter_time(iter)=toc;
     end

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                        Are we done yet?
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
     
         if comp_err && iter >=2
             if ((err_mse(iter-1)-err_mse(iter))/sigsize<STOPTOL);
                 if verbose
                    display(['Stopping. Approximation error changed less than ' num2str(STOPTOL)])
                 end
                done = 1; 
             elseif verbose && toc-t>10
                display(sprintf('Iteration %i. --- %i mse change',iter ,(err_mse(iter-1)-err_mse(iter))/sigsize)) 
                t=toc;
             end
         else
             if ((oldERR - ERR)/sigsize < STOPTOL) && iter >=2;
                 if verbose
                    display(['Stopping. Approximation error changed less than ' num2str(STOPTOL)])
                 end
                done = 1; 
             elseif verbose && toc-t>10
                display(sprintf('Iteration %i. --- %i mse change',iter ,(oldERR - ERR)/sigsize)) 
                t=toc;
             end
         end
         
    % Also stop if residual gets too small or maxIter reached
     if comp_err
         if err_mse(iter)<1e-16
             display('Stopping. Exact signal representation found!')
             done=1;
         end
     elseif iter>1 
         if ERR<1e-16
             display('Stopping. Exact signal representation found!')
             done=1;
         end
     end

     if iter >= MAXITER
         display('Stopping. Maximum number of iterations reached!')
         done = 1; 
     end
 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                    If not done, take another round
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   
     if ~done
        iter=iter+1; 
        oldERR=ERR;        
     end
end


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                  Only return as many elements as iterations
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

if nargout >=2
    err_mse = err_mse(1:iter);
end
if nargout ==3
    iter_time = iter_time(1:iter);
end
if verbose
   display('Done') 
end

MM 优化框架

Fri, 23 Jan 2026 00:00:00 +0000

基本原理

Majorization-Minimization优化框架在各类算法中很常见。在目标函数$J(x)$难优化时，可以寻找一个容易优化的目标函数$G(x)$，当$G(x)$满足一定条件时，$G(x)$的最优解能无限逼近$J(x)$的最优解。

$G(x)$应该满足的条件

容易优化
$G_k(x)\ge J(x)$
$G_k(x_k)=J(x_k)$

优化过程

考虑一个优化问题，$\mathscr X$为闭凸集、$J(\mathbf x)$连续

$$ \min_\mathbf x J(\mathbf x)\quad \text{s.t.} \quad x \in \mathscr X. $$

\begin{algorithm}
\caption{Majorization Minimization}
\begin{algorithmic}
    \STATE 寻找一个可行点$x^0 \in \mathscr X，k \gets 0$
    \REPEAT
        \STATE $\mathbf x^{k+1}=\arg\min_{x\in \mathscr X} G_k(\mathbf{x})$
        \STATE $k\gets k+1$
    \UNTIL{满足一些条件}
\end{algorithmic}
\end{algorithm}

压缩感知

Wed, 19 Nov 2025 00:00:00 +0000

利用信号的稀疏性作为先验知识，完成对信号的压缩。

基本公式

$x$ 为信号
$s$ 为稀疏系数，$x=\Psi s$
$y$ 为观测值，$y = \Phi\Psi s = Rs$
$\Phi$ 为观测矩阵
$\Psi$ 为稀疏矩阵

如何重建

$L_0$最小范数(理论解)

$$ \min_s \Vert s\Vert_0\quad s.t. \quad y=Rs $$

$L_1$最小范数

求解$L_0$最小范数是NP-hard问题，但如果信号足够稀疏（$M \gg O(Klog(N/K))$），可以使用$L_1$最小范数得到相同结果。

$$ \min_s \Vert s\Vert_1\quad s.t. \quad y=Rs $$

存在噪声的情况下

没有K的先验知识

$$ \hat s = \arg\min_s{\{\Vert y-Rs\Vert_2^2+\lambda_K\Vert s\Vert_1\}} $$

$\lambda_K$为拉格朗日乘子，取决于采样点数$N$和噪声水平$\sigma_\varepsilon$

有K的先验知识

$$ \hat s= \min_s \Vert s\Vert_1\quad s.t. \quad \Vert y-Rs\Vert_2 \lt \sigma_\varepsilon $$

性质

非相关性

通过最大化每次测量的信息效率，分散信号特征到所有采样点中。

$$ \mu(\mathrm{\bf{R}})=\mu(\bf{\Phi},\bf{\Psi})=\sqrt{N}\operatorname*{max}_{1\leq k,j\leq N}\frac{\vert\langle\varphi_{k},\psi_{j}\rangle\vert}{\Vert\varphi_{k}\Vert_2\Vert\psi_{j}\Vert_2} $$

典型测量基$\Phi$ 和稀疏基$\Psi$组合 1. 傅里叶矩阵+单位矩阵：当信号自身具有稀疏性时 2. 噪声基+小波基 3. 随机矩阵+任意固定基

限制等距条件(RIP)

保证测量矩阵对稀疏信号的几何结构保持稳定

$$ \left(1-\delta_{s}\right)\Vert \nu\Vert_{2}^{2}\ \leq\ \Vert\mathbf{R}\,\nu\Vert_{2}^{2}\ \leq\ \left(1+\delta_{s}\right)\Vert\nu\Vert_{2}^{2} $$

典型重建方法

基于松弛理论

凸优化

凸优化算法利用$L_1$范数最小化来求解$L_0$问题等效的凸优化问题问题，问题如下

$$ \hat s = \arg\min_s{\lVert s\rVert_1} \qquad \text{s.t.} \quad \lVert Rs-y\rVert_2\le\varepsilon. $$

该类算法可以分为内点法（Interior Point Method, IPM）和一阶算法。内点法的典型算法有基追踪（BP）、$L_1$正则化最小二乘（$L_1$-LS）、$L_1$-magic等。它们属于重构精度较高的高阶算法，但是它们对高阶问题的求解效率很低。一阶算法的代表有梯度投影（GPSR）、软阈值迭代（IST, TwIST, FIST）、Bregman、不动点延拓（FPC, FPC-AC）、基于Nesterov理论的算法（NESTA）、一阶锥求解器（TFOCS）、交替方向法（ADM/ADMM）等。这类方法对高维问题的求解非常有效。 $L_1$范数最小化算法可以用来重构稀疏性较差的信号，对加性噪声具有良好的鲁棒性。

非凸优化

非凸优化与之类似，主要的研究方向是基于迭代重加权原理和基于$L_q(0<q<1)$正则化理论。基于迭代重加权原理的算法由优化最小法（Majorization Minimization）框架推导而来。基本思想是利用与未知信号相关的加权矩阵求解一个优化问题序列。典型的算法有迭代重加权最小二乘（IRLS）和FOCUSS等。基于$L_q(0<q<1)$正则化理论的算法所要求解的问题如下

$$ \hat s = \arg\min_s \{{\lVert Rs-y\rVert_2}+\lambda\lVert s\rVert^q_q\}. $$

典型的算法有$L_{1/2}$阈值迭代算法和$L_{2/3}$阈值迭代算法。相比于凸优化算法，非凸优化算法通常对稀疏信号的重构精度更高，然而若参数选择不合适极易收敛到局部极小值，导致信号重构结果出现偏差。

基于贪婪追踪

贪婪追踪算法是一类搜索算法，此类算法主要包含两个步骤，即支集选择和系数更新。现有各种贪婪追踪算法主要由匹配追踪（MP）和正交匹配追踪（OMP）发展而来。典型的算法有OMP、IHT等，此类算法优势是在稀疏性较强时，能迅速重建所需信号。当信号稀疏性较弱或测量存在噪声时，运算效率低、重建结果与真实值之间会存在很大偏差

基于贝叶斯

贝叶斯重构算法是通过将信号稀疏性与先验信息关联，从概率论角度求解稀疏信号的一类算法。典型的有基于稀疏性贝叶斯学习和基于图模型的信息传递算法。

PID

Sun, 08 Dec 2024 00:00:00 +0000

PID概述

激励为系统输入与输出之差

$$ e(t)=R_{in}(t)-Y{out}(t) $$

PID公式

$$ u(t)=K_p\cdot err(t)+K_i\int_{0}^{t}err(t)\,dx+K_d\frac{d\,err(t)}{dt} $$$$ u(t)=K_c\,[e(t)+\frac{1}{T_i}\int_{0}^{t}e(t)\,dt+T_d\frac{d\,e(t)}{dt}] $$

有:

$$ K_i=\frac{K_c}{T_i} $$$$ K_d=K_c\cdot T_d $$

Ziegler-Nichols方法

先给定一个$K_{cr}$（控制器增益）值，使系统处于临界震荡。
测定系统震荡周期$T_{cr}$
计算得到$K_c$、$T_i$和$T_d$的值
基于上一步得到的值微调参数

控制方式	$K_c$	$T_i$	$T_d$
P	$0.5K_{cr}$	$\infty$	$0$
PI	$0.45K_{cr}$	$0.83T_{cr}$	$0$
PD	$0.8K_{cr}$	$\infty$	$0.12T_{cr}$
PID	$0.6K_{cr}$	$0.5T_{cr}$	$0.12T_{cr}$

以$0.5K_{cr}$为基础

增大$K_i$（即减小$T_i$），需微调增大$K_c$
增大$K_d$（即增大$T_d$），需微调减小$K_c$