Nesterov 牛顿动量

Fri, 06 Mar 2026 00:00:00 +0000

基本理论

动量法（Momentum）

普通梯度下降法$\theta\leftarrow\theta-\eta\nabla_\theta J(\theta)$在接近最优值时收敛速度会变慢，有事甚至陷入局部最优。这时如果考虑历史梯度，将会引导参数向最优值更快收敛，这就是动量算法基本思想。

$$ \begin{align*} m&\leftarrow\gamma m+\eta\nabla_\theta J(\theta) \\ \theta&\leftarrow\theta -m \end{align*}. $$

牛顿动量（Nesterov）

尽管动量法能最终找到最优解，但是在不同维度梯度差较大时容易震荡。牛顿动量法在超前一个动量单位处进行梯度更新，公式修正为

$$ \begin{align*} m&\leftarrow\gamma m+\eta\nabla_\theta J(\theta+\beta m) \\ \theta&\leftarrow\theta -m \end{align*}. $$

仿真

对二元函数$z=x^2-4xy+10y^2$分别用梯度下降法、动量法和牛顿动量法求最小值。$\eta=0.012, \gamma=0.8$。梯度下降方法速度慢，动量法震荡明显。

结果

迭代结果和迭代次数

1
2
3

Gradient 最优点: (0.0000, 0.0000), 迭代次数: 816
Momentum 最优点: (0.0000, -0.0000), 迭代次数: 176
Nesterov 最优点: (0.0000, 0.0000), 迭代次数: 137

优化路径

梯度下降

动量法

牛顿动量

代码

global A;   %#ok
% 此处仅作演示，实际编程过程中应避免使用全局变量。
A = [...
    1, -2; ...
    -2, 10 ...
];
x_0 = [150, 100].';
eta = 1.2e-2;
gamma = 0.8;
max_iter = 1000;

dim1_space = linspace(-200, 200, 1e2);
dim2_space = linspace(-200, 200, 1e2);
[X, Y] = meshgrid(dim1_space, dim2_space);
Z = arrayfun(@(x, y) dst_function([x; y]), X, Y);

% 三维曲面图
figure;
surf(X, Y, Z, 'EdgeColor', 'none');
colormap(jet);
colorbar;
xlabel('x_1');
ylabel('x_2');

[x_opt, x_history] = gradient_descent(x_0, eta, max_iter);
% 绘制优化路径
figure;
contour(X, Y, Z, 50);
colormap(jet);
colorbar;
hold on;
plot(x_history(1, :), x_history(2, :), 'ro-', 'LineWidth', 2, 'MarkerSize', 5);
xlabel('x_1');
ylabel('x_2');
% 绘制最优点
hold on;
plot(x_opt(1), x_opt(2), 'kx', 'LineWidth', 2, 'MarkerSize', 10);
legend('等高线', '优化路径', '最优点');
fprintf('Gradient 最优点: (%.4f, %.4f), 迭代次数: %d\n', x_opt(1), x_opt(2), size(x_history, 2) - 1);

[x_opt, x_history] = momentum_method(x_0, eta, gamma, max_iter);
% 绘制优化路径
figure;
contour(X, Y, Z, 50);
colormap(jet);
colorbar;
hold on;
plot(x_history(1, :), x_history(2, :), 'ro-', 'LineWidth', 2, 'MarkerSize', 5);
xlabel('x_1');
ylabel('x_2');
% 绘制最优点
hold on;
plot(x_opt(1), x_opt(2), 'kx', 'LineWidth', 2, 'MarkerSize', 10);
legend('等高线', '优化路径', '最优点');
fprintf('Momentum 最优点: (%.4f, %.4f), 迭代次数: %d\n', x_opt(1), x_opt(2), size(x_history, 2) - 1);

[x_opt, x_history] = nesterov_method(x_0, eta, gamma, max_iter);
% 绘制优化路径
figure;
contour(X, Y, Z, 50);
colormap(jet);
colorbar;
hold on;
plot(x_history(1, :), x_history(2, :), 'ro-', 'LineWidth', 2, 'MarkerSize', 5);
xlabel('x_1');
ylabel('x_2');
% 绘制最优点
hold on;
plot(x_opt(1), x_opt(2), 'kx', 'LineWidth', 2, 'MarkerSize', 10);
legend('等高线', '优化路径', '最优点');
fprintf('Nesterov 最优点: (%.4f, %.4f), 迭代次数: %d\n', x_opt(1), x_opt(2), size(x_history, 2) - 1);

function [x_opt, x_history] = gradient_descent(x_0, eta, max_iter)
    x_history = zeros(length(x_0), max_iter+1);
    x_history(:, 1) = x_0;
    x = x_0;
    for i = 1:max_iter
        grad = dst_function_gradient(x);
        x = x - eta * grad;
        x_history(:, i+1) = x;
        if norm(grad) < 1e-6  % 梯度足够小，认为收敛
            x_history = x_history(:, 1:i+1);  % 截断历史记录
            break;
        end
    end
    x_opt = x;
end

function [x_opt, x_history] = momentum_method(x_0, eta, gamma, max_iter)
    x_history = zeros(length(x_0), max_iter+1);
    x_history(:, 1) = x_0;
    x = x_0;
    m = zeros(size(x_0));
    for i = 1:max_iter
        grad = dst_function_gradient(x);
        m = gamma * m + eta * grad;
        x = x - m;
        x_history(:, i+1) = x;
        if norm(grad) < 1e-6  % 梯度足够小，认为收敛
            x_history = x_history(:, 1:i+1);  % 截断历史记录
            break;
        end
    end
    x_opt = x;
end

function [x_opt, x_history] = nesterov_method(x_0, eta, gamma, max_iter)
    x_history = zeros(length(x_0), max_iter+1);
    x_history(:, 1) = x_0;
    x = x_0;
    m = zeros(size(x_0));
    for i = 1:max_iter
        y = x - gamma * m;  % 计算预测点
        grad = dst_function_gradient(y);  % 在预测点计算梯度
        m = gamma * m + eta * grad;  % 更新动量
        x = x - m;  % 更新参数
        x_history(:, i+1) = x;
        if norm(grad) < 1e-6  % 梯度足够小，认为收敛
            x_history = x_history(:, 1:i+1);  % 截断历史记录
            break;
        end
    end
    x_opt = x;
end

%% 目标函数
function y = dst_function(x)
    global A;   %#ok
    y = x.' * A * x;
end

function grad = dst_function_gradient(x)
    global A;   %#ok
    grad = 2 * A * x;
end

PGD

Wed, 04 Mar 2026 00:00:00 +0000

基本原理

近端梯度下降（Proximal Gradient Descent）是梯度下降方法的一种，其适用于目标函数存在不可微部分的凸优化问题。问题描述为

$$ \mathbf x = \arg\min_\mathbf x {g(\mathbf x)+h(\mathbf x)}, $$

其中，$g(\mathbf x)$为可微凸函数、$h(\mathbf x)$为不可微凸函数。

求解方法

求解该类型凸优化问题可以通过近端映射（Proximal Mapping）和梯度迭代的手段实现。近端映射函数定义为

$$ \text{prox}_{h,t}(\mathbf x) = \arg\min_\mathbf z{\frac1{2t}\lVert\mathbf x-\mathbf z\rVert^2_2+h(\mathbf z)}, $$

梯度迭代方法为

$$ \begin{align*} &\mathbf z^{(k)}=\mathbf x^{(k)}-t_k\nabla g(\mathbf x^{(k)}) \\ &\mathbf x^{(k+1)}=\text{prox}_{h,t_k}(\mathbf z^{(k)}) \end{align*} $$

证明

对$g(\mathbf x)$在$\mathbf x^{(k)}$处做二阶泰勒展开，令$L=\nabla^2g(\mathbf x)=\frac1t$

$$ \begin{align*} \arg\min_\mathbf x g(\mathbf x)+h(\mathbf x) &\approx \arg\min_\mathbf x{g(\mathbf x^{(k)})+\nabla g(\mathbf x^{(k)})^\text T (\mathbf x -\mathbf x^{(k)})+\frac12(\mathbf x-\mathbf x^{(k)})^\text T\nabla^2g(\mathbf x^{(k)})(\mathbf x-\mathbf x^{(k)})}+h(\mathbf x) \\ \mathbf{\hat x}&= \arg\min_\mathbf x{\nabla g(\mathbf x^{(k)})^\text T (\mathbf x -\mathbf x^{(k)})+\frac1{2t}(\mathbf x-\mathbf x^{(k)})^\text T(\mathbf x-\mathbf x^{(k)})}+h(\mathbf x) \\ &= \arg\min_\mathbf x\sum_i{[\nabla g(\mathbf x^{(k)})]_i (x_i -x_i^{(k)})+\frac1{2t}(x_i-x_i^{(k)})^2}+h(\mathbf x) \\ &= \arg\min_\mathbf x\frac1{2t}\sum_i{2t[\nabla g(\mathbf x^{(k)})]_i (x_i -x_i^{(k)})+(x_i-x_i^{(k)})^2}+h(\mathbf x) \\ &= \arg\min_\mathbf x\frac1{2t}\sum_i{t^2[\nabla g(\mathbf x^{(k)})]_i^2+2t[\nabla g(\mathbf x^{(k)})]_i (x_i -x_i^{(k)})+(x_i-x_i^{(k)})^2}+h(\mathbf x) \\ &= \arg\min_\mathbf x\frac1{2t}\sum_i{[x_i-x_i^{(k)}+t[\nabla g(\mathbf x^{(k)})]_i]^2}+h(\mathbf x) \\ &= \arg\min_\mathbf x\frac1{2t}{\lVert\mathbf x-[\mathbf x^{(k)}-t\nabla g(\mathbf x^{(k)})]\rVert^2_2}+h(\mathbf x) \\ \end{align*}. $$

令$\mathbf z^{(k)}=\mathbf x^{(k)}-t\nabla g(\mathbf x^{(k)})$，原问题即可转换为近端映射

$$ \begin{align*} \mathbf{\hat x} &= \arg\min_\mathbf x\frac1{2t}{\lVert\mathbf x-\mathbf z\rVert^2_2}+h(\mathbf x) \\ &=\text{prox}_{h,t}(\mathbf z) \end{align*}. $$

设定合适的参数，即可使用迭代方法求得$\arg\min_\mathbf x g(\mathbf x)+h(\mathbf x)$的结果。

Lipschitz常数

假定目标凸函数满足梯度上的Lipschitz连续。当步长$t\le\frac1L$时，保证多次迭代后能收敛到梯度为0的点。梯度Lipschitz连续表示为

$$ \frac{\lVert\nabla f(a)-\nabla f(b)\rVert_2}{\lVert a-b\rVert_2} \le L(f) \quad \forall a,b\in\text{dom}f, $$

函数$\nabla f$的Lipschitz常数为

$$ L = \sup\frac{\lVert\nabla f(a)-\nabla f(b)\rVert_2}{\lVert a-b\rVert_2} $$

凸优化 on Detail's Blog