Differentiation along line search and its relationship with gradient
$begingroup$
I have a continuously differentiable function $f:mathbb{R}^nrightarrow mathbb{R}$. Now during line search part of optimization I need to differentiate $f(x+alpha d)$ wrt $alpha$, where $alpha in mathbb{R}_+$ and $d in mathbb{R}^n$. In other words, let $g:mathbb{R}rightarrow mathbb{R}$ be a function such that $g(alpha)=f(x+alpha d)$ that needs to be differentiated.
Is this always true: $g'(alpha)=nabla f(x+alpha d)^Td$.
derivatives optimization
asked Jan 14 at 5:35
user402940user402940
1128
$endgroup$
add a comment |
$begingroup$
I have a continuously differentiable function $f:mathbb{R}^nrightarrow mathbb{R}$. Now during line search part of optimization I need to differentiate $f(x+alpha d)$ wrt $alpha$, where $alpha in mathbb{R}_+$ and $d in mathbb{R}^n$. In other words, let $g:mathbb{R}rightarrow mathbb{R}$ be a function such that $g(alpha)=f(x+alpha d)$ that needs to be differentiated.
Is this always true: $g'(alpha)=nabla f(x+alpha d)^Td$.
derivatives optimization
asked Jan 14 at 5:35
user402940user402940
1128
$endgroup$
add a comment |
$begingroup$
I have a continuously differentiable function $f:mathbb{R}^nrightarrow mathbb{R}$. Now during line search part of optimization I need to differentiate $f(x+alpha d)$ wrt $alpha$, where $alpha in mathbb{R}_+$ and $d in mathbb{R}^n$. In other words, let $g:mathbb{R}rightarrow mathbb{R}$ be a function such that $g(alpha)=f(x+alpha d)$ that needs to be differentiated.
Is this always true: $g'(alpha)=nabla f(x+alpha d)^Td$.
derivatives optimization
asked Jan 14 at 5:35
user402940user402940
1128
$endgroup$
I have a continuously differentiable function $f:mathbb{R}^nrightarrow mathbb{R}$. Now during line search part of optimization I need to differentiate $f(x+alpha d)$ wrt $alpha$, where $alpha in mathbb{R}_+$ and $d in mathbb{R}^n$. In other words, let $g:mathbb{R}rightarrow mathbb{R}$ be a function such that $g(alpha)=f(x+alpha d)$ that needs to be differentiated.
Is this always true: $g'(alpha)=nabla f(x+alpha d)^Td$.
derivatives optimization
derivatives optimization
asked Jan 14 at 5:35
user402940user402940
1128
asked Jan 14 at 5:35
user402940user402940
1128
edited Jan 14 at 5:42
user402940
asked Jan 14 at 5:35
user402940user402940
1128
asked Jan 14 at 5:35
user402940user402940
1128
asked Jan 14 at 5:35
user402940user402940
1128
1128
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
Yes. You need the chain rule: for $ alpha in mathbb R$ let $h(alpha)=x+alpha d$. Then we have $g(alpha)=f(h( alpha))$. Since $f$ and $h$ are differentiable, $g$ is differentiable and
$$g'(alpha)=f'(h(alpha))h'(alpha)=nabla f(x+alpha d)^Td.$$
answered Jan 14 at 6:46
FredFred
45.4k1848
$endgroup$
add a comment |
$begingroup$
Yes. It follows from the chain rule, knowing that differentiability implies that all partial derivatives exist. Thus we obtain
begin{equation}
g'(alpha) = frac{df(boldsymbol{x}+alpha boldsymbol{d})}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) frac{d(x_i + alpha d_i)}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) d_i = nabla f(boldsymbol{x} + alpha boldsymbol{d}) cdot boldsymbol{d}.
end{equation}
The expression on the right-hand side is also called the directional derivative of $f$ in the direction $boldsymbol{d}$ at the point $boldsymbol{x}+alpha boldsymbol{d}$, which makes sense since $boldsymbol{d}$ is precisely the direction of your line search.
answered Jan 14 at 6:37
ChristophChristoph
50616
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3072883%2fdifferentiation-along-line-search-and-its-relationship-with-gradient%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Yes. You need the chain rule: for $ alpha in mathbb R$ let $h(alpha)=x+alpha d$. Then we have $g(alpha)=f(h( alpha))$. Since $f$ and $h$ are differentiable, $g$ is differentiable and
$$g'(alpha)=f'(h(alpha))h'(alpha)=nabla f(x+alpha d)^Td.$$
answered Jan 14 at 6:46
FredFred
45.4k1848
$endgroup$
add a comment |
$begingroup$
Yes. You need the chain rule: for $ alpha in mathbb R$ let $h(alpha)=x+alpha d$. Then we have $g(alpha)=f(h( alpha))$. Since $f$ and $h$ are differentiable, $g$ is differentiable and
$$g'(alpha)=f'(h(alpha))h'(alpha)=nabla f(x+alpha d)^Td.$$
answered Jan 14 at 6:46
FredFred
45.4k1848
$endgroup$
add a comment |
$begingroup$
Yes. You need the chain rule: for $ alpha in mathbb R$ let $h(alpha)=x+alpha d$. Then we have $g(alpha)=f(h( alpha))$. Since $f$ and $h$ are differentiable, $g$ is differentiable and
$$g'(alpha)=f'(h(alpha))h'(alpha)=nabla f(x+alpha d)^Td.$$
answered Jan 14 at 6:46
FredFred
45.4k1848
$endgroup$
Yes. You need the chain rule: for $ alpha in mathbb R$ let $h(alpha)=x+alpha d$. Then we have $g(alpha)=f(h( alpha))$. Since $f$ and $h$ are differentiable, $g$ is differentiable and
$$g'(alpha)=f'(h(alpha))h'(alpha)=nabla f(x+alpha d)^Td.$$
answered Jan 14 at 6:46
FredFred
45.4k1848
answered Jan 14 at 6:46
FredFred
45.4k1848
answered Jan 14 at 6:46
FredFred
45.4k1848
answered Jan 14 at 6:46
FredFred
45.4k1848
45.4k1848
add a comment |
add a comment |
$begingroup$
Yes. It follows from the chain rule, knowing that differentiability implies that all partial derivatives exist. Thus we obtain
begin{equation}
g'(alpha) = frac{df(boldsymbol{x}+alpha boldsymbol{d})}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) frac{d(x_i + alpha d_i)}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) d_i = nabla f(boldsymbol{x} + alpha boldsymbol{d}) cdot boldsymbol{d}.
end{equation}
The expression on the right-hand side is also called the directional derivative of $f$ in the direction $boldsymbol{d}$ at the point $boldsymbol{x}+alpha boldsymbol{d}$, which makes sense since $boldsymbol{d}$ is precisely the direction of your line search.
answered Jan 14 at 6:37
ChristophChristoph
50616
$endgroup$
add a comment |
$begingroup$
Yes. It follows from the chain rule, knowing that differentiability implies that all partial derivatives exist. Thus we obtain
begin{equation}
g'(alpha) = frac{df(boldsymbol{x}+alpha boldsymbol{d})}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) frac{d(x_i + alpha d_i)}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) d_i = nabla f(boldsymbol{x} + alpha boldsymbol{d}) cdot boldsymbol{d}.
end{equation}
The expression on the right-hand side is also called the directional derivative of $f$ in the direction $boldsymbol{d}$ at the point $boldsymbol{x}+alpha boldsymbol{d}$, which makes sense since $boldsymbol{d}$ is precisely the direction of your line search.
answered Jan 14 at 6:37
ChristophChristoph
50616
$endgroup$
add a comment |
$begingroup$
Yes. It follows from the chain rule, knowing that differentiability implies that all partial derivatives exist. Thus we obtain
begin{equation}
g'(alpha) = frac{df(boldsymbol{x}+alpha boldsymbol{d})}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) frac{d(x_i + alpha d_i)}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) d_i = nabla f(boldsymbol{x} + alpha boldsymbol{d}) cdot boldsymbol{d}.
end{equation}
The expression on the right-hand side is also called the directional derivative of $f$ in the direction $boldsymbol{d}$ at the point $boldsymbol{x}+alpha boldsymbol{d}$, which makes sense since $boldsymbol{d}$ is precisely the direction of your line search.
answered Jan 14 at 6:37
ChristophChristoph
50616
$endgroup$
Yes. It follows from the chain rule, knowing that differentiability implies that all partial derivatives exist. Thus we obtain
begin{equation}
g'(alpha) = frac{df(boldsymbol{x}+alpha boldsymbol{d})}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) frac{d(x_i + alpha d_i)}{dalpha} = sum_{i=1}^n frac{partial f}{partial x_i}(boldsymbol{x}+alphaboldsymbol{d}) d_i = nabla f(boldsymbol{x} + alpha boldsymbol{d}) cdot boldsymbol{d}.
end{equation}
The expression on the right-hand side is also called the directional derivative of $f$ in the direction $boldsymbol{d}$ at the point $boldsymbol{x}+alpha boldsymbol{d}$, which makes sense since $boldsymbol{d}$ is precisely the direction of your line search.
answered Jan 14 at 6:37
ChristophChristoph
50616
answered Jan 14 at 6:37
ChristophChristoph
50616
answered Jan 14 at 6:37
ChristophChristoph
50616
answered Jan 14 at 6:37
ChristophChristoph
50616
50616
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3072883%2fdifferentiation-along-line-search-and-its-relationship-with-gradient%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
