Derivative of sum of Gaussians wrt. sigma - How does this work?
$begingroup$
Let $varepsilon_i sim mathcal{N}(0,sigma_i^2)$ be samples from independent Gaussians, we know by additivity of independent Gaussians that $varepsilon_1 + varepsilon_2 = varepsilon'$ where $varepsilon' sim mathcal{N}(0,sigma_1^2+sigma_2^2)$.
We can also write this with $varepsilon^*_i sim mathcal{N}(0,1)$ as
$sigma_1 varepsilon^*_1 + sigma_2 varepsilon^*_2 = sqrt{sigma_1^2 + sigma_2^2}varepsilon^*_3$
What confuses me here if that while both terms describe the same distribution, their derivatives with respect to $sigma$ are quite different.
$frac{d}{dsigma_1} sigma_1 varepsilon^*_1 + sigma_2 varepsilon^*_2 = varepsilon^*_1$
whereas
$frac{d}{dsigma_1} sqrt{sigma_1^2 + sigma_2^2}varepsilon^*_3 = frac{varepsilon^*_3sigma_1}{sqrt{sigma_1^2 + sigma_2^2}}$
So I see something is going wrong here, I suspect this is because using standard differential calculus might not extend to cases where random variables are involved(?). Can someone here clarify for me, why these derivatives differ and what the right tools for computing derivatives involving random variables would be?
Also feel free to correct my notation, I'm not a mathematician.
Thanks!
calculus probability normal-distribution differential
asked Jan 17 at 11:49
Frederik HarderFrederik Harder
62
$endgroup$
|
show 2 more comments
$begingroup$
Let $varepsilon_i sim mathcal{N}(0,sigma_i^2)$ be samples from independent Gaussians, we know by additivity of independent Gaussians that $varepsilon_1 + varepsilon_2 = varepsilon'$ where $varepsilon' sim mathcal{N}(0,sigma_1^2+sigma_2^2)$.
We can also write this with $varepsilon^*_i sim mathcal{N}(0,1)$ as
$sigma_1 varepsilon^*_1 + sigma_2 varepsilon^*_2 = sqrt{sigma_1^2 + sigma_2^2}varepsilon^*_3$
What confuses me here if that while both terms describe the same distribution, their derivatives with respect to $sigma$ are quite different.
$frac{d}{dsigma_1} sigma_1 varepsilon^*_1 + sigma_2 varepsilon^*_2 = varepsilon^*_1$
whereas
$frac{d}{dsigma_1} sqrt{sigma_1^2 + sigma_2^2}varepsilon^*_3 = frac{varepsilon^*_3sigma_1}{sqrt{sigma_1^2 + sigma_2^2}}$
So I see something is going wrong here, I suspect this is because using standard differential calculus might not extend to cases where random variables are involved(?). Can someone here clarify for me, why these derivatives differ and what the right tools for computing derivatives involving random variables would be?
Also feel free to correct my notation, I'm not a mathematician.
Thanks!
calculus probability normal-distribution differential
asked Jan 17 at 11:49
Frederik HarderFrederik Harder
62
$endgroup$
$begingroup$
Note that while the random variables $varepsilon_1$ and $varepsilon_2$ do not depend on the choice of $sigma_1$ as they are assumed to just be independent variables both distributed by $mathcal N(0,1)$, the random variable $varepsilon_3$ does depend on the choice of $sigma_1$. Yes, its distribution is $mathcal N(0,1)$, but it is not just another independent random variable but defined in terms of $varepsilon_1$, $varepsilon_2$, $sigma_1$ and $sigma_2$. You would need to take $frac{mathrm dvarepsilon_3^*}{mathrm dsigma_1}$ into account.
$endgroup$
– Christoph
Jan 17 at 11:59
$begingroup$
Oh! this is interesting! Could you give me an idea, what $frac{dvarepsilon_3^*}{dsigma_1}$ would look like? I don't really have an idea, how to approach this.
$endgroup$
– Frederik Harder
Jan 17 at 12:06
$begingroup$
Well, by definition $e_3^* = frac{sigma_1 varepsilon_1^* + sigma_2 varepsilon_2^*}{sqrt{sigma_1^2 + sigma_2^2}}$, now you can just take the derivative with respect to $sigma_1$.
$endgroup$
– Christoph
Jan 17 at 12:08
1
$begingroup$
I see... that's of course trivial. So I guess my takeaway here is that $varepsilon_3^*$ is a function of $varepsilon_1^*$ and $varepsilon_2^*$ which happens to be Gaussian itself, but this shouldn't obscure the dependence. Thanks for clearing that up!
$endgroup$
– Frederik Harder
Jan 17 at 12:27
1
$begingroup$
@LoveTooNap29 The form with a correlation term is also a special case where the RVs are bivariate normally distributed. In general, the sum of two normal random variables need not even be normal.
$endgroup$
– spaceisdarkgreen
Jan 17 at 17:49
|
show 2 more comments
$begingroup$
Let $varepsilon_i sim mathcal{N}(0,sigma_i^2)$ be samples from independent Gaussians, we know by additivity of independent Gaussians that $varepsilon_1 + varepsilon_2 = varepsilon'$ where $varepsilon' sim mathcal{N}(0,sigma_1^2+sigma_2^2)$.
We can also write this with $varepsilon^*_i sim mathcal{N}(0,1)$ as
$sigma_1 varepsilon^*_1 + sigma_2 varepsilon^*_2 = sqrt{sigma_1^2 + sigma_2^2}varepsilon^*_3$
What confuses me here if that while both terms describe the same distribution, their derivatives with respect to $sigma$ are quite different.
$frac{d}{dsigma_1} sigma_1 varepsilon^*_1 + sigma_2 varepsilon^*_2 = varepsilon^*_1$
whereas
$frac{d}{dsigma_1} sqrt{sigma_1^2 + sigma_2^2}varepsilon^*_3 = frac{varepsilon^*_3sigma_1}{sqrt{sigma_1^2 + sigma_2^2}}$
So I see something is going wrong here, I suspect this is because using standard differential calculus might not extend to cases where random variables are involved(?). Can someone here clarify for me, why these derivatives differ and what the right tools for computing derivatives involving random variables would be?
Also feel free to correct my notation, I'm not a mathematician.
Thanks!
calculus probability normal-distribution differential
asked Jan 17 at 11:49
Frederik HarderFrederik Harder
62
$endgroup$
Let $varepsilon_i sim mathcal{N}(0,sigma_i^2)$ be samples from independent Gaussians, we know by additivity of independent Gaussians that $varepsilon_1 + varepsilon_2 = varepsilon'$ where $varepsilon' sim mathcal{N}(0,sigma_1^2+sigma_2^2)$.
We can also write this with $varepsilon^*_i sim mathcal{N}(0,1)$ as
$sigma_1 varepsilon^*_1 + sigma_2 varepsilon^*_2 = sqrt{sigma_1^2 + sigma_2^2}varepsilon^*_3$
What confuses me here if that while both terms describe the same distribution, their derivatives with respect to $sigma$ are quite different.
$frac{d}{dsigma_1} sigma_1 varepsilon^*_1 + sigma_2 varepsilon^*_2 = varepsilon^*_1$
whereas
$frac{d}{dsigma_1} sqrt{sigma_1^2 + sigma_2^2}varepsilon^*_3 = frac{varepsilon^*_3sigma_1}{sqrt{sigma_1^2 + sigma_2^2}}$
So I see something is going wrong here, I suspect this is because using standard differential calculus might not extend to cases where random variables are involved(?). Can someone here clarify for me, why these derivatives differ and what the right tools for computing derivatives involving random variables would be?
Also feel free to correct my notation, I'm not a mathematician.
Thanks!
calculus probability normal-distribution differential
calculus probability normal-distribution differential
asked Jan 17 at 11:49
Frederik HarderFrederik Harder
62
asked Jan 17 at 11:49
Frederik HarderFrederik Harder
62
edited Jan 18 at 9:56
Frederik Harder
asked Jan 17 at 11:49
Frederik HarderFrederik Harder
62
asked Jan 17 at 11:49
Frederik HarderFrederik Harder
62
asked Jan 17 at 11:49
Frederik HarderFrederik Harder
62
62
$begingroup$
Note that while the random variables $varepsilon_1$ and $varepsilon_2$ do not depend on the choice of $sigma_1$ as they are assumed to just be independent variables both distributed by $mathcal N(0,1)$, the random variable $varepsilon_3$ does depend on the choice of $sigma_1$. Yes, its distribution is $mathcal N(0,1)$, but it is not just another independent random variable but defined in terms of $varepsilon_1$, $varepsilon_2$, $sigma_1$ and $sigma_2$. You would need to take $frac{mathrm dvarepsilon_3^*}{mathrm dsigma_1}$ into account.
$endgroup$
– Christoph
Jan 17 at 11:59
$begingroup$
Oh! this is interesting! Could you give me an idea, what $frac{dvarepsilon_3^*}{dsigma_1}$ would look like? I don't really have an idea, how to approach this.
$endgroup$
– Frederik Harder
Jan 17 at 12:06
$begingroup$
Well, by definition $e_3^* = frac{sigma_1 varepsilon_1^* + sigma_2 varepsilon_2^*}{sqrt{sigma_1^2 + sigma_2^2}}$, now you can just take the derivative with respect to $sigma_1$.
$endgroup$
– Christoph
Jan 17 at 12:08
1
$begingroup$
I see... that's of course trivial. So I guess my takeaway here is that $varepsilon_3^*$ is a function of $varepsilon_1^*$ and $varepsilon_2^*$ which happens to be Gaussian itself, but this shouldn't obscure the dependence. Thanks for clearing that up!
$endgroup$
– Frederik Harder
Jan 17 at 12:27
1
$begingroup$
@LoveTooNap29 The form with a correlation term is also a special case where the RVs are bivariate normally distributed. In general, the sum of two normal random variables need not even be normal.
$endgroup$
– spaceisdarkgreen
Jan 17 at 17:49
|
show 2 more comments
$begingroup$
Note that while the random variables $varepsilon_1$ and $varepsilon_2$ do not depend on the choice of $sigma_1$ as they are assumed to just be independent variables both distributed by $mathcal N(0,1)$, the random variable $varepsilon_3$ does depend on the choice of $sigma_1$. Yes, its distribution is $mathcal N(0,1)$, but it is not just another independent random variable but defined in terms of $varepsilon_1$, $varepsilon_2$, $sigma_1$ and $sigma_2$. You would need to take $frac{mathrm dvarepsilon_3^*}{mathrm dsigma_1}$ into account.
$endgroup$
– Christoph
Jan 17 at 11:59
$begingroup$
Oh! this is interesting! Could you give me an idea, what $frac{dvarepsilon_3^*}{dsigma_1}$ would look like? I don't really have an idea, how to approach this.
$endgroup$
– Frederik Harder
Jan 17 at 12:06
$begingroup$
Well, by definition $e_3^* = frac{sigma_1 varepsilon_1^* + sigma_2 varepsilon_2^*}{sqrt{sigma_1^2 + sigma_2^2}}$, now you can just take the derivative with respect to $sigma_1$.
$endgroup$
– Christoph
Jan 17 at 12:08
1
$begingroup$
I see... that's of course trivial. So I guess my takeaway here is that $varepsilon_3^*$ is a function of $varepsilon_1^*$ and $varepsilon_2^*$ which happens to be Gaussian itself, but this shouldn't obscure the dependence. Thanks for clearing that up!
$endgroup$
– Frederik Harder
Jan 17 at 12:27
1
$begingroup$
@LoveTooNap29 The form with a correlation term is also a special case where the RVs are bivariate normally distributed. In general, the sum of two normal random variables need not even be normal.
$endgroup$
– spaceisdarkgreen
Jan 17 at 17:49
$begingroup$
Note that while the random variables $varepsilon_1$ and $varepsilon_2$ do not depend on the choice of $sigma_1$ as they are assumed to just be independent variables both distributed by $mathcal N(0,1)$, the random variable $varepsilon_3$ does depend on the choice of $sigma_1$. Yes, its distribution is $mathcal N(0,1)$, but it is not just another independent random variable but defined in terms of $varepsilon_1$, $varepsilon_2$, $sigma_1$ and $sigma_2$. You would need to take $frac{mathrm dvarepsilon_3^*}{mathrm dsigma_1}$ into account.
$endgroup$
– Christoph
Jan 17 at 11:59
$begingroup$
Note that while the random variables $varepsilon_1$ and $varepsilon_2$ do not depend on the choice of $sigma_1$ as they are assumed to just be independent variables both distributed by $mathcal N(0,1)$, the random variable $varepsilon_3$ does depend on the choice of $sigma_1$. Yes, its distribution is $mathcal N(0,1)$, but it is not just another independent random variable but defined in terms of $varepsilon_1$, $varepsilon_2$, $sigma_1$ and $sigma_2$. You would need to take $frac{mathrm dvarepsilon_3^*}{mathrm dsigma_1}$ into account.
$endgroup$
– Christoph
Jan 17 at 11:59
$begingroup$
Oh! this is interesting! Could you give me an idea, what $frac{dvarepsilon_3^*}{dsigma_1}$ would look like? I don't really have an idea, how to approach this.
$endgroup$
– Frederik Harder
Jan 17 at 12:06
$begingroup$
Oh! this is interesting! Could you give me an idea, what $frac{dvarepsilon_3^*}{dsigma_1}$ would look like? I don't really have an idea, how to approach this.
$endgroup$
– Frederik Harder
Jan 17 at 12:06
$begingroup$
Well, by definition $e_3^* = frac{sigma_1 varepsilon_1^* + sigma_2 varepsilon_2^*}{sqrt{sigma_1^2 + sigma_2^2}}$, now you can just take the derivative with respect to $sigma_1$.
$endgroup$
– Christoph
Jan 17 at 12:08
$begingroup$
Well, by definition $e_3^* = frac{sigma_1 varepsilon_1^* + sigma_2 varepsilon_2^*}{sqrt{sigma_1^2 + sigma_2^2}}$, now you can just take the derivative with respect to $sigma_1$.
$endgroup$
– Christoph
Jan 17 at 12:08
1
1
$begingroup$
I see... that's of course trivial. So I guess my takeaway here is that $varepsilon_3^*$ is a function of $varepsilon_1^*$ and $varepsilon_2^*$ which happens to be Gaussian itself, but this shouldn't obscure the dependence. Thanks for clearing that up!
$endgroup$
– Frederik Harder
Jan 17 at 12:27
$begingroup$
I see... that's of course trivial. So I guess my takeaway here is that $varepsilon_3^*$ is a function of $varepsilon_1^*$ and $varepsilon_2^*$ which happens to be Gaussian itself, but this shouldn't obscure the dependence. Thanks for clearing that up!
$endgroup$
– Frederik Harder
Jan 17 at 12:27
1
1
$begingroup$
@LoveTooNap29 The form with a correlation term is also a special case where the RVs are bivariate normally distributed. In general, the sum of two normal random variables need not even be normal.
$endgroup$
– spaceisdarkgreen
Jan 17 at 17:49
$begingroup$
@LoveTooNap29 The form with a correlation term is also a special case where the RVs are bivariate normally distributed. In general, the sum of two normal random variables need not even be normal.
$endgroup$
– spaceisdarkgreen
Jan 17 at 17:49
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3076882%2fderivative-of-sum-of-gaussians-wrt-sigma-how-does-this-work%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3076882%2fderivative-of-sum-of-gaussians-wrt-sigma-how-does-this-work%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Note that while the random variables $varepsilon_1$ and $varepsilon_2$ do not depend on the choice of $sigma_1$ as they are assumed to just be independent variables both distributed by $mathcal N(0,1)$, the random variable $varepsilon_3$ does depend on the choice of $sigma_1$. Yes, its distribution is $mathcal N(0,1)$, but it is not just another independent random variable but defined in terms of $varepsilon_1$, $varepsilon_2$, $sigma_1$ and $sigma_2$. You would need to take $frac{mathrm dvarepsilon_3^*}{mathrm dsigma_1}$ into account.
$endgroup$
– Christoph
Jan 17 at 11:59
$begingroup$
Oh! this is interesting! Could you give me an idea, what $frac{dvarepsilon_3^*}{dsigma_1}$ would look like? I don't really have an idea, how to approach this.
$endgroup$
– Frederik Harder
Jan 17 at 12:06
$begingroup$
Well, by definition $e_3^* = frac{sigma_1 varepsilon_1^* + sigma_2 varepsilon_2^*}{sqrt{sigma_1^2 + sigma_2^2}}$, now you can just take the derivative with respect to $sigma_1$.
$endgroup$
– Christoph
Jan 17 at 12:08
1
$begingroup$
I see... that's of course trivial. So I guess my takeaway here is that $varepsilon_3^*$ is a function of $varepsilon_1^*$ and $varepsilon_2^*$ which happens to be Gaussian itself, but this shouldn't obscure the dependence. Thanks for clearing that up!
$endgroup$
– Frederik Harder
Jan 17 at 12:27
1
$begingroup$
@LoveTooNap29 The form with a correlation term is also a special case where the RVs are bivariate normally distributed. In general, the sum of two normal random variables need not even be normal.
$endgroup$
– spaceisdarkgreen
Jan 17 at 17:49