Proving that the conditional entropy of a probability measure is concave












0












$begingroup$


Let $mu$ be a probability measure on $mathcal{X}$ and let $mathcal{E}, mathcal{F}$ be countable partitions of the space. Define the entropy of $mu$ with respect to the partition $mathcal{E}$ as
$$
H(mu, mathcal{E}) = - sum_{E in mathcal{E}} mu(E) log mu(E)
$$
and the conditional entropy as
$$
H(mu, mathcal{E} | mathcal{F}) = sum_{F in mathcal{F}} mu(F) H(mu_F, mathcal{E}) = - sum_{F in mathcal{F}} sum_{E in mathcal{E}} mu_{|F}(E) log (dfrac{1}{mu(F)} mu_{|F}(E)),
$$
where $mu_{|F}(cdot) = mu(cdot cap F)$ and $mu_F$ is the normalized restriction of $mu$ on $F in mathcal{F}$.



Now it is easy to see by concavity of $x mapsto -x log x$ that the entropy $mu mapsto H(mu, mathcal{E})$ is a concave function, but what about the conditional entropy? How can I prove its concavity? The normalizing coefficient $dfrac{1}{mu(F)}$ seems to make the function slightly more complicated.






probability-theory measure-theory ergodic-theory







share|cite|improve this question









$endgroup$

















    0












    $begingroup$


    Let $mu$ be a probability measure on $mathcal{X}$ and let $mathcal{E}, mathcal{F}$ be countable partitions of the space. Define the entropy of $mu$ with respect to the partition $mathcal{E}$ as
    $$
    H(mu, mathcal{E}) = - sum_{E in mathcal{E}} mu(E) log mu(E)
    $$
    and the conditional entropy as
    $$
    H(mu, mathcal{E} | mathcal{F}) = sum_{F in mathcal{F}} mu(F) H(mu_F, mathcal{E}) = - sum_{F in mathcal{F}} sum_{E in mathcal{E}} mu_{|F}(E) log (dfrac{1}{mu(F)} mu_{|F}(E)),
    $$
    where $mu_{|F}(cdot) = mu(cdot cap F)$ and $mu_F$ is the normalized restriction of $mu$ on $F in mathcal{F}$.



    Now it is easy to see by concavity of $x mapsto -x log x$ that the entropy $mu mapsto H(mu, mathcal{E})$ is a concave function, but what about the conditional entropy? How can I prove its concavity? The normalizing coefficient $dfrac{1}{mu(F)}$ seems to make the function slightly more complicated.






    probability-theory measure-theory ergodic-theory







    share|cite|improve this question









    $endgroup$















      0












      0








      0





      $begingroup$


      Let $mu$ be a probability measure on $mathcal{X}$ and let $mathcal{E}, mathcal{F}$ be countable partitions of the space. Define the entropy of $mu$ with respect to the partition $mathcal{E}$ as
      $$
      H(mu, mathcal{E}) = - sum_{E in mathcal{E}} mu(E) log mu(E)
      $$
      and the conditional entropy as
      $$
      H(mu, mathcal{E} | mathcal{F}) = sum_{F in mathcal{F}} mu(F) H(mu_F, mathcal{E}) = - sum_{F in mathcal{F}} sum_{E in mathcal{E}} mu_{|F}(E) log (dfrac{1}{mu(F)} mu_{|F}(E)),
      $$
      where $mu_{|F}(cdot) = mu(cdot cap F)$ and $mu_F$ is the normalized restriction of $mu$ on $F in mathcal{F}$.



      Now it is easy to see by concavity of $x mapsto -x log x$ that the entropy $mu mapsto H(mu, mathcal{E})$ is a concave function, but what about the conditional entropy? How can I prove its concavity? The normalizing coefficient $dfrac{1}{mu(F)}$ seems to make the function slightly more complicated.






      probability-theory measure-theory ergodic-theory







      share|cite|improve this question









      $endgroup$




      Let $mu$ be a probability measure on $mathcal{X}$ and let $mathcal{E}, mathcal{F}$ be countable partitions of the space. Define the entropy of $mu$ with respect to the partition $mathcal{E}$ as
      $$
      H(mu, mathcal{E}) = - sum_{E in mathcal{E}} mu(E) log mu(E)
      $$
      and the conditional entropy as
      $$
      H(mu, mathcal{E} | mathcal{F}) = sum_{F in mathcal{F}} mu(F) H(mu_F, mathcal{E}) = - sum_{F in mathcal{F}} sum_{E in mathcal{E}} mu_{|F}(E) log (dfrac{1}{mu(F)} mu_{|F}(E)),
      $$
      where $mu_{|F}(cdot) = mu(cdot cap F)$ and $mu_F$ is the normalized restriction of $mu$ on $F in mathcal{F}$.



      Now it is easy to see by concavity of $x mapsto -x log x$ that the entropy $mu mapsto H(mu, mathcal{E})$ is a concave function, but what about the conditional entropy? How can I prove its concavity? The normalizing coefficient $dfrac{1}{mu(F)}$ seems to make the function slightly more complicated.






      probability-theory measure-theory ergodic-theory




      probability-theory measure-theory ergodic-theory






      share|cite|improve this question













      share|cite|improve this question











      share|cite|improve this question




      share|cite|improve this question










      asked Jan 20 at 8:23









      UunoUuno

      555




      555






















          1 Answer
          1






          active

          oldest

          votes


















          1












          $begingroup$

          Note that if we define $$mu_F(X)=frac{mu(Xcap F)}{mu(F)}$$ where $X$ is a measurable subset of $mathcal{X}$ and $Fin mathcal{F}$, then $mu_F$ is also a probability measure. Let $omega =cmu +(1-c)nu$ where $omega,mu,nu$ are probability measures and $cin (0,1)$. By convexity of the entropy and the fact that $$begin{eqnarray}omega_F (X)&=&frac{cmu(Xcap F)+(1-c)nu(Xcap F)}{cmu(F)+(1-c)nu(F)}\&=&frac{cmu(F)cdot mu_F(X)}{cmu(F)+(1-c)nu(F)}+frac{(1-c)nu(F)cdotnu_F(X)}{cmu(F)+(1-c)nu(F)}
          end{eqnarray}$$           i.e. $omega_F$ is a convex combination of $mu_F$ and $nu_F$, we have
          $$begin{eqnarray}
          H(omega_F, mathcal{E})&ge& frac{cmu(F)}{cmu(F)+(1-c)nu(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{cmu(F)+(1-c)nu(F)}H(nu_F, mathcal{E})\
          &=&frac{cmu(F)}{omega(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{omega(F)}H(nu_F, mathcal{E})
          end{eqnarray}$$           for $omega(F)>0$. This implies
          $$begin{eqnarray}
          H(omega, mathcal{E} | mathcal{F}) &=& sum_{F in mathcal{F}} omega(F) H(omega_F, mathcal{E}) \
          &ge&sum_{F in mathcal{F}}cmu(F)H(mu_F, mathcal{E})+(1-c)nu(F)H(nu_F, mathcal{E})\
          &=&cH(mu, mathcal{E} | mathcal{F})+(1-c)H(nu, mathcal{E} | mathcal{F})
          end{eqnarray}$$           as desired.






          share|cite|improve this answer











          $endgroup$













            Your Answer





            StackExchange.ifUsing("editor", function () {
            return StackExchange.using("mathjaxEditing", function () {
            StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
            StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
            });
            });
            }, "mathjax-editing");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "69"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            noCode: true, onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3080320%2fproving-that-the-conditional-entropy-of-a-probability-measure-is-concave%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            1












            $begingroup$

            Note that if we define $$mu_F(X)=frac{mu(Xcap F)}{mu(F)}$$ where $X$ is a measurable subset of $mathcal{X}$ and $Fin mathcal{F}$, then $mu_F$ is also a probability measure. Let $omega =cmu +(1-c)nu$ where $omega,mu,nu$ are probability measures and $cin (0,1)$. By convexity of the entropy and the fact that $$begin{eqnarray}omega_F (X)&=&frac{cmu(Xcap F)+(1-c)nu(Xcap F)}{cmu(F)+(1-c)nu(F)}\&=&frac{cmu(F)cdot mu_F(X)}{cmu(F)+(1-c)nu(F)}+frac{(1-c)nu(F)cdotnu_F(X)}{cmu(F)+(1-c)nu(F)}
            end{eqnarray}$$             i.e. $omega_F$ is a convex combination of $mu_F$ and $nu_F$, we have
            $$begin{eqnarray}
            H(omega_F, mathcal{E})&ge& frac{cmu(F)}{cmu(F)+(1-c)nu(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{cmu(F)+(1-c)nu(F)}H(nu_F, mathcal{E})\
            &=&frac{cmu(F)}{omega(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{omega(F)}H(nu_F, mathcal{E})
            end{eqnarray}$$             for $omega(F)>0$. This implies
            $$begin{eqnarray}
            H(omega, mathcal{E} | mathcal{F}) &=& sum_{F in mathcal{F}} omega(F) H(omega_F, mathcal{E}) \
            &ge&sum_{F in mathcal{F}}cmu(F)H(mu_F, mathcal{E})+(1-c)nu(F)H(nu_F, mathcal{E})\
            &=&cH(mu, mathcal{E} | mathcal{F})+(1-c)H(nu, mathcal{E} | mathcal{F})
            end{eqnarray}$$             as desired.






            share|cite|improve this answer











            $endgroup$


















              1












              $begingroup$

              Note that if we define $$mu_F(X)=frac{mu(Xcap F)}{mu(F)}$$ where $X$ is a measurable subset of $mathcal{X}$ and $Fin mathcal{F}$, then $mu_F$ is also a probability measure. Let $omega =cmu +(1-c)nu$ where $omega,mu,nu$ are probability measures and $cin (0,1)$. By convexity of the entropy and the fact that $$begin{eqnarray}omega_F (X)&=&frac{cmu(Xcap F)+(1-c)nu(Xcap F)}{cmu(F)+(1-c)nu(F)}\&=&frac{cmu(F)cdot mu_F(X)}{cmu(F)+(1-c)nu(F)}+frac{(1-c)nu(F)cdotnu_F(X)}{cmu(F)+(1-c)nu(F)}
              end{eqnarray}$$               i.e. $omega_F$ is a convex combination of $mu_F$ and $nu_F$, we have
              $$begin{eqnarray}
              H(omega_F, mathcal{E})&ge& frac{cmu(F)}{cmu(F)+(1-c)nu(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{cmu(F)+(1-c)nu(F)}H(nu_F, mathcal{E})\
              &=&frac{cmu(F)}{omega(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{omega(F)}H(nu_F, mathcal{E})
              end{eqnarray}$$               for $omega(F)>0$. This implies
              $$begin{eqnarray}
              H(omega, mathcal{E} | mathcal{F}) &=& sum_{F in mathcal{F}} omega(F) H(omega_F, mathcal{E}) \
              &ge&sum_{F in mathcal{F}}cmu(F)H(mu_F, mathcal{E})+(1-c)nu(F)H(nu_F, mathcal{E})\
              &=&cH(mu, mathcal{E} | mathcal{F})+(1-c)H(nu, mathcal{E} | mathcal{F})
              end{eqnarray}$$               as desired.






              share|cite|improve this answer











              $endgroup$
















                1












                1








                1





                $begingroup$

                Note that if we define $$mu_F(X)=frac{mu(Xcap F)}{mu(F)}$$ where $X$ is a measurable subset of $mathcal{X}$ and $Fin mathcal{F}$, then $mu_F$ is also a probability measure. Let $omega =cmu +(1-c)nu$ where $omega,mu,nu$ are probability measures and $cin (0,1)$. By convexity of the entropy and the fact that $$begin{eqnarray}omega_F (X)&=&frac{cmu(Xcap F)+(1-c)nu(Xcap F)}{cmu(F)+(1-c)nu(F)}\&=&frac{cmu(F)cdot mu_F(X)}{cmu(F)+(1-c)nu(F)}+frac{(1-c)nu(F)cdotnu_F(X)}{cmu(F)+(1-c)nu(F)}
                end{eqnarray}$$                 i.e. $omega_F$ is a convex combination of $mu_F$ and $nu_F$, we have
                $$begin{eqnarray}
                H(omega_F, mathcal{E})&ge& frac{cmu(F)}{cmu(F)+(1-c)nu(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{cmu(F)+(1-c)nu(F)}H(nu_F, mathcal{E})\
                &=&frac{cmu(F)}{omega(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{omega(F)}H(nu_F, mathcal{E})
                end{eqnarray}$$                 for $omega(F)>0$. This implies
                $$begin{eqnarray}
                H(omega, mathcal{E} | mathcal{F}) &=& sum_{F in mathcal{F}} omega(F) H(omega_F, mathcal{E}) \
                &ge&sum_{F in mathcal{F}}cmu(F)H(mu_F, mathcal{E})+(1-c)nu(F)H(nu_F, mathcal{E})\
                &=&cH(mu, mathcal{E} | mathcal{F})+(1-c)H(nu, mathcal{E} | mathcal{F})
                end{eqnarray}$$                 as desired.






                share|cite|improve this answer











                $endgroup$



                Note that if we define $$mu_F(X)=frac{mu(Xcap F)}{mu(F)}$$ where $X$ is a measurable subset of $mathcal{X}$ and $Fin mathcal{F}$, then $mu_F$ is also a probability measure. Let $omega =cmu +(1-c)nu$ where $omega,mu,nu$ are probability measures and $cin (0,1)$. By convexity of the entropy and the fact that $$begin{eqnarray}omega_F (X)&=&frac{cmu(Xcap F)+(1-c)nu(Xcap F)}{cmu(F)+(1-c)nu(F)}\&=&frac{cmu(F)cdot mu_F(X)}{cmu(F)+(1-c)nu(F)}+frac{(1-c)nu(F)cdotnu_F(X)}{cmu(F)+(1-c)nu(F)}
                end{eqnarray}$$                 i.e. $omega_F$ is a convex combination of $mu_F$ and $nu_F$, we have
                $$begin{eqnarray}
                H(omega_F, mathcal{E})&ge& frac{cmu(F)}{cmu(F)+(1-c)nu(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{cmu(F)+(1-c)nu(F)}H(nu_F, mathcal{E})\
                &=&frac{cmu(F)}{omega(F)}H(mu_F, mathcal{E})+frac{(1-c)nu(F)}{omega(F)}H(nu_F, mathcal{E})
                end{eqnarray}$$                 for $omega(F)>0$. This implies
                $$begin{eqnarray}
                H(omega, mathcal{E} | mathcal{F}) &=& sum_{F in mathcal{F}} omega(F) H(omega_F, mathcal{E}) \
                &ge&sum_{F in mathcal{F}}cmu(F)H(mu_F, mathcal{E})+(1-c)nu(F)H(nu_F, mathcal{E})\
                &=&cH(mu, mathcal{E} | mathcal{F})+(1-c)H(nu, mathcal{E} | mathcal{F})
                end{eqnarray}$$                 as desired.







                share|cite|improve this answer














                share|cite|improve this answer



                share|cite|improve this answer








                edited Jan 20 at 15:32

























                answered Jan 20 at 8:58









                SongSong

                14.1k1633




                14.1k1633






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Mathematics Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    Use MathJax to format equations. MathJax reference.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3080320%2fproving-that-the-conditional-entropy-of-a-probability-measure-is-concave%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    Popular posts from this blog

                    Mario Kart Wii

                    Image
                    Please consider supporting The Cutting Room Floor on Patreon. Thanks for all your support! Mario Kart Wii From The Cutting Room Floor Jump to: navigation, search Mario Kart Wii Developer : Nintendo Publisher : Nintendo Platform : Wii Released in JP : April 10, 2008 Released in US : April 27, 2008 Released in EU : April 11, 2008 Released in AU : April 24, 2008 Released in KR : April 30, 2009 This game has unused animations. This game has unused areas. This game has unused code. This game has unused objects. This game has unused modes / minigames. This game has unused graphics. This game has unused models. This game has unused items. This game has unused music. This game has regional differences. This game has a notes page This game has a prerelease article Mario Kart Wii is the sixth game in the series (not counting Arcade editions), which intro...
                    Read more

                    Understanding the size os this class of aleatory events

                    Image
                    0 1 $begingroup$ Hi I am having problems to understand this: I have: $Omega = [0,1]$ , $Lambda =$ all subsets with defined length Let $A_{0}$ = {A $subset$ [0,1]: A is finite union of intervals} 1º: My doubt is that $A_{0}$ is a algebra but its not a sigma algebra. Why? Also, A = $(0,1/2) cup (1/2,3/4) cup (3/4,7/8) cup ... cup(1-1/2^{n},1-1/2^{n+1})...$ 2º: So the class of aleatory events would be bigger than $A_{0}$ . Why? How the set A is greater than $A_{0}$ ? measure-theory share | cite | improve this question asked Jan 8 at 19:17 Diogo Bastos Diogo Bastos 116 6 ...
                    Read more

                    Partial Derivative Guidance.

                    Image
                    0 $begingroup$ I was given this tricky problem in which I'm confuse on how to solve it and I would like some help: Let $v(x,y)$ be continuously differentiable function. Let the function $u(s,t)$ be given by $u(s,t) = sin(s)*v(s-2t,t^2)$ . Compute $du/ds$ and $du/dt$ in terms of $v$ and it's partial derivatives. My attempt (which is wrong): Using chain rule: $du/ds=(du/dx)(dx/ds) + (du/dy)(dy/ds)$ $du/dt=(du/dx)(dx/dt) + (du/dx)(dy/dt)$ So, $du/ds = sin(s)*v(x,y)*s + sin(s)*v(x,y)*0$ $du/dt = 2t*sin(s)*v(x,y) - 2*sin(s)*v(x,y)$ Thanks, calculus multivariable-calculus share | cite | improve this question edited Jan 19 at 19:38...
                    Read more