Vtgyjfy

Let $G$ be a group, $H le G$ and $f colon G times G rightarrow G$ the group operation. We know that $complement_GH$ (the complement of $H$ in $G$) contains the inverse of any of its elements, so, whatever $G$ and $H$ are, $lbrace e rbrace subseteq f(complement_GH times complement_GH)$.

On the other hand, if we take $G=(mathbb{Z},+)$ and $H=2 mathbb{Z}$, we get that $f(complement_GH times complement_GH)=H$, because by summing pairwise all the odd integers we get all the even integers (and them, only).

This makes me conclude that, in general, at least the following holds: $lbrace e rbrace subseteq f(complement_GH times complement_GH) cap H subseteq H$.

I ask the following:

1. what's the characterization of $H$ and/or $G$ to get $f(complement_GH times complement_GH)=lbrace e rbrace$, if any?


2. what's the characterization of $H$ and/or $G$ to get $f(complement_GH times complement_GH)=H$?

(by "characterization of $H$ and/or $G$" I mean something like, e.g., "$H$ normal in $G$", or the like).

Note that you should begin with $H<G$ instead of $Hle G$, as $H=G$ certainly makes $f(complement_GH times complement_GH)=emptyset$.

So assume $H<G$. If we pick $ain complement_GH$, then for any $hin H$, we have $ha^{-1}notin H$ and hence $h=ha^{-1}cdot ain f(complement_GH times complement_GH)$. This make
$$Hsubseteq f(complement_GH times complement_GH)qquad text{if }H<G. $$

Consequently, the situation in your first question occurs iff $H={e}$ and the situation in your second question occurs.

For the second part, in order to obtain only $H$, the case $ha^{-1}cdot a$ we used above must be "essentially" the only one. Indeed,
$$H= f(complement_GH times complement_GH)iff[G:H]=2.$$
Proof:

• If $H$ is of index 2, pick $ain Gsetminus H$ such that Then if $x,yincomplement_GH$, we have $h_1:=xain H$, $h_2:=a^{-1}yin H$ and so $xy=xaa^{-1}y=h_1h_2in H$.




• On the other hand, if $f(complement_GH times complement_GH)=H$, then we already know $Hne G$. If $a,bincomplement_GH$, it follows that $a^2in H$ and $abin H$, hence $a^2H=abH$ and thereby $aH=bH$, i.e., there are only two cosets

Back to the first question: We have $f(complement_GH times complement_GH)={e}$ iff $G$ is of order $2$ and $H$ the trivial subgroup.

Generalization: If there exist $a,bincomplement_GH$ with $abnotin H$, then $f(complement_GH times complement_GH)=G$. Indeed, If $gin G$, then at least one of $ag$, $b^{-1}g$ is $notin H$ because otherwise also $ab=ag(b^{-1}g)^{-1}in H$. We conclude $g=a^{-1}cdot ag=bcdot b^{-1}agin f(complement_GH times complement_GH)$.
Such $a,b$ exist whenever $[G:H]>2$. Therefore,
$$f(complement_GH times complement_GH)=begin{cases}emptyset&text{if }H=G\H&text{if }[G:H]=2\G&text{otherwise}end{cases}$$