Kirillov–Lipsman orbit method of a class of Gelfand pairs: part I

Let \(G:=K\ltimes N\) be the semidirect product with Lie algebra \(\mathfrak {g},\) where N is a simply connected nilpotent Lie group, and K is a subgroup of the automorphisms group, Aut(N),  of N. We say that the pair (K, N) is a nilpotent Gelfand pair when the set \(L_K^1(N)\) of integrable K-invariant functions on N forms an abelian algebra under convolution. According to Lipsman, the unitary dual \(\widehat{G}\) of G is in one-to-one correspondence with the space of admissible coadjoint orbits \(\mathfrak {g}^\ddag /G\) of G. Under some assumptions on the pair (K, N) we will show in this paper and its sequel (part II), that the Kirillov–Lipsman bijection

$$\widehat{G}\simeq \mathfrak {g}^\ddagger /G$$

is a homeomorphism for a class of Lie groups associated with the nilpotent Gelfand pairs (K, N). Part I (this paper) concerns generalities and the study of the convergence in the quotient space \(\mathfrak {g}^\ddag /G.\) More precisely, we give a necessary and sufficient conditions when a sequence of admissible coadjoint orbits converges in \(\mathfrak {g}^\ddag /G.\)