General rogue waves and modulation instability of the generalized coupled nonlinear Schrödinger system in optical pulses.

We focus on rogue waves and modulation instability (MI) of the generalized coupled nonlinear Schrödinger (GCNLS) system in optical pulses. Through the Kadomtsev-Petviashvili hierarchy reduction method, general high-order rogue wave solutions in Gram determinant form at p=p0 are constructed, which contain derivative operators with respect to parameters p and q. We reduce solutions to purely algebraic expressions with the aid of the elementary Schur polynomials. The multiplicity of p0 determines the structures of rogue waves and generates diverse patterns. The structures of Nth-order rogue waves are composed of N(N+1)/2 fundamental ones while p0 is a simple root. Free parameters aj play an important part in the patterns of Nth-order rogue waves, large values of a3 lead to triangle structures while large values of a5 yield pentagonal shapes. When p0 is a double root, rogue waves are given by 2×2 block determinants. They are degenerate solutions with N1=0 or N2=0, and they are non-degenerate solutions under the constraint N1,N2>0. Dynamics of degenerate and non-degenerate rogue waves exhibit significant difference from the former case. MI of the GCNLS system is investigated by linear stability analysis since it is closely associated with the excitation of rogue waves. Effects of different parameters on distributions of the growth rate G for MI are considered. Numerical results suggest that amplitudes Aj and wave numbers kj(j=1,2) of the background fields control the widths and positions of MI areas. The results can help us better understand some specific physical issues, especially the propagation in optical fibers.