One of the challenges is how effectively utilize the bandwidth of the transmission media. Wavelength division multiplexing is common used method of multiplexing to create high-speed full-optical networks. Increase of transmission capacity can be achieved by using high order modulation formats. Moreover, considering the growing demand for broadband of access networks, the present research study investigated various modulation formats as a way of increasing the performance of optical transmission systems. This chapter deals with comparison of bit error rate for the same symbol rate of phase-shift keying and quadrature amplitude modulation technique by overcoming the problem of Four Wave Mixing nonlinear effect, which occurring in WDM. The VPI photonic software was used as a simulation tool. The simulation results showed how high-order modulation formats can increase the transmission capacity and allow better adaptation of transmission signal to transmission channel. Also, it has been observed that quadrature amplitude modulation has better performance in comparison with on/off keying and phase-shift keying modulation.