Study of interfacial instabilities in a three-layer fluid system