Classical and quantum chaos in a three-mode bosonic system