Readers: Do you get annoyed if there are two POV's in a novel?

I don't like it when they switch off every chapter to another person. But in my novel, I'm writing maybe one or two critical chapters from the secondary characters point of view. It really gives the reader insight as to how the character feels, not just how the main character sees them.