Would it be easier/harder for me to learn the polish language?

I live in a house with a grandmother and aunt who speak polish fluently due to the fact that my grandmother moved here when she was in her twenties. I can understand their conversations and put meanings to certain words I hear. But, I'm not sure if it would be more difficult because of how my grandmother speaks in polish and English to my brother and I. I haven't learned the language because our doctor at a young age advised our aunt and grandmother not to teach us the language to make learning English easier. I'm just curious, and would like any opinions you guys have!