The 25-year-old emerged as a mainstay in the United starting line-up under Sir Alex Ferguson but, after a season with David Moyes in charge at Old Trafford, started just six league games following the appointment of the former Netherlands boss.
Van Gaal has clashed with the likes of Rivaldo while at Barcelona in the past and Rafael, who left United for Lyon this summer, acknowledged the Dutchman's track record with Brazilian players.
"I don't know, sincerely," he told Globo Esporte when asked about Van Gaal's dislike of Brazilian players. "I don't know what to say about that because I was the only one [at United], since Anderson left before.
"I know this history that he doesn't like Brazilians but I can't say: 'Oh, he doesn't like Brazilians'. I know that he didn't like me but, if that's because I'm Brazilian, I don't know."
Rafael described Ferguson's departure from Old Trafford in 2013 as "fatal" to his United career.
He remarked: "I did the complete season, when [Ferguson] left. In 2012-13 I played almost every game, it was my best season at United - we were even champions - and that's when they changed the manager. That change was fatal for me. I was going well, then Ferguson left.
"That's what changed, so [Antonio] Valencia started playing as a right-back for Van Gaal. He's also a great player but I was doing well in that position.
"Ferguson even called to wish me good luck. I had a very good relationship with him, he helped me a lot in Manchester, and always did everything to help me. He signed me when I was 16; I don't even know what to say."
Rafael also indicated that he holds no grudges over his exit from United, having lost his place in the side.
"[Van Gaal] had his opinions, his thoughts," the defender said. "Everyone has their opinion. A manager is a manager. I always respected it and I will always respect it."
