U.S. mutual funds are placing bigger bets on privately held companies to get a head start finding the next IPO superstar, a strategy that has yielded some dramatic payoffs and flameouts.