The regulator is ascertaining whether the exchange allowed Americans to trade unregistered securities on its platform.