Since the mass of the black hole in nearby galaxies appears to be proportional to the spheroid mass, the mass function of black holes must be similar in shape to the spheroid mass function. The mean black hole mass is therefore that appropriate to an L* galaxy, or about 3 x 108 M. The Eddington limit of such a black hole is about 3 x 1046 erg s-1 and its mass doubling (Salpeter) time is about 3 x 107 yr. If the typical mass black hole has therefore grown from say a million solar mass one in 3 x 109 yr (i.e. by z ~ 2), then we probably need L > 0.05 LEdd ~ 1045 erg s-1. This means that the typical growing black hole was powerful and of quasar-like luminosity (indeed housing a quasar at the centre).
Such an obscured powerful object would locally be classified as a ULIRG (see Sanders & Mirabel 1996), although the distant ones need not be the same as the local ones, which are perhaps mainly fuelled by mergers.
Of course it is possible that massive black holes grew inside galaxies which themselves were merging back at z ~ 2. Nevertheless, unless they were all assembled from smaller holes just before accretion switched off, it is probable that they emit for a reasonable fraction of the last doubling time as a single object.