Basically. You could say there were two Covenant groups at the end of Halo 3; the first, led by the Arbiter, were the ones who accepted humanity's truce and agreed to the ceasefire. Then, in the months following Halo 3's conclusion, there were those fanatics who still followed the false ways of the prophets. They were their own group called the "Servants of Abiding Truth". Jul M'dama and the Covenant we see in Halo 4 basically started from that group. And we don't necessarily know where the Arbiter is in all of this yet.
As for the Brutes, some of them returned to their homeworld, very few remained in service to the Elites, and the Elites basically wiped out whoever didn't fall under either of those categories.
IRON MAIDEN IS MY RELIGION! Gamertag: Demigod Elessar