Yes, I'd say definitely leave them longer than 4 weeks! The only time I wean bunnies at 4 weeks is if something has gone wrong. They can survive at that age, but usually one would hope for better than just survival. I have found that 6-8 weeks works the best, both for their health and their growth rates. You can re-breed the doe at 4-6 weeks after kindling, wean the bunnies at 6-8 weeks, and she'll have 2 weeks to recover and gestate.
If you have a large litter, you might think about weaning the biggest bunnies first and leaving the smaller ones for another week of milk; that can even things out a bit.
Another option for weaning, which I like and use sometimes depending on how many does and growouts I have in play, is to move the mother and leave the bunnies where they are. That definitely minimizes the stress of weaning. Of course that means that you have the doe kindle in the grow-out pen, but I find it works very well when you have the capacity to do it. The doe can go back to a smaller pen while she waits for her next litter.