The earliest I will wean is 4 weeks and I will only wean 1/3 of the litter at a time. I did have a doe die in December that had a litter that wasn't quite 3 1/2 weeks. The litter survived, but they didn't have the benefits of learning much from their mom. They aren't clean, it took a while to get them to drink out of a water bottle (I had to make the switch back), and they are very unruly with eachother (like they don't know how to find a pecking order). They are all extremely sweet, but it is definitely nothing I would do intentionally. This weekend, they will be butchered.
Now, if I think that I'm going to want some out of a particular doe for showing or breeding, sometimes I will leave them with the doe for 5-6 weeks so that they will learn her mothering skills. There are only 2 does in my herd I've done this with and it is because the does are, IMO, exceptional at what they do.