Buying younger stock might work out well, since you'll be able to work with them and get them accustomed to your routine and handling expectations. I do believe there are nasty-tempered rabbits out there- I've owned some! But for the most part what I've seen is that scared rabbits - scared animals of any sort, really - can appear aggressive when they're fearful. And a normal adult rabbit (one that has not been selectively bred for docility so has a typically fearful prey mindset) that hasn't been handled, is set in its ways, and is abruptly moved to a new environment (and has a mite infestation!) is a pretty reasonable candidate for fear aggression behaviors. So, getting young rabbits and gentling them is often a good way to go. Also, buying from a smaller rabbitry, where the animals are more likely to have been selected for docility at some level, increases your chances of actually being able to gentle the animals to begin with.
I don't agree to buy rabbits sight-unseen; if I get to where I'm considering buying a rabbit and don't like what I see (the animal or the owner), I do not hesitate to walk away empty-handed. But regarding wanting to tour someone's rabbitry: not all folks who do not invite you to their property are hiding something. They don't know you any more than you know them. It's one thing if you're talking about an actual business, in which case an expectation of transparency is reasonable. But I am a small show and meat breeder, and we live in a rural area not terribly far from a big city, which hosts a tremendous range of different sorts of people. Not all of those people agree with the way we live (I'm talking about PETA types, or those who believe Big Brother should have a say in every area of our lives, or those who live a life powered by drugs, among others). We have a lot to lose if someone comes to our property and cases the place, or decides that he or she has a bone to pick with how we're doing things. A large part of my sales is through word-of-mouth, but when I use craigslist, I nearly always arrange to meet buyers somewhere in town. Even though my experiences on craigslist are about 99% positive, it's that 1% that keeps me from inviting strangers to our home. This is not to hide anything, rather it's an attempt to protect ourselves and our property.
I actually have some good "rabbit friends" that I initially met via craigslist ads, but I have many more as a result of going to shows, working with 4-H families, and cooperating with other breeders. I really enjoy helping folks get started in rabbits, and often that includes a tour of our set-up, but that only happens when a certain level of trust has developed. So the moral of the story, I guess, goes back to the suggestion to get to know some of the breeders in your area. Going to shows can be a great way to meet them, as well as to get yourself established as trustworthy, and to learn which breeders are well-respected by others. There is a gamut of personalities, to be sure, but there are always certain people that rise to the top and are admired by most of the community.