I suggested overcomingbiaschan as a joke, but I am warming to the idea of anonymity. It seems to me that most forum software is a bad comprise between identity and anonymity, since it allows people with no lives to create alternate online personalities that they become hugely invested in, and useful discussions become swamped by people’s need to nurture these electronic alter-egos. So rather than compromising, pick an extreme. Either ask people to pay a token fee to join the forum and ask that they use their real names, or use an ultra-minimalist anonymous BBS such as kareha.
A fee-and-name-based system would not require karma or reputation tracking, since it is easy to remember someone’s name and whether that person is an idiot or not, the fee would keep out time-wasters, and the name would make someone less willing to wreck their own reputation by trolling. Though it would be impossible to enforce people using their real names, the fee would hopefully encourage people to play by the rules.
There’s a brief argument for anonymous systems here which makes the point that usernames encourage vanity, and you can see kareha in action here.
In the end though, what is really important is that you need a smart, respectful community and an explicit set of rules rigorously enforced by excellent moderators, and that goes for any messaging system you use.
Some more thoughts:
The poor quality of discussion on Digg, Reddit and Slashdot demonstrate, I think, that user-moderation is ultimately a bad system that tends toward in-groups and lowest-common-denominator discussion topics. A notable exception is Hacker News, and I think this is mostly to do with the community emphasis on simple rules like “be polite”, and the close link between community and moderators. I remember recently the administrator of HN, Paul Graham, made a thread asking whether links to election stories should be banned from HN. A sensible discussion followed, and the community voted to ban election stories. I think this shows that voting works best on the meta-level, rather than voting on specific comments and articles, it is better to vote on rules about comments and articles, which moderators then enforce. The best example of a self-regulating community is probably Wikipedia.
Further to my point about paying to use a discussion area, an example of a successful forum that actually does this is Something Awful. On Something Awful anyone can view currently active threads, but you have to pay the tenbux if you want to post or view the archives. When you join it is made clear that if you get banned, your money is not refunded. This gives an incentive for good behaviour. The people who run Something Awful also use the forums as source material for articles on the main site. Overcoming Bias could do something similar. If an OB admin sees somthing they like on the forum, they could e-mail the poster and ask them if they would like to write up the forum comment as a blog post. Or alternatively there could be a community-edited, publically-viewable wiki. The possibilities are endless.
Finally as far as forum software goes, I hear Vanilla has excellent moderation tools, which I think is the most important thing.