Here’s a smart and funny bit about culture fit. I think it’s brutal and by and large true. 
When I first heard the term “culture fit”, I thought intuitively, yes. Great way to build a company. Companies are cultures and we should aim to build them.
Then, not long after, I thought, wait. “Culture fit” among highly-educated-20-something-males-on-the-spectrum might not actually make for a great culture, if it’s going to be of any size.
That last bit is key: “culture fit” is probably unavoidable in early days, but rapidly transitions to creepy when your company starts growing.
When you start from scratch, you choose partners. To partner up, by definition, means you probably share a vision. Otherwise, why would you be partners?
In all likelihood, sharing a vision means sharing priors. So your early team will not be diverse. It will likely be a tight group of people whose cultures, um, fit.
(It doesn’t have to be this way. But odds are it will.)
Now, if your company’s goal are to be a tight, focused lifestyle business, great. That’s a good way to go. You don’t need much diversity.
However, if you are funded, your goals are probably a big audience and a big exit. Your ability to reach new and diverse audiences requires diversity in your company. I don’t just mean skin color and gender. I mean, people of varying experiences who see opportunities that you don’t.
You need type-A glad-handing salespeople. You need neckbeard sysadmins. You need touchy-feely designers. You need nagging accountants. They will grow the business. And you won’t get them if they need to be like you.
(If you achieve the above, you’ll get the skin-tone-and-gender diversity for free, btw.)
So, start with culture fit. Then know when to lay off.
 See also