Even in the olden olden days there were a few scholarships, but yes, most people paid for college and mostly the well off went... and met the children of other well off people, married well, and were well off themselves and sent their kids to college and on and on. The big change was after World War II when the vets, who included many people from poor families, got their college paid for in large part by the government. Then, psychologically, college became not just for the rich, and a much larger segment of the country started going to college.
Back in the day when only the well off went to college a high school degree was all you needed for a good job. Now most jobs do require higher level skills, but even the ones that really don't are asking for a college degree, because these days if you don't have one it says something about you, your ambition, ability, etc. And because of affirmative action rules that have forced employers to spell out what they want in an employee much more clearly, and then not deviate from that, the occasional non-BA who would be great in the job doesn't get considered for legal reasons.