Raising the retirement age is a common recommendation to end the financial woes facing Social Security. The idea is simple: Because people are living longer and healthier lives, they can afford to work well past 66 and retire later in life.

A new report, however, finds that this would create significant hardships for people in physically demanding jobs, especially minorities, less educated people and those at the low end of the wage scale, according to "Still Working Hard," by the Center for Economic and Policy Research.