Disadvantages of IRAs One drawback of using IRAs to save for retirement is that the annual contribution limits are relatively low. In the case of traditional IRAs, the distributions you make will be taxed according to your income tax rate at the time the withdrawal takes place. If distributions are made before age 59 and a half, a federal tax penalty of 10% applies. However, in certain situations, such as Gold IRA Investments, you may be allowed to make early withdrawals from an IRA without a 10% penalty. Expenses that may exempt you from the penalty include, but are not limited to, buying a home for the first time, medical bills and higher education payments.