The IRS is no stranger to adding penalties to existing debts for a multitude of reasons. If you come into a situation where you feel that a penalty was unjustly added to your balance due, you can apply for a penalty abatement, or a removal of that penalty towards your debt due.
To qualify for penalty abatement of any kind, you will need to provide proof to the IRS that there was a reasonable cause for filing or paying late. A few examples of reasonable cause are a death in the family, an unavoidable absence due to being in jail or rehab, destruction of records due to environmental damage (fire, flood, etc.), a civil disturbance that delayed ability to make a payment, or incorrect advice was dispensed from a reputable and competent tax professional.
The IRS also has a program called the First Time Penalty Abatement (FTA). The program is designed to remove three kinds of penalties: Failure to File, Failure to Pay, and Failure to Deposit. Failure to File is a late filing penalty that is 5% of your balance per month. Failure to Pay is a penalty incurred when you pay your taxes late. This penalty is 0.5% of your outstanding debt per month. The Failure to Deposit penalty is related to payroll taxes, and that penalty is dependent on how many calendar days a deposit is late or if there was a direct payment or not.
To qualify for the FTA one must meet three pieces of criteria. The first is that you have not incurred any penalties for the previous three years. The IRS ignores penalties that are under $100 and penalties for underpaying estimated tax in previous years. The second criteria is that you have filed all required returns and extensions. The final criteria is that you have made payment arrangements for any outstanding debt, either by paying in full or through an installment agreement. The IRS will only remove penalties incurred during the first year, and up to $10,000.