This set of posts comes from Pastor Mark Driscoll and his wife Grace. The sermon series The Peasant Princess brought up dozens of questions that they felt they should respond to. The result is this blog series: Christian Sex: Frank Answers to Honest Questions.
Answer: Yes, this is a serious issue and you are sinning against them. Your sin is affecting other aspects of your marriage and causing your spouse to suffer, though they may not know why. Right now your sin is growing because it is hiding in darkness (a lie), and by confessing it to them and asking for their forgiveness you will be bringing it into the light, where it can be forgiven and healed over time. When you do confess to your spouse, please be careful to not blame them for your sin, but accept responsibility. In addition, both of you will most likely need to get help from a pastor or biblical counselor because this often does more damage than you are able to recognize at first.