does it matter if a baby watches some TV?

Baby Doodle is 5 1/2 months old. He just started eating solid food, he rolls over, creeps, but doesn't crawl, and can sit up on his own for a few seconds at a time. And he loves television. We try not to let him watch it when it's on in the same room as him. We distract him with books, toys, superbaby, tickling, just about anything to keep him from staring at the fast moving objects on the screen. Sometimes he falls for our tricks, but sometimes he is completely mesmorized by the TV and won't look at anything else until one of us picks him up and physically moves him so he can't see the TV. Even when we do that sometimes he cranes his neck around and rolls his eyes so he can see that evil television.

I know there are studies that say that kids who watch too much TV are more likely to get ADD or ADHD. But baby Doodle is only 5 months old. He might not even be able to make out the actors on the screen let alone the commercials (though most of what we watch is on TiVo so we don't watch the ads). I would say BD watches about 3 full minutes of TV every day. Those usually occur while I'm running on my trampoline and he has managed to move himself to a place where he can watch too. When I realize what he has done I move him to another position or spot in the room.

Am I dooming my baby to have a short attention span or is the relatively short time he spends staring at the TV harmless?