1. Never. As Robert Filmer infamously argued, parents are within their rights to treat their children however they please.
2. Only if the parent has perpetrated a severe crime (attempted murder, rape, etc.) against the child.
3. If the parent is likely to perpetrate a severe crime against the child.
4. If loss of custody increases the expected welfare of (parent + child). Or in other words, if the gain to the child exceeds the loss to the parent.
5. If the loss of custody increases the expected welfare of the child. Notice: On this standard, losses to the parent don't count.
6. If the loss of custody increases the welfare of children in general. Notice: On this standard, a child could be taken from his parents even if he'd be worse off as a result - as long as the deterrent effect on other potentially abusive parents made up the difference.
7. If the loss of custody increases expected utility of all non-child-abusers. Notice: This standard doesn't just count the deterrent effect on potentially abusive parents; it also counts third parties' desire for retribution against abusive parents.
8. Zero tolerance: Loss of custody is the "mandatory minimum" sentence for any clear-cut act of child abuse.
I could be wrong, but I'd guess that current U.S. policy is somewhere between #3 and #4. But where is the morally right place to draw the line?
Somehow I doubt that many people will take position #8. It's expressively appealing, but even people who hate economics can't easily evade the fact that a kid's imperfect parents are usually his least-bad alternative. But how much lower are you willing to go?