Keldor would never part them from their children if they really want them to stay. He knows what that sort of thing feels like; at least to a child. Being deprived of some of your family.

Not knowing your real mother...

Dammit! Now he's gonna get mopey if we let him!

Focus on something positive...

"I would appreciate that."