Well, I have a boolean field for gender, and the values possible for that kind of field are true/false/unchecked, right? In what way I can set 0 for not selected and 1 & 2 for two genders as you said? Is it possible at all to use 3-value option group in order to cause the gender field to be either Yes, No, or Not selected?
It seems to me you've already tested that and found it to be impossible (post #1). That would have been the solution for you if it had been possible.
Firstly then, CheckBoxes can either be boolean fields (TRUE/FALSE) or they can be three-way (TRUE/FALSE/Unset). The idea of a boolean field with three possible settings is contradictory.
If an Option Group cannot take a Null value (Post #1) then another value must be assigned specifically in your logic.
I think your options are laid out clearly in post #2.