I can kind of see their point of view. Particularly if the original intent involved a Christian gathering of boys for certain activities, etc.
Imagine setting up an A&A club, but making it open for all people. You rent a hall, specifying that people can come play A&A every saturday from noon until 10, supply tables, chairs, etc.
Now you get a guy coming in with MtG cards, occupies a table, and starts to play. He has some friends join, and unlike the A&A slackers, they are there from noon until 10 every day. Eventually all of the chairs and tables are used up by these people who have no intention of playing A&A, but rather have usurped the meeting area and time for their own purposes.
You, my friend, appear to be a MtG player in the A&A club. The BSA has certain principals, goals, and properties, some of which you clearly wish to hijack for your own political/selfish reasons. If you can’t adhere/abide by the principals of the BSA, then one might well wonder why you’d become apart of this organization. (incidently i never did participate in the Boy Scouts, however i did do a lot of camping with my friends growing up).