The petting is literally the only thing removed.
The North American version of Fire Emblem Fates has arrived in the hands of reviewers, and with it new details about the removal of a controversial feature.
There was some concern when it was revealed that the "petting" minigame was removed from My Castle mode, specifically that conversations and statistical bonuses would be lost. Information from a source at GoNintendo has confirmed that characters can still be invited to the room, and the stat bonuses are still available; only the touchscreen "petting" has been removed.