I was under the impression that Retro Studios, like Nintendo, used their own engines. I don't know for sure or even care but that was what I always thought.
Even though i'd love to see something new from Retrp Studios, I'd be pretty happy if they were working on a new 2D Mario for the Wii U launch. I absolutely loved Super Mario 3D Land and EAD Tokyo should certainly keep making Mario games but if puting Mario im the hands of Retro Studios is what it takes to launch with Mario, I'd fully support that decision.