Madworld is not the sort of game that should ever be expected to sell huge numbers. It wouldn't have sold big on the other consoles either or on the PS2 last gen. It's just not that kind of big blockbuster hit kind of game. It stands out more on the Wii because of a lack of mature core games, but it isn't a big title. Combine that with an iffy price point and there you go.
For GTA... well I didn't even know it had come out. But I don't see videogame ads on TV very much at all these days. I just don't seem to watch the shows that videogames are advertised during. When I used to watch wrestling I used to see game ads all the time. So I can't say if it was marketed enough or not. But I didn't really see big hype for the title on the internet. With the really big games you just KNOW when the game is coming out, even if you're not specifically paying attention to it. There's buzz on game forums and such. I haven't seen that really with GTA: Chinatown Wars.
I wonder if part of it is how the title was revealed. When it was "debuted" all we had was a logo. No screenshots at all. There was nothing to start the buzz aside from the GTA name. Don't forget it was also debuted at last year's E3, which went over like a fart in church with the internet core gamer audience. Instead of starting the hype train it just lumped GTA: Chinatown Wars with Wii Music and Animal Crossing as part of "that crappy E3".
I think there's also a portable vs. console game thing going here. It's like GTA IV is the REAL GTA and there's that natural assumption that portables are given half-assed spinoffs. That's not necessarily the case here but, with American third party publishers, there's a mostly fair assumption that only suckers get the portable cash-ins. Portable versions of American franchises usually just rely on a popular name to attract sales. If you got burned by that too many times you might see GTA for the DS and assume "Ah, it's not going to be that good. It'll just be scaled down and half-assed."
I think it may also be the GTA is VERY overexposed. You had three console games last gen, a GBA game, the two Stories games (so a PS2 owner might own FIVE GTA games for one console) plus GTA IV's recent release. I think we're at a point where it isn't essential to own EVERY GTA game and a gamer with that attitude is more likely going to cut out the portable games first.
With mature games though it really relies a lot of Nintendo's example. The reason their systems are considered "kiddie" is entirely because Nintendo puts in no real effort to providing mature games aside from a token title here and there. The first party really sets the tone. On the flipside the Xbox has never really been seen as a valid option for kids because despite a few token efforts of MS, the first party lineup is all targetted at an older audience and the image follows it.