First off, it is pretty frail. Only collectors and those who really took care of their PSP games would avoid this. I mean, just taking it out of it's case would slowly cause the UMD to split. Imagine if it was handled by kids. Games would break, people would stop playing their games, and interest in PSP would decline. And this is a portable, where games are meant to be carried around. This just increases the problem ten-fold.
Second, it's a tad bit too large. Even if you have a case for it, you wouldn't be able to carry around more than a few in your pocket. You can certainly pack away a few more in a knapsack or bag, but this seriously reduces the amount of accessibility. Not everyone wants to carry a bag, which leads to the PSP not even being taken.
Third, UMD reader. The UMD reader was no doubt a reason why the PSP cost a lot. It had a lot of small components, and if it broke, repairing it probably was not cheap. having a component like this as part of a portable system was definitely not a good idea. If it broke, for a lot of people, it might've meant never owning a PSP again. It also took up a large portion of the PSP itself, a portion that could've been better used.
Fourth, load times and battery. I know you said it's possible, but not everyone are awesome programmers. A lot of games had bad load times, and the UMD spinning drained the battery. Having a system that is easy to deal with for developers is a huge plus for software creation. A spinning disk definitely was not the best idea for a portable system.
Lastly, it's only 1.8GB. Though the DS proved that capacity doesn't matter, the PSP was clearly marketed as having games that were PS2 quality, and and thus set itself up for this limitation. Developers had to create something that was high quality, while limiting themselves to 1.8GB. This may not seem like a big problem, as the PSP definitely achieved great PS2 quality games with it's 1.8GB limit, but at the same time, it would not be untrue to say that developers could've achieved more with bigger space. Right off the bat, the PSP limited itself, setting a max space of 1.8GB, a limit which could not be increased without some sort of new UMD reader.
Btw, if I sound like I hate the PSP, I'm gonna say right now that I love it. Favourite system this gen. I'm just hating on the UMD. I seriously can't understand what Sony was thinking when they thought it up.