Law: Complex software is hard, and increasing complexity leads to exponential increases in difficulty (and corresponding likelihood of failure).
Everyone needs to understand this law. It is a law.
This law, or more correctly, forgetting this law, is the reason for “massive budget overruns”. Its the reason for all failures in IT (except the one where the dev team get abducted by aliens).
Corollary: It is impossible to completely predict complexity before starting development.
Bugger. In other words, the difficulty of your project is impossible to accurately estimate. But it will get more accurate as development proceeds.
Therefore: Keep projects small. Build small. Then add bits. If you need to build a system that appears big, build bits at a time. “Impossible!” you cry at my foolish words, “We have to do the whole system, not just bits!”. That may be true. It probably isn’t, but it may be. But you’re forgetting the 1st law. Its a law dude. It doesnt go away because the client says “You have to do the whole system!”. You will do the whole system. Just bit by bit. This process takes a lot of up-front thought, always focussed on reducing complexity. Its a harder way to develop. But, because of the 1st law, its the way that leads to the maximum chance of success.
PS. Don’t use IBM products. Anything from IBM will suck your project under faster than a greased titanic. And … sorry J2EE, .Net, RubyOnRails et al. There is no silver bullet. Not even a shiny bronze one.