There is an appl-alert-boxes attribute on the session system handle that will control whether an alert-box pops up or the message is displayed in the message area. But I think that is not exactly what you want / need. I always use a method / function invocation whenever I produce an application specific message. It is the method / function that decides how the message is handled. Usually it is always written to the client log file of the ABL session and based on other parameters an alert-box might be popped up.
I also "try" to have an error handling in place that holds as much water as possible to not present ugly ABL error messages to the user.
Heavy Regards, RealHeavyDude.