My experience of directio (and it's small I must admit) is that all it does is mask other issues that should/could be resolved to much greater effect. As Tom suggests, it's shifting responsibility.
Am I reading it correctly that your database is around 5TB? A decent size then! I'd be interested to know what % of that is actually active data and how much of it is essentially just there for old time sake.
3 APWs is verging on the upper limit of what I'd consider implementing without serious thought. But then I've not worked with such a large DB in the past.
Tom's question re the storage system is really important. I might have missed it, but do you have info on how fast a bigrow is on the disk? The so-called Furgal Test: Furgal Test?
Yeah 5TB, We have another database too which is starting to catch up. But this particular database has one table with 5 billion records in it. Were looking at reducing this as its just not cricket.
The furgal test; i've done a variety of tests on most servers using this and Crystal Disk; the script has the following:
call proutil sports -C truncate bi -bi 32768
call proutil sports -C bigrow 4
G: Drive (mainly indexes, 4096 blocksize disk) - 17 seconds to grow
R: Drive (Mainly data 4096 blocksize disk) - 17 seconds to grow
B: Drive (BI 16384 blocksize disk) - 22 seconds to grow