Roll Forward fails with timestamp error

[cj_brandt]

While applying AI logs to 2 separate databases in the last couple months, the roll foward has hit a timestamp error and we are unable to continue applying AI logs. The scan of the AI log shows it is a legit AI log, but the timestamp the next AI file expects is wrong - it doesn't match the timestamp in the database. The timestamp in the database is correct because the database can be started after the error is given.

We have opened a couple tickets, but I was curious if anyone else has ever seen this.

Server 1
RH Linux 5.3; 10.2B03; applied ~ 200 AI logs then next AI had wrong timestamp reported.
Server 2
HP Itanium 11.31; 10.2B0606; applied ~ 33,000 AI logs then next AI had wrong timestamp reported.

The only fix is to take a new bkup of db and ship to HA / DR. Because the db on server 2 is over 1 TB in size, that is a pain.

I am confident of our procedures of applying AI logs so I'm not looking for help in that area, I just find it hard to believe we are the only site that has noticed this.

We apply AI logs to over 300 dbs in HA and DR - we have been doing this for over 7 years and now we have had 2 issues in a couple months on 10.2B.

 

[RealHeavyDude]

I only had an issue with the After Image time stamps once: When I was a newbie and just had taken over a production machine running a Progress V6 database at the beginning of the 1990s, the database crashed on a Friday and we were not able to start it again. Because of the database being backed up offline every Sunday and after image was enabled I thought that our butts were covered.

How wrong I was: Before I was able to review the backup strategy and the involved scripts in detail I had to find out the hard way that one does not mess with the database between backup and beginning after image. As it turned out my predecessor that had written the script thought that it was a good idea to remove old log data from the database immediately after the backup had finished but before after image was switched on! He thought that having after image enabled during this lengthy purge process was just an unnecessary performance overhead. I was just wondering how he was able to successfully verify that after image was working at all (I never met this guy in person ...)!

The reason for the database not being able to start was that it contained 2 bad blocks. Lucky me, I did backup the damaged database before I tried the restore and the roll forward recovery. As almost all hopes were gone I restored the copy of the damaged database and succeeded in repairing it using the dbrpr option on proutil. I was able to delete the 2 bad blocks and the database started successfully! Plus, we found out that the 2 bad blocks contained log data that was due to be purged ...

Needless to say that I did change the scripts and thoroughly tested them. A week later the disk on which the database was located was gone ...
Heavy Regards, RealHeavyDude.

 

[cj_brandt]

One other item we just noticed. In both cases the AI with the incorrect timestamp was produced - AI notes were written as a result of work being done in the data dictionary involving schema changes. There was no other users in the database - only AI notes produced were a result of schema work.