Error Error 1124

[Cringer]

Progress 11.2.1 32 bit (yes I know! ) on Windows Server 2003 Enterprise. We've been getting Error 1124 overnight on Tuesday nights for the last 3 weeks and trying to establish what is causing it. We can't see anything obvious from the various log files (Progress and OS), and was wondering if anyone has any ideas what we should be looking at for causes?

 

[Cringer]

Running dbtool on the area that's broken we get loads of:

Warning - first record fragment of 331796673 area 25 is only 6 bytes, LOB 1.

 

[Cringer]

From the event log, just before error 1124
Message from PROGRESS database f:\database\live\icmasliv (5199)
APW 169: bkwrite: write to disk failed errno 22. (3645)

 

[TomBascom]

"Write to disk failed" seems kind of obvious to me...

 

[Cringer]

Yep thanks Tom. Sorry I was posting info as I uncovered it. No evidence of those errors for previous crashes which is a little disturbing.

 

[TomBascom]

FWIW errno 22 is "invalid argument"

 

[TomBascom]

Is there a virus scanner active on that drive?

 

[Cringer]

Kaspersky is there. It doesn't scan, just monitors, and should be excluding Progress related directories. We've not had any control over that side of things due to owning company IT regime, but we've just been sold. Should we be removing kaspersky? (rhetorical question really!)

 

[Rob Fitzpatrick]

Check the OS logs if you can (event viewer). Look for errors related to physical disks. You should look at your DB log as 1124s are written there. Maybe finding the first instance of it will given you a time frame to concentrate on in the Windows logs which will turn up further errors.

Have you established, from an application or operations perspective, what is special about Tuesdays? Scheduled IT maintenance maybe?

 

[Cringer]

No physical disk errors. Can't pinpoint anything in particular about tuesdays either. Which is odd.

 

[TomBascom]

I bet you a beer that it is the virus scanner malware.

 

[Cringer]

Agreed. Could you rephrase that in a way I can pass it to the powers that be to add weight to our constant pestering please?

 

[TomBascom]

Running dbtool on the area that's broken we get loads of:

Warning - first record fragment of 331796673 area 25 is only 6 bytes, LOB 1.

Is this happening now (not just Tuesday nights...)

Which dbtool options?

Have you done a scan with dbrpr?

This sounds more like a bad thing happened -- rather than a transient write error. It still could have been caused by malware (I consider virus scanners to be malware).

 

[cj_brandt]

Does the windows server get backed up to tape on Tuesday nights ? I've seen errors like that from backing up live db extents to tape.

 

[Cringer]

We've been in touch with Progress and they have advised that it looks like the server's resources are being used up at that point as the database doesn't shut down. It just completely crashes out. They said to get Kaspersky off, quick!
After further investigations we found that our IT company in their wisdom moved out kaspersky management server to a new machine and have not pulled across all the exclusions and rules that had been set up on the old one. That's probably the issue.

@TomBascom , what is dbrpr? How do we use it? The dbtool output was from option 4. Just been informed that this is because of a 32bit/64bit issue from when we upgraded something, and we have been advised how to fix it by Progress.

Thanks everyone for the help.

 

[TomBascom]

dbrpr is a utility for scanning (and maybe fixing) a database that you think might be corrupted. There is an article on its use in the kbase

 

[Cringer]

Fab thanks Tom

 

[Cringer]

Urgh Ka$p€rsky is gone from the server and yet we had another db crash at 1:30 this morning. So that's not the problem. Different errors, but seemingly all to do with resource contention. :/

 

[Cringer]

Been doing some more poking and thinking. I've noticed that the directory that is the parent to all the database files is a shared directory. I'm guessing it's possible that another machine has a drive mapped to there which is then being scanned by that machine's anti-virus or backup client. Does anyone know if it's possible to determine who might have a drive mapped? Or who was accessing the share at 01:30 this morning? I can see who is on currently, but that's not much use!

 

[cj_brandt]

maybe a batch file that runs every 15 minutes and uses "net session" to list the users connected to the shared folder.

Won't help this time, but next time you would have a history of users on the shared folder.