Text File Manipulation

[ank123]

Hi,

Please provide me the solution on my query below :

datafile.txt

AcctNum,code,Region,,,,
12345451,AN ,abaab
12345452,AN ,xccxc
76677545,RP ,acxcc
43567878,RP ,afghh
32190900,AN ,afrfrf
87312345,AN ,aqaw

I have a text file (datafile.txt) which has data with above format.
This file has many more columns and can have huge number of the records.

I have to process the above file and need to send the records into another text file (a1.txt) but this file can have maximum 10k records for a particular 'code' say 'AN' and other records should be moved to another file a2.txt. i.e if there are records more than 10k for a particular 'code' then the records number 10001 onwards will be moved to file a2.txt

Thanks in advance.

 

[MaximMonin]

like this

define temp-table filedata no-undo
field acctnum as character
field code as character
field region as character
index i0 code.

input from value (filename)
repeat:
create filedata.
import filedata.acctnum filedata.code filedata.region.
end.


for each filedata break by filedata.code:
if first-of (filedata.code) then
do:
i = 0.
j = 1.
end.
i = i + 1.
if i modulo 10000 = 0 then
do :
j = j + 1.
end.
write record to filename + filedata.code + string(j).
end.

 

[j4n]

if it's a 4gb datafile you might don't wanna use a temp-table to store all the data. just read a couple, analyse them and write them to a file.

 

[tamhas]

I suspect we don't really have all the requirements. If the only requirement is not writing more than 10,000 of one code per file, then create counters, either simple variables if there are a small number of fixed codes or a temp-table of code/count if it is more open. As you read and write each record, increment the counter. If a counter reaches 9999 or 10000, depending on the real limit, then close that file and open a new one, zeroing all the counters.

Of course, if there are unspoken requirements, you may have to modify the algorithm.

One might also wonder why you are handling what appears to be relational data in massive text files. Consider sucking them into a table and doing all your processing there.

 

[TomBascom]

Or consider processing them with OS tools...

 

[Cringer]

With that amount of data in text files you will definitely run into problems in Progress. It's not the best tool for the job. I would suggest that some Unix scripts would do the job a lot more efficiently. If Unix/Linux isn't available then there must be some similar text file processing applications for Windows.