Resolved How to load and manage a big text file ?

Pioux

New Member
Hi everyone,

I would like to load in a memptr a big text file (more than 3 GB) and then manage the content (I'm using OpenEdge v10).

If I do the following, I get a runtime error because the SET-SIZE statement only work with INTEGER data-type and the value of the variable is too big to fit in an INTEGER.
Code:
    DEFINE VARIABLE mm-g-file-binary  AS MEMPTR  NO-UNDO.
    DEFINE VARIABLE mc-g-source-file  AS CHARACTER  NO-UNDO.
    DEFINE VARIABLE mi-g-source-size  AS INT64  NO-UNDO.

    DEFINE STREAM mst-import.
    
    ASSIGN
        mc-g-source-file  = fin-source-file:SCREEN-VALUE
        FILE-INFO:FILE-NAME = mc-g-source-file
        mi-g-source-size  = FILE-INFO:FILE-SIZE.

    SET-SIZE(mm-g-file-binary) = mi-g-source-size.

    INPUT STREAM mst-import FROM VALUE(mc-g-source-file) BINARY NO-MAP NO-CONVERT.
    IMPORT STREAM mst-import mm-g-file-binary.
    INPUT STREAM mst-import CLOSE.
How can I do to use a INT64 instead an INTEGER to set a size to my MEMPTR ?
 

Marian EDU

Member
check copy-lob for that although don't see how would you 'manage' a 3GB content... if it's a somehow well defined binary format you might still want to manage it in chunks instead of loading everything in memory.
 

TomBascom

Curmudgeon
Ditto COPY-LOB
 

TomBascom

Curmudgeon
Are you running a 64 bit Progress executable?
 

Pioux

New Member
In fact I have to read each line and search for specific bytes patterns, that's what I wanted to say by "manage".
And yes it would be better if I can load the file by chunks, but I don't know how to do that with the IMPORT statement, that's why I've tried the way above.

I will check the COPY-LOB option, thanks for the help ! I'm not familiar with this method, I have to read the documentations.

Tom, no I run a 32 bit executable for our frontend, we use the 64 bits only for the application core.
 

TomBascom

Curmudgeon
You are not going to be able to work with a 3GB LOB in memory within the context of a 32 bit executable.
 

Pioux

New Member
doh! And do you know which method can I use to load it by chunks (as Marian says) and avoid to load the 3 GB in memory ?
 

TomBascom

Curmudgeon
Have you looked at the documentation for COPY-LOB?

(It supports STARTING AT and FOR options.)
 

Pioux

New Member
Yes it's done. I've succeded with the COPY-LOB method, and used the STARTING AT and FOR options to load le file by chunks. It takes some time to proceed but it works !
Thank you all for your support !
 
Top