Resolved How to load and manage a big text file ?

[Pioux]

Hi everyone,

I would like to load in a memptr a big text file (more than 3 GB) and then manage the content (I'm using OpenEdge v10).

If I do the following, I get a runtime error because the SET-SIZE statement only work with INTEGER data-type and the value of the variable is too big to fit in an INTEGER.

    DEFINE VARIABLE mm-g-file-binary  AS MEMPTR  NO-UNDO.
    DEFINE VARIABLE mc-g-source-file  AS CHARACTER  NO-UNDO.
    DEFINE VARIABLE mi-g-source-size  AS INT64  NO-UNDO.

    DEFINE STREAM mst-import.
    
    ASSIGN
        mc-g-source-file  = fin-source-file:SCREEN-VALUE
        FILE-INFO:FILE-NAME = mc-g-source-file
        mi-g-source-size  = FILE-INFO:FILE-SIZE.

    SET-SIZE(mm-g-file-binary) = mi-g-source-size.

    INPUT STREAM mst-import FROM VALUE(mc-g-source-file) BINARY NO-MAP NO-CONVERT.
    IMPORT STREAM mst-import mm-g-file-binary.
    INPUT STREAM mst-import CLOSE.

How can I do to use a INT64 instead an INTEGER to set a size to my MEMPTR ?

 

[Marian EDU]

check copy-lob for that although don't see how would you 'manage' a 3GB content... if it's a somehow well defined binary format you might still want to manage it in chunks instead of loading everything in memory.

 

[TomBascom]

Ditto COPY-LOB

 

[TomBascom]

Are you running a 64 bit Progress executable?

 

[Pioux]

In fact I have to read each line and search for specific bytes patterns, that's what I wanted to say by "manage".
And yes it would be better if I can load the file by chunks, but I don't know how to do that with the IMPORT statement, that's why I've tried the way above.

I will check the COPY-LOB option, thanks for the help ! I'm not familiar with this method, I have to read the documentations.

Tom, no I run a 32 bit executable for our frontend, we use the 64 bits only for the application core.

 

[TomBascom]

You are not going to be able to work with a 3GB LOB in memory within the context of a 32 bit executable.

 

[Pioux]

doh! And do you know which method can I use to load it by chunks (as Marian says) and avoid to load the 3 GB in memory ?

 

[TomBascom]

Have you looked at the documentation for COPY-LOB?

(It supports STARTING AT and FOR options.)

 

[Pioux]

Yes it's done. I've succeded with the COPY-LOB method, and used the STARTING AT and FOR options to load le file by chunks. It takes some time to proceed but it works !
Thank you all for your support !