Virtual Grids can view large text files?

General Help regarding HMG, Compilation, Linking, Samples

Moderator: Rathinagiri

edk
Posts: 999
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Virtual Grids can view large text files?

Post by edk »

edk wrote: Mon Nov 29, 2021 2:32 pm While working on this task, it turned out that the Grid can be very memory-hungry, especially when we read any property in the QueryData code, be it the Grid control itself, the form window, etc.

To visualize it, just select "Virtual Grid (test memory leaking)" in the "File" menu.
Just try to move the mouse pointer over the Grid and you will see how quickly the available memory is "thinning". Uncheck all three boxes to disable properies polling in QueryData and memory will no longer be consumed at that rate.
I guess generaly there is a problem with a memory leak in the UNICODE version :shock: . In ANSI, there is no leakage in the Grid. ;)
User avatar
srvet_claudio
Posts: 2224
Joined: Thu Feb 25, 2010 8:43 pm
Location: Uruguay
Contact:

Re: Virtual Grids can view large text files?

Post by srvet_claudio »

edk wrote: Thu Dec 02, 2021 10:10 am I guess generaly there is a problem with a memory leak in the UNICODE version . In ANSI, there is no leakage in the Grid.
Hi see this post:
https://www.hmgforum.com/viewtopic.php?p=62584#p62584
Best regards.
Dr. Claudio Soto
(from Uruguay)
http://srvet.blogspot.com
User avatar
Claudio Ricardo
Posts: 367
Joined: Tue Oct 27, 2020 3:38 am
DBs Used: DBF, MySQL, MariaDB
Location: Bs. As. - Argentina

Re: Virtual Grids can view large text files?

Post by Claudio Ricardo »

Dr. Claudio, en su opinión, cual opción conviene más en unicode ?
En un temporizador (en main.prg cada uno o dos minutos) usar:
hb_gcAll (), hb_gcCollectAll () o (de HMG) RELEASE MEMORY, cual de las tres sería más efectiva ?
O antes de salir de la función que demanda mucha memoria usar:
__mvXRelease ( <variable_name> ) ?

Desde yá muy agradecido !
Corrige al sabio y lo harás más sabio, Corrige al necio y lo harás tu enemigo.
WhatsApp / Telegram: +54 911-63016162
User avatar
AUGE_OHR
Posts: 2117
Joined: Sun Aug 25, 2019 3:12 pm
DBs Used: DBF, PostgreSQL, MySQL, SQLite
Location: Hamburg, Germany

Re: Virtual Grids can view large text files?

Post by AUGE_OHR »

hi,

as Claudio say :
in unicode, and is due to the lack of explicit memory release when at c code the strings are transformed into utf8 to utf16 which is the Windows standard
wehn use a "BIG" File (7 GB) you will see that it first grab Size of File but than it become twice Size
when use RELEASE MEMORY i got back 2nd Part of RAM but 1st Part "stay" in Memory until close App

---

now i use Code by Edward to read "BIG" Files.
i (still) wonder that it can handle 7 GB File also under 32 Bit OS :idea:

Question : can i "search" in a "virtual" GRID :?: if Yes how :?:
have fun
Jimmy
edk
Posts: 999
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Virtual Grids can view large text files?

Post by edk »

AUGE_OHR wrote: Fri Dec 03, 2021 2:00 am Question : can i "search" in a "virtual" GRID :?: if Yes how :?:
Check this example:
demo.zip
(9.66 KiB) Downloaded 166 times
I added the "SeekLine" method and an example of how to search in a classic way (Do While loop) - slower version and using the "SeekLine" method - faster version.
User avatar
AUGE_OHR
Posts: 2117
Joined: Sun Aug 25, 2019 3:12 pm
DBs Used: DBF, PostgreSQL, MySQL, SQLite
Location: Hamburg, Germany

Re: Virtual Grids can view large text files?

Post by AUGE_OHR »

hi,

i have a funny Effect with Mouse in that "virtual" GRID

i have add this Code into "QueryFile"

Code: Select all

Function QueryFile( oFile, cDelimiter )
   ...
   IF ! EMPTY(oFile:CurrentLine()) .AND. ! EMPTY(oFile:nLastLine)
      Form_1.StatusBar.Item( 1 ) := "Rec. " + STR( oFile:CurrentLine() ) + "/" + STR( oFile:nLastLine )
   ENDIF
Return Nil
now click into GRID to "select" any Item and than move Mouse over GRID
you will "see" that Statusbar change ... how is this possible when just "hover" :o
have fun
Jimmy
HGAutomator
Posts: 202
Joined: Thu Jul 16, 2020 5:42 pm
DBs Used: DBF

Re: Virtual Grids can view large text files?

Post by HGAutomator »

edk,

Thanks so much for this, sorry I haven't checked the forum for a while.

I'll test this during the week.
edk
Posts: 999
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Virtual Grids can view large text files?

Post by edk »

Updated version of the SeekLine method (by optimizing the code, the search process accelerated by approx. 50%; bugs related to the record map update were removed):
demo.zip
(9.89 KiB) Downloaded 149 times
BTW. Interesting behavior of method and variable in class objects.

First, I read a variable from one object and write its value to a variable in another object.

Then I update the value in the second object, and to my surprise, the value in the first object also changed to take the value from the second object.

I don't know why this is happening, can someone who knows the classes well explain it to me?

You can see this effect by searching for e.g. 11222333 in incremental browsing by calling menu "File" -> Open file (without counting the number of lines) - incremental reading file.

Code snippet showing this situation:

Code: Select all

//I read aRecordsMap from the oFileGrid object and save it to the oFileMT object.
oFileMT:PutRecordsMap ( oFileGrid:GetRecordsMap() )		// <--- If I do not execute this methods before changing aRecordsMap in the oFileMT object, then in the oFileGrid object there is the original value of aRecordsMap.

//Why does this result in "::aRecordsMap" being synchronized between two different objects later in the code?

//aRecordsMap should only be updated in the oFileMT object.
lFound := oFileMT:SeekLine( AllTrim( Form_1.tb_Seek.Value ), Form_1.Grid_1.Value, .T. , { | nPos, nLastPos | Form_1.PBar.Value := nPos / nLastPos * 100 } )

//Why is aRecordsMap the same in both oFileMT and oFileGrid?
MsgInfo ( "Items in 'aRecordsMap' of 'oFileGrid' object: " + Str( Len ( oFileGrid:GetRecordsMap() ) ) + CRLF + ;
	"Items in 'aRecordsMap' of 'oFileMT' object: " + Str( Len ( oFileMT:GetRecordsMap() ) ) )
edk
Posts: 999
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Virtual Grids can view large text files?

Post by edk »

AUGE_OHR wrote: Sun Dec 05, 2021 5:45 am hi,

i have a funny Effect with Mouse in that "virtual" GRID

i have add this Code into "QueryFile"

Code: Select all

Function QueryFile( oFile, cDelimiter )
   ...
   IF ! EMPTY(oFile:CurrentLine()) .AND. ! EMPTY(oFile:nLastLine)
      Form_1.StatusBar.Item( 1 ) := "Rec. " + STR( oFile:CurrentLine() ) + "/" + STR( oFile:nLastLine )
   ENDIF
Return Nil
now click into GRID to "select" any Item and than move Mouse over GRID
you will "see" that Statusbar change ... how is this possible when just "hover" :o
Virtual Grid calls DataQuery even on mouse over.
Maybe try to change "Rec. " + STR( oFile:CurrentLine() ) to "Rec. " + STR( This.value )
User avatar
AUGE_OHR
Posts: 2117
Joined: Sun Aug 25, 2019 3:12 pm
DBs Used: DBF, PostgreSQL, MySQL, SQLite
Location: Hamburg, Germany

Re: Virtual Grids can view large text files?

Post by AUGE_OHR »

hi Edward,
edk wrote: Mon Dec 06, 2021 12:56 pm Virtual Grid calls DataQuery even on mouse over.
Maybe try to change "Rec. " + STR( oFile:CurrentLine() ) to "Rec. " + STR( This.value )
Ok, understand.

STR( This.value ) work fine, Thx
have fun
Jimmy
Post Reply