Virtual Grids can view large text files?

General Help regarding HMG, Compilation, Linking, Samples

Moderator: Rathinagiri

HGAutomator
Posts: 202
Joined: Thu Jul 16, 2020 5:42 pm
DBs Used: DBF

Re: Virtual Grids can view large text files?

Post by HGAutomator »

Hi edk,

What command line are you using to compile this? I'm getting some unresolved externals, including GETPROCESSMEMORYINFO and GETPARENT, and a couple of others.


edk wrote: Mon Dec 06, 2021 12:35 pm Updated version of the SeekLine method (by optimizing the code, the search process accelerated by approx. 50%; bugs related to the record map update were removed): demo.zip

BTW. Interesting behavior of method and variable in class objects.

First, I read a variable from one object and write its value to a variable in another object.

Then I update the value in the second object, and to my surprise, the value in the first object also changed to take the value from the second object.

I don't know why this is happening, can someone who knows the classes well explain it to me?

You can see this effect by searching for e.g. 11222333 in incremental browsing by calling menu "File" -> Open file (without counting the number of lines) - incremental reading file.

Code snippet showing this situation:

Code: Select all

//I read aRecordsMap from the oFileGrid object and save it to the oFileMT object.
oFileMT:PutRecordsMap ( oFileGrid:GetRecordsMap() )		// <--- If I do not execute this methods before changing aRecordsMap in the oFileMT object, then in the oFileGrid object there is the original value of aRecordsMap.

//Why does this result in "::aRecordsMap" being synchronized between two different objects later in the code?

//aRecordsMap should only be updated in the oFileMT object.
lFound := oFileMT:SeekLine( AllTrim( Form_1.tb_Seek.Value ), Form_1.Grid_1.Value, .T. , { | nPos, nLastPos | Form_1.PBar.Value := nPos / nLastPos * 100 } )

//Why is aRecordsMap the same in both oFileMT and oFileGrid?
MsgInfo ( "Items in 'aRecordsMap' of 'oFileGrid' object: " + Str( Len ( oFileGrid:GetRecordsMap() ) ) + CRLF + ;
	"Items in 'aRecordsMap' of 'oFileMT' object: " + Str( Len ( oFileMT:GetRecordsMap() ) ) )
HGAutomator
Posts: 202
Joined: Thu Jul 16, 2020 5:42 pm
DBs Used: DBF

Re: Virtual Grids can view large text files?

Post by HGAutomator »

Never mind. This is using MingW. I'm using Borland.
User avatar
AUGE_OHR
Posts: 2117
Joined: Sun Aug 25, 2019 3:12 pm
DBs Used: DBF, PostgreSQL, MySQL, SQLite
Location: Hamburg, Germany

Re: Virtual Grids can view large text files?

Post by AUGE_OHR »

hi,
HGAutomator wrote: Tue Dec 07, 2021 7:01 pm Never mind. This is using MingW. I'm using Borland.
there will be a Sample in next MiniGUI Release with "missing" Function.
have fun
Jimmy
HGAutomator
Posts: 202
Joined: Thu Jul 16, 2020 5:42 pm
DBs Used: DBF

Re: Virtual Grids can view large text files?

Post by HGAutomator »

Ok, maybe I should wait then. I cannabilized edk's code to just open a selected file in static mode. Right now, I'm trying to correct a bug where it shows the header line 2 times.

I'll hold off on this. But when the sample is built into the next release, can we make sure that they test a delimited file that already includes headers?
AUGE_OHR wrote: Tue Dec 07, 2021 10:13 pm hi,
HGAutomator wrote: Tue Dec 07, 2021 7:01 pm Never mind. This is using MingW. I'm using Borland.
there will be a Sample in next MiniGUI Release with "missing" Function.
edk
Posts: 999
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Virtual Grids can view large text files?

Post by edk »

HGAutomator wrote: Wed Dec 08, 2021 4:45 pm Ok, maybe I should wait then. I cannabilized edk's code to just open a selected file in static mode. Right now, I'm trying to correct a bug where it shows the header line 2 times.

I'll hold off on this. But when the sample is built into the next release, can we make sure that they test a delimited file that already includes headers?
It's quite simple ;)
demo.zip
(6.94 KiB) Downloaded 103 times
HGAutomator
Posts: 202
Joined: Thu Jul 16, 2020 5:42 pm
DBs Used: DBF

Re: Virtual Grids can view large text files?

Post by HGAutomator »

Ah, man. I didn't realize you posted this, until a few minutes ago. I've been tinkering with the code since last night, trying to get it to interpret a header row correctly, but it kept doing things like repeating the header row in the data grid.

Ok, thanks edk. This is immediately useful.
edk wrote: Wed Dec 08, 2021 7:44 pm
HGAutomator wrote: Wed Dec 08, 2021 4:45 pm Ok, maybe I should wait then. I cannabilized edk's code to just open a selected file in static mode. Right now, I'm trying to correct a bug where it shows the header line 2 times.

I'll hold off on this. But when the sample is built into the next release, can we make sure that they test a delimited file that already includes headers?
It's quite simple ;) demo.zip
HGAutomator
Posts: 202
Joined: Thu Jul 16, 2020 5:42 pm
DBs Used: DBF

Re: Virtual Grids can view large text files?

Post by HGAutomator »

Hi edk,

How would you define a button that does something, like going to a specific line number? oFile is only in the scope of the function that creates the grid, not the main form, so we can't put

Code: Select all

oFile:GoTo( nLineNumber )
in the main form.

If we put something like

Code: Select all

@10, 155 BUTTON b_GoTo CAPTION "Go to Line Number" ACTION GotoRecord( oFile, nLineNumber )
in the OpenFile() function, the app complains about not having a parent.

Alternatively, we could define a hot key like <Ctrl>G, or <Ctrl>F for a general search, but there still seem to be some scope issues.


edk wrote: Wed Dec 08, 2021 7:44 pm
HGAutomator wrote: Wed Dec 08, 2021 4:45 pm Ok, maybe I should wait then. I cannabilized edk's code to just open a selected file in static mode. Right now, I'm trying to correct a bug where it shows the header line 2 times.

I'll hold off on this. But when the sample is built into the next release, can we make sure that they test a delimited file that already includes headers?
It's quite simple ;) demo.zip
franco
Posts: 921
Joined: Sat Nov 02, 2013 5:42 am
DBs Used: DBF
Location: Canada

Re: Virtual Grids can view large text files?

Post by franco »

I found to do actions before activate window I need to create a function for the action to do EG: Action function()
I also use the button has no action but I use on enter creates the action.
You could try

Code: Select all

@10, 155 BUTTON b_GoTo CAPTION "Go to Line Number"   ACTION  Function()
 OR
@10, 155 BUTTON b_GoTo CAPTION "Go to Line Number" 
          On Enter  GotoRecord( oFile, nLineNumber )
All The Best,
Franco
Canada
edk
Posts: 999
Joined: Thu Oct 16, 2014 11:35 am
Location: Poland

Re: Virtual Grids can view large text files?

Post by edk »

HGAutomator wrote: Fri Dec 10, 2021 3:16 pm Hi edk,

How would you define a button that does something, like going to a specific line number? oFile is only in the scope of the function that creates the grid, not the main form, so we can't put

Code: Select all

oFile:GoTo( nLineNumber )
in the main form.

If we put something like

Code: Select all

@10, 155 BUTTON b_GoTo CAPTION "Go to Line Number" ACTION GotoRecord( oFile, nLineNumber )
in the OpenFile() function, the app complains about not having a parent.

Alternatively, we could define a hot key like <Ctrl>G, or <Ctrl>F for a general search, but there still seem to be some scope issues.
It all depends on what your goal is.
If you want the Grid to jump to a specific line, you need to refer to it directly and not a file, because QueryData will take care of that.
However, if you want to read the value of a specific row regardless of Grid, then you refer to the reachable object, in my example it will be the "oGridFile" object.

Also ON KEY is quite simple ;).

See this example:

Code: Select all

* :encoding=windows-1250:	ĄĆĘŁŃÓŚŻŹ ąćęłóńśżź

#pragma -w3
#pragma -es2

#include "hmg.ch"
#Include "Fileio.CH"

MemVar _HMG_SYSDATA

REQUEST HB_LANG_DEWIN
REQUEST HB_CODEPAGE_DEWIN

Function Main
Local cFileToOpen := 'TestBIGwithHeaders.dat'
Local cDelimiter  := CHR ( 9 ), oGridFile

set( _SET_CODEPAGE, "DEWIN" )
hb_LangSelect( 'DEWIN')
	
DEFINE WINDOW Form_1 ;
	AT 0,0 ;
	WIDTH 1100 ;
	HEIGHT 660 ;
	TITLE 'Large text file viewer.' ;
	MAIN ;
	ON INIT Prepare( cFileToOpen, cDelimiter )

	DEFINE MAIN MENU
		DEFINE POPUP 'File'
			MENUITEM 'Open file' ACTION oGridFile := OpenFile ( cFileToOpen, cDelimiter )
		END POPUP
	END MENU
	
	@20, 870 TEXTBOX tb_goto WIDTH 100 VALUE 1 NUMERIC
	@20, 980 BUTTON bGoto CAPTION "Go to line" ACTION Form_1.Grid_1.Value := Form_1.tb_goto.Value
	
	@70, 870 TEXTBOX tb_record WIDTH 100 VALUE 1 NUMERIC
	@70, 980 BUTTON bRecord CAPTION "Read Line" ACTION ( oGridFile:GoTo( Form_1.tb_record.Value + 1 /* "+ 1" first line is the header */ ), MsgInfo ( oGridFile:ReadLine() ) )
	
	@530, 10 TEXTBOX tb_Seek WIDTH 800 VALUE "DB_Analoganzeigen_Pult.Sandmenge_t_tag_Visu"
	@530, 840 BUTTON bSeekMethod CAPTION "Seek Method" ACTION MT_SeekLine( oGridFile )
	
	Form_1.tb_goto.Visible := .F.
	Form_1.bGoto.Visible := .F.
	
	Form_1.tb_record.Visible := .F.
	Form_1.bRecord.Visible := .F.
	
	Form_1.tb_Seek.Visible := .F.
	Form_1.bSeekMethod.Visible := .F.
	
	DEFINE STATUSBAR
		STATUSITEM "" WIDTH Form_1.Width / 2
		STATUSITEM "" WIDTH Form_1.Width / 2 
	END STATUSBAR

	ShowProgressBar( 1, , , (Form_1.Width / 2) + 5 , Form_1.Width / 2 - 30 )		//init progress bar
	ShowProgressBar( 0 )												//hide progress bar
		
END WINDOW

ACTIVATE WINDOW Form_1

Return Nil
***************************************************
Function QueryFile( oFile, cDelimiter )
Local nRecord          := This.QueryRowIndex
Local nCol             := This.QueryColIndex

Static aFields         := {}

IF oFile == Nil
	Return Nil
ENDIF

IF oFile:CurrentLine() <> nRecord + 1 /* "+ 1" first line is the header */ .OR. Len ( aFields ) == 0		//Prevents the current row from being reread when query to display the next column (speeding up)
	oFile:GoTo (  nRecord + 1 /* "+ 1" first line is the header */ )
	aFields   := hb_ATokens ( oFile:ReadLine(), cDelimiter )
ENDIF

This.QueryData := IF ( Len ( aFields ) < nCol , "", aFields [ nCol ] )


Return Nil

**************************************************************
Function OpenFile ( cFileToOpen, cDelimiter )

Local aColumns, aHeaders := {}, aWidths := {}, i, oFile

IF File ( cFileToOpen )

	oFile := vfFileRead():New( cFileToOpen, , , )
	oFile:Open()

	IF oFile:Error()
		MsgStop ("Error " + Str( oFile:ErrorNo() ), "Error" )
		RETURN Nil
	ENDIF
		
	IF IsControlDefined ( Grid_1, Form_1 )
  		Form_1.Grid_1.Release
  	ENDIF
  	
 	//Preparing columns (based on the first row from the file)
	aColumns := hb_ATokens ( oFile:ReadLine(), cDelimiter )
		
	For i := 1 TO Len( aColumns )
		AADD( aHeaders, aColumns [i] )
		AADD( aWidths, Max(20, Len( aColumns [ i ] ) * 19 ) )
	Next i
	
	//Calculate the number of rows before displaying the Grid
	WAIT WINDOW "Counting the number of records ..."  NOWAIT
	ShowProgressBar( 2 )		//show progress bar
	Form_1.StatusBar.Item(1) := "Counting the number of records ... This may take some time .... "
	
	//code block executed while the CountLines() method is running
	oFile:exGauge := { | nPos, nLastPos, nLines, cState | ( Form_1.StatusBar.Item(1) := "Counting the number of records: " + AllTrim ( Str ( nLines ) ) + "... This may take some time ... " + cState, Form_1.PBar.Value := nPos / nLastPos * 100, doEvents() ) }
	oFile:CountLines()
	Form_1.StatusBar.Item(1) := ""
	ShowProgressBar( 0 )		//hide progress bar

	WAIT CLEAR
	
  	@ 10,10 GRID Grid_1 ;
  		OF Form_1 ;
		WIDTH 850 ;
		HEIGHT 500 ;
		VALUE 1 ;
		HEADERS aHeaders ;
		WIDTHS aWidths;
		VIRTUAL ;
		ITEMCOUNT 0 ;
		ON QUERYDATA QueryFile( oFile, cDelimiter )
		
	Form_1.Grid_1.PaintDoubleBuffer := .T.
	
	Form_1.Grid_1.ItemCount := oFile:nLastLine - 1 /* "- 1" first line is the header */
	
	Form_1.tb_goto.Visible := .T.
	Form_1.bGoto.Visible := .T.
	
	Form_1.tb_record.Visible := .T.
	Form_1.bRecord.Visible := .T.
	
	Form_1.tb_Seek.Visible := .T.
	Form_1.bSeekMethod.Visible := .T.
		
	Form_1.Grid_1.Value := 1
	Form_1.Grid_1.Setfocus
	
	ON KEY CONTROL+F OF Form_1 ACTION Eval ( _GetControlAction ( 'bSeekMethod' , 'Form_1' ) )
	ON KEY CONTROL+G OF Form_1 ACTION Eval ( _GetControlAction ( 'bGoto' , 'Form_1' ) )
	ON KEY CONTROL+R OF Form_1 ACTION Eval ( _GetControlAction ( 'bRecord' , 'Form_1' ) )
	
ENDIF

RETURN oFile

********************************************
Function MT_SeekLine( oFile )

Local lFound

IF !IsControlDefined ( Grid_1, Form_1 )
	RETURN Nil
ENDIF
		
Form_1.StatusBar.Item(1) := "Seeking ... This may take some time .... "
ShowProgressBar( 2 )		//show progress bar
	
lFound := oFile:SeekLine( AllTrim( Form_1.tb_Seek.Value ), Form_1.Grid_1.Value + 1 /* "+ 1" first line is the header */ , .T. , { | nPos, nLastPos | Form_1.PBar.Value := nPos / nLastPos * 100 } ) 	
  	
Form_1.StatusBar.Item(1) := ""
ShowProgressBar( 0 )		//hide progress bar
	
IF !lFound
	MsgStop ("Not found " + AllTRim( Form_1.tb_Seek.Value ) )
ELSE
	Form_1.Grid_1.Value := oFile:CurrentLine() - 1 /* "- 1" first line is the header */
ENDIF

Form_1.Grid_1.Setfocus
	
RETURN Nil
*********************************************
Function Prepare ( cFileToOpen, cDelimiter )
Local hTestFile, i, x, nMaxRec := 7000000, aTestRec, cLines := ""

Release Memory

IF !File ( cFileToOpen )
	Msginfo ("A test file will be prepared, this may take some time, please be patient ... ")
	nMaxRec  := Val(InputBox('Number of records','Prepare a sample data file.',AllTrim(Str(nMaxRec))))
	aTestRec := {"DP/DP_Kamat_HD.Soll_Umin_HD_Bediener", "DB_Baggerpumpe_1.Ist_Strom_BP1", "DB_Analoganzeigen_Pult.Sandmenge_m3_h_Visu", ;
			"DP/DP_Kamat_HD.Soll_Umin_HD_Kamat", "DB_Analoganzeigen_Pult.Druck_Schneidrad_REAL_VI", "Füllstand_Druck_Land.Fuell_Sumpf_min_unter", "DB_Analoganzeigen_Pult.Speisedruck_REAL_Visu", ;
			"DB_Analoganzeigen_Pult.Foerdermenge_t_h_Visu" ,"DB_Analoganzeigen_Pult.Druck_vor_BP2_REAL_Visu ", "DB_Analoganzeigen_Pult.Druck_SW_BB_REAL_VISU", ;
			"DB_Analoganzeigen_Pult.Foerderdichte_REAL_Visu", "Füllstand_Druck_Land.Fuell_Sumpf_Land_Differe", "DB_Analoganzeigen_Pult.Druck_nach_BP1_real_Visu" ,;
			"DB_Analoganzeigen_Pult.Druck_Hyd_Pumpe_1_Visu", "Füllstand_Druck_Land.Fuell_Sumpf_zeit1", "DP/DP_Kamat_HD.Hochdruck_bar", "DB_Zähler Winden.Zaehler_STP_Mitte_7", ;
			"DB_Zähler Winden.Zaehler_SW_StB_7", "DB_Baggerpumpe_2.Ist_U_min_BP2", "DB_Analoganzeigen_Pult.Saugdruck_BP1_Real_Visu", "DB_Baggerpumpe_1.Ist_U_min_BP1", ;
			"DB_Analoganzeigen_Pult.Foerdergeschw_REAL_Visu", "DB_Zähler Winden.Zaehler_SW_BB_7", "DB_Analoganzeigen_Pult.Druck_Hyd_Pumpe_2_3_Visu", "DB_Baggerpumpe_2.Ist_Strom_BP2", ;
			"DB_Analoganzeigen_Pult.Seillast_REAL_VISU", "DP/DP_Kamat_HD.Ist_U_min_HD_PUMPE", "DB_Analoganzeigen_Pult.Druck_nach_BP2_real_Visu", "DB_Analoganzeigen_Pult.Tiefe_Leiter_PED550_Visu", ;
			"Füllstand_Druck_Land.Druck_vor_BP2_Land_bar_r", "Füllstand_Druck_Land.Fuell_Vorlage_maximu_aus", "DB_Analoganzeigen_Pult.Druck_SW_STB_REAL_VISU", "DB_Zähler Winden.Zaehler_STP_BB_7", "DP/DP_Kamat_HD.Kamat_HD_PUMPE_Soll_Umin",;
			"DB_Leiterwinde.Baggertiefe_WSP", "Füllstand_Druck_Land.Fuell_Sumpf_minimu_ein", "DB_Analoganzeigen_Pult.Sandmenge_t_tag_Visu", "Füllstand_Druck_Land.Fuell_Sumpf_Land_real", ;
			"DB_Zähler Winden.Zaehler_STP_StB_7", "Füllstand_Druck_Land.Fuell_Sumpf_Land_U_min", "Füllstand_Druck_Land.Druck_nach_BP2_La_bar_r", "DB_Analoganzeigen_Pult.Druck_vor_BP1_REAL_Visu", "Tiefenberechnung.Baggertiefe" }

	WAIT WINDOW "Preparing a test file. This may take some time .... "  NOWAIT 

	ShowProgressBar( 2 )		//show progress bar
	Form_1.StatusBar.Item(1) := "Preparing a test file. This may take some time .... "
	
	hTestFile := hb_FCreate( cFileToOpen, FC_NORMAL, FO_EXCLUSIVE)

	cLines += "Rec." + cDelimiter + "Measure variable" + cDelimiter + "Date/Time" + cDelimiter + "Value" + cDelimiter + "Valid" + cDelimiter + "Stamp" + CRLF
	
	x := 0
	FOR  i := 1 TO nMaxRec
		x++
		IF x > Len( aTestRec )
			x := 1
		ENDIF
		cLines += StrZero (i, 8) + cDelimiter + '"' + aTestRec [ x ] + '"' + cDelimiter + '"' + hb_TSToStr(hb_DateTime()) + '"' + cDelimiter + AllTrim (Str ( hb_Rand32() ) ) + cDelimiter + AllTrim( Str( hb_RandomInt() ) ) + cDelimiter + AllTrim( Str( hb_TtoN(hb_DateTime()) ) ) + CRLF
		IF Len ( cLines ) > 512 * 1025 ^ 2 .OR. i % Int(nMaxRec/100) == 0 
			FWrite( hTestFile, cLines )
			cLines := ""
			WAIT WINDOW "Preparing a test file. This may take some time .... " + Str (i / nMaxRec * 100) + " % " NOWAIT
			Form_1.PBar.Value := i / nMaxRec * 100
			//Release Memory
			EmptyWorkingSet()
			Do Events
		ENDIF
	Next i
	FWrite( hTestFile, cLines )
	FClose( hTestFile )
	
	Form_1.StatusBar.Item(1) := ""
	ShowProgressBar( 0 )		//hide progress bar
	
	WAIT CLEAR
ENDIF

RETURN Nil
***************************************************************************
Function ShowProgressBar( nMode, nMin, nMax, nCol, nLenght )
Default nMode := 1		//1 = init, 2 = set/show, 3 = close, 0/other = hide
Default nMin:=1, nMax:=100
Default nCol:=20, nLenght:=740
DO CASE 
	Case nMode = 1 
		DEFINE PROGRESSBAR PBar
			PARENT Form_1
			ROW    5 
			COL    nCol
			WIDTH  nLenght
			HEIGHT 12
			RANGEMIN nMin
			RANGEMAX nMax
			VALUE nMin
			TOOLTIP ""
			HELPID Nil
			VISIBLE .F.
			SMOOTH .T.
			VERTICAL .F. 
			BACKCOLOR Nil
			FORECOLOR Nil
		END PROGRESSBAR
		
		/* put the progress bar in the status bar */
		SETPARENT(Form_1.PBar.Handle, Form_1.STATUSBAR.Handle)
		
	Case nMode = 3
		Form_1.PBar.Release
		
	Case nMode = 2
		Form_1.PBar.RangeMin :=  nMin
		Form_1.PBar.RangeMax :=  nMax
		Form_1.PBar.Value :=  nMin
		Form_1.PBar.Visible := .T.
		DO EVENTS
	
	Other
		Form_1.PBar.Visible := .F.
		DO EVENTS
ENDCASE
RETURN Nil


*******************************************************************
*   vfFileRead Class                                              *
*******************************************************************

#include "hbclass.ch"
#include "fileio.ch"

#define vf_DEF_READ_SIZE  	4096
#define vf_DELIMITER		hb_eol()
#define vf_REC_MAP_STEP		100000

CREATE CLASS vfFileRead

   VAR cFile                   
   VAR pHandle                 
   VAR nError                  
   VAR cDelim                  
   VAR nReadSize               
   VAR lEOF
   VAR lBOF
   VAR nCurrLine
   VAR nLastLine
   VAR exGauge
   VAR aRecordsMap INIT {} 		//PROTECTED
   VAR nRecordsMapStep PROTECTED
   
   METHOD New( cFile, nSize, cDelimiter, nRecMapStep )		// Create a new object (file name, buffer size , delimiter, map record indexed step)
   METHOD Open( nMode )								// Open the file ( nMode )
   METHOD Close()									// Close the file
   METHOD ReadLine()								// Read line of current record (when opened, current record is first)
   METHOD GoTop()									// Go to first record
   METHOD GoBottom()								// Go to last record
   METHOD GoTo( nLine )								// Go to <nLine> record
   METHOD Skip( nSkip )								// Skip by <nSkip> records
   METHOD IsEOF()									// Returns .T. if the current record is the last
   METHOD IsBOF()									// Returns .T. if the current record is the first
   METHOD Error()									// Returns .T. an error occurred
   METHOD ErrorNo()									// Returns the error number
   METHOD GetPointer()								// Reads the current position of the file pointer
   METHOD SetPointer( nPointer )						// Sets the file pointer to the indicated position
   METHOD GetLastLinePointer()						// Reads the file pointer position for the last record
   METHOD CurrentLine()								// Reads the current record
   METHOD CountLines( xGauge )						// Calculates the number of lines (records), as a parameter you can pass a block of code executed during the counting operation
   METHOD SeekLine( cSeekString, nFromRecord, lCaseSensitive, xGauge )	//Seek <cSeekString> in line, begin from <nFromRecord> and if <lCaseSensitive> (default is true)		//Rerturn true or false, set ::CurrentLine() to matched line
   METHOD GetRecordsMap()							// Gets a map of records
   METHOD PutRecordsMap( aMap )						// Puts a map of records
     
END CLASS

*******************************************************************

METHOD New( cFile, nSize, cDelimiter, nRecMapStep ) CLASS vfFileRead

Default nRecMapStep := vf_REC_MAP_STEP

nRecMapStep := Max ( nRecMapStep, 10000 )

IF nSize == NIL .OR. nSize < 1
	nSize := vf_DEF_READ_SIZE
ENDIF

::cFile     := cFile
::pHandle   := NIL
::nError    := 0
::nReadSize := nSize
::cDelim    := cDelimiter
::lEOF	  := .F.
::lBOF	  := .F.
::nCurrLine := 0
::nLastLine := NIL
::aRecordsMap:={}
::nRecordsMapStep := nRecMapStep
   
RETURN Self

*******************************************************************

METHOD Open( nMode ) CLASS vfFileRead

Local cLine
IF ::pHandle == NIL

	IF nMode == NIL
		nMode := FO_READ + FO_SHARED   // Default to shared read-only mode
	ENDIF
	::pHandle := hb_vfOpen( ::cFile, nMode )
	::aRecordsMap:={}
	IF ::pHandle == NIL
		::nError    := FError()       
		::lBOF      := .F.
		::nCurrLine := 0
	ELSE
		::nError    := 0
		::lBOF      := .T.
		::nCurrLine := 1
		
		//delimiter detection
		IF ::cDelim == NIL
			::cDelim := vf_DELIMITER
			cLine := hb_vfReadLen( ::pHandle, 64*1024 /* ::nReadSize */ )
			::GoTop()
			DO CASE
				CASE hb_AT( CHR (13) + CRLF, cLine ) > 0		//CRCRLF
					::cDelim := CHR (13) + CRLF
				CASE hb_AT( CRLF, cLine ) > 0					//CRLF
					::cDelim := CRLF
				CASE hb_AT( CHR (13), cLine ) > 0				//CR
					::cDelim := CHR (13)
				CASE hb_AT( CHR (10), cLine ) > 0				//LF
					::cDelim := CHR (10)
			ENDCASE
		ENDIF                      

	ENDIF
ELSE
      // The file is already open, so rewind to the beginning.
	IF !::GoTop()
      	::lBOF      := .F.
      	::nCurrLine := 0
      	::nError    := FError()
	ELSE
		::lBOF      := .T.
		::nCurrLine := 1
	ENDIF
ENDIF
   
RETURN Self

*******************************************************************

METHOD Close() CLASS vfFileRead

IF .NOT. ::pHandle == NIL
	hb_vfClose( ::pHandle )
	::pHandle := NIL           // The file is no longer open
ENDIF
::nError    := FError()
::lEOF      := .F.
::lBOF      := .F.
::nCurrLine := 0
::nLastLine := NIL

RETURN Self

*******************************************************************

METHOD ReadLine() CLASS vfFileRead

Local cLine       := ""
Local nCurrentPos := ::GetPointer()

DO WHILE !hb_vfEof( ::pHandle )
	cLine += hb_vfReadLen( ::pHandle, ::nReadSize )
	IF hb_AT( ::cDelim, cLine ) > 0
		cLine := Left ( cLine, hb_AT( ::cDelim, cLine ) - 1 )
		EXIT
	ENDIF
ENDDO

::SetPointer ( nCurrentPos )
::nError := FError()

RETURN cLine

*******************************************************************

METHOD GoTop() CLASS vfFileRead

::lEOF      := .F.
::lBOT      := .T.
::nCurrLine := 1

RETURN hb_vfSeek( ::pHandle, 0, 0 ) == 0

*******************************************************************

METHOD GoBottom() CLASS vfFileRead

hb_vfSeek( ::pHandle, 0, 2 )
::Skip( -1 )
::lEOF      := .T.
::lBOT      := .F.
::nCurrLine := ::nLastLine
::nError    := FError()
      
RETURN FError() == 0   

*******************************************************************

METHOD GoTo( nLine ) CLASS vfFileRead

Local nToSkip ,nIntStep, nPos

DEFAULT nLine := ::nCurrLine

IF !hb_isNumeric (nLine)
	RETURN Self
ENDIF

IF ABS ( nLine - ::nCurrLine ) >= ::nRecordsMapStep		//only if we skip by indexed steps
	nIntStep := ( INT( nLine / ::nRecordsMapStep ) + IF ( nLine % ::nRecordsMapStep < (::nRecordsMapStep / 2), 0, 1 ) ) * ::nRecordsMapStep
	DO WHILE nIntStep >= 0
		nPos := hb_AScan( ::aRecordsMap, { |x|  Bin2ULL ( ULLUnShrink ( x [ 1 ] ) ) == nIntStep } )
		IF nPos > 0
			::nCurrLine := Bin2ULL ( ULLUnShrink ( ::aRecordsMap [ nPos ] [ 1 ] ) )
			::SetPointer( Bin2ULL ( ULLUnShrink ( ::aRecordsMap [ nPos ] [ 2 ] ) ) )
			EXIT
		ENDIF
		//Try with previous indexed step
		nIntStep -= ::nRecordsMapStep
	ENDDO
ENDIF

nToSkip := nLine - ::nCurrLine

DO CASE
	CASE hb_isNil( ::nCurrLine )			//We do not know the current line number, so start at the beginning of the file
		::GoTop()
		::Skip ( nLine - 1 )
	CASE nLine <= 1					//Line number less than 1, so start at the beginning of the file
		::GoTop()
	CASE hb_isNumeric( ::nLastLine ) .AND. nLine >= ::nLastLine		//Go beyond the known last line number, so go to the end.
		::GoBottom()
	CASE hb_isNumeric( ::nLastLine ) .AND. ::nLastLine - nLine < ABS ( nToSkip ) 	//We know the number of the last record, the nToSkip is greater than the difference between the last and the target record - so skip backward from the end (less skips)
		::GoBottom()
		::Skip( 0 - (::nLastLine - nLine ) )
	CASE nLine < 0 - nToSkip				//The target line is closer to the start than the nToSkip, so start at the beginning of the file.
		::GoTop()
		::Skip ( nLine - 1 )
	OTHER							
		::Skip ( nToSkip )             
ENDCASE
::nError := FError()

RETURN Self

*******************************************************************

METHOD Skip ( nSkip ) CLASS vfFileRead

Local cLine     := ""
Local lRun      := .T.
Local nSkips    := 0
Local nBuffSize := 64 * 1024	//::nReadSize  //A larger buffer speeds up the process
Local nLenDelim := Len (::cDelim)
Local nStart, nEnd, nBackOf, nPos, nPointer
Default nSkip   := 1

IF nSkip == 0
	RETURN Self
ENDIF

::lEOF := .F.
::lBOF := .F.

DO WHILE lRun
                                                            
	IF nSkip > 0
		IF hb_vfEof( ::pHandle )
			::Skip ( -1 )
			::nLastLine := ::nCurrLine
			::lEOF      := .T.
			EXIT
		ENDIF
		nStart   := 1
		nPointer := ::GetPointer()
		cLine    := hb_vfReadLen( ::pHandle, nBuffSize )
		DO WHILE Right (cLine, 1) $ ::cDelim .AND. !hb_vfEof( ::pHandle )		//Avoid splitting of end-of-line marks in the buffer.
			cLine += hb_vfReadLen( ::pHandle, nLenDelim )
		ENDDO
		DO WHILE ( nPos := hb_AT( ::cDelim, cLine, nStart ) ) > 0
			nSkips ++
			IF hb_isNumeric( ::nCurrLine )
				::nCurrLine ++
				IF ::nCurrLine % ::nRecordsMapStep == 0 .AND. hb_AScan( ::aRecordsMap, { |x|  Bin2ULL ( ULLUnShrink ( x [ 1 ] ) ) == ::nCurrLine } ) == 0
					AADD( ::aRecordsMap, { ULLShrink( ULL2Bin( ::nCurrLine ) ), ULLShrink( ULL2Bin( nPointer + nPos  + nLenDelim - 1 ) ) } )
				ENDIF
				
			ENDIF
			nStart := nPos + nLenDelim
			IF nSkips == nSkip
				hb_vfSeek( ::pHandle, 0 - ( Len ( cLine ) - nStart + 1), 1 )
				lRun := .F.
				IF hb_vfEof( ::pHandle )
					::Skip ( -1 )
					::nLastLine := ::nCurrLine
					::lEOF      := .T.
				ENDIF
				EXIT
			ENDIF
		ENDDO
	ELSE			// -skip
	
		IF ::GetPointer() == 0		//top of file
			::lBOF      := .T.
			::nCurrLine := 1
			EXIT
		ENDIF
		cLine := ""
		
		DO WHILE ( nPos := hb_AT( ::cDelim, cLine ) ) == 0 .AND. hb_vfSeek( ::pHandle, 0, 1 ) <> 0		//Avoid splitting of end-of-line marks in the buffer.
			nBackOf  := Min( nBuffSize, ::GetPointer() )
			hb_vfSeek( ::pHandle, 0 - nBackOf, 1 )
			cLine    := hb_vfReadLen( ::pHandle, nBackOf ) + cLine
			hb_vfSeek( ::pHandle, 0 - nBackOf, 1 )
		ENDDO
		IF nPos == 0
			nPos := 1
		ENDIF
		
		nStart := nPos
		
		cLine  := SubStr ( cLine, nStart )
		hb_vfSeek( ::pHandle, nStart - 1, 1 )
		
		nPointer := ::GetPointer()
		nEnd     := Len( cLine )
		
		IF Right( cLine, nLenDelim) == ::cDelim
			nEnd -= nLenDelim
		ENDIF
	
		DO WHILE (nPos := hb_RAT( ::cDelim, cLine, 1, nEnd ) ) > 0
			nSkips --
			IF hb_isNumeric( ::nCurrLine )
				::nCurrLine --
				IF ::nCurrLine % ::nRecordsMapStep == 0 .AND. hb_AScan( ::aRecordsMap, { |x|  Bin2ULL ( ULLUnShrink ( x [ 1 ] ) ) == ::nCurrLine } ) == 0
					AADD( ::aRecordsMap, { ULLShrink( ULL2Bin( ::nCurrLine ) ), ULLShrink( ULL2Bin( nPointer + nPos + nLenDelim -1 ) ) } )
				ENDIF
			ENDIF
			
			nEnd := nPos - 1		//nLenDelim

			IF nSkips == nSkip
				hb_vfSeek( ::pHandle, 0 + nPos + nLenDelim - 1 , 1 )
				lRun := .F.
				EXIT
			ENDIF
		ENDDO
	ENDIF
ENDDO

cLine := ''
::nError := FError()

RETURN Self

*******************************************************************

METHOD IsEOF() CLASS vfFileRead

IF !::lEOF .AND. hb_IsNumeric ( ::nCurrLine ) .AND. hb_IsNumeric ( ::nLastLine ) .AND. ::nCurrLine == ::nLastLine
	::lEOF := .T.
ENDIF

IF !::lEOF .AND. ::GetPointer() == ::GetLastLinePointer()
	::lEOF := .T.
ENDIF

RETURN hb_vfEof( ::pHandle ) .OR. ::lEOF

*******************************************************************

METHOD IsBOF() CLASS vfFileRead

RETURN ::GetPointer() == 0 .OR. ::lBOF

*******************************************************************

METHOD Error() CLASS vfFileRead

RETURN FError() != 0

*******************************************************************

METHOD ErrorNo() CLASS vfFileRead

RETURN ::nError

*******************************************************************

METHOD GetPointer() CLASS vfFileRead
	
RETURN hb_vfSeek( ::pHandle, 0, 1 )

*******************************************************************

METHOD SetPointer( nPointer ) CLASS vfFileRead

hb_vfSeek( ::pHandle, nPointer, 0 )
::nError := FError()

RETURN ::nError == 0

*******************************************************************

METHOD GetLastLinePointer() CLASS vfFileRead

Local nCurrentPointer := ::GetPointer()
Local nCurrentLine    := ::nCurrLine
Local lCurrentEOF     := ::lEOF
Local nLastLinePointer
::GoBottom()
nLastLinePointer := ::GetPointer()
::SetPointer ( nCurrentPointer )	//roll back pointer
::nCurrLine := nCurrentLine			//roll back ::nCurrLine
::lEOF := lCurrentEOF				//roll back ::lEOF
::nError := FError()

RETURN nLastLinePointer

*******************************************************************

METHOD CurrentLine() CLASS vfFileRead

RETURN ::nCurrLine

*******************************************************************

METHOD CountLines( xGauge ) CLASS vfFileRead

Local cLine       := ""
Local nCurrentPos := ::GetPointer()
Local nLastPos    := hb_vfSeek( ::pHandle, 0, 2 )
Local nSkips      := 0
Local nlDelim     := Len ( ::cDelim )
Local nPrevPos    := 0
Local nBuffSize   := 64 * 1024
Local exCurrentGauge := ::exGauge
Local nPointer    := 0
Local nStart, nPos

IF HB_ISEVALITEM( xGauge )
	::exGauge := xGauge
ENDIF

::aRecordsMap := {}
::GoTop()

DO WHILE !hb_vfEof( ::pHandle )
	nStart   := 1
	nPointer := ::GetPointer()
	cLine    := hb_vfReadLen( ::pHandle, nBuffSize )
	DO WHILE Right (cLine, 1) $ ::cDelim .AND. !hb_vfEof( ::pHandle )	//Avoid splitting of end-of-line marks in the buffer.
		cLine += hb_vfReadLen( ::pHandle, nlDelim )
	ENDDO
	DO WHILE (nPos := hb_AT( ::cDelim, cLine, nStart )) > 0
		nSkips ++
		IF nSkips % ::nRecordsMapStep == 0
			AADD( ::aRecordsMap, { ULLShrink( ULL2Bin( nSkips ) ), ULLShrink( ULL2Bin( nPointer + nPrevPos - 1 /* pointer starts from 0 not from 1 like AT( ) */ ) ) } )
			
			IF HB_ISEVALITEM( ::exGauge )
				Eval( ::exGauge, nPointer + nPrevPos - 1, nLastPos, nSkips , 'Processing', Self )
			ENDIF
		
		ENDIF
			
		nStart   := nPos + nlDelim
		nPrevPos := nStart

	ENDDO
ENDDO

IF .NOT. Right( cLine, nlDelim ) == ::cDelim .AND. nLastPos > 0		//The last line does not end with a delimiter. Make it the last record.
	nSkips ++
ENDIF

IF HB_ISEVALITEM( ::exGauge )
	Eval( ::exGauge, nPointer + nPrevPos - 1, nLastPos, nSkips , 'Done', Self )
ENDIF

::nLastLine := nSkips
::SetPointer ( nCurrentPos )
::nError    := FError()
::exGauge   := exCurrentGauge

RETURN nSkips
*******************************************************************
METHOD SeekLine( cSeekString, nFromRecord, lCaseSensitive, xGauge ) CLASS vfFileRead

Local cLine
Local nCurrentPos := ::GetPointer()
Local nLastPos    := hb_vfSeek( ::pHandle, 0, 2 )
Local nSkips      := 0
Local nlDelim     := Len ( ::cDelim )
Local nPrevPos    := 0
Local nBuffSize   := 64 * 1024
Local exCurrentGauge := ::exGauge
Local nFoundLine  := 0
Local nStart, nPos, nPointer

Default cSeekString := "", nFromRecord := 1, lCaseSensitive := .T.

::SetPointer ( nCurrentPos )

IF Empty ( cSeekString ) .OR. cSeekString == ::cDelim
	RETURN nFoundLine <> 0
ENDIF

IF !lCaseSensitive
	cSeekString := HMG_Upper ( cSeekString )
ENDIF

IF HB_ISEVALITEM( xGauge )
	::exGauge := xGauge
ENDIF

::GoTo( nFromRecord )
DO WHILE !hb_vfEof( ::pHandle ) .AND. nFoundLine == 0
	nStart   := 1
	nPointer := ::GetPointer()
	cLine    := hb_vfReadLen( ::pHandle, nBuffSize )
	
	//Avoid splitting of whole line in the buffer.
	nPos := hb_RAt ( ::cDelim, cLine )
	IF nPos > 0
		cLine := Left ( cLine, nPos + nlDelim - 1)
		::SetPointer ( nPointer + Len ( cLine ) )
	ENDIF
	
	IF !lCaseSensitive
		cLine := HMG_Upper ( cLine )
	ENDIF
	
	DO WHILE (nPos := hb_AT( ::cDelim, cLine, nStart )) > 0
		
		IF hb_AT( cSeekString, cLine, nStart, nPos ) > 0
			nFoundLine := ::nCurrLine + nSkips
			EXIT
		ENDIF
		
		IF (::nCurrLine + nSkips) % ::nRecordsMapStep == 0
		
			IF hb_AScan( ::aRecordsMap, { |x|  Bin2ULL ( ULLUnShrink ( x [ 1 ] ) ) == (::nCurrLine + nSkips) } ) == 0
				AADD( ::aRecordsMap, { ULLShrink( ULL2Bin( (::nCurrLine + nSkips) ) ), ULLShrink( ULL2Bin( nPointer + nPrevPos - 1 /* pointer starts from 0 not from 1 like AT( ) */ ) ) } )
			ENDIF
			
			IF HB_ISEVALITEM( ::exGauge )
				Eval( ::exGauge, nPointer + nPrevPos - 1, nLastPos, ::nCurrLine + nSkips , 'Processing', Self )
			ENDIF
		
		ENDIF
			
		nStart   := nPos + nlDelim
		nPrevPos := nStart
		
		nSkips ++

	ENDDO
	
ENDDO

IF HB_ISEVALITEM( ::exGauge )
	Eval( ::exGauge, nPointer + nPrevPos - 1, nLastPos, ::nCurrLine + nSkips , 'Done', Self )
ENDIF

::SetPointer ( nCurrentPos )
IF nFoundLine <> 0
	::GoTo ( nFoundLine )
ENDIF

::nError    := FError()
::exGauge   := exCurrentGauge

RETURN nFoundLine <> 0

*******************************************************************

METHOD GetRecordsMap() CLASS vfFileRead

RETURN ::aRecordsMap

*******************************************************************

METHOD PutRecordsMap( aMap ) CLASS vfFileRead

IF aMap == NIL .OR. !hb_isArray( aMap )
	RETURN Self
ENDIF
::aRecordsMap := aMap

RETURN Self 

*******************************************************************

Function ULLShrink( cBin )
RETURN REMRIGHT( cBin, CHR(0) )

*******************************************************************

Function ULLUnShrink( cBin )
RETURN PADRIGHT( cBin, 8, CHR(0) )

*******************************************************************  

//Support Unsigned Long Long
#pragma BEGINDUMP   
 

#include "hbapi.h"
#include "hbapiitm.h"

HB_FUNC( BIN2ULL )
{
   PHB_ITEM pItem    = hb_param( 1, HB_IT_STRING );
   HB_U64   uiResult = 0;

   if( pItem )
   {
      HB_SIZE nLen = hb_itemGetCLen( pItem );
      if( nLen )
      {
         const char * pszString = hb_itemGetCPtr( pItem );
         uiResult = HB_GET_LE_UINT64( pszString );
          }
   }
   hb_retnint( uiResult );
}

HB_FUNC( ULL2BIN )
{
   char   szResult[ 8 ];
   HB_U64 uiValue = ( HB_U64 ) hb_parnint( 1 );

   HB_PUT_LE_UINT64( szResult, uiValue );
   hb_retclen( szResult, 8 );
}

#pragma ENDDUMP

************************ THE END *********************
HGAutomator
Posts: 202
Joined: Thu Jul 16, 2020 5:42 pm
DBs Used: DBF

Re: Virtual Grids can view large text files?

Post by HGAutomator »

franco,

Thanks, but your first example

Code: Select all

@10, 155 BUTTON b_GoTo CAPTION "Go to Line Number"   ACTION  Function()
is pretty much the same as what I supplied above, except for replacing

Code: Select all

GotoRecord( oFile, nLineNumber )
with

Code: Select all

Function()
It returns a runtime error that Window:Parent is not defined, if it's placed Before or After the GRID definition. If it's placed where the Main Form or Main Menu are defined, then there's a runtime error (understandably) that variable oFile doesn't exist.

You second example, the one with AS ENTER, doesn't compile at all. It returns a standard "syntax error at '@'".

The scope problem, as I understand it, is that oFile is in the scope of QueryFile() and OpenFile(). But the buttons don't share that scope.
Post Reply