HMGforum.com

Posted: **Wed Nov 17, 2021 5:32 pm**

Hi,

I read in another post that Virtual Grids can look at large text files, if we apply the HB_F() functions for navigation.

Is that correct? Would it be possible to load delimited large tab files into a grid?

I have a license for Delimit http://delimitware.com/ , but can't distribute it freely. If it's possible to view and scroll a file with 3 gigabytes and about 17 million lines, with the virtual grid?

I know how to do the parsing, I wrote a utility that I'll share on the forum soon. But I just need to confirm it's possible to view large text files in a harbour environment.

Posted: **Thu Nov 18, 2021 8:16 am**

hi,

you can use VIRTUAL and GRID with every TEXT File ... when hole ROW is "one Cell"

Problem are "Delimiter" of CSV which you have to split into single Element for Column.
IMHO to "display" Data is always "slow" ... even a Progressbar (if used more that 100 x )

---

i do use MEMOREAD(), MLCOUNT() and MEMOLINE() and this Function

Code: Select all

FUNCTION AtInside( cText , cMarker)
LOCAL nPos := 1
LOCAL aRet := {}
DEFAULT cMarker TO ","

   DO WHILE .T.
      nPos := AT( cMarker, cText )
      IF nPos > 0
         AADD( aRet, LTRIM( SUBSTR( cText, 1, ( nPos - 1 ) ) ) )
         cText := SUBSTR( cText, ( nPos + 1 ), LEN( cText ) - nPos )
      ELSE
         AADD( aRet, LTRIM( cText ) )
         EXIT
      ENDIF
   ENDDO

RETURN aRet

it will to split Line, "delimited" with cMarker, and add Element to Array

when LEN(Array) is OK i can FIELDPUT() Element to DBF

Posted: **Thu Nov 18, 2021 3:26 pm**

Thanks Auge,

I don't think I'd use Memoread(). Wouldn't that have problems loading huge files?

I'd prefer to use the HB functions in Readonly mode. The import utility I wrote is based on code that originally imported the entire text file into a single dbf table. That has the 2 billion record RDD limitation, so I use the HB functions to bypass that.

Posted: **Thu Nov 18, 2021 9:49 pm**

hi,

HGAutomator wrote: ↑Thu Nov 18, 2021 3:26 pm I don't think I'd use Memoread(). Wouldn't that have problems loading huge files?

i had Problem with 32 Bit e.g. under Xbase++ but with 64 Bit no Problem.

Code: Select all

HB_MEMOREAD() // not limited to 64 KB as MEMOREAD()

if i "read" with File with F-Function it work on every "Cell"
if i extract a "Line" i can call a Thread for every "Line" to extract Element and add to DBF

as HMG 64 Bit Version can use Unicode i´#m not limited any more

Posted: **Fri Nov 19, 2021 12:56 am**

Ok, I'll play with it and see what happens, thanks A.O.

Posted: **Mon Nov 29, 2021 2:32 pm**

HGAutomator wrote: ↑Wed Nov 17, 2021 5:32 pm Hi,

I read in another post that Virtual Grids can look at large text files, if we apply the HB_F() functions for navigation.

Is that correct? Would it be possible to load delimited large tab files into a grid?

I have a license for Delimit http://delimitware.com/ , but can't distribute it freely. If it's possible to view and scroll a file with 3 gigabytes and about 17 million lines, with the virtual grid?

I know how to do the parsing, I wrote a utility that I'll share on the forum soon. But I just need to confirm it's possible to view large text files in a harbour environment.

Hi.

You intrigued me with your question.

At first I thought it wouldn't be a complicated task, I wanted to use the hb_F* function to process text files, but found out that they don't quite work properly on large files.
I did a simple test which consisted of going to the desired records, going back and calculating the number of records.
Depending on the order of the actions I got different results.
This can be seen by selecting the "Test hb_F* functions" option from the "Test" menu.

So I decided to build my own class. There were performance issues, especially with large files (eg, 70,000,000 records and a size of more than 7 GB) and the jump between hundreds of thousands of records (this is the case when using the vertical slider).
So I thought it might be worth indexing the file you were browsing, but the index file was again taking up a lot of disk space.
I came up with the idea to do something like a record map in my class, which will not be a file but a matrix.
In order to reduce the size of the matrix again, I used 64-bit words to write the values of record numer and file pointer (don't worry it works in the 32-bit also), which I was shrinking again.

The record map is created on the fly when navigating the file using the GoTo and Skip methods, and is completely created when the CountLines method is called.

The class is probably not without errors, but it may be helpful for you.

I used the Virtual Grid control to browse the files. To be able to read the entire file into the Grid, you need to know the number of records. The counting process may take some time with large files.
So, I did three browsing variants:
First: only the beginning of the file is loaded. The moment cell navigation is moved near the end of the list, the next part of the file is loaded - incremental loading.
Second: records are counted before browsing.
Third: counting the number of records is performed in a separate thread and is constantly updated while browsing the Grid.

While working on this task, it turned out that the Grid can be very memory-hungry, especially when we read any property in the QueryData code, be it the Grid control itself, the form window, etc.

To visualize it, just select "Virtual Grid (test memory leaking)" in the "File" menu.
Just try to move the mouse pointer over the Grid and you will see how quickly the available memory is "thinning". Uncheck all three boxes to disable properies polling in QueryData and memory will no longer be consumed at that rate.

demo.7z: (1.19 MiB) Downloaded 136 times

Posted: **Wed Dec 01, 2021 7:12 am**

hi,

i have "play" with CLASS vfFileRead and found out that it can "read" file > 4 GB under 32 Bit App / OS

how is this possible with harbour / HMG

(HB_PUT_LE_UINT64() / HB_GET_LE_UINT64()

p.s. have change hb_parnl(1) to HMG_parnl(1) in Listview Function and use

Code: Select all

#define COMPILE_HMG_UNICODE
#include "HMG_UNICODE.h"

Posted: **Wed Dec 01, 2021 10:14 am**

hi,

where does these File I/O Function

Code: Select all

hb_vfOpen()
hb_vfClose()
hb_vfEof()
hb_vfSize()
hb_vfRead()
hb_vfReadLen()
hb_vfSeek()
hb_vfWrite()
hb_vfErase()
hb_vfAttrGet()
hb_vfConfig()
hb_vfDirectory()
hb_vfExists()
hb_vfRename()
hb_vfOpenProcess()

come from

Posted: **Wed Dec 01, 2021 10:32 am**

AUGE_OHR wrote: ↑Wed Dec 01, 2021 10:14 am hi,

where does these File I/O Function
come from

Hi Jimmy,

Try the following command

hbmk2.exe -find hb_vf* > out.txt

and will get the result:

Harbour core (installed):
hb_vfAttrGet()
hb_vfAttrSet()
hb_vfClose()
hb_vfCommit()
hb_vfConfig()
hb_vfCopyFile()
hb_vfDirectory()
hb_vfDirBuild()
hb_vfDirExists()
hb_vfDirMake()
hb_vfDirRemove()
hb_vfDirSpace()
hb_vfDirUnbuild()
hb_vfEof()
hb_vfErase()
hb_vfExists()
hb_vfFlush()
hb_vfHandle()
hb_vfLink()
hb_vfLinkRead()
hb_vfLinkSym()
hb_vfLoad()
hb_vfLock()
hb_vfLockTest()
hb_vfMoveFile()
hb_vfNameExists()
hb_vfOpen()
hb_vfRead()
hb_vfReadAt()
hb_vfReadLen()
hb_vfRename()
hb_vfSeek()
hb_vfSize()
hb_vfTempFile()
hb_vfTimeGet()
hb_vfTimeSet()
hb_vfTrunc()
hb_vfUnlock()
hb_vfWrite()
hb_vfWriteAt()
hbpipeio.hbc (installed):
hb_vfFromPipes()
hb_vfOpenProcess()
hbtcpio.hbc (installed):
hb_vfFromSocket()

Posted: **Wed Dec 01, 2021 10:41 am**

AUGE_OHR wrote: ↑Wed Dec 01, 2021 7:12 am hi,

i have "play" with CLASS vfFileRead and found out that it can "read" file > 4 GB under 32 Bit App / OS

how is this possible with harbour / HMG
(HB_PUT_LE_UINT64() / HB_GET_LE_UINT64()

They come from HARBOUR\include\hbdefs.h :

Code: Select all

#define HB_GET_LE_UINT64( p )    ( ( UINT64 ) \
                                      ( ( ( UINT64 ) (( BYTE * )( p ))[0] ) | \
                                        ( ( UINT64 ) (( BYTE * )( p ))[1] <<  8 ) | \
                                        ( ( UINT64 ) (( BYTE * )( p ))[2] << 16 ) | \
                                        ( ( UINT64 ) (( BYTE * )( p ))[3] << 24 ) | \
                                        ( ( UINT64 ) (( BYTE * )( p ))[4] << 32 ) | \
                                        ( ( UINT64 ) (( BYTE * )( p ))[5] << 40 ) | \
                                        ( ( UINT64 ) (( BYTE * )( p ))[6] << 48 ) | \
                                        ( ( UINT64 ) (( BYTE * )( p ))[7] << 56 ) ) )

   #define HB_PUT_LE_UINT64( p, w )    do { \
                                         (( BYTE * )( p ))[0] = ( BYTE )( w ); \
                                         (( BYTE * )( p ))[1] = ( BYTE )( (w) >>  8 ); \
                                         (( BYTE * )( p ))[2] = ( BYTE )( (w) >> 16 ); \
                                         (( BYTE * )( p ))[3] = ( BYTE )( (w) >> 24 ); \
                                         (( BYTE * )( p ))[4] = ( BYTE )( (w) >> 32 ); \
                                         (( BYTE * )( p ))[5] = ( BYTE )( (w) >> 40 ); \
                                         (( BYTE * )( p ))[6] = ( BYTE )( (w) >> 48 ); \
                                         (( BYTE * )( p ))[7] = ( BYTE )( (w) >> 56 ); \
                                       } while ( 0 )

AUGE_OHR wrote: ↑Wed Dec 01, 2021 7:12 am p.s. have change hb_parnl(1) to HMG_parnl(1) in Listview Function and use
Code: Select all
#define COMPILE_HMG_UNICODE
#include "HMG_UNICODE.h"

You can remove these Listview function completely, I left them by mistake because I was checking the cause of the memory leak.
Below is the current version of the class, with improved handling of zero byte files.

Code: Select all

*******************************************************************
*   vfFileRead Class                                              *
*******************************************************************

#include "hbclass.ch"
#include "fileio.ch"

#define vf_DEF_READ_SIZE  	4096
#define vf_DELIMITER		hb_eol()
#define vf_REC_MAP_STEP		100000

CREATE CLASS vfFileRead

   VAR cFile                   
   VAR pHandle                 
   VAR nError                  
   VAR cDelim                  
   VAR nReadSize               
   VAR lEOF
   VAR lBOF
   VAR nCurrLine
   VAR nLastLine
   VAR exGauge
   VAR aRecordsMap INIT {} PROTECTED
   VAR nRecordsMapStep PROTECTED
   
   METHOD New( cFile, nSize, cDelimiter, nRecMapStep )		// Create a new object (file name, buffer size , delimiter, map record indexed step)
   METHOD Open( nMode )								// Open the file ( nMode )
   METHOD Close()									// Close the file
   METHOD ReadLine()								// Read line of current record (when opened, current record is first)
   METHOD GoTop()									// Go to first record
   METHOD GoBottom()								// Go to last record
   METHOD GoTo( nLine )								// Go to <nLine> record
   METHOD Skip( nSkip )								// Skip by <nSkip> records
   METHOD IsEOF()									// Returns .T. if the current record is the last
   METHOD IsBOF()									// Returns .T. if the current record is the first
   METHOD Error()									// Returns .T. an error occurred
   METHOD ErrorNo()									// Returns the error number
   METHOD GetPointer()								// Reads the current position of the file pointer
   METHOD SetPointer( nPointer )						// Sets the file pointer to the indicated position
   METHOD GetLastLinePointer()						// Reads the file pointer position for the last record
   METHOD CurrentLine()								// Reads the current record
   METHOD CountLines( xGauge )						// Calculates the number of lines (records), as a parameter you can pass a block of code executed during the counting operation
   METHOD GetRecordsMap()							// Gets a map of records
   METHOD PutRecordsMap( aMap )						// Puts a map of records
     
END CLASS

*******************************************************************

METHOD New( cFile, nSize, cDelimiter, nRecMapStep ) CLASS vfFileRead

Default nRecMapStep := vf_REC_MAP_STEP

nRecMapStep := Max ( nRecMapStep, 10000 )

IF nSize == NIL .OR. nSize < 1
	nSize := vf_DEF_READ_SIZE
ENDIF

::cFile     := cFile
::pHandle   := NIL
::nError    := 0
::nReadSize := nSize
::cDelim    := cDelimiter
::lEOF	  := .F.
::lBOF	  := .F.
::nCurrLine := 0
::nLastLine := NIL
::aRecordsMap:={}
::nRecordsMapStep := nRecMapStep
   
RETURN Self

*******************************************************************

METHOD Open( nMode ) CLASS vfFileRead

Local cLine
IF ::pHandle == NIL

	IF nMode == NIL
		nMode := FO_READ + FO_SHARED   // Default to shared read-only mode
	ENDIF
	::pHandle := hb_vfOpen( ::cFile, nMode )
	::aRecordsMap:={}
	IF ::pHandle == NIL
		::nError    := FError()       
		::lBOF      := .F.
		::nCurrLine := 0
	ELSE
		::nError    := 0
		::lBOF      := .T.
		::nCurrLine := 1
		
		//delimiter detection
		IF ::cDelim == NIL
			::cDelim := vf_DELIMITER
			cLine := hb_vfReadLen( ::pHandle, 64*1024 /* ::nReadSize */ )
			::GoTop()
			DO CASE
				CASE hb_AT( CHR (13) + CRLF, cLine ) > 0		//CRCRLF
					::cDelim := CHR (13) + CRLF
				CASE hb_AT( CRLF, cLine ) > 0					//CRLF
					::cDelim := CRLF
				CASE hb_AT( CHR (13), cLine ) > 0				//CR
					::cDelim := CHR (13)
				CASE hb_AT( CHR (10), cLine ) > 0				//LF
					::cDelim := CHR (10)
			ENDCASE
		ENDIF                      

	ENDIF
ELSE
      // The file is already open, so rewind to the beginning.
	IF !::GoTop()
      	::lBOF      := .F.
      	::nCurrLine := 0
      	::nError    := FError()
	ELSE
		::lBOF      := .T.
		::nCurrLine := 1
	ENDIF
ENDIF
   
RETURN Self

*******************************************************************

METHOD Close() CLASS vfFileRead

IF .NOT. ::pHandle == NIL
	hb_vfClose( ::pHandle )
	::pHandle := NIL           // The file is no longer open
ENDIF
::nError    := FError()
::lEOF      := .F.
::lBOF      := .F.
::nCurrLine := 0
::nLastLine := NIL

RETURN Self

*******************************************************************

METHOD ReadLine() CLASS vfFileRead

Local cLine       := ""
Local nCurrentPos := ::GetPointer()

DO WHILE !hb_vfEof( ::pHandle )
	cLine += hb_vfReadLen( ::pHandle, ::nReadSize )
	IF hb_AT( ::cDelim, cLine ) > 0
		cLine := Left ( cLine, hb_AT( ::cDelim, cLine ) - 1 )
		EXIT
	ENDIF
ENDDO

::SetPointer ( nCurrentPos )
::nError := FError()

RETURN cLine

*******************************************************************

METHOD GoTop() CLASS vfFileRead

::lEOF      := .F.
::lBOT      := .T.
::nCurrLine := 1

RETURN hb_vfSeek( ::pHandle, 0, 0 ) == 0

*******************************************************************

METHOD GoBottom() CLASS vfFileRead

hb_vfSeek( ::pHandle, 0, 2 )
::Skip( -1 )
::lEOF      := .T.
::lBOT      := .F.
::nCurrLine := ::nLastLine
::nError    := FError()
      
RETURN FError() == 0   

*******************************************************************

METHOD GoTo( nLine ) CLASS vfFileRead

Local nToSkip ,nIntStep, nPos

DEFAULT nLine := ::nCurrLine

IF !hb_isNumeric (nLine)
	RETURN Self
ENDIF

IF ABS ( nLine - ::nCurrLine ) >= ::nRecordsMapStep		//only if we skip by indexed steps
	nIntStep := ( INT( nLine / ::nRecordsMapStep ) + IF ( nLine % ::nRecordsMapStep < (::nRecordsMapStep / 2), 0, 1 ) ) * ::nRecordsMapStep
	DO WHILE nIntStep >= 0
		nPos := hb_AScan( ::aRecordsMap, { |x|  Bin2ULL ( ULLUnShrink ( x [ 1 ] ) ) == nIntStep } )
		IF nPos > 0
			::nCurrLine := Bin2ULL ( ULLUnShrink ( ::aRecordsMap [ nPos ] [ 1 ] ) )
			::SetPointer( Bin2ULL ( ULLUnShrink ( ::aRecordsMap [ nPos ] [ 2 ] ) ) )
			EXIT
		ENDIF
		//Try with previous indexed step
		nIntStep -= ::nRecordsMapStep
	ENDDO
ENDIF

nToSkip := nLine - ::nCurrLine

DO CASE
	CASE hb_isNil( ::nCurrLine )			//We do not know the current line number, so start at the beginning of the file
		::GoTop()
		::Skip ( nLine - 1 )
	CASE nLine <= 1					//Line number less than 1, so start at the beginning of the file
		::GoTop()
	CASE hb_isNumeric( ::nLastLine ) .AND. nLine >= ::nLastLine		//Go beyond the known last line number, so go to the end.
		::GoBottom()
	CASE hb_isNumeric( ::nLastLine ) .AND. ::nLastLine - nLine < ABS ( nToSkip ) 	//We know the number of the last record, the nToSkip is greater than the difference between the last and the target record - so skip backward from the end (less skips)
		::GoBottom()
		::Skip( 0 - (::nLastLine - nLine ) )
	CASE nLine < 0 - nToSkip				//The target line is closer to the start than the nToSkip, so start at the beginning of the file.
		::GoTop()
		::Skip ( nLine - 1 )
	OTHER							
		::Skip ( nToSkip )             
ENDCASE
::nError := FError()

RETURN Self

*******************************************************************

METHOD Skip ( nSkip ) CLASS vfFileRead

Local cLine     := ""
Local lRun      := .T.
Local nSkips    := 0
Local nBuffSize := 64 * 1024	//::nReadSize  //A larger buffer speeds up the process
Local nLenDelim := Len (::cDelim)
Local nStart, nEnd, nBackOf, nPos, nPointer
Default nSkip   := 1

IF nSkip == 0
	RETURN Self
ENDIF

::lEOF := .F.
::lBOF := .F.

DO WHILE lRun
                                                            
	IF nSkip > 0
		IF hb_vfEof( ::pHandle )
			::Skip ( -1 )
			::nLastLine := ::nCurrLine
			::lEOF      := .T.
			EXIT
		ENDIF
		nStart   := 1
		nPointer := ::GetPointer()
		cLine    := hb_vfReadLen( ::pHandle, nBuffSize )
		DO WHILE Right (cLine, 1) $ ::cDelim .AND. !hb_vfEof( ::pHandle )		//Avoid splitting of end-of-line marks in the buffer.
			cLine += hb_vfReadLen( ::pHandle, nLenDelim )
		ENDDO
		DO WHILE ( nPos := hb_AT( ::cDelim, cLine, nStart ) ) > 0
			nSkips ++
			IF hb_isNumeric( ::nCurrLine )
				::nCurrLine ++
				IF ::nCurrLine % ::nRecordsMapStep == 0 .AND. hb_AScan( ::aRecordsMap, { |x|  Bin2ULL ( ULLUnShrink ( x [ 1 ] ) ) == ::nCurrLine } ) == 0
					AADD( ::aRecordsMap, { ULLShrink( ULL2Bin( ::nCurrLine ) ), ULLShrink( ULL2Bin( nPointer + nPos  + nLenDelim - 1 ) ) } )
				ENDIF
				
			ENDIF
			nStart := nPos + nLenDelim
			IF nSkips == nSkip
				hb_vfSeek( ::pHandle, 0 - ( Len ( cLine ) - nStart + 1), 1 )
				lRun := .F.
				IF hb_vfEof( ::pHandle )
					::Skip ( -1 )
					::nLastLine := ::nCurrLine
					::lEOF      := .T.
				ENDIF
				EXIT
			ENDIF
		ENDDO
	ELSE			// -skip
	
		IF ::GetPointer() == 0		//top of file
			::lBOF      := .T.
			::nCurrLine := 1
			EXIT
		ENDIF
		cLine := ""
		
		DO WHILE ( nPos := hb_AT( ::cDelim, cLine ) ) == 0 .AND. hb_vfSeek( ::pHandle, 0, 1 ) <> 0		//Avoid splitting of end-of-line marks in the buffer.
			nBackOf  := Min( nBuffSize, ::GetPointer() )
			hb_vfSeek( ::pHandle, 0 - nBackOf, 1 )
			cLine    := hb_vfReadLen( ::pHandle, nBackOf ) + cLine
			hb_vfSeek( ::pHandle, 0 - nBackOf, 1 )
		ENDDO
		IF nPos == 0
			nPos := 1
		ENDIF
		
		nStart := nPos
		
		cLine  := SubStr ( cLine, nStart )
		hb_vfSeek( ::pHandle, nStart - 1, 1 )
		
		nPointer := ::GetPointer()
		nEnd     := Len( cLine )
		
		IF Right( cLine, nLenDelim) == ::cDelim
			nEnd -= nLenDelim
		ENDIF
	
		DO WHILE (nPos := hb_RAT( ::cDelim, cLine, 1, nEnd ) ) > 0
			nSkips --
			IF hb_isNumeric( ::nCurrLine )
				::nCurrLine --
				IF ::nCurrLine % ::nRecordsMapStep == 0 .AND. hb_AScan( ::aRecordsMap, { |x|  Bin2ULL ( ULLUnShrink ( x [ 1 ] ) ) == ::nCurrLine } ) == 0
					AADD( ::aRecordsMap, { ULLShrink( ULL2Bin( ::nCurrLine ) ), ULLShrink( ULL2Bin( nPointer + nPos + nLenDelim -1 ) ) } )
				ENDIF
			ENDIF
			
			nEnd := nPos - 1		//nLenDelim

			IF nSkips == nSkip
				hb_vfSeek( ::pHandle, 0 + nPos + nLenDelim - 1 , 1 )
				lRun := .F.
				EXIT
			ENDIF
		ENDDO
	ENDIF
ENDDO

cLine := ''
::nError := FError()

RETURN Self

*******************************************************************

METHOD IsEOF() CLASS vfFileRead

IF !::lEOF .AND. hb_IsNumeric ( ::nCurrLine ) .AND. hb_IsNumeric ( ::nLastLine ) .AND. ::nCurrLine == ::nLastLine
	::lEOF := .T.
ENDIF

IF !::lEOF .AND. ::GetPointer() == ::GetLastLinePointer()
	::lEOF := .T.
ENDIF

RETURN hb_vfEof( ::pHandle ) .OR. ::lEOF

*******************************************************************

METHOD IsBOF() CLASS vfFileRead

RETURN ::GetPointer() == 0 .OR. ::lBOF

*******************************************************************

METHOD Error() CLASS vfFileRead

RETURN FError() != 0

*******************************************************************

METHOD ErrorNo() CLASS vfFileRead

RETURN ::nError

*******************************************************************

METHOD GetPointer() CLASS vfFileRead
	
RETURN hb_vfSeek( ::pHandle, 0, 1 )

*******************************************************************

METHOD SetPointer( nPointer ) CLASS vfFileRead

hb_vfSeek( ::pHandle, nPointer, 0 )
::nError := FError()

RETURN ::nError == 0

*******************************************************************

METHOD GetLastLinePointer() CLASS vfFileRead

Local nCurrentPointer := ::GetPointer()
Local nCurrentLine    := ::nCurrLine
Local lCurrentEOF     := ::lEOF
Local nLastLinePointer
::GoBottom()
nLastLinePointer := ::GetPointer()
::SetPointer ( nCurrentPointer )	//roll back pointer
::nCurrLine := nCurrentLine			//roll back ::nCurrLine
::lEOF := lCurrentEOF				//roll back ::lEOF
::nError := FError()

RETURN nLastLinePointer

*******************************************************************

METHOD CurrentLine() CLASS vfFileRead

RETURN ::nCurrLine

*******************************************************************

METHOD CountLines( xGauge ) CLASS vfFileRead

Local cLine       := ""
Local nCurrentPos := ::GetPointer()
Local nLastPos    := hb_vfSeek( ::pHandle, 0, 2 )
Local nSkips      := 0
Local nlDelim     := Len ( ::cDelim )
Local nPrevPos    := 0
Local nBuffSize   := 64 * 1024
Local exCurrentGauge := ::exGauge
Local nPointer    := 0
Local nStart, nPos

IF HB_ISEVALITEM( xGauge )
	::exGauge := xGauge
ENDIF

::aRecordsMap := {}
::GoTop()

DO WHILE !hb_vfEof( ::pHandle )
	nStart   := 1
	nPointer := ::GetPointer()
	cLine    := hb_vfReadLen( ::pHandle, nBuffSize )
	DO WHILE Right (cLine, 1) $ ::cDelim .AND. !hb_vfEof( ::pHandle )	//Avoid splitting of end-of-line marks in the buffer.
		cLine += hb_vfReadLen( ::pHandle, nlDelim )
	ENDDO
	DO WHILE (nPos := hb_AT( ::cDelim, cLine, nStart )) > 0
		nSkips ++
		IF nSkips % ::nRecordsMapStep == 0
			AADD( ::aRecordsMap, { ULLShrink( ULL2Bin( nSkips ) ), ULLShrink( ULL2Bin( nPointer + nPrevPos - 1 /* pointer starts from 0 not from 1 like AT( ) */ ) ) } )
			
			IF HB_ISEVALITEM( ::exGauge )
				Eval( ::exGauge, nPointer + nPrevPos - 1, nLastPos, nSkips , 'Processing', Self )
			ENDIF
		
		ENDIF
			
		nStart   := nPos + nlDelim
		nPrevPos := nStart

	ENDDO
ENDDO

IF .NOT. Right( cLine, nlDelim ) == ::cDelim .AND. nLastPos > 0		//The last line does not end with a delimiter. Make it the last record.
	nSkips ++
ENDIF

IF HB_ISEVALITEM( ::exGauge )
	Eval( ::exGauge, nPointer + nPrevPos - 1, nLastPos, nSkips , 'Done', Self )
ENDIF

::nLastLine := nSkips
::SetPointer ( nCurrentPos )
::nError    := FError()
::exGauge   := exCurrentGauge

cLine := ""

RETURN nSkips

*******************************************************************

METHOD GetRecordsMap() CLASS vfFileRead

RETURN ::aRecordsMap

*******************************************************************

METHOD PutRecordsMap( aMap ) CLASS vfFileRead

IF aMap == NIL .OR. !hb_isArray( aMap )
	RETURN Self
ENDIF
::aRecordsMap := aMap

RETURN Self 

*******************************************************************

Function ULLShrink( cBin )
RETURN REMRIGHT( cBin, CHR(0) )

*******************************************************************

Function ULLUnShrink( cBin )
RETURN PADRIGHT( cBin, 8, CHR(0) )

*******************************************************************  

//Support Unsigned Long Long
#pragma BEGINDUMP   
 

#include "hbapi.h"
#include "hbapiitm.h"

HB_FUNC( BIN2ULL )
{
   PHB_ITEM pItem    = hb_param( 1, HB_IT_STRING );
   HB_U64   uiResult = 0;

   if( pItem )
   {
      HB_SIZE nLen = hb_itemGetCLen( pItem );
      if( nLen )
      {
         const char * pszString = hb_itemGetCPtr( pItem );
         uiResult = HB_GET_LE_UINT64( pszString );
          }
   }
   hb_retnint( uiResult );
}

HB_FUNC( ULL2BIN )
{
   char   szResult[ 8 ];
   HB_U64 uiValue = ( HB_U64 ) hb_parnint( 1 );

   HB_PUT_LE_UINT64( szResult, uiValue );
   hb_retclen( szResult, 8 );
}

#pragma ENDDUMP

************************ THE END *********************

HMGforum.com

Virtual Grids can view large text files?

Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?

Re: Virtual Grids can view large text files?