Best way to get the last nth lines from a large text file

[HGAutomator]

Hi,

I'm working with tab-delimited text files that are half a gig in length. I need e.g. the last 10,000 lines of the text file, but I don't want to read each line to get there. I'm going to extract those last 10,000 lines into another file, which we will then open & manipulate.

So after I go to the end of file, what's the best way to count backwards for 10,000 lines? is there an existing function for something like that?

[franco]

This is how I would do it. May be old way but works' Can build with build.bat

Code: Select all

#include "hmg.ch"

Function Main()
     CRTEMP()
Return



Function CrTemp
Local CF:= {}, FL1 := "FL1.DBF", YOURFILE := 'YOURFILENAME.TXT'          //PUT YOUR FILE NAME HERE
IF ! FILE(FL1)
	aADD(CF,{'FLD1', 'C', 200, 0})          // MAY NEED MORE OR LESS THEN 200 DEPENDS ON TEXT FILE.
	DBCREATE( "FL1", CF)
	USE
ENDIF
USE FL1 NEW
ZAP
APPEND FROM &YOURFILE DELIMITED
DELETE FOR RECNO() < 10000
PACK
GO TOP
SET PRINTER TO 'NEWFILE.TXT'
LIST FLD1 TO PRINTER 
SET PRINTER TO
USE

RETURN

[HGAutomator]

Franco,

Thanks, but definitely no way. If it were a smaller file, then of course no problem.

But some of them are 3 gigabytes in size, millions of lines.

I would need to open it as a text file, jump right to the end, then start counting back. APPEND FROM would take hours.

The part that I'm trying to work out, is how to count back 10,000 lines from the end of the file.

[AUGE_OHR]

hi,

does it Matter if have 9999 Lines instead of 10000 Lines

what about "get last 10 %" of File

Code: Select all

LOCAL nHandle := FOpen("DUMMY.TXT", FO_READWRITE+FO_DENYWRITE) 

   nEnd := FSeek( nHandle, 0 , FS_END ) 
   FSeek( nHandle, 0 , FS_SET ) 

   nBytes := nEnd/100*90
   FSeek( nHandle, nBytes, FS_RELATIVE )

now find 1st CRLF and start "read" Rest of file

[HGAutomator]

This is closer to what I'm looking for, yes.

I would actually have the exact number of lines that needed to be extracted, when we execute the code. So preferably, we'd like to hit the end of the file first, like you're doing here ; and count the number of carriage returns by skipping backwards.

There was a function, FMove2Prev(nHandle), in the Super.lib distribution that might work here. If not, your solution might be enough of an approximation, if we overestimate the percentage in order to avoid missing some important lines.

Ok, thanks Auge.

Thanks, Franco.

hi,

does it Matter if have 9999 Lines instead of 10000 Lines

what about "get last 10 %" of File

Code: Select all

LOCAL nHandle := FOpen("DUMMY.TXT", FO_READWRITE+FO_DENYWRITE) 

   nEnd := FSeek( nHandle, 0 , FS_END ) 
   FSeek( nHandle, 0 , FS_SET ) 

   nBytes := nEnd/100*90
   FSeek( nHandle, nBytes, FS_RELATIVE )

now find 1st CRLF and start "read" Rest of file

[franco]

Sorry for first try. I did not use very big file.
This works faster. 6 seconds 1,044,600 lines and add 1 second for every 500,000 more.

Code: Select all

Function CrTemp
Local CF:= {}, FL1 := "FL1.DBF", YOURFILE := 'YOURFILENAME.TXT'          //PUT YOUR FILE NAME HERE
Local rec := 0
IF ! FILE(FL1)
	aADD(CF,{'FLD1', 'C', 200, 0})          // MAY NEED MORE OR LESS THEN 200 DEPENDS ON TEXT FILE.
	DBCREATE( "FL1", CF)
	USE
ENDIF
USE FL1 NEW
ZAP
APPEND FROM &YOURFILE DELIMITED
go bottom
msgbox(RECNO())
rec := recno()
SET PRINTER TO 'NEWFILE.TXT'
LIST FLD1 TO PRINTER for recno() > rec-10000   //you could create var for 10,000  then rec-var
SET PRINTER TO
ZAP
APPEND FROM 'NEWFILE.TXT' DELIMITED
GO BOTTOM
MSGBOX(RECNO())

USE

RETURN

[edk]

Maybe something like this (10 000 000 lines/records 2.5 GB: looking for bottom of file takes in 6 sec on my SSD)

Code: Select all

/*
 * HMG - Harbour Win32 GUI library
 * Copyright 2002-2008 Roberto Lopez <mail.box.hmg@gmail.com>
 * http://www.hmgforum.com//
*/

#include "hmg.ch"

Function Main()

   DEFINE WINDOW Form_1 ;
      AT 0,0 ;
      WIDTH 430 ;
      HEIGHT 270 ;
      MAIN;
      TITLE 'Demo'

      DEFINE MAIN MENU

			POPUP 'File'

				ITEM 'Make file BigSize.txt' ACTION makefile()
				ITEM 'Read Last 10 lines'  	 ACTION readFile(10)
				SEPARATOR
				ITEM 'Exit' 		         ACTION ThisWindow.Release
			END POPUP
	  END MENU
     

   END WINDOW

   CENTER WINDOW Form_1

   ACTIVATE WINDOW Form_1

Return
**************************************
FUNCTION MakeFile()
Local nRecords := 10000000, i, cTab := Chr ( 9 )
Local cAnyString := "any string", cLine := ""
Local hFile:=FCREATE("BigSize.txt")

FOR i:= 32 TO 255
	cAnyString := cAnyString + Chr ( i )
NEXT i
WAIT WINDOW "Please wait ...." NOWAIT  

FOR i :=  1 TO nRecords
	cLine := strZero(i, 8) + cTab + "Line number " + str (i) + cTab + cAnyString + CRLF
	FWRITE(hFile, cLine, Len( cLine ))
	IF i%100000 = 0
		WAIT WINDOW "Please wait ("+ AllTrim(str( i/nRecords * 100 )) +"%) ...." NOWAIT
	ENDIF
NEXT i

FCLOSE(hFile)
WAIT CLEAR
MsgInfo ("Done")
RETURN
************************************
Function ReadFile( nLinesToRead )
Local nLastLine, nSec
Default nLinesToRead := 10
IF !File( "BigSize.txt" )
	MsgInfo ( "First make file BigSize.txt" )
	RETURN
ENDIF
WAIT WINDOW "Opening the BigSize.txt ...." NOWAIT 
hb_FUse ( "BigSize.txt" )

//msgdebug ( hb_FInfo() )
WAIT WINDOW "Looking for the bottom of file ...." NOWAIT

/*
// It is slower ;-(

nSec:=Seconds()
nLastLine := hb_FLastRec()
msginfo("Total lines:" + Str( nLastLine) + "  Seconds:" + Str( Seconds() - nSec ) )
nSec:=Seconds()
hb_FGoto ( nLastLine - nLinesToRead ) 
msginfo("Line No:" + Str( hb_FRecno()) + "  Seconds:" + Str( Seconds() - nSec ) )

*/

//  It is faster when we go bottom and skip backwards. ;-)
nSec:=Seconds()
hb_FGoBottom () 
msginfo("Last line No:" + Str( hb_FRecno()) + "  Take seconds:" + Str( Seconds() - nSec ) )
WAIT CLEAR

//nSec:=Seconds()
hb_FSkip ( nLinesToRead * (-1)) 
//msginfo("Line No:" + Str( hb_FRecno()) + "  Take seconds:" + Str( Seconds() - nSec ) )

DO WHILE ! hb_FEof()
	msginfo ( hb_FReadLN() )
	hb_FSkip()
ENDDO
hb_FUse ()		//Close file
RETURN

[HGAutomator]

That looks doable, thanks edk.

Franco,

Thanks, but even aside from the speed, I don't want to create another multi-gig file, in the form of the dbf. I'll sometimes execute this on the workstation where the speed is max ; but sometimes this might be executed on a server.

The only thing we want to create is an output file that contains only the final lines from the original file.


Thanks everyone,

[martingz]

I do not know if I understood you correctly, I hope that is what you ask
text file with 5292052 lines


#include <hmg.ch>
#include "simpleio.ch"
#include "fileio.ch"
#include "inkey.ch"

Function Main
set browsesync on
set softseek on
set multiple off warning
set language to spanish
set navigation extended
set tooltipstyle balloon
Load Window Main
Main.Center
Main.Activate
Return nil


function inicio
cstring := hb_memoread('fact.txt')
arch:='fact1.txt'
if (nmanejador := fcreate(arch, fc_normal)) == -1
msginfo('no se puede crear el fichero:','mensaje del sistema')
return nil
endif
nlines:= mlcount(cstring,100)
msgbox('lineas :' + transform(nlines,'999,999,999'))
for h = (nlines - 10000) to nlines
registro:=alltrim(memoline(cstring , 100, h))
fwrite(nmanejador, registro + crlf )
next
fclose(nmanejador)
fclose(nmanejador)
return nil

[serge_girard]

Edward,

Question: is there no hb_FBof() and then hb_FSkip() -1 ?

Serge