Help needed Edward(edk) or anyone who can help

[RPC]

Hi Edward,
Hope you are doing well.
Long back you have kindly helped me download files from internet especially www.nseindia.com website.
Though there were some changes after that in the website I could manage to download the files.
Now from 8 jul 24 they have stopped the support from link from where I was downloading the files.

Pls see link www.nseindia.com/all-reports > Archives date > 8 jul 2024 (SS enclosed).
In the SS you will see on top left side in red letters it is written "Discontinued w.e.f. July 08, 2024..."
and they have asked to download CM-UDiFF common bhavcopy Final (zip) . This file downloads if
clicked. When I try to replace the name of the downloaded file in download link in program or even in address bar it
doesn't download.
for eg

https://archives.nseindia.com//content/ ... av.csv.zip

the above work till 5 jul 24. From 8 Jul 24 they have stopped working.

I have tried
https://archives.nseindia.com//content/ ... 00.csv.zip

it doesn't work I get 404 error.

Pls help downloading the file.
Many thanks
Rajeev

[edk]

Links like before will no longer work.
The link to the CM-UDiFF Common Bhavcopy Final(zip) is as follows:

Code: Select all

https://www.nseindia.com/api/reports?archives=[{"name":"CM-UDiFF Common Bhavcopy Final (zip)","type":"daily-reports","category":"capital-market","section":"equities"}]&date=08-Jul-2024&type=equities&mode=single

As you can see, there is a date= parameter in the link where you must post the date, e.g. 08-Jul-2024, 09-Jul-2024, 10-Jul-2024, etc.

try:

• 08-Jul-2024:

  Code: Select all

  https://www.nseindia.com/api/reports?archives=[{"name":"CM-UDiFF Common Bhavcopy Final (zip)","type":"daily-reports","category":"capital-market","section":"equities"}]&date=08-Jul-2024&type=equities&mode=single

• 09-Jul-2024:

  Code: Select all

  https://www.nseindia.com/api/reports?archives=[{"name":"CM-UDiFF Common Bhavcopy Final (zip)","type":"daily-reports","category":"capital-market","section":"equities"}]&date=09-Jul-2024&type=equities&mode=single

• 10-Jul-2024:

  Code: Select all

  https://www.nseindia.com/api/reports?archives=[{"name":"CM-UDiFF Common Bhavcopy Final (zip)","type":"daily-reports","category":"capital-market","section":"equities"}]&date=10-Jul-2024&type=equities&mode=single

[RPC]

Hi Edward
Thanks for the quick reply.
I am unable to download the file. When I Copy paste link in address bar I get message "Resource not found".
Pls let me know what I am doing wrong.
Thanks
Rajeev

[edk]

Oh, the site seems to check cookies and if they are valid, it allows the data to be retrieved. Try opening the website https://www.nseindia.com/all-reports in your browser first and then paste the links I provided in a new tab. It works for me.
We will need to look at the mechanism of how the platform authorizes data downloads. I don't have much time at the moment, but maybe I'll try to analyze it tomorrow.

[RPC]

Yes Edward, it works that way.
Pls take your time to figure out about cookies.
Many thanks for devoting your time in solving my problem.
Rajeev

[edk]

Okay, I've got it

Code: Select all

#include "HMG.CH"

FUNCTION main()

SET DATE BRIT
SET CENTURY ON

DEFINE WINDOW testunzip MAIN AT 0,0 WIDTH 200 HEIGHT 200
       DEFINE DATEPICKER datepicker_1
	   ROW 20
	   COL 20
	   WIDTH 100
	   HEIGHT 20
	   ONLOSTFOCUS getDay()
	   VALUE Date()
       END DATEPICKER

       DEFINE BUTTON Button_1
          ROW 120
          COL 20
          WIDTH 100
          HEIGHT 30
          CAPTION "Download" 
          ACTION Download()
      END BUTTON	

      DEFINE LABEL Lbl_1
         ROW 45
         COL 20
         WIDTH 150
         HEIGHT 20
	 VALUE ""
      END LABEL

      DEFINE LABEL Lbl_2
         ROW 70
         COL 20
         WIDTH 150
         HEIGHT 50
         VALUE ""
      END LABEL		 
	  
	  
END WINDOW
testunzip.center
testunzip.activate
RETURN nil	  

****************************************************************

/*  url syntax
https://www.nseindia.com/api/reports?archives=[{"name":"CM-UDiFF Common Bhavcopy Final (zip)","type":"daily-reports","category":"capital-market","section":"equities"}]&date=10-Jul-2024&type=equities&mode=single
*/


FUNCTION Download()
LOCAL cUrl, cFileZIP, dDate, cHost, cDateQuery
      dDate := testunzip.datepicker_1.value
      cDateQuery := NtoC( DAY( dDate ), 10, 2, '0' ) + '-' + UPPER( LEFT( CMONTH( dDate ), 1 ) ) + LOWER( SubStr( CMONTH( dDate ), 2, 2 ) ) + '-' + NtoC( YEAR( dDate ), 10, 4, '0' )
      cHost:= 'https://www.nseindia.com'
      cUrl := '/api/reports?archives=[{"name":"CM-UDiFF Common Bhavcopy Final (zip)","type":"daily-reports","category":"capital-market","section":"equities"}]&date=' + cDateQuery + '&type=equities&mode=single'

      cFileZIP := downloadfile( cHost, cUrl )

      IF Empty ( cFileZIP )
         msginfo( {"CM-UDiFF Common Bhavcopy Final (zip) file not found for ", dDate })
      ELSE
         IF FILE( cFileZIP )
              UNCOMPRESS cFileZIP
              MsgInfo ("File " + hb_FNameNameExt( cFileZIP ) + " uncompressed")
         ELSE
              MsgStop ( "File " + hb_FNameNameExt( cFileZIP ) + " not found!" )
         ENDIF
     ENDIF

RETURN nil

****************************************************************
FUNCTION GetDay( dDate )
testUnZip.Lbl_1.value := cdow(testUnZip.datepicker_1.value)
RETURN nil

****************************************************************
FUNCTION DownloadFile( cHost, cUrl )

LOCAL ohttp
LOCAL cAttTAG := 'Content-Disposition: attachment; filename="'
Local cHeader, cHeaders, recv
Local cUrlFile := "", cFile := ""
Local cCookie := ""
Local cCurrentUrl 

   BEGIN SEQUENCE WITH { | e | BREAK( e ) }
   	oHttp := Win_OleCreateObject( "WinHttp.WinHttpRequest.5.1" )
   	
        /* First, we connect to http://www.nseindia.com/all-reports to download the access cookies. */

	cCurrentUrl := "http://www.nseindia.com/all-reports" 
	testUnZip.Lbl_2.value := "Connecting to " + cCurrentUrl
   	
        // Connect to the HTTPs server
        ohttp:Open( "GET", cCurrentUrl, .T. )
	// Set Headers
	ohttp:setRequestHeader("Content-Type", "application/x-www-form-urlencoded")
	ohttp:setRequestHeader("Accept", "*/*")
	ohttp:setRequestHeader("X-Requested-With", "XMLHttpRequest")
	ohttp:setRequestHeader("User-Agent", "Mozilla/5.0 (Linux; Android 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Mobile Safari/537.36")
	ohttp:setRequestHeader("DNT", "1")
	ohttp:Send()
	ohttp:WaitForResponse()

        testUnZip.Lbl_2.value := "Getting response headers ..."
	// download the response headers
	cHeaders := ohttp:getAllResponseHeaders()
	IF ohttp:Status <> 200
	     testUnZip.Lbl_2.value := "Error"
             BREAK "HTTP Server " + cCurrentUrl + " status is " + hb_NToS( ohttp:Status ) + " " + ohttp:statusText
        ENDIF
     
        testUnZip.Lbl_2.value := "Getting cookies ..."
        //get cookies
	cCookie := ""
	FOR EACH cHeader IN hb_ATokens(cHeaders, .T.)	
		IF Left(cHeader,12) == "Set-Cookie: "
			cCookie += IF (!EMPTY( cCookie ), '; ', '') + hb_ATokens(SubStr( cHeader, 13 ), ';') [1]
		ENDIF
	NEXT
	
//	msgdebug (cCookie)

        /* With an access cookies, we download the report attachment */
     
	cCurrentUrl := cHost + cUrl
        testUnZip.Lbl_2.value := "Connecting to " + cCurrentUrl
	//get file from link
	ohttp:Open( "GET", cCurrentUrl, .T. )
	ohttp:setRequestHeader("Content-Type", "application/x-www-form-urlencoded")
	ohttp:setRequestHeader("Accept", "*/*")
	ohttp:setRequestHeader("X-Requested-With", "XMLHttpRequest")
	ohttp:setRequestHeader("User-Agent", "Mozilla/5.0 (Linux; Android 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Mobile Safari/537.36")
	IF !Empty( cCookie )
		ohttp:setRequestHeader("Cookie", cCookie)	//required if exist
	ENDIF
	ohttp:setRequestHeader("DNT", "1")
	ohttp:Send()
	ohttp:WaitForResponse()
	
        testUnZip.Lbl_2.value := "Getting contents of the attachment ..."
	// download the response
	recv := ohttp:responseBody()
	testUnZip.Lbl_2.value := "Getting cookies ..."
	cHeaders := ohttp:getAllResponseHeaders()

        IF ohttp:Status <> 200
            testUnZip.Lbl_2.value := "Error"
            BREAK "HTTP Server " + cCurrentUrl + " status is " + hb_NToS( ohttp:Status ) + " " + ohttp:statusText
        ENDIF
     
        ohttp := Nil

	testUnZip.Lbl_2.value := "Getting attachment name ..."
	//get attachment filename
	FOR EACH cHeader IN hb_ATokens(cHeaders, .T.)	
	     IF Left(cHeader, Len ( cAttTAG ) ) == cAttTAG
                  cUrlFile := SubStr( cHeader, Len ( cAttTAG ) + 1, Len ( cHeader ) - Len ( cAttTAG ) - 1 )
                  EXIT
             ENDIF
	NEXT

	IF !Empty( cUrlFile )
             cFile := getcurrentfolder() + '\' + cUrlFile
             IF FILE ( cFile ) //old file exist
   	          DELETEFILE ( cFile )
             ENDIF
     
             IF EMPTY( recv ) .OR. '!DOCTYPE' $ recv
                  testUnZip.Lbl_2.value := "Error"
    	          BREAK "Invalid response "
    	     ENDIF
    	
             testUnZip.Lbl_2.value := "Saving attachment file ..."
    	     StrFile ( recv, cFile )
             testUnZip.Lbl_2.value := "Attachment file saved."

         ELSE
             testUnZip.Lbl_2.value := "There is no attachment."
             MsgStop ( "Attachment name not received!" )
         ENDIF
     
   RECOVER USING oError
      MsgStop( IF( VALTYPE( oError ) = 'O', oError:Description, oError ) )

   END SEQUENCE

   ohttp := Nil

RETURN cFile

[RPC]

Hi Edward,
Thank you for the code.
Right now I have to rush to hospital. I will check it later today.
Thanks once again
Rajeev

[RPC]

Hi Edward,
It works now ! thank you very much.
Can you pls tell me how you got the api link for download ?
I want to download similar file from Derivatives section.
If I know how to get api link I will be able to download that file also.
Many thanks
Rajeev

[edk]

It's quite simple. To download a file, the browser must know its source/link. Once you have downloaded the file, select "Copy download link" from the list of downloads from the context menu. Such a copied link can be in URL encoded format, so it only needs to be decoded to the native one.

Record_2024_07_12_09_30_24_137-ezgif.com-optimize.gif (1.84 MiB) Viewed 1189 times

https://www.hmgforum.com/download/file.php?id=11971

A little tip: once you download a cookie, it has its own validity period, which is quite long. So you only need to download cookies once and authorize yourself with them when downloading subsequent archived data.

[RPC]

Hi Edward
Thank you very much for your effort in explaining this.
I was not aware of this at all especially www.urldecoder.org website.
Now I will try to download derivatives also.
Thanks a lot.
Rajeev