Welcome, Guest
Username: Password: Remember me
General discussions, feature requests for CodeTyphon Project and discussions that don't fit in any of the other specific CodeTyphon forum categories.
  • Page:
  • 1

TOPIC:

In reference of Reading and Writing Large csv file of my Post 5 years 3 months ago #12963

  • Md. Shariful Alam Khan
  • Md. Shariful Alam Khan's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 111
  • Thank you received: 0
I post this 4 days ago but yet to receive a suitable answer. That's why I copied Delphi's TStreamReader and TStreamWriter class in one pas file. But I can't compile it because of some error. Can anyone here to help me with this? I need this unit for CSV file read/write line by line, I believe this will help a lot of you.

Please see attached pas file as zip.
Attachments:

Please Log in or Create an account to join the conversation.

In reference of Reading and Writing Large csv file of my Post 5 years 3 months ago #12964

  • Sternas Stefanos
  • Sternas Stefanos's Avatar
  • Offline
  • Moderator
  • Moderator
  • Ex Pilot, M.Sc, Ph.D
  • Posts: 4506
  • Thank you received: 1100
Sir
the BIG problem is Delphi Windows Unicode strings.
For this @Matis suggest to use TFileStream class

CodeTyphon use UTF-8 Unicode encoding for all OSes,
but can handle any Unicode encoding strings.

So, It's NOT simple to Read and Write Delphi String Data files, like CSV files.

Sorry, but we will NOT solve the Delphi strings problem on NO Windows OSes
it's NOT our problem :)

PS: Please look and TCSVDataset (codetyphon/fpcsrc/packages/fcl-db/src\base/csvdataset.pp) for more help
PilotLogic Architect and Core Programmer

Please Log in or Create an account to join the conversation.

Last edit: by Sternas Stefanos.

In reference of Reading and Writing Large csv file of my Post 5 years 3 months ago #12965

  • Md. Shariful Alam Khan
  • Md. Shariful Alam Khan's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 111
  • Thank you received: 0
Sorry, I think you don't understand my point. On my unit which I attached doesn't have Windows unit it only has classes. Please check. I tried with that unit but have few compile error and runtime error.

TFilestream does not have line by line read which is very important on my situation as I have to read one line and add a hashcode of that line to the end of that line and put it to other csv file. I can't load a big at least 4-5GB file in memory which will cause my application a fault.

Please Log in or Create an account to join the conversation.

In reference of Reading and Writing Large csv file of my Post 5 years 3 months ago #12966

  • Sternas Stefanos
  • Sternas Stefanos's Avatar
  • Offline
  • Moderator
  • Moderator
  • Ex Pilot, M.Sc, Ph.D
  • Posts: 4506
  • Thank you received: 1100
Yes
but
1) the line you read from your CSV data file what Unicode encoding has ?
2) is this data file for Linux Unicode encoding or from Windows Unicode encoding.
3) When convert this line to CT UTF-8, all will be OK ?

Please look and TCSVDataset (codetyphon/fpcsrc/packages/fcl-db/src\base/csvdataset.pp) for more help

.... it's time for my second coffee :woohoo: ...
PilotLogic Architect and Core Programmer

Please Log in or Create an account to join the conversation.

Last edit: by Sternas Stefanos.

In reference of Reading and Writing Large csv file of my Post 5 years 3 months ago #12967

  • Md. Shariful Alam Khan
  • Md. Shariful Alam Khan's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 111
  • Thank you received: 0
1. it's ANSI
2. 100% Linux
3. Yes

I started in CodeTyphon because my application was built for Linux and it failed to run in CentOS 6 only because of GLIB C. Delphi has support for GLIBC >2.14 but CentOS has GLIBC 2.12. This is why I come to CodeTyphon. Although my application is running well on all Linux distros.

Just see my output CSV file as an attachment which has only 2 rows though.

Please Log in or Create an account to join the conversation.

In reference of Reading and Writing Large csv file of my Post 5 years 3 months ago #12968

  • Sternas Stefanos
  • Sternas Stefanos's Avatar
  • Offline
  • Moderator
  • Moderator
  • Ex Pilot, M.Sc, Ph.D
  • Posts: 4506
  • Thank you received: 1100
My suggestion
try to use TCSVDataset or modify TCSVDataset comp
PilotLogic Architect and Core Programmer

Please Log in or Create an account to join the conversation.

In reference of Reading and Writing Large csv file of my Post 5 years 3 months ago #12970

  • Md. Shariful Alam Khan
  • Md. Shariful Alam Khan's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 111
  • Thank you received: 0
Thanks for your suggestion but it also failed. When I tried to call LoadFromCSVFile or Active := True of TCSVDataset application hanged for infinite times don't when it will release.

Delphi's TSreamReader works perfectly with this situation as it's read line by line without loading to full file in memory. Isn't there any way to solve it in FreePascal?

Please Log in or Create an account to join the conversation.

In reference of Reading and Writing Large csv file of my Post 5 years 3 months ago #12971

  • Sanogo
  • Sanogo's Avatar
  • Offline
  • New Member
  • New Member
  • Posts: 8
  • Thank you received: 0
You have to use a TFileStream and implement a read line by line method to achieve your goals, (I have done this before), you can find an example here Read File Line by Line or make your ripped streamrw.pas unit compile with CodeTyphoon although I would dissuade you from that because of copyright issues, (by the way, it should require very little work to make it compile with CT, you just need to put in a little effort).

Please Log in or Create an account to join the conversation.

  • Page:
  • 1