Welcome, Guest
Username: Password: Remember me
Components and Libraries for Database Development, discussions, problems and suggestions
  • Page:
  • 1

TOPIC:

Problem with BLOB stream 9 years 8 months ago #5764

  • Nicola Sperotto
  • Nicola Sperotto's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 5
  • Thank you received: 0
I'm trying to load very large BLOBS to SQLServer database, using Typhon & ZeosLib (last version).
I need to upload the files in chunks, so I've tryed this source, wich works perfectly in Delphi:

procedure TForm1.SaveFileToBlob(Filename: TFilename);
const
bufSize = 16384;
var
sStream : TFileStream;
pBuf : Pointer;
cnt,totCnt: Integer;
begin
// total bytes reset
totCnt:=0;
// open the file stream for reading
sStream:=TFileStream.Create(Filename, fmOpenRead or fmShareDenyWrite);
try
try
// reserve space for buffer
GetMem(pBuf, bufSize);
// read first chunk
cnt:=sStream.Read(pBuf^, bufSize);
totCnt:=cnt;

while (cnt > 0) do
begin
Application.ProcessMessages;
qry_Blob.Close;
qry_Blob.ParamByName('id').AsInteger:=1;
qry_Blob.ParamByName('data').SetBlobData(pBuf,cnt);
//
qry_Blob.ExecSQL;
// read next chunk
cnt:=sStream.Read(pBuf^, bufSize);
inc(totCnt,cnt);
// show total bytes processed
Label1.Caption:=Format('Bytes caricati: %d - %d MB',[totCnt, totCnt div (1024*1204)]);
Form1.Refresh;
Application.ProcessMessages;
end;
finally
FreeMem(pBuf, bufSize);
end;
finally
sStream.Free;
end;

ShowMessage('Done');
end;

When I put this in Lazarus I get a SEGMENTATION FAULT error.

Someone can help? Any idea?

Please Log in or Create an account to join the conversation.

Problem with BLOB stream 9 years 7 months ago #5989

  • Michael Hiergeist
  • Michael Hiergeist's Avatar
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 24
  • Thank you received: 7
Hi,

how do you access the SQL-Server (MSSSQL?)

AFAIK sending chunks for MSSQL isn't supported for Delphi or FPC neither. MSSQL/Sybase are the !only! RDBM's we support who have no real prepared stmts!
There are BCP_functions available but i did gave up since i allways get a SQL-Server Error German: mssql dberror S 'DBError : [10007] : General SQL Server error: Check messages from the SQL Server.'#$D#$A'Das verwendete Front-End-Tool unterstützt kein Massenladen vom Host. Verwenden Sie die unterstützten Tools für diesen Befehl.'

Which means a normal user/not admin isn't able to get this running.
I'm a bit surpriced it would work with Delphi.. Can't see no differences in this area. So it might not be a Zeos issue?

Please Log in or Create an account to join the conversation.

Problem with BLOB stream 9 years 6 months ago #6043

  • Nicola Sperotto
  • Nicola Sperotto's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 5
  • Thank you received: 0
Hi,

thank you for your reply. In delphi I use ZeosLib 6.6 + ntwdblib.dll - It works... in Typhon / FreePascal I get a SIGSEGV when I try to use SetBlobData...

nicola

Please Log in or Create an account to join the conversation.

Problem with BLOB stream 9 years 6 months ago #6045

  • Sternas Stefanos
  • Sternas Stefanos's Avatar
  • Offline
  • Moderator
  • Moderator
  • Ex Pilot, M.Sc, Ph.D
  • Posts: 4508
  • Thank you received: 1100
Sir, my suggestion for ZeosLib
is to post on ZeosLib Forum
is very active forum.
and Zeos it's a BIG library.
PilotLogic Architect and Core Programmer
The following user(s) said Thank You: Nicola Sperotto

Please Log in or Create an account to join the conversation.

Last edit: by Sternas Stefanos.

Problem with BLOB stream 9 years 6 months ago #6046

  • Nicola Sperotto
  • Nicola Sperotto's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 5
  • Thank you received: 0
Ok, thank you for your suggestion... In any case, I leave here an information I've just found, maybe it could be useful for someone.
The Blob data are sent to the database not as string but as an hexadecimal string, preceded by 0x and with the HEX code of any
"char" read from the stream... like this:

0x25504446

that is 0x + $25 $50 $44 $46 -> %PDF

In my case I'm trying to load a .pdf file... so I wil try do compose the query without using parameter binding...

Best regards

Please Log in or Create an account to join the conversation.

Problem with BLOB stream 9 years 6 months ago #6047

  • Sternas Stefanos
  • Sternas Stefanos's Avatar
  • Offline
  • Moderator
  • Moderator
  • Ex Pilot, M.Sc, Ph.D
  • Posts: 4508
  • Thank you received: 1100
Thanks Sir
I read for this Zeos Error on many forums... thanks for the solution :)
PilotLogic Architect and Core Programmer

Please Log in or Create an account to join the conversation.

Problem with BLOB stream 9 years 6 months ago #6068

  • Michael Hiergeist
  • Michael Hiergeist's Avatar
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 24
  • Thank you received: 7
@sternas
can you pont me to some more such weird threads?

@nicola

we've some problems with the FPC-TDataSet. It's less maintained since SQLDB uses the TBufDataSet.

IMHO is the issue in your code:

qry_Blob.Close;
qry_Blob.Edit/Insert; <<< is missing or am i'm blind? Btw. on Delphi the TdataSet-descendants automatically fall into Edit mode...
qry_Blob.ParamByName('id').AsInteger:=1;
qry_Blob.ParamByName('data').SetBlobData(pBuf,cnt);
//
qry_Blob.ExecSQL;

Does it resolve your issue?
And please .. use the Zeos-Forum instead. I just have a look to CodeTyphoon from time to time..

EDIT: did check the Code on our side. Delphi doesn't support
ParamByName('data').SetBlobData(pBuf,cnt);
syntax. I did a research if we've a test to check what's going on. Nope there is no test for this FPC abnormailty, i'm affraid. :S

All test including DBLIB are using:
ParamByName('data').LoadFromStream(AStream, ftBlob/ftMemo/ftWideMemo);

To avoid further issues in this area i'll add a FPC test now.. Anyway i can't believe sending lobs in chunks is working with the TabularDataStream-Protocol the DBLIB/NTWDBLIB/FreeTDS-Libs are using. Sending binary data is not supported for the frontend API. This protokol supports RawEncoded-Strings only(The reason for the BinToHex conversion). But TDS has NO Limit for such strings whreas Firbird is limited to 32KB f.e..
OR is there a trick i don't know about?

Please Log in or Create an account to join the conversation.

Last edit: by Michael Hiergeist.

Problem with BLOB stream 9 years 6 months ago #6076

  • Nicola Sperotto
  • Nicola Sperotto's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 5
  • Thank you received: 0
Sir, thank you... but there's no need of Edit/Insert, since I use ExecSQL - To illustrate the situation, I post the entire code (wich is now functioning). I hope it would
be useful in future to someone...

// pascal code
function TForm1.SaveFileToBlob(Filename: TFilename; bufSize: integer = 256): integer;
type
  TBuffer = string;
var
  sStream: TFileStream;
  pBuf: TBuffer;
  cnt, i: integer;
  sSegment: string;
begin
  // chunks size
  bufSize := bufSize * 1024;

  Result:=0;
  // open the file stream for reading
  sStream := TFileStream.Create(Filename, fmOpenRead or fmShareDenyWrite);
  try
    // set buffer length and read first chunk
    SetLength(pBuf, bufSize);
    cnt := sStream.Read(PChar(pBuf)^, bufSize);
    // re-set buffer length with bytes effectively read
    SetLength(pBuf, cnt);
     
    while (cnt > 0) do
    begin
      Application.ProcessMessages;
    
      qry_Blob.Close;
      // build binary chunk
      sSegment := '0x';
      for i := Low(pBuf) to high(pBuf) do
        sSegment := sSegment + IntToHex(Ord(pBuf[i]), 2);
      // create the SaveBlob query and execute parameters binding
      qry_Blob.SQL.Text := Format('exec SaveBlob %d, %s, %s', [1, sSegment, 'NULL']);
      qry_Blob.ExecSQL;

      // read next chunk
      cnt := sStream.Read(PChar(pBuf)^, bufSize);
      SetLength(pBuf, cnt);
      // increase result 
      Inc(Result, cnt);        
      {$ifdef debug}
        Label1.Caption := Format('Bytes caricati: %d - %d MB', [Result, Result div (1024 * 1204)]);
        Form1.Refresh;
        Application.ProcessMessages;
      {$endif}
    end;
  finally
    sStream.Free;
  end;

  {$ifdef debug}
  ShowMessage('Done');
  {$endif}
end;

-- MS SQL Code:
-- table where to save file
CREATE TABLE BlobTest
(
	Id int,
	Data image
);
GO

-- procedure to update BlobTest table
create procedure SaveBlob @id int, @data image, @append bit = 1
as
begin
  declare @ptr binary(16)
  -- TESTING -> append always to record id = 1 
  -- (must be modified...)
  set @id=1;
  set @Append = 1;
  -- 
  
  -- create a record (if not exists) with an empty BLOB (not null!)
  if (select COUNT(*) from BlobTest where Id=@id) = 0
    insert into BlobTest
       select @id, ''

  -- sets the text pointer
  select @ptr = TEXTPTR(data) from  BlobTest where id = @id

  -- append the data to the record (using @ptr)
  if @Append = 1
    updatetext BlobTest.data @ptr NULL 0 @data
  else -- never reached (in testing @Apped always = 1)
     writetext BlobTest.data @ptr @data
end;

Regards
Nicola

Please Log in or Create an account to join the conversation.

  • Page:
  • 1