How to read FoxPro Memo with PHP? - php

Question

I have to convert .DBF and .FPT files from Visual FoxPro to MySQL. Right now my script works for .DBF files, it opens and reads them with dbase_open() and dbase_get_record_with_names() and then executes the MySQL INSERT commands.
However, some fields of these .DBF files are of type MEMO and therefore stored in a separate files ending in .FPT. How do I read this file?
I have found the specifications of this filetype in MSDN, but I don't know how I can read this file byte-wise with PHP (also, I would really prefer a simplier solution).
Any ideas?

Answer 1

Alright, I have carefully studied the MSDN specifications of DBF and FPT file structures and the outcome is a beautiful PHP class which can open a DBF and (optional) an FPT memo file at the same time. This class will give you record after record and thereby fetch any memos from the memo file - if opened.
class Prodigy_DBF {
private $Filename, $DB_Type, $DB_Update, $DB_Records, $DB_FirstData, $DB_RecordLength, $DB_Flags, $DB_CodePageMark, $DB_Fields, $FileHandle, $FileOpened;
private $Memo_Handle, $Memo_Opened, $Memo_BlockSize;
private function Initialize() {
if($this->FileOpened) {
fclose($this->FileHandle);
}
if($this->Memo_Opened) {
fclose($this->Memo_Handle);
}
$this->FileOpened = false;
$this->FileHandle = NULL;
$this->Filename = NULL;
$this->DB_Type = NULL;
$this->DB_Update = NULL;
$this->DB_Records = NULL;
$this->DB_FirstData = NULL;
$this->DB_RecordLength = NULL;
$this->DB_CodePageMark = NULL;
$this->DB_Flags = NULL;
$this->DB_Fields = array();
$this->Memo_Handle = NULL;
$this->Memo_Opened = false;
$this->Memo_BlockSize = NULL;
}
public function __construct($Filename, $MemoFilename = NULL) {
$this->Prodigy_DBF($Filename, $MemoFilename);
}
public function Prodigy_DBF($Filename, $MemoFilename = NULL) {
$this->Initialize();
$this->OpenDatabase($Filename, $MemoFilename);
}
public function OpenDatabase($Filename, $MemoFilename = NULL) {
$Return = false;
$this->Initialize();
$this->FileHandle = fopen($Filename, "r");
if($this->FileHandle) {
// DB Open, reading headers
$this->DB_Type = dechex(ord(fread($this->FileHandle, 1)));
$LUPD = fread($this->FileHandle, 3);
$this->DB_Update = ord($LUPD[0])."/".ord($LUPD[1])."/".ord($LUPD[2]);
$Rec = unpack("V", fread($this->FileHandle, 4));
$this->DB_Records = $Rec[1];
$Pos = fread($this->FileHandle, 2);
$this->DB_FirstData = (ord($Pos[0]) + ord($Pos[1]) * 256);
$Len = fread($this->FileHandle, 2);
$this->DB_RecordLength = (ord($Len[0]) + ord($Len[1]) * 256);
fseek($this->FileHandle, 28); // Ignoring "reserved" bytes, jumping to table flags
$this->DB_Flags = dechex(ord(fread($this->FileHandle, 1)));
$this->DB_CodePageMark = ord(fread($this->FileHandle, 1));
fseek($this->FileHandle, 2, SEEK_CUR); // Ignoring next 2 "reserved" bytes
// Now reading field captions and attributes
while(!feof($this->FileHandle)) {
// Checking for end of header
if(ord(fread($this->FileHandle, 1)) == 13) {
break; // End of header!
} else {
// Go back
fseek($this->FileHandle, -1, SEEK_CUR);
}
$Field["Name"] = trim(fread($this->FileHandle, 11));
$Field["Type"] = fread($this->FileHandle, 1);
fseek($this->FileHandle, 4, SEEK_CUR); // Skipping attribute "displacement"
$Field["Size"] = ord(fread($this->FileHandle, 1));
fseek($this->FileHandle, 15, SEEK_CUR); // Skipping any remaining attributes
$this->DB_Fields[] = $Field;
}
// Setting file pointer to the first record
fseek($this->FileHandle, $this->DB_FirstData);
$this->FileOpened = true;
// Open memo file, if exists
if(!empty($MemoFilename) and file_exists($MemoFilename) and preg_match("%^(.+).fpt$%i", $MemoFilename)) {
$this->Memo_Handle = fopen($MemoFilename, "r");
if($this->Memo_Handle) {
$this->Memo_Opened = true;
// Getting block size
fseek($this->Memo_Handle, 6);
$Data = unpack("n", fread($this->Memo_Handle, 2));
$this->Memo_BlockSize = $Data[1];
}
}
}
return $Return;
}
public function GetNextRecord($FieldCaptions = false) {
$Return = NULL;
$Record = array();
if(!$this->FileOpened) {
$Return = false;
} elseif(feof($this->FileHandle)) {
$Return = NULL;
} else {
// File open and not EOF
fseek($this->FileHandle, 1, SEEK_CUR); // Ignoring DELETE flag
foreach($this->DB_Fields as $Field) {
$RawData = fread($this->FileHandle, $Field["Size"]);
// Checking for memo reference
if($Field["Type"] == "M" and $Field["Size"] == 4 and !empty($RawData)) {
// Binary Memo reference
$Memo_BO = unpack("V", $RawData);
if($this->Memo_Opened and $Memo_BO != 0) {
fseek($this->Memo_Handle, $Memo_BO[1] * $this->Memo_BlockSize);
$Type = unpack("N", fread($this->Memo_Handle, 4));
if($Type[1] == "1") {
$Len = unpack("N", fread($this->Memo_Handle, 4));
$Value = trim(fread($this->Memo_Handle, $Len[1]));
} else {
// Pictures will not be shown
$Value = "{BINARY_PICTURE}";
}
} else {
$Value = "{NO_MEMO_FILE_OPEN}";
}
} else {
$Value = trim($RawData);
}
if($FieldCaptions) {
$Record[$Field["Name"]] = $Value;
} else {
$Record[] = $Value;
}
}
$Return = $Record;
}
return $Return;
}
function __destruct() {
// Cleanly close any open files before destruction
$this->Initialize();
}
}
The class can be used like this:
$Test = new Prodigy_DBF("customer.DBF", "customer.FPT");
while(($Record = $Test->GetNextRecord(true)) and !empty($Record)) {
print_r($Record);
}
It might not be an almighty perfect class, but it works for me. Feel free to use this code, but note that the class is VERY tolerant - it doesn't care if fread() and fseek() return true or anything else - so you might want to improve it a bit before using.
Also note that there are many private variables like number of records, recordsize etc. which are not used at the moment.

Answer 2

I replaced this code:
fseek($this->Memo_Handle, (trim($RawData) * $this->Memo_BlockSize)+8);
$Value = trim(fread($this->Memo_Handle, $this->Memo_BlockSize));
with this code:
fseek($this->Memo_Handle, (trim($RawData) * $this->Memo_BlockSize)+4);
$Len = unpack("N", fread($this->Memo_Handle, 4));
$Value = trim(fread($this->Memo_Handle, $Len[1] ));
this helped for me

Answer 3

Although not PHP, VFP is 1-based references and I think PHP is zero-based references so you'll have to decypher and adjust accordingly, but this works and hopefully you'll be able
to post your version of this portion when finished.
FILETOSTR() in VFP will open a file, and read the entire content into
a single memory variable as a character string -- all escape keys, high byte characters, etc, intact. You'll probably need to rely on an FOPEN(), FSEEK(), FCLOSE(), etc.
MemoTest.FPT was my sample memo table/file
fpt1 = FILETOSTR( "MEMOTEST.FPT" )
First, you'll have to detect the MEMO BLOCK SIZE used when the file was created. Typically this would be 64 BYTES, but per the link you had in your post.
The header positions 6-7 identify the size (VFP, positions 7 and 8). The first byte is the high-order
nBlockSize = ASC( SUBSTR( fpt1, 7, 1 )) * 256 + ASC( SUBSTR( fpt1, 8, 1 ))
Now, at your individual records. Wherever in your DBF structure has the memo FIELD (and you can have many per single record structure) there will be 4 bytes. In THE RECORD field, it identifies the "block" in the memo file where the content is stored.
MemoBytes = 4 bytes at your identified field location. These will be stored as ASCII from 0-255. This field is stored with the FIRST byte as low-order and the 4th byte as 256^3 = 16777216. The first "Block" ever used will be starting in position offset of 512 per the memo .fpt file spec that the header takes up positions 0-511.
So, if your first memo field has a content of "8000" where the 8 is the actual 0x08, not number "8" which is 0x38, and the zeros are 0x00.
YourMemoField = "8000" (actually use ascii, but for readability showing hex expected value)
First Byte is ASCII value * 1 ( 256 ^ 0 )
Second Byte is ASCII value * 256 (256 ^ 1)
Third Byte is ASCII value * 65536 (256 ^ 2)
Fourth Byte is ASCII value * 16777216 (256 ^ 3)
nMemoBlock = byte1 + ( byte2 * 256 ) + ( byte3 * 65536 ) + ( byte4 * 16777216 )
Now, you'll need to FSEEK() to the
FSEEK( handle, nMemoBlock * nBlockSize +1 )
for the first byte of the block you are looking for. This will point to the BLOCK header. In this case, per the spec, the first 4 bytes identify the Block SIGNATURE, the second 4 bytes is the length of the content. For these two, the bytes are stored with HIGH-BYTE first.
From your FSEEK(), its REVERSE of the nMemoBlock above with the high-byte. The "Byte1-4" here are from your FSEEK() position
nSignature = ( byte1 * 16777216 ) + ( byte2 * 65536 ) + ( byte3 * 256 ) + byte4
nMemoLength = ( byte5 * 16777216 ) + ( byte6 * 65536 ) + ( byte7 * 256 ) + byte8
Now, FSEEK() to the 9th byte (1st actual character of the data AFTER the 8 bytes of the header you just read for signature and memo length). This is the beginning of your data.
Now, read the rest of the content...
FSEEK() +9 characters to new position
cFinalMemoData = FREAD( handle, nMemoLength )
I know this isn't perfect, nor PHP script, but its enough of pseudo-code on hOW things are stored and hopefully gets you WELL on your way.
Again, PLEASE take into consideration as you are stepping through your debug process to ensure 0 or 1 offset basis. To help simplify and test this, I created a simple .DBF with 2 fields... a character field and a memo field, added a few records and some basic content to confirm all content, positions, etc.

Answer 4

I think it's unlikely there are FoxPro libraries in PHP.
You may have to code it from scratch. For byte-wise reading, meet fopen() fread() and colleagues.
Edit: There seems to be a Visual FoxPro ODBC driver. You may be able to connect to a FoxPro database through PDO and that connector. How the chances of success are, and how much work it would be, I don't know.

Answer 5

The FPT file contains memo data. In the DBF you have columns of type memo, and the information in this column is a pointer to the entry in the FPT file.
If you are querying the data from the table you only have to reference the memo column to get the data. You do not need to parse the data out of the FPT file separately. The OLE DB driver (or the ODBC driver if your files are VFP 6 or earlier) should just give you this information.
There is a tool that will automatically migrate your Visual FoxPro data to MySQL. You might want to check it out to see if you can save some time:
Go to http://leafe.com/dls/vfp
and search for "Stru2MySQL_2" for the tool to migrate the structures of the data and "VFP2MySQL Data Upload program" for tools to help with the migration.
Rick Schummer VFP MVP

Answer 6

You also might want to check the PHP dbase libraries.They work quite well with DBF files.

Answer 7

<?
class Prodigy_DBF {
private $Filename, $DB_Type, $DB_Update, $DB_Records, $DB_FirstData, $DB_RecordLength, $DB_Flags, $DB_CodePageMark, $DB_Fields, $FileHandle, $FileOpened;
private $Memo_Handle, $Memo_Opened, $Memo_BlockSize;
private function Initialize() {
if($this->FileOpened) {
fclose($this->FileHandle);
}
if($this->Memo_Opened) {
fclose($this->Memo_Handle);
}
$this->FileOpened = false;
$this->FileHandle = NULL;
$this->Filename = NULL;
$this->DB_Type = NULL;
$this->DB_Update = NULL;
$this->DB_Records = NULL;
$this->DB_FirstData = NULL;
$this->DB_RecordLength = NULL;
$this->DB_CodePageMark = NULL;
$this->DB_Flags = NULL;
$this->DB_Fields = array();
$this->Memo_Handle = NULL;
$this->Memo_Opened = false;
$this->Memo_BlockSize = NULL;
}
public function __construct($Filename, $MemoFilename = NULL) {
$this->Prodigy_DBF($Filename, $MemoFilename);
}
public function Prodigy_DBF($Filename, $MemoFilename = NULL) {
$this->Initialize();
$this->OpenDatabase($Filename, $MemoFilename);
}
public function OpenDatabase($Filename, $MemoFilename = NULL) {
$Return = false;
$this->Initialize();
$this->FileHandle = fopen($Filename, "r");
if($this->FileHandle) {
// DB Open, reading headers
$this->DB_Type = dechex(ord(fread($this->FileHandle, 1)));
$LUPD = fread($this->FileHandle, 3);
$this->DB_Update = ord($LUPD[0])."/".ord($LUPD[1])."/".ord($LUPD[2]);
$Rec = unpack("V", fread($this->FileHandle, 4));
$this->DB_Records = $Rec[1];
$Pos = fread($this->FileHandle, 2);
$this->DB_FirstData = (ord($Pos[0]) + ord($Pos[1]) * 256);
$Len = fread($this->FileHandle, 2);
$this->DB_RecordLength = (ord($Len[0]) + ord($Len[1]) * 256);
fseek($this->FileHandle, 28); // Ignoring "reserved" bytes, jumping to table flags
$this->DB_Flags = dechex(ord(fread($this->FileHandle, 1)));
$this->DB_CodePageMark = ord(fread($this->FileHandle, 1));
fseek($this->FileHandle, 2, SEEK_CUR); // Ignoring next 2 "reserved" bytes
// Now reading field captions and attributes
while(!feof($this->FileHandle)) {
// Checking for end of header
if(ord(fread($this->FileHandle, 1)) == 13) {
break; // End of header!
} else {
// Go back
fseek($this->FileHandle, -1, SEEK_CUR);
}
$Field["Name"] = trim(fread($this->FileHandle, 11));
$Field["Type"] = fread($this->FileHandle, 1);
fseek($this->FileHandle, 4, SEEK_CUR); // Skipping attribute "displacement"
$Field["Size"] = ord(fread($this->FileHandle, 1));
fseek($this->FileHandle, 15, SEEK_CUR); // Skipping any remaining attributes
$this->DB_Fields[] = $Field;
}
// Setting file pointer to the first record
fseek($this->FileHandle, $this->DB_FirstData);
$this->FileOpened = true;
// Open memo file, if exists
if(!empty($MemoFilename) and file_exists($MemoFilename) and preg_match("%^(.+).fpt$%i", $MemoFilename)) {
$this->Memo_Handle = fopen($MemoFilename, "r");
if($this->Memo_Handle) {
$this->Memo_Opened = true;
// Getting block size
fseek($this->Memo_Handle, 6);
$Data = unpack("n", fread($this->Memo_Handle, 2));
$this->Memo_BlockSize = $Data[1];
}
}
}
return $Return;
}
public function GetNextRecord($FieldCaptions = false) {
$Return = NULL;
$Record = array();
if(!$this->FileOpened) {
$Return = false;
} elseif(feof($this->FileHandle)) {
$Return = NULL;
} else {
// File open and not EOF
fseek($this->FileHandle, 1, SEEK_CUR); // Ignoring DELETE flag
foreach($this->DB_Fields as $Field) {
$RawData = fread($this->FileHandle, $Field["Size"]);
// Checking for memo reference
if($Field["Type"] == "M" and $Field["Size"] == 4 and !empty($RawData)) {
// Binary Memo reference
$Memo_BO = unpack("V", $RawData);
if($this->Memo_Opened and $Memo_BO != 0) {
fseek($this->Memo_Handle, $Memo_BO[1] * $this->Memo_BlockSize);
$Type = unpack("N", fread($this->Memo_Handle, 4));
if($Type[1] == "1") {
$Len = unpack("N", fread($this->Memo_Handle, 4));
$Value = trim(fread($this->Memo_Handle, $Len[1]));
} else {
// Pictures will not be shown
$Value = "{BINARY_PICTURE}";
}
} else {
$Value = "{NO_MEMO_FILE_OPEN}";
}
} else {
if($Field["Type"] == "M"){
if(trim($RawData) > 0) {
fseek($this->Memo_Handle, (trim($RawData) * $this->Memo_BlockSize)+8);
$Value = trim(fread($this->Memo_Handle, $this->Memo_BlockSize));
}
}else{
$Value = trim($RawData);
}
}
if($FieldCaptions) {
$Record[$Field["Name"]] = $Value;
} else {
$Record[] = $Value;
}
}
$Return = $Record;
}
return $Return;
}
function __destruct() {
// Cleanly close any open files before destruction
$this->Initialize();
}
}
?>