Writing File in Chunks using Ajax and FileReader - php

Question

I am working on creating a file uploader, where the contents will be written to a remote server in chunks using a 3rd party API. The API provides a WriteFileChunk() method that takes 3 parameters, the target file path, the start position (Int64) and data (string) of bytes.
Each time FileReader receives a chunk of the maximum supported size (16kb), I need to use Ajax to pass this to a PHP file and write it using the API. I suspect that this should be done in the onprogress event of FileReader, however I am at somewhat of a loss given that I cannot find any similar examples.
What would be the best way to implement this using FileReader, ensuring that each chunk is uploaded before writing the next? If onprogress is the best choice, how can I get the current chunk data?
$(document).ready(function()
{
function uploadFile()
{
var files = document.getElementById('file').files;
if (!files.length)
{
alert('Please select a file!');
return;
}
var file = files[0];
var first_byte = 0;
var last_byte = file.size - 1;
// File Reader
var reader = new FileReader();
reader.onerror = function(evt)
{
switch(evt.target.error.code)
{
case evt.target.error.NOT_FOUND_ERR:
alert('File Not Found!');
break;
case evt.target.error.NOT_READABLE_ERR:
alert('File is not readable');
break;
case evt.target.error.ABORT_ERR:
break;
default:
alert('An error occurred reading this file.');
};
};
reader.onprogress = function(evt)
{
if (evt.lengthComputable)
{
var percentLoaded = Math.round((evt.loaded / evt.total) * 100);
console.log(percentLoaded + "%");
if (percentLoaded < 100)
{
$("#upload_progress").progressbar('value', percentLoaded);
}
}
};
reader.onabort = function(evt)
{
alert('File Upload Cancelled');
};
reader.onloadstart = function(evt)
{
$("#upload_progress").progressbar({
value: 0,
max: 100
});
};
reader.onload = function(evt)
{
$("#upload_progress").progressbar('value', 100);
};
reader.onloadend = function(evt)
{
if (evt.target.readyState == FileReader.DONE) // DONE == 2
{
alert("Upload Complete!");
//console.log(evt.target.result);
}
};
var blob = file.slice(first_byte, last_byte + 1);
reader.readAsBinaryString(blob);
}
fileupload_dialog = $( "#dialog-fileupload" ).dialog(
{
autoOpen: false,
height: 175,
width: 350,
modal: true,
buttons:
{
"Upload File": uploadFile
},
close: function()
{
form[ 0 ].reset();
}
});
form = fileupload_dialog.find( "form" ).on( "submit", function( event )
{
event.preventDefault();
uploadFile();
});
$("#file_upload a").click(function()
{
event.preventDefault();
fileupload_dialog.dialog( "open" );
});
});

Answer 1

The main challenge here is that the FileReader will need to read the entire file into memory before it return any usable data to us via the result property, meaning you can't grab chunks from the file while the file is being read (and the progress event won't provide/point to any data):
This property [result] is only valid after the read operation is
complete [...]
Source
As the complete file is loaded into memory there wouldn't really be any benefits chunking the read process (if it was possible), besides from reducing a little lag perhaps between the two processes.
I would suggest the following approach based on the above:
Load the entire file into memory, but as ArrayBuffer
Calculate number of segments required (Math.ceil(fileLength/chunkSize))
Create a chunk using a Uint8Array view for the ArrayBuffer using the offset and chunk length arguments
Send the chunk, wait for response asynchronously, continue to next chunk until left length is <= 0 byte.
The chunk can be converted to a Blob before sending it, if needed:
var chunkBlob = new Blob([chunk], {type: "application/octet-stream"});
Example process
A pseudo server example waiting arbitrarily 100ms between each block of 16kb:
file.onchange = function(e) {
var fr = new FileReader();
fr.onprogress = function(e) {progress.value = e.loaded / e.total};
fr.onload = startUpload.bind(fr);
progress.style.display = "inline-block";
fr.readAsArrayBuffer(e.target.files[0]);
}
// Main upload code
function startUpload() {
// calculate sizes
var chunkSize = 16<<10;
var buffer = this.result;
var fileSize = buffer.byteLength;
var segments = Math.ceil(fileSize / chunkSize);
var count = 0;
progress.value = 0;
// start "upload"
(function upload() {
var segSize = Math.min(chunkSize, fileSize - count * chunkSize);
if (segSize > 0) {
var chunk = new Uint8Array(buffer, count++ * chunkSize, segSize); // get a chunk
progress.value = count / segments;
// send chunk to server (here pseudo cycle for demo purpose)
setTimeout(upload, 100); // when upload OK, call function again for the next block
}
else {
alert("Done");
progress.style.display = "none";
}
})()
}
body {font:16px sans-serif;margin:20px 0 0 20px}
<label>Select any file: <input type=file id="file"></label><br>
<progress id="progress" value=0 max=1 style="display:none" />