“Transfer-Encoding: chunked” header in PHP. What is "padding" for? - php

As suggested in a reply to a previous question (PHP External Oauth : how to displaying a waiting message while waiting for callback (not using AJAX) ), I am using transfer encoding: chunked to display a waiting message while some tasks are performed. My first attempts failed, and I found a solution in this question “Transfer-Encoding: chunked” header in PHP. There is a "padding" of 1024 blank space. Without this padding it doesn't work. I have googled around but I can't find what this padding is for. Here is the sample code (from the related question).
<?php
header('Content-Encoding', 'chunked');
header('Transfer-Encoding', 'chunked');
header('Content-Type', 'text/html');
header('Connection', 'keep-alive');
ob_flush();
flush();
$p = ""; //padding
for ($i=0; $i < 1024; $i++) {
$p .= " ";
};
echo $p;
ob_flush();
flush();
for ($i = 0; $i < 10000; $i++) {
echo "string";
ob_flush();
flush();
sleep(2);
}
?>
Does anybody have an explanation why it works with and doesn't work without the "padding" ?

The padding is for filling the server buffer, as I understand.
Without it the server will wait until PHP will fill it and after this will flush it - even if in PHP code you do flush().
Related:
PHP Output buffer flush issue on Apache/Linux (May 2010; on Server Fault)

I have no idea what this padding is supposed to do, and actually it shouldn't work (someone may enlighten me if I'm wrong on this). The idea with chunked encoding is that you send your data in chunks. Each chunk consists of a line containing the length of a chunk, followed by a newline and then the data of the chunk. A response can contain as many chunks as you want to have. So basically the response of 3 chunks containing "Hello" would look like this:
5 <--- this is the length of the chunk, that is "Hello" == 5 chars
Hello <--- This is a the actual data
<-- an empty line is between the chunks
5
Hello
5
Hello
<-- send two empty lines to end the transmission
So I'd rewrite this to something like:
<?php
header('Content-Encoding', 'chunked');
header('Transfer-Encoding', 'chunked');
header('Content-Type', 'text/html');
header('Connection', 'keep-alive');
ob_flush();
flush();
for ($i = 0; $i < 10000; $i++) {
$string = "string";
echo strlen($string)."\r\n"; // this is the length
echo $string."\r\n"; // this is the date
echo "\r\n"; // newline between chunks
ob_flush(); // rinse and repeat
flush();
sleep(2);
}
echo "\r\n"; // send final empty line
ob_flush();
flush();
?>
The code above will not work at all circumstances (e.g. with strings containing newlines or non-ascii encodings), so you will have to adapt it to your use case.

Related

PHP - output text one character at a time

Trying to create a old school terminal text effect (one character at a time with a small delay) in PHP - without javascript if possible.
All text written to the screen should go through this function.
I was thinking something like a buffer you can dynamically append text to make sure it would finish one line, before starting on the next.
Not sure how to preceed or if it's even possible without using Javascript.
Inefficient, but to achieve the goal you set (without javascript), you could use PHP's output buffering to achieve a small delay between characters output:
<?php
ob_start();
$buffer = str_repeat(" ", 4096); // fill the buffer
$string = 'Hello World';
$len = strlen($string);
$sleep = 0.5; // sleep half a second between output chars
for($i=0; $i < $len; $i++) {
echo $buffer . $string[$i];
ob_flush();
flush();
usleep($sleep * 1000000);
}

Encoding puzzles with sockets in different languages

I have this below code written in PHP responsible for the server socket, specifically by writing messages to certain sockets:
header('Content-Type: text/html; charset=utf-8');
const PAYLOAD_LENGTH_16 = 126;
const PAYLOAD_LENGTH_63 = 127;
const OPCODE_CONTINUATION = 0;
for ($i = 0; $i < $frameCount; $i++) {
// fetch fin, opcode and buffer length for frame
$fin = $i != $maxFrame ? 0 : self::FIN;
$opcode = $i != 0 ? self::OPCODE_CONTINUATION : $opcode;
$bufferLength = $i != $maxFrame ? $bufferSize : $lastFrameBufferLength;
// set payload length variables for frame
if ($bufferLength <= 125) {
$payloadLength = $bufferLength;
$payloadLengthExtended = '';
$payloadLengthExtendedLength = 0;
}
elseif($bufferLength <= 65535) {
$payloadLength = self::PAYLOAD_LENGTH_16;
$payloadLengthExtended = pack('n', $bufferLength);
$payloadLengthExtendedLength = 2;
} else {
$payloadLength = self::PAYLOAD_LENGTH_63;
$payloadLengthExtended = pack('xxxxN', $bufferLength); // pack 32 bit int, should really be 64 bit int
$payloadLengthExtendedLength = 8;
}
// set frame bytes
$buffer = pack('n', (($fin | $opcode) << 8) | $payloadLength).$payloadLengthExtended.substr($message, $i * $bufferSize, $bufferLength);
And below I have the code in Objective-C responsible for receiving these messages from the socket server:
NSInteger len = 0;
uint8_t buffer[4096];
while ([inputStream hasBytesAvailable]) {
len = [inputStream read:buffer maxLength:sizeof(buffer)];
if (len > 0) {
[self.data appendBytes:buffer length:len];
[self.log insertText:[NSString stringWithFormat:#"Log: Received a message from server:\n\n"]];
NSLog(#"Received a message from server...");
}
}
when all bytes are received I run the following command to turn the data into a file:
[self.data writeToFile:#"dataComes.txt" options:NSDataWritingAtomic error:nil]
The Problem
We will send a large file in JSON format for objective-c, with that he will receive that information and will generate a file called dataComes.txt, I can see the JSON file normally but except for some strange characters such as:
~ or ~Â or â-Û
These strange characters always shows at the beginning of each block messages that Objective-C receives (Yes, the socket server and TCP divide large messages into blocks of messages).
What is the cause of this problem and how it could solve this?
SOLUTION 1: Filtering
I can filter out unwanted characters that may come, but it will also filter out some words that have accentuation:
NSCharacterSet *notAllowedChars = [[NSCharacterSet characterSetWithCharactersInString:#"0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ[]{}:,'"] invertedSet];
NSString *resultString = [[total componentsSeparatedByCharactersInSet:notAllowedChars] componentsJoinedByString:#" "];
SOLUTION 2: Stop using sockets
I have tried many ways to send data to my app, the only one that worked was to send the data separately (a loop of one JSON), but to works I had to put my code (PHP) to sleep using sleep(1) (and I believe this is not good) because if not Objective-C recognizes that this data is a single package.
In this case, or my code have problems, or the programming of socket in objective-c was not very well done and has inconsistencies (bug). What remains for me to do with my connections through normal requests via web server (which I do not think it's a good idea, since I have to do this every 3 seconds in a 5 minute time interval).
SOLUTION 3: FILTERING + UNICODE
On the server side I can filter all special characters and create a specific combination for it example:
Hello é world to Hello /e001/ world
And in my app I can filter this combination and change to the real format....

ob_implicit_flush(), flush(), ob_flush() - not working on remote server

If I load this script in chrome from my local server on XAMPP:
header("Content-Type:text/plain");
set_time_limit(0);
$max = 40;
for ($i = 0; $i < $max; $i++) {
$response = array( 'server time: ' . date("h:i:s", time()), 'progress' => round($i/$max*100));
echo json_encode($response);
ob_flush();
flush();
sleep(1);
}
ob_clean();
It works as you would expect, every second the page displays a new response.
However, when I upload it to my remote server (running the same version of php), it waits until the entire script finishes before it displays the output.
On very long scripts, it updates the output every 30-60 seconds or so.
As the title suggests, I've tried using all the different flush functions, but nothing works.
There is likely some difference in the php.ini of my local server and my remote server, but I don't know what.
Please help.
---EDIT---
I've been doing some more testing. I've noticed that exactly it only updates every 4096 bytes, which happens to be what my remote server's php ini value for 'output_buffering' is.
However, for some reason, if I change output_buffering to '1' or 'off', nothing changes. It still only updates every 4096 bytes.
I'm testing the 2 identical scripts on different servers on the same browser.
I didn't take into account nginx, which has it's own output buffer.
I simply added 'header("X-Accel-Buffering: no");' to the top of the php script and it all works fine now.
For me adding header('Content-Encoding: none'); did the trick. This is needed when using PHP-FPM.
This works fine in Apache + PHP
header('Content-Encoding: none');
ob_implicit_flush(1);
echo "<br>PROCESSING bla bla bla";
Optionally you can add the following line (after every small piece of data) if you want to throw out really small chunks too.
echo str_repeat(' ',1024*64);
Before you need use ob_start() and ob_end_clean(). And add header Content-Length or Transfer-Encoding: chunked. And check if «implicit_flush» is On in your php.ini
Add padding for response. Check this code:
<?php
set_time_limit(0);
ob_start();
header('Content-Type: text/plain');
define("PADDING", 16);
//+padding
for($i=0;$i<PADDING;$i++){
//64 spaces (1 block)
echo str_repeat(' ', 64);
}
$max = 40;
for ($i = 0; $i < $max; $i++) {
$response = array( 'server time: ' . date("h:i:s", time()), 'progress' => round($i/$max*100));
echo json_encode($response);
ob_flush();
flush();
sleep(1);
}
ob_end_clean();
?>

How to split content into equal parts then write to file in PHP

I have been reading/testing examples since last night, but the cows never came home.
I have a file with (for example) approx. 1000 characters in one line and want to split it into 10 equal parts then write back to the file.
Goal:
1. Open the file in question and read its content
2. Count up to 100 characters for example, then put a line break
3. Count 100 again and another line break, and so on till it's done.
4. Write/overwrite the file with the new split content
For example:
I want to turn this => KNMT2zSOMs4j4vXsBlb7uCjrGxgXpr
Into this:
KNMT2zSOMs
4j4vXsBlb7
uCjrGxgXpr
This is what I have so far:
<?php
$MyString = fopen('file.txt', "r");
$MyNewString;
$n = 100; // How many you want before seperation
$MyNewString = substr($MyString,0,$n);
$i = $n;
while ($i < strlen($MyString)) {
$MyNewString .= "\n"; // Seperator Character
$MyNewString .= substr($MyString,$i,$n);
$i = $i + $n;
}
file_put_contents($MyString, $MyNewString);
fclose($MyString);
?>
But that is not working quite the way I anticipated.
I realize that there are other similiar questions like mine, but they were not showing how to read a file, then write back to it.
<?php
$str = "aonoeincoieacaonoeincoieacaonoeincoieacaonoeincoieacaonoeincoieacaon";
$pieces = 10;
$ch = chunk_split($str, $pieces);
$piece = explode("\n", $ch);
foreach($piece as $line) {
// write to file
}
?>
http://php.net/manual/en/function.chunk-split.php
Hold on here. You're not giving a file name/path to file_put_contents();, you're giving a file handle.
Try this:
file_put_contents("newFileWithText.txt", $MyNewString);
You see, when doing $var=fopen();, you're giving $var a value of a handle, which is not meant to be used with file_put_contents(); as it doesnt ask for a handle, but a filename instead. So, it should be: file_put_contents("myfilenamehere.txt", "the data i want in my file here...");
Simple.
Take a look at the documentation for str_split. It will take a string and split it into chunks based on length, storing each chunk at a separate index in an array that it returns. You can then iterate over the array adding a line break after each index.

Clear PHP CLI output

I'm trying to get a "live" progress indicator working on my php CLI app. Rather than outputting as
1Done
2Done
3Done
I would rather it cleared and just showed the latest result. system("command \C CLS") doesnt work. Nor does ob_flush(), flush() or anything else that I've found.
I'm running windows 7 64 bit ultimate, I noticed the command line outputs in real time, which was unexpected. Everyone warned me that out wouldn't... but it does... a 64 bit perk?
Cheers for the help!
I want to avoid echoing 24 new lines if I can.
Try outputting a line of text and terminating it with "\r" instead of "\n".
The "\n" character is a line-feed which goes to the next line, but "\r" is just a return that sends the cursor back to position 0 on the same line.
So you can:
echo "1Done\r";
echo "2Done\r";
echo "3Done\r";
etc.
Make sure to output some spaces before the "\r" to clear the previous contents of the line.
[Edit] Optional: Interested in some history & background? Wikipedia has good articles on "\n" (line feed) and "\r" (carriage return)
I came across this while searching for a multi line solution to this problem. This is what I eventually came up with. You can use Ansi Escape commands. http://www.inwap.com/pdp10/ansicode.txt
<?php
function replaceOut($str)
{
$numNewLines = substr_count($str, "\n");
echo chr(27) . "[0G"; // Set cursor to first column
echo $str;
echo chr(27) . "[" . $numNewLines ."A"; // Set cursor up x lines
}
while (true) {
replaceOut("First Ln\nTime: " . time() . "\nThird Ln");
sleep(1);
}
?>
I recently wrote a function that will also keep track of the number of lines it last output, so you can feed it arbitrary string lengths, with newlines, and it will replace the last output with the current one.
With an array of strings:
$lines = array(
'This is a pretty short line',
'This line is slightly longer because it has more characters (i suck at lorem)',
'This line is really long, but I an not going to type, I am just going to hit the keyboard... LJK gkjg gyu g uyguyg G jk GJHG jh gljg ljgLJg lgJLG ljgjlgLK Gljgljgljg lgLKJgkglkg lHGL KgglhG jh',
"This line has newline characters\nAnd because of that\nWill span multiple lines without being too long",
"one\nmore\nwith\nnewlines",
'This line is really long, but I an not going to type, I am just going to hit the keyboard... LJK gkjg gyu g uyguyg G jk GJHG jh gljg ljgLJg lgJLG ljgjlgLK Gljgljgljg lgLKJgkglkg lHGL KgglhG jh',
"This line has newline characters\nAnd because of that\nWill span multiple lines without being too long",
'This is a pretty short line',
);
One can use the following function:
function replaceable_echo($message, $force_clear_lines = NULL) {
static $last_lines = 0;
if(!is_null($force_clear_lines)) {
$last_lines = $force_clear_lines;
}
$term_width = exec('tput cols', $toss, $status);
if($status) {
$term_width = 64; // Arbitrary fall-back term width.
}
$line_count = 0;
foreach(explode("\n", $message) as $line) {
$line_count += count(str_split($line, $term_width));
}
// Erasure MAGIC: Clear as many lines as the last output had.
for($i = 0; $i < $last_lines; $i++) {
// Return to the beginning of the line
echo "\r";
// Erase to the end of the line
echo "\033[K";
// Move cursor Up a line
echo "\033[1A";
// Return to the beginning of the line
echo "\r";
// Erase to the end of the line
echo "\033[K";
// Return to the beginning of the line
echo "\r";
// Can be consolodated into
// echo "\r\033[K\033[1A\r\033[K\r";
}
$last_lines = $line_count;
echo $message."\n";
}
In a loop:
foreach($lines as $line) {
replaceable_echo($line);
sleep(1);
}
And all lines replace each other.
The name of the function could use some work, just whipped it up, but the idea is sound. Feed it an (int) as the second param and it will replace that many lines above instead. This would be useful if you were printing after other output, and you didn't want to replace the wrong number of lines (or any, give it 0).
Dunno, seemed like a good solution to me.
I make sure to echo the ending newline so that it allows the user to still use echo/print_r without killing the line (use the override to not delete such outputs), and the command prompt will come back in the correct place.
i know the question isn't strictly about how to clear a SINGLE LINE in PHP, but this is the top google result for "clear line cli php", so here is how to clear a single line:
function clearLine()
{
echo "\033[2K\r";
}
function clearTerminal () {
DIRECTORY_SEPARATOR === '\\' ? popen('cls', 'w') : exec('clear');
}
Tested on Win 7 PHP 7. Solution for Linux should work, according to other users reports.
something like this :
for ($i = 0; $i <= 100; $i++) {
echo "Loading... {$i}%\r";
usleep(10000);
}
Use this command for clear cli:
echo chr(27).chr(91).'H'.chr(27).chr(91).'J'; //^[H^[J
Console functions are platform dependent and as such PHP has no built-in functions to deal with this. system and other similar functions won't work in this case because PHP captures the output of these programs and prints/returns them. What PHP prints goes to standard output and not directly to the console, so "printing" the output of cls won't work.
<?php
error_reporting(E_ERROR | E_WARNING | E_PARSE);
function bufferout($newline, $buffer=null){
$count = strlen(rtrim($buffer));
$buffer = $newline;
if(($whilespace = $count-strlen($buffer))>=1){
$buffer .= str_repeat(" ", $whilespace);
}
return $buffer."\r";
};
$start = "abcdefghijklmnopqrstuvwxyz0123456789";
$i = strlen($start);
while ($i >= 0){
$new = substr($start, 0, $i);
if($old){
echo $old = bufferout($new, $old);
}else{
echo $old = bufferout($new);
}
sleep(1);
$i--;
}
?>
A simple implementation of #dkamins answer. It works well. It's a bit- hack-ish. But does the job. Wont work across multiple lines.
function (int $count = 1) {
foreach (range(1,$count) as $value){
echo "\r\x1b[K"; // remove this line
echo "\033[1A\033[K"; // cursor back
}
}
See the full example here
Unfortunately, PHP 8.0.2 does not has a function to do it. However, if you just want to clear console try this: print("\033[2J\033[;H"); or use : proc_open('cls', 'w');
It works in php 8.0.2 and windows 10. It is the same that system('cls') using c language programing.
Tried some of solutions from answers:
<?php
...
$messages = [
'11111',
'2222',
'333',
'44',
'5',
];
$endlines = [
"\r",
"\033[2K\r",
"\r\033[K\033[1A\r\033[K\r",
chr(27).chr(91).'H'.chr(27).chr(91).'J',
];
foreach ($endlines as $i=>$end) {
foreach ($messages as $msg) {
output()->write("$i. ");
output()->write($msg);
sleep(1);
output()->write($end);
}
}
And \033[2K\r seems like works correct.

Categories