Apologies in advance for any mistakes as I'm very new to Node-Red. I'm trying to set up a simple HTTP server to update my ESP8266 node.
How it works is: the ESP8266 will send an HTTP request to a specific URL, and await a response of a binary folder. The binary file is stored in a folder locally on my Windows PC where the Node-Red is running. How can I setup a Node-Red flow that will do this?
This is my current setup but it doesn't work. When I request the URL through a web browser, it just downloads a folder with no file type.
These are the documentations for the ESP8266 side. The documentation recommends to setup a PHP server with the following code. However, this code is a little more complex which helps check the version before updating with the right binary file which I plan to implement later on in Node-Red.
header('Content-type: text/plain; charset=utf8', true);
function check_header($name, $value = false) {
if(!isset($_SERVER[$name])) {
return false;
}
if($value && $_SERVER[$name] != $value) {
return false;
}
return true;
}
function sendFile($path) {
header($_SERVER["SERVER_PROTOCOL"].' 200 OK', true, 200);
header('Content-Type: application/octet-stream', true);
header('Content-Disposition: attachment; filename='.basename($path));
header('Content-Length: '.filesize($path), true);
header('x-MD5: '.md5_file($path), true);
readfile($path);
}
if(!check_header('HTTP_USER_AGENT', 'ESP8266-http-Update')) {
header($_SERVER["SERVER_PROTOCOL"].' 403 Forbidden', true, 403);
echo "only for ESP8266 updater!\n";
exit();
}
if(
!check_header('HTTP_X_ESP8266_STA_MAC') ||
!check_header('HTTP_X_ESP8266_AP_MAC') ||
!check_header('HTTP_X_ESP8266_FREE_SPACE') ||
!check_header('HTTP_X_ESP8266_SKETCH_SIZE') ||
!check_header('HTTP_X_ESP8266_CHIP_SIZE') ||
!check_header('HTTP_X_ESP8266_SDK_VERSION') ||
!check_header('HTTP_X_ESP8266_VERSION')
) {
header($_SERVER["SERVER_PROTOCOL"].' 403 Forbidden', true, 403);
echo "only for ESP8266 updater! (header)\n";
exit();
}
$db = array(
"18:FE:AA:AA:AA:AA" => "DOOR-7-g14f53a19",
"18:FE:AA:AA:AA:BB" => "TEMP-1.0.0"
);
if(isset($db[$_SERVER['HTTP_X_ESP8266_STA_MAC']])) {
if($db[$_SERVER['HTTP_X_ESP8266_STA_MAC']] !=
$_SERVER['HTTP_X_ESP8266_VERSION']) ) {
sendFile("./bin/".$db[$_SERVER['HTTP_X_ESP8266_STA_MAC']]."bin");
} else {
header($_SERVER["SERVER_PROTOCOL"].' 304 Not Modified', true, 304);
}
exit();
}
header($_SERVER["SERVER_PROTOCOL"].' 500 no version for ESP MAC', true,
500);
Any help will be much appreciated. Thank you.
First things first. Let´s try to understand why your flow does not work and try to have it fixed.
There is a flow in Node-RED library that performs exactly what you want to achieve. If you have not tried yet please do it asap.
Link: https://cookbook.nodered.org/http/serve-a-local-file
Second point to consider: When we want to serve local content we need to set up an http static path in Node-RED configuration file called: settings.js
You have to open this file, find the line that start with httpStatic:, modify it to your fit your setup and leave it uncommented. Here is how my setup looks like (don´t simply copy and paste, you need to configure your directory here).
// following property can be used to identify a directory of static content
// that should be served at http://localhost:1880/.
//httpStatic: '/home/nol/node-red-static/',
httpStatic: "C://Users/OCM/.node-red/static",
Be careful when selecting the file to edit. According to the words of the experts:
"Most often, when your changes to the settings file are not taking effect, it means you have edited the wrong file. There is one delivered with the installed code, which is then copied into your user directory (typically ~/.node-red). Rickus"
Finally, don´t forget to copy your binary file to this static directory.
After checking all those points it is likely that the outcome of your flow will be the binary file and this can be checked by adding a debug node (configured to display complete msg object) right after the file in node. If you see a buffer in your msg.payload it means that your flow is serving the binary file.
EDIT: Searching Node-RED library I see that there is already an example flow that is compliant with ESP8266httpUpdate HTTP requests.
Link: https://flows.nodered.org/flow/888b4cd95250197eb429b2f40d188185
Related
There's a lot of code in each file, too much to post, so I'm giving you a general idea of what's happening in each file.
index.php
[html dropdown menu code etc.]
scripts.js
[AJAX detects user selection from dropdown, grabs fetch.php which pulls database to generate html code for secondary dropdown selections to put in index.php]
fetch.php
[Generates secondary dropdown code based on user selection and query of the database]
I need to see what exactly is being queried to debug, so I'd like to echo the sql select statement:
$query = "SELECT * FROM databasename WHERE.."
That is in fetch.php when user makes a selection from index.php - How do I do this?
When I deal with AJAX, that I return as JSON, one trick I use is to take advantage of output buffering. You can't just echo or output anything you want because it will mess up the JSON data so for an example,
ob_start(); //turn on buffering at beginning of script.
.... other code ...
print_r($somevar);
.... other code ...
$debug = ob_get_clean(); //put output in a var
$data['debug'] = $debug;
header('Content-Type: application/json');
echo json_encode($data); //echo JSON data.
What this does, is wrap any output from you script into your JSON data so that it's format is not messed up.
Then on the javascript side you can use console.log
$.post(url, input, function(data){
if(data.debug) console.log(data.debug);
});
If you are not used to debugging with console.log(), you can usually hit F12 and open the debugger in most browsers. Then in there the output will be sent to the "console". IE9 had a bit of an issue with console.log() if I recall, but I don't want to go to far off track.
NOTE: Just make sure to not leave this stuff in the code when you move it to production, its very simple to just comment this line out,
//$data['debug'] = $debug;
And then your debug information wont be exposed in production. There are other ways to automatically do this, but it depends on if you do development local then publish to the server. For example you can switch it on the $_SERVER['SERVER_ADDR']; which will be ::1 or 127.0.0.1 when it's local. This has a few drawbacks, mainly the server address is not available from the Command Line Interface (CLI). So typically I will tie it into a global constant that says what "mode" the site is in (included in the common entry point, typically index.php).
if(!defined('ENV_DEVELOPMENT')) define('ENV_DEVELOPMENT','DEVELOPMENT');
if(!defined('ENV_PRODUCTION')) define('ENV_PRODUCTION','PRODUCTION');
if(!defined('ENVIRONMENT')) define('ENVIRONMENT',ENV_DEVELOPMENT);
//site is in Development mode, uncomment for production
//if(!defined('ENVIRONMENT')) define('ENVIRONMENT',ENV_DEVELOPMENT);
Then it is a simple matter to check it:
if(ENVIRONMENT == ENV_PRODUCTION ) $data['debug'] = $debug;
If you know how to use error reporting you can even tie into that using
if(ini_get('display_errors') == 1) $data['debug'] = $debug;
Which will only show the debug when display errors is on.
Hope that helps.
UPDATE
Because I mentioned it in the comments, here is an example of it wrapped in a class (this is a simplified version, so I didn't test this)
class LibAjax{
public static function respond($callback, $options=0, $depth=32){
$result = ['userdata' => [
'debug' => false,
'error' => false
]];
ob_start();
try{
if(!is_callable($callback)){
//I have better exception in mine, this is just more portable
throw new Exception('Callback is not callable');
}
$callback($result);
}catch(\Exception $e){
//example 'Exception[code:401]'
$result['userdata']['error'] = get_class($e).'[code:'.$e->getCode().']';
//if(ENVIRONMENT == ENV_DEVELOPMENT){
//prevents leaking data in production
$result['userdata']['error'] .= ' '.$e->getMessage();
$result['userdata']['error'] .= PHP_EOL.$e->getTraceAsString();
//}
}
$debug = '';
for($i=0; $i < ob_get_level(); $i++){
//clear any nested output buffers
$debug .= ob_get_clean();
}
//if(ENVIRONMENT == ENV_DEVELPMENT){
//prevents leaking data in production
$result['userdata']['debug'] = $debug;
//}
header('Content-Type: application/json');
echo self::jsonEncode($result, $options, $depth);
}
public static function jsonEncode($result, $options=0, $depth=32){
$json = json_encode($result, $options, $depth);
if(JSON_ERROR_NONE !== json_last_error()){
//debug is not passed in this case, because you cannot be sure that, that was not what caused the error. Such as non-valid UTF-8 in the debug string, depth limit, etc...
$json = json_encode(['userdata' => [
'debug' => false,
'error' => json_last_error_msg()
]],$options);
}
return $json;
}
}
Then when you make a AJAX response you just wrap it like this (note $result is pass by reference, this way we don't have to do return, and in the case of an exception we update $result in "real time" instead of on completion)
LibAjax::respond( function(&$result){
$result['data'] = 'foo';
});
If you need to pass additional data into the closure don't forget you can use the use statement, like this.
$otherdata = 'bar';
LibAjax::respond( function(&$result) use($otherdata){
$result['data'][] = 'foo';
$result['data'][] = $otherdata;
});
Sandbox
This handles catching any output and puts it in debug, if the environment is correct (commented out). Please pleas make sure to implement some kind of protection so that the output is not sent to clients on production, I cant stress that enough. It also catches any exceptions puts it in error. And it also handles the header and encoding.
One big benefit to this is consistent structure to your JSON, you will know (on the client side) that if if(data.userdata.error) then you have an exception on the back end. It gives you one place to tweak your headers, JSON encoding etc...
One note in PHP7 you'll have to or should add the Throwable interface (instead of Exception). If you want to catch Error and Exception classes Or do two catch blocks.
Let's just say I do a lot of AJAX and got sick of re-writing this all the time, my actual class is more extensive then this, but that's the gist of it.
Cheers.
UPDATE1
One thing I had to do for things to display was to parse the data variable before I console.log() it
This is typically because you are not passing the correct header back to the browser. If you send (just before calling json_encode)
header('Content-Type: application/json');
This just lets the browser know what type of data it is getting back. One thing most people forget is that on the web all responses are done in text. Even images or file download and web pages. It's all just text, what makes that text into something special is the Content-Type that the browser thinks it is.
One thing to note about header is you cannot output anything before sending the headers. However this plays well with the code I posted because that code will capture all the output and send it after the header is sent.
I updated the original code to have the header, I had it in the more complex class one I posted later. But if you add that in it should get rid of the need to manually parse the JSON.
One last thing I should mention I do is check if I got JSON back or text, you could still get text in the event that some error occurs before the output buffering is started.
There are 2 ways to do this.
If Data is a string that needs to be parsed
$.post(url, {}, function(data){
if( typeof data == 'string'){
try{
data = $.parseJSON(data);
}catch(err){
data = {userdata : {error : data}};
}
}
if(data.userdata){
if( data.userdata.error){
//...etc.
}
}
//....
}
Or if you have the header and its always JSON, then its a bit simpler
$.post(url, {}, function(data){
if( typeof data == 'string'){
data = {userdata : {error : data}};
}
if(data.userdata){
if( data.userdata.error){
//...etc.
}
}
//....
}
Hope that helps!
UPDATE2
Because this topic comes up a lot, I put a modified version of the above code on my GitHub you can find it here.
https://github.com/ArtisticPhoenix/MISC/blob/master/AjaxWrapper/AjaxWrapper.php
Echo the contents and do a die() or exit; afterwards... then in the Network tab of your browser, start it recording, run the Ajax request (it'll fail) but check the resource/name and then view the Response, and it'll show you what was echo'd in the script
Taken from: Request Monitoring in Chrome
Chrome currently has a solution built in.
Use CTRL+SHIFT+I (or navigate to Current Page Control > Developer > Developer Tools.
In the newer versions of Chrome, click the Wrench icon > Tools > Developer Tools.) to enable the Developer Tools.
From within the developer tools click on the Network button. If it isn't already, enable it for the session or always.
Click the "XHR" sub-button.
Initiate an AJAX call.
You will see items begin to show up in the left column under "Resources".
Click the resource and there are 2 tabs showing the headers and return content.
Other browsers also have a Network tab, but you will need to use what I commented to get the string value of the query.
ArtisticPhoenix solution above is delightful.
I have a website/app that can operate in a standard web hosting environment and on the Google App Engine. So I develop on Cloud9, deploy test versions (via GitHub) to an appspot.com URL, and ultimately to the GAE project that's associated with the custom domain.
I wanted web crawlers to be entirely disallowed on the development sites and to have specified access on the production site. But I didn't want to have the task of managing different versions of a robot.txt file.
What I developed is posted as an answer below. Perhaps it will be helpful to others or perhaps someone has a more elegant solution.
p.s. I read up on the most appropriate etiquette for someone asking and immediately answering their own question. There was a difference of opinion primarily divided between those who believed the answer should remain in the question (so as not to appear as if one was trying to increase their own reputation) versus posting an official answer. I'm choosing the latter based on the logic that if a person is looking for an answer and sees a posting with 0 answers, they won't know that there might be a solution they could consider.
My solution is to have PHP generate a response.
Since I have no other reason to serve text files, app.yaml file is modified so that a request to any .txt extension triggers the primary script (which is included by index.php)...
- url: /(.*\.(txt$))
script: index.php
The PHP script parses the URL...
$argument = basename(parse_url($_SERVER["REQUEST_URI"], PHP_URL_PATH));
if ($argument !== "" && $argument !== "index.php") {
if ($argument == "robots.txt") {
$action = "answerRobot";
}
elseif ($argument == "_________") {
// Other things I look for such as translating directory
// requests into actions. For instance website.com/admin
// (a non-existent directory) triggers an admin function
// in the main script.
}
}
I then check to see what environment I'm in — first whether I'm running in GAE and then which project...
if (strpos(getcwd(), "/base/data/home/apps/s~____") > -1) {
$appEngine = true;
if (strpos($_SERVER["APPLICATION_ID"], "dev") > -1) {
$crawlable = false;
}
else {
$crawlable = true;
}
}
Finally, I generate robots.txt content if required...
if ($action == "answerRobot") {
header("Content-type: text/plain");
if ($crawlable) {
$content = "User-agent: *"
. "\nDisallow: /code/"
. "\nDisallow: /icons/"
. "\nDisallow: /specific_file.ext"
. "\nDisallow: /specific_file.ext"
;
}
else {
$content = "User-agent: *"
. "\nDisallow: /"
;
}
echo $content;
exit();
}
There are 8000 html files in my data directory.I parsed all the html files via traversing the target directory and to call the parse function.The filename will be displayed when to be parsed in my program.
<?php
base_dir="c:\data";
function parse($fname){
//to parse data from the file,omitted codes
echo $fname;
}
foreach(new RecursiveDirectoryIterator(base_dir) as $item){
parse($item);
}
?>
I found that the filenames will be displayed on the web page until my whole php file was executed,the filenames will be displayed in 10 minutes ,filename is not displayed one by one when one file is pared over.
How can i display the result instantly on the web page one by one,not wait until the whole php file was finished?
It is important for me to get the file name dispalyed that i know if there is problem in my code,if long time passed ,nothing displayed in the web page will make me nervous.
If you need just to trace your program execution you could use error_log($fname) command instead of 'echo'. Less effort than AJAX and you could track execution 'live' by paste in Ubuntu terminal:
tail -F /var/log/apache2/error.log
(on other *nix path may be different).
1. PLAIN
The simple way is to disable any output buffering, and use flush for good measure. Also, disabling compression.
<?php
while (ob_get_level()) {
ob_end_clean(); // or ob_end_flush() if you want to display previous output from this script.
}
...
print "Whatever\n";
flush();
The above allows for minimal modification to your current code. It has few guarantees since the actual workflow between the parse process and your browser might contain entities (proxies, etc.) on which you have little or no control; if you execute external programs, those might be fully buffered instead of line buffered; and so on.
2. AJAX
A more complicated (but satisfying) way is, if possible, break this into two chunks. This also poses a security risk that needs addressing.
In the main PHP file you output a Javascript variable (inside a Javascript block).
print "var filesToProcess = [\n";
print implode(",", array_map(
function($rawFileName) {
return '"' . /*javascriptQuote*/($rawFileName) . '"';
},
$filesToProcess
));
print "];\n";
This will make available to the client side Javascript an array with all the files.
You can now do the processing in AJAX:
function processOneFile() {
if (!window.filesToProcess.length) {
return;
}
// jQuery
$.post('/path/to/processor.php',
{ file: window.filesToProcess.pop() },
function(result) {
// ... see below
}
).always(function(){
window.setTimeout('processOneFile', 100);
});
}
window.setTimeout('processOneFile', 100);
This will call a PHP file with one file to process after the other. The result must be returned in JSON:
Header("Content-Type: application/json;charset=UTF-8");
die(json_encode(array( "status" => "OK", "file" => $file, "problem" => null )));
Security risk: the client is sending along a file name, any file name, and the script is executing something on that. In general you do not know who the client is, or who it may be, so you need to validate the file name (e.g. ascertain its basename does indeed exist in the target directory you sent in the first place):
if (!file_exists($yourDir . DIRECTORY_SEPARATOR . basename($file)) {
Header("Content-Type: application/json;charset=UTF-8");
die(json_encode(array(
"status" => "FAILED",
"file" => $file,
"problem" => "file could not be found",
/* "real-problem" => "Sorry, friend, it didn't work." */
)));
}
The array will be returned to the Javascript function above:
function(result) {
var msg;
if (result.status == "OK") {
msg = result.file + " OK";
} else {
msg = result.file + ": " + result.problem
}
$('#logDiv').append($('<p>').text(msg));
}
The above will transform a HTML entity
<div id="logDiv"></div>
in, say,
<div id="logDiv">
<p>file1.pdf OK</p>
<p>file2.pdf: missing UUID entry</p>
...
</div>
Since you know the initial filesToProcess.length, you can also display a progress bar (there are even jQuery plugins to do that as easy as $('#bar').progress(n*100/initialLength) ).
Or you can run lengthy processes by sending out two log lines, one just before the $.post
$('#logDiv').append($('<p>').text("Parsing " + file));
Do you mean showing results without refreshing webpage?
IF that, you can use ajax to send request.
I am having roblems with locating a PHP script to allow me to obtain the contents of a txt file on a remote server, then output to a variable. Outputting something to a variable is not the hard part. It's the picking up and reading the contents of the file that's the hard part. Anyone have any ideas?
I have trawled the forum and can only locate a method that works locally. Not ideal as the target is remote.
The objective really is, how do I find out if a file exists on the remote server and output a status in html.
Ideas?
Assuming your remote server is accessible by http or ftp you can use file_exists():
if (file_exists("http://www.example.com/somefile.txt")) {
echo "Found it!;
}
or
if (file_exists("ftp:user:password#www.example.com/somefile.txt")) {
echo "Found it!;
}
Use this:
$url = 'http://php.net';
$file_headers = #get_headers($url);
if($file_headers[0] == 'HTTP/1.1 404 Not Found') {
echo "URL does not exist";
}
else {
echo "URL exists";
}
Source: http://www.php.net/manual/en/function.file-exists.php#75064
You can try to use this code:
if (file_exists($path)) {
echo "it exists";
} else {
echo "it does not exist";
}
As you can see $path is the path of your file. Of course you can write anything else instead of those echo.
Accessing files on other servers can be quite tricky! If you have access to the file via ftp, you can use ftp to fetch the file, for example with ftp_fget().
If you do not have access to the file-system via ssh, you only can check the response the server gives when requesting the file. If the server responds with an error 404, the file is either not existent or it is not accessible via http due to the server configuration.
You can check this through curl, see this tutorial for a detailled explanation of obtaining the response code through curl.
I know this is an old thread, but as Lars Ebert points out, checking for the existence of a file on a remote server can be tricky, so checking the server response, using cURL, is how I have been able to do it on our big travel site. Using file_exists() threw an error every time, but checking for a "200 OK" has proved quite successful. Here is the code we are using to check for images for our hotel listings page:
$media_url = curl_init("http://pathto/remote_file.png");
curl_setopt($media_url, CURLOPT_RETURNTRANSFER, true);
$media_img = curl_exec($media_url);
$server_response = curl_getinfo($media_url, CURLINFO_HTTP_CODE);
if($server_response != 200){
echo "pathto/graphics/backup_image.png";
}else{
echo "http://pathto/remote_file.png";
}
Where "http://pathto/remote_file.png" is the remote image we seek, but we need to know whether it is really there. And "pathto/graphics/backup_image.png" is what we display if the remote image does not exist.
I know it's awfully verbose, compared to file_exists(), but it's also more accurate, at least so far.
I am getting an Error when I try to upload data using xmlrpc in wordpress. The code used to work fine but all of a sudden this error started appearing. I have not changed anything in the code.
Error (-32300): transport error - HTTP status code was not 200
Also, I know my script works because google chrome returns an 'ok' status on the GET request.
php.ini has 128mb of memory allocated.
Here is the code that is used to make post
/**
* Make Posts using the XMLRPC classes
*/
function makePosts() {
$data_set = $this->getMovieLinks();
$xml_client = new XMLRPClientWordPress();
foreach ($data_set as $key) {
echo '<pre>';
echo 'This is title movie about to be added ======== : ' . $key['title'];
echo '</pre>';
//new_post($title,$summary,$category,$image_url,$internal_links)
if ($xml_client->new_post($key['title'], $key['summary'], $key['category'], $key['image'], $key['internal_links']) ) {
$status=1;
} else {
$status=0;
}
if (isset($status)) {
echo ' ====== ADDED';
} else {
echo ' ====== ERROR ADDING';
}
}
} // Function makePosts endes here
You can do few things to debug the error.
Take a look in the server logs, maybe they include the real reason for the problem.
Look for "memory_limit" in your php.ini. Try higher number and see whether that's the problem.
Try deactivating one plugin at a time, May be one plugin may causing the error.
I received the same error, finally I found out the reason was I that I enabled such code in .htaccess (XML-PRC server side); I was blocked myself.
order deny,allow
deny from all
allow from 211.111.0.0/16
The server hosted "http://example.com/xmlrpc.php" was blocked post script IP source.
You should:
Add the XML-RPC client script IP to XML-PRC server side; even the client and server in same site.
Or simply remove "deny from all" from .htaccess
I have the same error but i solved it :
I typed http://www.example.com/xmlrpc.php but the good is http://example.com/xmlrpc.php because if it have "www" prefix it redirected with staus cod
If none of the above solutions work:
Make sure you are not being white-listed on the hosting provider. Our client uses wp-engine and we had this exact issue when posting media items.
After making the same request on an outside network, different IP, we got a 200 (OK) status code.