python how to write data into a single file with different call - php

I have a python script and a PHP page on my website. I have to create a csv file and write the data onto it in following manner-
PHP processes the GET requests and calls the python script by shell_exec, passing the GET variables into it, is JSON format. The script here looks like-
...
$var_a = $_GET['a'];
$var_b = $_GET['b'];
$var_c = $_GET['c'];
$post_data = array('item' => 'get-list',
'var a' => "$var_a",
'var b' => "$var_b",
'var c' => "$var_c"
);
$json_data = json_encode($post_data);
$temp = shell_exec('python process_data.py "' . $json_data .'"');
echo $temp;
...
This script is quite straight forward, which does simply collects the GET variables and forms JSON object of it. Now this data is passed to process_data.py, which processes the data and saves the data into a csv file, the code for which is-
import sys, json, time, csv
json_data = json.dumps(sys.argv[1])
def scr(json_data):
json_data = json_data
count = 0
json_list = json_data.split(',')
for i in json_list:
count=count+1
if(count==5):
lati = json_list[0].strip('{')
longi = json_list[1]
datestmp = json_list[2]
timestmp = json_list[3]
out = open('file.csv', 'w')
out.write('%s;' % lati.split(':')[1])
out.write('\n')
out.write('%s;' % longi.split(':')[1])
out.write('\n')
out.write('%s;' % datestmp.split(':')[1])
out.write('\n')
out.write('%s;' % timestmp.split(':',1)[1])
out.write('\n')
out.close()
print 'Latitude: ',lati.split(':')[1],', Longitude: ',longi.split(':')[1],', Datestamp: ',datestmp.split(':')[1],', Time: ', timestmp.split(':',1)[1]
else:
print 'Wrong data received'
scr(json_data)
All the things are working well. The data is processed and saved. The PHP script also saves the current data into database. So PHP does two operations-
1. Saves the data into database
2. Pass the data to python script which saves the data into CSV file.
However, the GET data is coming at regular time-interval(say 1 or 2 seconds). So, the call to Python file gets regular for these intervals. It will have to open the CSV file every 1-2 seconds, write the data there. This will slow down the processing.
I want the CSV file to remain open till the data is coming, and close it when the GET request is not made for a long time. Is there any way to do so?

Related

Export php array to json with a specific structure (google charts)

I'm trying to use the google charts api to present power (watt) over time. I have a sqlite database in which i store my data. A php script then gathers the data and outputs it into a json file. For the charts to work i need to keep a specific structure. Whenever i run the phpscript the json file gets overwritten with the new data. I need help with the php script to always output the data according to googles parameters.
I'm aiming to end up with an area chart that plots power on the y axis and the datestamps on the x axis.
I've read theese documents in googles documentation but i can't figure out how to output the data the way they do.
https://developers.google.com/chart/interactive/docs/php_example
https://developers.google.com/chart/interactive/docs/reference#dataparam
while ($res = $stmt->fetch(PDO::FETCH_ASSOC)){
$voltage[] = $res['voltage'];
$current[] = $res['current'];
$datestamp[] = $res['datestamp'];
}
$unixtimestamp = array();
foreach ($datestamp as $nebulosa){
$unixtimestamp[] = strtotime($nebulosa);
}
$power = array();
foreach ($voltage as $key => $door){
$power[] = $door * $current[$key];
}
//echo "<pre>", print_r($power, true), "</pre>";
$fp = fopen('data.json', 'w');
fwrite($fp, json_encode($power));
fwrite($fp, json_encode($datestamp));
fclose($fp);
The json file has this format after running the php script.
[3.468,5]["2016-10-14 14:56:22","2016-10-14 14:56:23"]

Getting all from json api array using php

I wanna improve on how to fetch data from an API. In this case I want to fetch every app-id from the Steam API, and list them one per line in a .txt file. Do I need an infinite (or a very high-number) loop (with ++ after every iteration) to fetch everyone? I mean, counting up from id 0 with for example a foreach-loop? I'm thinking it will take ages and sounds very much like bad practice.
How do I get every appid {"appid:" n} from the response of http://api.steampowered.com/ISteamApps/GetAppList/v0001?
<?php
//API-URL
$url = "http://api.steampowered.com/ISteamApps/GetAppList/v0001";
//Fetch content and decode
$game_json = json_decode(curl_get_contents($url), true);
//Define file
$file = 'steam.txt';
//This is where I'm lost. One massive array {"app": []} with lots of {"appid": n}.
//I know how to get one specific targeted line, but how do I get them all?
$line = $game_json['applist']['apps']['app']['appid'][every single line, one at a time]
//Write to file, one id per line.
//Like:
//5
//7
//8
//and so on
file_put_contents($file, $line, FILE_APPEND);
?>
Any pointing just in the right direction will be MUCH appreciated. Thanks!
You don't need to worry about counters with foreach loops, they are designed to go through and work with each item in the object.
$file = "steam.txt";
$game_list = "";
$url = "http://api.steampowered.com/ISteamApps/GetAppList/v0001";
$game_json = file_get_contents($url);
$games = json_decode($game_json);
foreach($games->applist->apps->app as $game) {
// now $game is a single entry, e.g. {"appid":5,"name":"Dedicated server"}
$game_list .= "$game->appid\n";
}
file_put_contents($file, $game_list);
Now you have a text file with 28000 numbers in it. Congratulations?

PHP Write Posted Data to File

I'm relatively new to PHP and I'm trying to get a small script running. I have a VB .net program that posts data using the following function.
Public Sub PHPPost(ByVal User As String, ByVal Score As String)
Dim postData As String = "user=" & User & "&" & "score=" & Score
Dim encoding As New UTF8Encoding
Dim byteData As Byte() = encoding.GetBytes(postData)
Dim postReq As HttpWebRequest = DirectCast(WebRequest.Create("http://myphpscript"), HttpWebRequest)
postReq.Method = "POST"
postReq.KeepAlive = True
postReq.ContentType = "application/x-www-form-urlencoded"
postReq.ContentLength = byteData.Length
Dim postReqStream As Stream = postReq.GetRequestStream()
postReqStream.Write(byteData, 0, byteData.Length)
postReqStream.Close()
End Sub
Where "myphpscript" is acutally the full URL to the PHP script. Basically I'm trying to POST the "User" variable and the "Score" variable to the PHP script. The script I've tried is as follows:
<?php
$File = "scores.rtf";
$f = fopen($File,'a');
$name = $_POST["name"];
$score = $_POST["score"];
fwrite($f,"\n$name $score");
fclose($f);
?>
The "scores.rtf" does not change. Any help would be appreciated. Thanks ahead of time, I'm new to PHP.
Ensure that your script is receiving the POST variables.
http://php.net/manual/en/function.file-put-contents.php
You can try file_put_contents, it combines the use of fopen ,fwrite & fclose.
It could be wise to use something like isset/empty to check that there is something to write before writing.
<?php
$file = 'scores.rtf';
// Open the file to get existing content
$current = file_get_contents($file);
// Append a new person to the file
$current .= print_r($_POST);
//Once confirmed remove the above line and use below
$current .= $_POST['name'] . ' ' . $_POST['score'] . "\n";
// Write the contents back to the file
file_put_contents($file, $current);
?>
Also, totally overlooked the RTF part, definitely look into what Mahan mentioned. I'd suggest the above if you have no need for that specific file type.
The "scores.rtf" does not change.
well RTF files is handled differently because its not really a pure text file it contains meta-data and tags that controls how text is displayed on the rtf file. please have a time to read to the following sources
http://www.webdev-tuts.com/generate-rtf-file-using-php.html
http://b-l-w.de/phprtf_en.php
http://paggard.com/projects/doc.generator/doc_generator_help.html
if in any case you want a normal text file you can use the code below, don't use fwrite(), please use file_put_contents()
file_put_contents("scores.txt", "\n$name $score");

FLASH AS2 and PHP variables

My Flash movie reads and sends data to a PHP file in a free server. It seems ok for Flash to read variable values from a text file (which is managed by a PHP file) if they are wrote in this way: &variable = value&, I have no problem with that.
But my PHP file, pre-treated (by some mathematical functions) data sent by Flash and then, updates the values in the text file, that is my intention but I can't accomplish it.
Suppose I want to update a counter ( it counts how many times the data were updated):
in the text file I have &counter=0& (initial value) and if I put in the PHP file:
<?php
$fp = fopen("jose_stats.txt", "r");// I guess with it, I've read all the variables and values
// one of them is the variable &counter.
fclose($fp);
$toSave = "&counter=&counter+1&\n";
$fp = fopen("jose_stats.txt", "w");
if(fwrite($fp, "$toSave")) {
echo "&verify=success&"; //prints to screen &verify=success which flash will read
//and store as myVars.verify
} else { // simple if statement
echo "&verify=fail&"; //prints to screen &verify=fail which flash will read and
//store as myVars.verify
}
fclose($fp);
?>
but then, I check my text file and it has &counter=&counter+1& line :( and not the expected &counter =1&.
Please, give me and advise. Thank you.
Why not use JSON?
Just store the data in JSON format:
$count = 1;
$toWrite = array( 'count' => $count );//put other data into this array if you want
//encode it
$toWrite = json_encode( $toWrite );
//and now write the data
To decode it in flash, import the JSON class:
An example of JSON in as2 using the JSON.as class:
try {
var o:Object = JSON.parse(jsonStr);
var s:String = JSON.stringify(obj);
} catch(ex) {
trace(ex.name + ":" + ex.message + ":" + ex.at + ":" + ex.text);
}
So just import the class, and run JSON.parse( yourPhpResponse );.
Also, the reason for why you're seeing &counter=& in the text file is because you're storing it like that: $toSave = "&counter=&counter+1&\n";.

PHP MYSQL BLOB export to csv format

I am trying to create a script that can export data from a mysql database containing files (pdf mainly). This data is then imported onto the local version of the system.
However I have a problem with the BLOB field, when I export and import using PHPMYADMIN, it works fine, however when using my script, the BLOB field has additional code added to the top. Almost as if it is code to instruct programs how to deal with it. When i try to import this version into PHPMYADMIN, it doesnt work.
Is is a formatting error, do i need to convert it to a type. At present it is simply pulled from the database as a row['content'] item and then outputted to the csv
Any help would be much appreciated
Regards
///////////////// source code of my script
$file = 'supportingFiles'; // csv name.
$result = mysql_query("SHOW COLUMNS FROM files");
$a = 0;
if (mysql_num_rows($result) > 0) { //print column titles based on number of columns in the two files
while ($row = mysql_fetch_assoc($result)) {
$a++;
}
}
$values = mysql_query("SELECT * FROM files "); //select client details wehere required
while ($rows = mysql_fetch_array($values)) { //print client information
for ($k=0;$k<$a;$k++) {
$csv_output .= $rows[$k].",";
}
$csv_output .= "\n"; //end of line
}
$filename = $file."_".date("d-m-Y_H-i",time());
header("Content-type: application/csv");
header("Content-disposition: csv" . date("Y-m-d") . ".csv");
header( "Content-disposition: filename=supportingFiles.csv");
echo $csv_output; //output data file
when an export is ran using this script the following proceeds the actual content outputted when a csv export of the table is run from phpmyadmin...
%âãÏÓ
%%ISIS AfpToPdf-V.6.2/h3 '2008-05-19 (build:6.20.0.08205)'
4 0 obj
[
/DeviceRGB
]
endobj
5 0 obj
[/Pattern 4 0 R]
endobj
6 0 obj
[
/DeviceCMYK
]
endobj
7 0 obj
[/Pattern 6 0 R]
endobj
14 0 obj
<</Length 1221/Filter/FlateDecode>>
stream
xÚ•WÛnã6ýÿümÌ%)’¢úo6Š6Û„ûTÈckW–RYÎ6ýüh‡CJ¶Ûi€\à™áp.gÇ‹|Æ!ÿ6{wÃA#~‚ãçgø3Í %ph×3ü^ïÿæ[P2cB‚É4Käw?
R<¦èØo-\ïðs³_oüqŽ.yL11°Ø·uåžà}ÕìÜTûáÏüã-\-ó©.ÿ„ÅŸ^þ!Ÿý=óõ‘YÆRƒ=±ÌÀ¬Á$´îÊ”)•TÉ ±6a¶×JŸúXËY¦ÇJµÖ²äìQ«/]k&ÍyÇœyV›¦gµši{^›0c†¨\c-jÏÆlì!ªÚQ! Íä^#YrÁóQ™_juÊ5x¶“˜µaâÂYÉôùZ)ëQÛk³ÉY•2~>_¥Ž1×{þ4[ä’ŒÉÅv2<¾ÿâ{Œ_ïªr…ŒžÂð©©„9øú™FFÄZÄ«ª†1Ö ŒÎ’58Tj˜â‡$áb¯D`²H‡¾
üí</m›ºÛ|¿ƒÕ¦h×n²ã¯—ˆøQc¿¼Ç‰ãvûå·ê k<ÙyJ+¦
"2¦75^‚Ó ãg&x M„ñ4–êØå wÈ°\äå%BÄËØŒ4쉷è÷çßASÃsb¢ñ(¾t_bO¼Gî…›7]Qõe„û¦=ªo:9ø| Ö”3¾2õØéPü74–š 2qÔYle%Ûn‹ö š{xjö-«U³¯»·5Wáhz‹ª¨Wˆ¿¶ÙÂCëËf¿ƒeœ¥ Ÿ”Ã\{IÆðE 9i=î… ð3ŠŠß‹§­«;DõÊ•îî¸ 4ñþYöóÓÎ^¯<ÁeN$…ö±öG ¥§÷kÔå”^]¥Xò2¼cˆ}\>¢LdQÅSA7=6RN¾‘¥B— dXêjë[w{z¿Uâ‡WIÈxVÊK¥`ܬpЗֈx—þ! U¦¨¿ús ½%ÁBŒœãÃ%؃ï¤ïž«p†ËzMÁaÞ6Ê»†"
iç\lMÏ>ÈL®v‘€šû8ê:Ú<kwh
‰h¶l#ì†ý"ô·+ÊjÇN½ü§7\2¥éû=0å‹¡¾ô½&±HW´ƒ„J^…–ǯ"?
that is because CSV is not well standardized as a format (not even the "comma" as in "comma seperated values") and it is because phpmyadmin does some encoding/decoding on the values when exporting/importing.
You need this encoding/decoding part, because your binary BLOB (as you said, mostly PDF) can easily contain commas, line breaks, quotes, and all kinds of stuff that breaks a CSV parser.
To import your files using phpmyadmin, you would have to replicate the encoding machanism used there - lucky you that it's open source and you can have a look at the code.
Alternatively, if you want your own export/import mechanism (lets say: you write your own importer that matches your exporter) then you could make good use of base64 encoding here to ensure your CSV (which is intended as a plain-text format by the way) stores binary data correctly.
exporter:
// convert binary blob to text format
$plaintextdata_for_csv = base64_encode($binarydata_from_blob);
importer:
// decode text format to binary blob
$binarydata_for_blob = base64_decode($plaintextdata_from_csv);

Categories