My other two questions didn't go down too well (here and here), due to my confusion and noob-ness; I'll have a final bash to clarify my problem.
I need to send historical trades and signals from my trading terminal. The code is in MQL (C variant) and uses the Wininet.dll. I can send data to my server using this:
string sData, url;
sData = "abc123,etc,etc";
url = "webname.com/PHP/insert.php?testdata="+sData;
int request = InternetOpenUrl (open, url, NULL, 0, 0, 0);
I want to use the insert.php script on my site to read the string that comes after [testdata=] and then insert it into my database for further analysis. This string could be thousands of characters long, which causes concern for URL length limitation.
People have mentioned cURL and jQuery but I don't understand how the above code can be used to simulate a POST request as the data string can get very large depending on the dates I select from my trade journal.
I want to try to do it the correct way but it's just machines talking to each other without any forms, so that's what's confusing me.
If I use this cURL example, how do I pass a long string to the $data variable?
Thanks in advance.
You can use Wininet's HttpSendRequest to accomplish the POST while keeping the URL short with as little GET/query data as possible. You'll also be using HttpOpenRequest. Some sample c++ code is available from another stackoverflow post. You can change the dataPayload to suit your needs.
Success :)))))))))))))
I've managed to write to my database with tens of thousands of characters. I had problems connecting because I was using http:// in the domain name, but when i removed it the code worked and I was able to talk to my script.
MQL (C language)
string myData = "testdata=abc123etc...";
string header = "Content-Type: application/x-www-form-urlencoded";
int open = InternetOpen("HTTP_Client_Sample", 0, "", "", 0);
int connect = InternetConnect(open, "website.com", 80, "", "", 3, 0, 1);
int request = HttpOpenRequest(connect, "POST", "/PHP/insert.php", NULL, NULL, acceptTypes, 0, 1);
HttpSendRequest(request, header, StringLen(header), myData , StringLen(myData));
PHP
Notice how i'm now able to use $_POST method which has a limitation of:
suhosin.post.max_value_length 1000000
More than enough for my needs. Before, my PHP script used $_GET which can only read 512 chars.
include("connect.php"); //Connect to the database
$data = $_POST['testdata'];
$result = mysql_query("INSERT INTO test (testdata) VALUES ('$data')");
if ($result) echo $data;
else echo "Error ".$mysqli->error;
mysql_close();
Thank you for everyone's help, it's greatly appreciated :)
Related
I have a MySQL table which can contain up to 500 000 rows and I am calling them on my site without any LIMIT clause; when I do it this without AJAX, it works normally, but with AJAX , again without setting LIMIT, no data is returned. I checked the AJAX code and there is no mistake there. The thing is , when I write a limit, for example 45 000 , it works perfectly; but above this, ajax returns nothing.
With limit
witohut the limit :
Can this be a ajax issue because i found nothing similar on the web or something else?
EDIT
here is the sql request
SELECT ans.*, quest.inversion, t.wave_id, t.region_id, t.branch_id, quest.block, quest.saleschannelid, b.division, b.regionsid, quest.yes, quest.no FROM cms_vtb as ans
LEFT JOIN cms_vtb_question as quest ON ans.question_id=quest.id
LEFT JOIN cms_task as t ON t.id=ans.task_id
LEFT JOIN cms_wave as w ON w.id=t.wave_id
LEFT JOIN cms_branchemployees as b ON b.id=t.branchemployees_id WHERE t.publish='1' AND t.concurent_id='' AND ans.answer<>'3' AND w.publish='1' AND quest.questhide<>1 ORDER BY t.concurent_id DESC LIMIT 44115
the php :
var url='&module=ajax_typespace1&<?=$base_url?>';
$.ajax({
url: 'moduls_ajax.php?'+url,
cache: false,
dataType:'html',
success: function(data)
{
$("#result").html(data);
}
});
Apparently it was a server error, adding ini_set('memory_limit', '2048M'); helped a lot
The reason this happens has to do with how you format the data sent to the client. Not having seen the code of moduls_ajax.php, I can only suspect that you are probably assembling the query result into a variable - possibly in order to json_encode it properly?
But doing so may result in a huge memory allocation, whereas if you send the data piece by piece to the Web server, you may need a fraction of the memory only.
The same happens on your web page where the same query is either output straight on, or is not being encoded. In the latter case, you'll discover that when the row number grows to about two or three times the current value, the working Web page will stop also.
For example:
$result = array();
while ($tuple = $resultset->fetch()) {
$result[] = $tuple;
}
print json_encode($result);
Instead - of course, it's more complicated than before -
// Since we know it is an array with numeric keys, the JSON
// will be of the format [ <item>, <item>,...,<item> ]
$sep = '[';
while ($tuple = $resultset->fetch()) {
print $sep . json_encode($tuple);
$sep = ',';
}
print ']';
Pros and cons
This is about three times as expensive as a single function call, and can also yield a slightly worse compression performance (the web browser may receive the data in chunks of different size and find more difficulty in compressing them optimally; it's a matter of tenths of one percent, usually). On the other hand, in some setups the output will arrive much more quickly to the client browser and possibly prevent browser timeouts.
The memory requirements, if the tuples are all more or less of the same size, is around two to three N-ths of before - if you have one thousand rows, and needed one gigabyte to be able to process the query, now three-four megabytes ought to suffice. Of course, this also means that the more rows, the better... and the less rows, the less point there is in doing this.
More of the same
The same approach holds for other kind of assembling (to HTML, CSV and so on).
In some cases it may be helpful to dump the data into an external temporary file and send a Location header to have it loaded by the browser. Sometimes it is possible (if PHP is compiled as an Apache module on a Unix system) to output the file after having deleted it, so that it's not necessary to do garbage collection on the temporary files:
$fp = fopen($temporary_file, 'r');
unlink($temporary_file); // The file is deleted, the handle remains valid
fpassthru($fp); // On some platforms this results in the browser being "short-circuited" to the file descriptor, so that the PHP script may terminate while output continues normally.
die();
I am having an issue converting an image stored as base64 in a PostgreSQL database into an image to display on a website. The data type is bytea and I need to get the data via cURL.
I am working with an API to connect to a client's stock system which returns XML data.
I know storing images this way in a DB is not a great idea but that's how the client's system works and it can't be changed as it is a part of an enterprise solution provided by a 3rd Party.
I'm using the following to query the DB for the PICTURE field from the PICTURE table where the PART = 01000015
$ch = curl_init();
$server = 'xxxxxx';
$select = 'PICTURE';
$from = 'picture';
$where = 'part';
$answer = '01000015';
$myquery = "SELECT+".$select."+FROM+".$from.'+WHERE+'.$where."+=+'".$answer."'";
//Define curl options in an array
$options = array(CURLOPT_URL => "http://xx.xxx.xx.xx/GetSql?datasource=$server&query=$myquery+limit+1",
CURLOPT_PORT => "82",
CURLOPT_HEADER => "Content-Type:application/xml",
CURLOPT_RETURNTRANSFER => TRUE
);
//Set options against curl object
curl_setopt_array($ch, $options);
//Assign execution of curl object to a variable
$data = curl_exec($ch);
//Close curl object
curl_close($ch);
//Pass results to the SimpleXMLElement function
$xml = new SimpleXMLElement($data);
//Return String
echo $xml->row->picture;
The response I get from this is: System.Byte[]
Thus if I use base64_decode() in PHP I am obviously just decoding the string "System.Byte[]".
I am guessing that I need to use the DECODE() function in PostgreSQL to convert the data in the query? However, I've tried loads of combinations but I'm stuck. I've had a few downvotes for questions and I'm not too sure why so if this is a bad question I'm sorry, I just really need some help with this one.
(nb:I've replaced the IP and $server with xxxxx for security)
To explain further:
The client has a POS system which is based on ASP.NET and saves the data as XML files on the remote server. I have access to this data via an API which includes a SQL query function using HTTP/cURL defined as follows:
http://remoteserver:82/pos.asmx.GetSql?datasource=DATASOURCE&query=MYQUERY
So to get the field that contains the picture data I am currently usingthe above code.
The query is in the CURL URL i.e. http://remoteserver:82/pos.asmx.GetSql?datasource=12345&query=SELECT+*+FROM+picture+WHERE+part+=+'01000015'";
However, this returns System.Byte[] instead of encoded data which I can then decode in PHP.
Additional info:
PostgreSQL version: PostgreSQL 9.1.3 on i686-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-51), 32-bit
Table Schema:
Available here: http://i.stack.imgur.com/sc8Gw.png
You should preferably have the server storing the data in PostgreSQL as a bytea field, then encoding to base64 to send to the client, but it sounds like you don't control the server.
The string System.Byte[] suggests it's an app using .NET, like ASP.NET or similar, and it's not correctly handling a bytea array. Instead of formatting it as base64 for output it's embedding the type name in the output.
You can't fix that on the client side, because the server is sending the wrong data.
You'll need to show the server-side tables and queries.
Update after query amended:
You're storing a bytea and returning it directly. The client doesn't seem to understand byte arrays and tries to output it naïvely, probably something like casting it to a string. Since the documentation says it expects "base64" you should probably provide that, instead of a byte array.
PostgreSQL has a handy function to base64-encode bytea data: encode.
Try:
SELECT
account, company, date_amended,
depot, keyfield, part,
encode(picture, 'base64') AS picture,
picture_size, source
FROM picture
WHERE part = '01000015'
The formating isn't significant, it just makes it easier to read here
I'm creating a C# to PHP Data Connector to allow for a standardized connection to a web server to host data from a database to a C# WinForm application. Everything is working with this one small exception.
The basic of use is this.
C# sends an AES encrypted command to the server. The server parses the command and performs the SQL query and returns an AES encrypted string. This string is then converted to a DataTable in C#.
When the SQL contains a column that is a BLOB I'm only getting back a small part of the full data. It seems that the field is being limited to only the first 2792 bytes.
Is there a setting that is preventing the full contents of the BLOB to be returned?
I'm not sure if it will be helpful, but here is the code that does the work.
$DataConnection = new PDO('mysql:host=10.10.100.102;dbname=jmadata', "root", "nbtis01");
$DataConnection->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
if (isset($Parameters['SQLQuery'])) { // Default List
$SQLQuery = $Parameters['SQLQuery'];
unset($Parameters['SQLQuery']);
}
if (isset($Parameters['LimitOverride'])) {
if (!strpos(strtoupper($SQLQuery), "LIMIT"))
$SQLQuery = rtrim($SQLQuery, ';') . " LIMIT " . $Parameters['LimitOverride'];
unset($Parameters['LimitOverride']);
}
$QueryParams = array();
foreach ($Parameters as $key => $value)
if ($key !== '')
$QueryParams[$key] = $value;
$Query = $DataConnection->prepare($SQLQuery);
$Query->execute($QueryParams);
$ReturnArray = $Query->fetchAll(PDO::FETCH_ASSOC);
if (!$ReturnArray)
$ReturnArray[0] = array("NoResults" => "");
EDIT -- ANSWER
I found my issue. The problem had nothing to do with PDO, PHP or MySQL. I was taking the BLOB data and doing a Base64 to it before putting it in the array, as the split characters I was using to build the result string that would be converted to datatable in c# used non-printable characters and the binary data as a string might have included these characters. The issue was when I was doing a convert in c# to get the original string so that could convert that to a byte array. I was using System.Text.Encoding.ASCII.GetString to convert the Base64 byte array to the original string. This was working on everything but the binary data from the BLOB fields.
The suggestion that it might be a terminating character is what made me find it. Once the Base64 was converted to string using ASCII there was something that was turning into a terminator and it was stopping the convert at that point. Once I found this I changed to System.Text.Encoding.Default.GetString and now it works perfect.
Posted the answer in case anyone else might be trying to do this and having this same issue.
More details in the Edit of the question.
Changed from System.Text.Encoding.ASCII.GetString to System.Text.Encoding.Default.GetString and the issue was resolved.
Thank you crush for pointing me in the right direction to find the solution.
Is it possible to run a Python script within PHP and transferring variables from each other ?
I have a class that scraps websites for data in a certain global way. i want to make it go a lot more specific and already have pythons scripts specific to several website.
I am looking for a way to incorporate those inside my class.
Is safe and reliable data transfer between the two even possible ? if so how difficult it is to get something like that going ?
You can generally communicate between languages by using common language formats, and using stdin and stdout to communicate the data.
Example with PHP/Python using a shell argument to send the initial data via JSON
PHP:
// This is the data you want to pass to Python
$data = array('as', 'df', 'gh');
// Execute the python script with the JSON data
$result = shell_exec('python /path/to/myScript.py ' . escapeshellarg(json_encode($data)));
// Decode the result
$resultData = json_decode($result, true);
// This will contain: array('status' => 'Yes!')
var_dump($resultData);
Python:
import sys, json
# Load the data that PHP sent us
try:
data = json.loads(sys.argv[1])
except:
print "ERROR"
sys.exit(1)
# Generate some data to send to PHP
result = {'status': 'Yes!'}
# Send it to stdout (to PHP)
print json.dumps(result)
You are looking for "interprocess communication" (IPC) - you could use something like XML-RPC, which basically lets you call a function in a remote process, and handles the translation of all the argument data-types between languages (so you could call a PHP function from Python, or vice versa - as long as the arguments are of a supported type)
Python has a builtin XML-RPC server and a client
The phpxmlrpc library has both a client and server
There are examples for both, Python server and client, and a PHP client and server
Just had the same problem and wanted to share my solution. (follows closely what Amadan suggests)
python piece
import subprocess
output = subprocess.check_output(["php", path-to-my-php-script, input1])
you could also do: blah = input1 instead of just submitting an unnamed arg... and then use the $_GET['blah'].
php piece
$blah = $argv[1];
if( isset($blah)){
// do stuff with $blah
}else{
throw new \Exception('No blah.');
}
The best bet is running python as a subprocess and capturing its output, then parsing it.
$pythonoutput = `/usr/bin/env python pythoncode.py`;
Using JSON would probably help make it easy to both produce and parse in both languages, since it's standard and both languages support it (well, at least non-ancient versions do). In Python,
json.dumps(stuff)
and then in PHP
$stuff = json_decode($pythonoutput);
You could also explicitly save the data as files, or use sockets, or have many different ways to make this more efficient (and more complicated) depending on the exact scenario you need, but this is the simplest.
For me the escapeshellarg(json_encode($data)) is giving not exactly a json-formatted string, but something like { name : Carl , age : 23 }.
So in python i need to .replace(' ', '"') the whitespaces to get some real json and be able to cast the json.loads(sys.argv[1]) on it.
The problem is, when someone enters a name with already whitespaces in it like "Ca rl".
I have a torrent hash from the magnet link. For example: fda164e7af470f83ea699a529845a9353cc26576
When I try to get information about leechers and peers I should request: http://tracker.publicbt.com/scrape?info_hash=???
How should I convert info hash for this request? Is it url encoding or becoding? how? In PHP.
It's a raw hexadecimal representation. Use pack() with H to convert it. Then URL encode it.
Got this python snippet from a colleague,
r = ''
s = 'fda164e7af470f83ea699a529845a9353cc26576'
for n in range(0, len(s), 2):
r += '%%%s' % s[n:n+2].upper()
print r
Output: %FD%A1%64%E7%AF%47%0F%83%EA%69%9A%52%98%45%A9%35%3C%C2%65%76
Works like a charm.
Edit: Works like a charm for getting the tracker to give back status 200 (ok) but still doesn't work for retrieving the torrent details...
In case someone is having trouble and comes across this thread in the future: the trick to this whole issue is to use the bool $raw_output argument of the PHP: sha1 function, setting it to "true".
The BDecode/DEncode classes can be found HERE. This project, called Trackon, also includes many other helpful classes to interact with torrent trackers and files.
So, in PHP, something like this will work to obtain the correct info hash for scraping the tracker for details:
include('./path/to/BDecode.php');
include('./path/to/BEncode.php');
function getHash($torFile){
$tfile = BDecode(file_get_contents($torFile));
$infohash = sha1(BEncode($tfile["info"]), TRUE);
return urlencode($infohash);
}
Then merely call it like so:
$hash = getHash('./path/to/.torrent');
Hope this helps someone out there. I was still scratching my head after reading many posts about how to obtain the correct info hash. I understand why this wasn't mentioned anywhere now though, this argument was added in PHP 5. If you're not running PHP 5, you will have to convert the sha1 hash to raw binary after you calculate it.