Im working on a project that downloads up to 5000 individual pieces of data from a server. It basically is a PHP page that takes POST variable, gets the data from the DB and sends it back to the .NET client.
It is slow. It takes about 1 second per request. I've googled a lot and tried all sorts of tweaks to the code, like the famous proxy-setting etc. But nothing speeds it up.
Any idea's? All solutions that make this super fast are welcome. Even C-written DLL's or anything you can think of. This just needs to be a lot faster.
Public Function askServer(oCode As String) As String
oBytesToSend = Encoding.ASCII.GetBytes("cmd=" & System.Web.HttpUtility.UrlEncode(oCode))
Try
oRequest = WebRequest.Create(webServiceUrl)
oRequest.Timeout = 60000
oRequest.Proxy = WebRequest.DefaultWebProxy
CType(oRequest, HttpWebRequest).UserAgent = "XXXXX"
oRequest.Method = "POST"
oRequest.ContentLength = oBytesToSend.Length
oRequest.ContentType = "application/x-www-form-urlencoded"
oStream = oRequest.GetRequestStream()
oStream.Write(oBytesToSend, 0, oBytesToSend.Length)
oResponse = oRequest.GetResponse()
If CType(oResponse, HttpWebResponse).StatusCode = Net.HttpStatusCode.OK Then
oStream = oResponse.GetResponseStream()
oReader = New StreamReader(oStream)
oResponseFromServer = oReader.ReadToEnd()
oResponseFromServer = System.Web.HttpUtility.UrlDecode(oResponseFromServer)
Return oResponseFromServer
Else
MsgBox("Server error", CType(vbOKOnly + vbCritical, MsgBoxStyle), "")
Return ""
End If
Catch e As Exception
MsgBox("Oops" & vbCrLf & e.Message, CType(vbOKOnly + vbCritical, MsgBoxStyle), "")
Return ""
End Try
End Function
Some ideas :
Run the http requests in parallel. (Client)
If the data response size allows it get all data needed in one request (you need change your server implementation).
Caching data. (Server)
Related
i am a beginner. I'm starting to learn Adobe Flash. I followed the code on Google on AS2:
l = new LoadVars();
l.onLoad = function(ok) {
ttl0=l.ttlslv0; atc0=l.atcslv0;
ttl1=l.ttlslv1; atc1=l.atcslv1;
};
l.sendAndLoad("http:localhost/getdata.php", l, "POST");
with php like this:
<?php
include_once("koneksi.php");
$qr = mysqli_query($con,"SELECT title, article from $tabel");
$nr = 0;
while($kolom=mysqli_fetch_row($qr) )
{
$ttl=$kolom[0];
$atc=$kolom[1];
echo "&ttlslv$nr=$ttl&atcslv$nr=$atc";
$nr++;
}
echo "&nr=$nr&";
?>
with the results that I can specify "what row" and "what column" will I take.
can this be changed to AS3 with the same results?
I have difficulty studying AS3
anyone want to give me a solution? thanx...
In AS3 you use the URLLoader class to load text/binary data. To work with URL-encoded strings you need the URLVariables class.
Something like that (not tested, no error handling either, just a common guideline):
// URLRequest instance holds various information
// about what, where and how you send.
var aRequest:URLRequest = new URLRequest;
// URL address.
aRequest.url = "http://localhost/getdata.php";
// Request method (POST or GET mostly).
aRequest.method = URLRequestMethod.POST;
// URLLoader instance performs the requested operations:
// uploads the request and relevant data, reads the answer,
// and dispatches all the events about any status changes.
var aLoader:URLLoader = new URLLoader;
// Tell the URLLoader that the answer is
// an URL-encoded text of key=value pairs.
aLoader.dataFormat = URLLoaderDataFormat.VARIABLES;
// This event handler function will be triggered
// upon successfully completed operation.
aLoader.addEventListener(Event.COMPLETE, onData);
// Start loading.
aLoader.load(aRequest);
// The COMPLETE event handler function.
function onData(e:Event):void
{
// Unsubscribe from the event. For the most part it is
// a GOOD idea NOT to re-use any network-related
// class instances once they've done their job.
// Just unsubscribe from all their events,
// dismantle their data structures, purge
// any references to them and let the
// Garbage Collector do his job.
aLoader.removeEventListener(Event.COMPLETE, onData);
// Retrieve the object that contains key=value pairs.
var anAnswer:URLVariables = aLoader.data as URLVariables;
// The data you wanted to get.
trace(anAnswer.ttlslv0);
trace(anAnswer.atcslv0);
trace(anAnswer.ttlslv1);
trace(anAnswer.atcslv1);
}
UPD: It seems that you are dealing with escaped text there. I composed a simple script that explains how it works:
import flash.net.URLVariables;
var V:URLVariables = new URLVariables;
var S:String = "a=1&b=2";
V.decode(S);
trace(V.a); // 1
trace(V.b); // 2
S = escape(S);
trace(S); // a%3D1%26b%3D2
trace(unescape(S)); // a=1&b=2
V.decode(S); // Error #2101
So, there are two options you can work on:
Figure out why server passes the escaped string and prevent it.
Load the server's answer as a plain text (URLLoaderDataFormat.TEXT instead of VARIABLES) so the URLLoader.data will be just a simple String, then unescape(...) it if necessary and feed to the URLVariables.decode(...).
I have some Session values that I am constantly changing via Ajax calls. I can't seem to get a handle on the POST data to process it and set the values.
What I am passing to it here is an array of strings like is shown in my code below.
Here is where AJAX calls:
var sessionValues = [];
str = {"PID": "1", "Level": "Main", "MenuName": "Kitchen", "State": "CHECKED"}
sessionValues.push(str);
var postObj = {"sessionData": sessionValues};
$.ajax({
type: 'POST',
data: {'data': postObj},
url: 'setSession.asp'
}).done(function(response){
console.log(response);
})
I have this working fine in a PHP version of the program but my ASP version is not grabbing the data. Here is my PHP ver and the ASP ver as best as I could convert it.
<-- php setSession.php works fine -->
$data = $_POST['data'];
foreach ($data['sessionData'] as $key => $value) {
$projectProduct = "1";
$level = $value["Level"];
$menuName = $value["MenuName"];
$state = $value["State"];
$_SESSION['PID:'.$projectProduct][$level][$menuName]['menu_state'] = $state;
echo "[PID:".$projectProduct."][".$level."][".$menuName."][".$state."]<br>";
}
0 =>>>>> Array<br>[PID:1][Main][Kitchen][CHECKED]
Here I want to do the same thing in ASP
' setSession.asp
data = Request.Form("data")
For Each part In data("sessionData")
projectProduct = part("PID")
level = part("Level")
menuName = part("MenuName")
state = part("State")
Session("PID:" & projectProduct).Item(level).Item(menuName).Remove("menu_state")
Session("PID:" & projectProduct).Item(level).Item(menuName).Add "menu_state", state
response.write("[PID:" & projectProduct&"]["&level&"]["&menuName&"]["&state&"]<br>")
Next
outputs blank
It looks like it never has any data but doesn't throw any errors. Am I reading the POST object correctly?
[edit]
Here is the RAW POST data captured from Fiddler:
data%5BsessionData%5D%5B0%5D%5BPID%5D=1&data%5BsessionData%5D%5B0%5D%5BLevel%5D=Main&data%5BsessionData%5D%5B0%5D%5BMenuName%5D=Kitchen&data%5BsessionData%5D%5B0%5D%5BState%5D=CHECKED
here I used a URL Decode on that string-
data[sessionData][0][PID]=1&data[sessionData][0][Level]=Main Level Plan&data[sessionData][0][MenuName]=Kitchen&data[sessionData][0][State]=CHECKED
This looks like I should be able to loop through the strings now by using
For Each part In Request.Form("data[sessionData]")
but nothing happens. I added a simple loop to look at the request.form and here is what it is seeing:
for each x in Request.Form
Response.Write(x)
Next
' outputs -> data[sessionData][0][PID]data[sessionData][0][Level]data[sessionData][0][MenuName]data[sessionData][0][State]
I guess what this comes down to is just reading through and processing that string correctly, or multiple if more than one is sent. Correct?
The RAW output definitely helps work out what is going on.
What is happening is jQuery is translating the JSON structure into HTTP POST parameters but during the process, it creates some overly complex key names.
If you break down the key value pairs you have something like
data[sessionData][0][PID]=1
data[sessionData][0][Level]=Main Level Plan
data[sessionData][0][MenuName]=Kitchen
data[sessionData][0][State]=CHECKED
As far as Classic ASP is concerned the this is just a collection of string key and value pairs and nothing more.
The correct approach to work out what these keys are is to do what you have done in the question, but with some minor alternations.
For Each x In Request.Form
Response.Write(x) & "=" & Request.Form(x) & "<br />"
Next
Which when outputted as HTML will look similar to the break down shown above.
Armed with the knowledge of what the keys are you should be able to reference them directly from the Request.Form() collection.
Dim pid: pid = Request.Form("data[sessionData][0][PID]")
Response.Write pid
Output:
1
Very simply, I have a program that needs to perform a large process (anywhere from 5 seconds to several minutes) and I don't want to make my page wait for the process to finish to load.
I understand that I need to run this gearman job as a background process but I'm struggling to identify the proper solution to get real-time status updates as to when the worker actually finishes the process. I've used the following code snippet from the PHP examples:
do {
sleep(3);
$stat = $gmclient->jobStatus($job_handle);
if (!$stat[0]) // the job is known so it is not done
$done = true;
echo "Running: " . ($stat[1] ? "true" : "false") . ", numerator: " . $stat[2] . ", denomintor: " . $stat[3] . "\n";
} while(!$done);
echo "done!\n";
and this works, however it appears that it just returns data to the client when the worker finished telling the job what to do. Instead I want to know when the literal process of the job finished.
My real-life example:
Pull several data feeds from an API (some feeds take longer than others)
Load a couple of the ones that always load fast, place a "Waiting/Loading" animation on the section that was sent off to a worker queue
When the work is done and the results have been completely retrieved, replace the animation with the results
This is a bit late, but I stumbled across this question looking for the same answer. I was able to get a solution together, so maybe it will help someone else.
For starters, refer to the documentation on GearmanClient::jobStatus. This will be called from the client, and the function accepts a single argument: $job_handle. You retrieve this handle when you dispatch the request:
$client = new GearmanClient( );
$client->addServer( '127.0.0.1', 4730 );
$handle = $client->doBackground( 'serviceRequest', $data );
Later on, you can retrieve the status by calling the jobStatus function on the same $client object:
$status = $client->jobStatus( $handle );
This is only meaningful, though, if you actually change the status from within your worker with the sendStatus method:
$worker = new GearmanWorker( );
$worker->addFunction( 'serviceRequest', function( $job ) {
$max = 10;
// Set initial status - numerator / denominator
$job->sendStatus( 0, $max );
for( $i = 1; $i <= $max; $i++ ) {
sleep( 2 ); // Simulate a long running task
$job->sendStatus( $i, $max );
}
return GEARMAN_SUCCESS;
} );
while( $worker->work( ) ) {
$worker->wait( );
}
In versions of Gearman prior to 0.5, you would use the GearmanJob::status method to set the status of a job. Versions 0.6 to current (1.1) use the methods above.
See also this question: Problem With Gearman Job Status
MongoDB + PHP: How do you query the size of a document stored in a mongodb object? The document limit, at the time of this writing (recently raised from 4MB), is 16MB.
How can I query the size of a Document using the ObjectId?
Object.bsonsize(document) is the function you're after. I'm reading this thread and finding a few suggestions about how to do it in php. The easiest is probably to query for the document object and run strlen on the bson_encode'd object giving you the size in bytes.
$byteSize = strlen( bson_encode( $yourDocument ) );
If you're wanting to do this check on a lot of documents, say in a loop, without reading back every document you'll need to run a command execute (Mardix had posted this little function to do just that - note: it could easily be rewritten a little better to avoid the hardcoded and global variables):
$DBName = "MyDBName";
$MongoDB = new MongoDB(new Mongo(),$DBName);
function documentSize(Array $Criteria){
global $MongoDB;
$collectionName = "MyCollection";
$jsonCriteria = json_encode($Criteria);
$code = "function(){
return Object.bsonsize(db.{$collectionName}.findOne({$jsonCriteria}))
}";
$resp = $MongoDB->execute($code);
return $resp["retval"];
}
PHP example where $id is the document id
$myDocSize = documentSize(array("_id"=>$id));
Hi I have a php script which tries to do a post.
I use Snoopy class and also I use proxy.
I managed to post but when I use a proxy the posting is extremely slow.
I mean it can take till 30 minutes.
I don't want to block my script for 30 minutes waiting for a post.
Any idea how could I solve this?
The code looks like:
require('../includes/Snoopy.class.php');
$snoopy = new Snoopy();
$snoopy->proxy_host = "my.proxy.host";
$snoopy->proxy_port = "8080";
$p_data['color'] = 'Red';
$p_data['fruit'] = 'apple';
$snoopy->cookies['vegetable'] = 'carrot';
$snoopy->cookies['something'] = 'value';
$snoopy->submit('http://phpstarter.net/samples/118/data_dump.php', $p_data);
echo '' . htmlspecialchars($snoopy->results) . '';
var $read_timeout = 0; // timeout on read operations, in seconds
// set to 0 to disallow timeouts
So, you could try to set $snoopy->read_timeout to any reasonable value.