I need detect user's country and show website's language by him / her country . (Turkish for Turkish people, English for all others)
How can i do this fastest way ? Performance is important for me .
I'm looking IPInfoDB' API , are there any better alternative ?
(I'm using PHP)
Well for people who might visit in 2017 this is a solution this extremely simple to use
<button class="btn dropdown-toggle" style="cursor:pointer;z-index:1000!important;margin-top:-67px;background:none!important;font-size:1.4em;" onclick="window.location.href='language'">
(a) <?php
if (!empty($_SERVER['HTTP_CLIENT_IP'])) {
$ip = $_SERVER['HTTP_CLIENT_IP'];}
elseif (!empty($_SERVER['HTTP_X_FORWARDED_FOR'])) {
$ip = $_SERVER['HTTP_X_FORWARDED_FOR'];
} else {
$ip = $_SERVER['REMOTE_ADDR'];
}
$url = "http://api.wipmania.com/".$ip;
(h) $country = file_get_contents($url); //ISO code for Nigeria it is NG check your country ISO code
?>
<?php if ($country == ""){
echo "Country not found";
} else { ?>
<script>
var map = "<?php echo $country ?>";
$.ajax({
type : 'POST',
url : 'http://www.mowemy.com/countryflag.php',
data : {map: map},
success : function(r) {
//console.log("success: " + r);
$('#mumil').html(r);
} })
</script>
<?php } ?>
<div id ="mumil" style="font-size:13px!important;margin-top:5px;"></div>
</button>
let me beak it down letter A - H is the script to detect your ISO number for the country for my country Nigeria it is NG you can google search your countries ISO number, with this script it is auto detected. Then i created a page with some info you fire AJAX to that page which it sends back the country flag color and name simple and easy PLEASE CALL JQUERY BEFORE AJAX TO RUN THE AJAX UNLESS IT WONT WORK GOODLUCK
You could use the API here http://www.hostip.info/use.html if you're okay with relying on an external site.
You can also use the GeoIP PHP API
Of course, implementing the script Orbit linked might just save you the hassle of going through the API's.
Good luck.
The best way to do this I have found is using the "GAE-IP-TO-COUNTRY" library:
https://github.com/andris9/GAE-IP-TO-COUNTRY/tree/master/src/ip_files
Example of use (you have to copy the "ip_files" directory to your server):
function iptocountry($ip) {
$numbers = preg_split( "/\./", $ip);
include("./ip_files/".$numbers[0].".php");
$code=($numbers[0] * 16777216) + ($numbers[1] * 65536) + ($numbers[2] * 256) + ($numbers[3]);
foreach($ranges as $key => $value){
if($key<=$code){
if($ranges[$key][0]>=$code){$country=$ranges[$key][1];break;}
}
}
return $country;
}
$country=iptocountry($_SERVER['REMOTE_ADDR']);
echo $country;
As others have pointed out, it would probably a better idea to check the Accept-Language HTTP Header for Turkish. If it's the preferred language, serve it. Otherwise serve English.
Here's some code.
I coded the next stuff using Accept-Language as other users pointed:
function GetAcceptedLangs()
{
$res=array();
$a=getallheaders();
if(isset($a["Accept-Language"]))
{
$aceptlangs=explode(",",str_replace(array(';','0','1','2','3','4','5','6','7','8','9','.',"q="),array(',','','','','','','','','','','','',""),$a["Accept-Language"]));
foreach($aceptlangs as $i=>$v)
{
if(trim($v)!="")
$res[]=trim($v);
}
}
return $res;
}
A simple
print_r(GetAcceptedLangs());
return in my case:
Array ( [0] => es-ES [1] => es [2] => en )
You can after define an array like this one to change to your internal language value, for example:
$al={"ES-es"=>"es","es"=>"es","en"=>"en"......}
They are already sorted by user preferences.
If all languages don't exists in the array you can go to the default language of your website. This is valid also if the browser don't send the Accept-Language header.
Another version removing the sub-region values
function GetAcceptedLangs2()
{
$res=array();
$a=getallheaders();
if(isset($a["Accept-Language"]))
{
$aceptlangs=explode(",",str_replace(array(';','0','1','2','3','4','5','6','7','8','9','.',"q="),array(',','','','','','','','','','','','',""),$a["Accept-Language"]));
foreach($aceptlangs as $i=>$v)
{
$va=trim($v);
if(($pos=strpos($va,"-"))!==false)
$l=substr($va,0,$pos);
else
$l=$va;
if($l!="" && !isset($check[$l]))
{
$check[$l]=1;
$res[]=$l;
}
}
}
return $res;
}
It would return in my case
Array ( [0] => es [1] => en )
Here is the straight forward way I like to use
<?php
class GeoIp
{
protected $api = 'https://ipapi.co/%s/json/';
protected $properties = [];
//------------------------------------------------------
/**
* __get
*
* #access public
* - #param $key string
* - #return string / bool
*/
public function __get($key)
{
if( isset($this->properties[$key]))
{
return $this->properties[$key];
}
return null;
}
//------------------------------------------------------
/**
* request_geoip
*
* #access public
* - #return void
*/
public function request_geoip($ip)
{
$url = sprintf($this->api, $ip);
$data = $this->send_geoip_request($url);
$this->properties = json_decode($data, true);
//var_dump($this->properties);
}
//------------------------------------------------------
/**
* send_geoip_request
*
* #access public
* - #param $url string
* - #return json
*/
public function send_geoip_request($url)
{
$loc_content = file_get_contents($url);
return $loc_content;
}
}
//You can access it like this:
$geo = new GeoIp;
$geo->request_geoip('foo');
echo $geo->country;
?>
Visit this website for more info
Related
I am working on an application that I need to post a secret variable. I wrote this code.
<form target="_blank" action="/validate/lista.php" method="POST">
<input type="hidden" name="evento" value="<?php echo $pname ?>" />
<button class="btn btn-block btn-md btn-outline-success">Lista</button>
</form>
My problem is that if the user inspect the element with chrome or whatever, he can see the value and change it before POST.
I could use SESSION but every user has a different session ID and this way I would need to POST the session ID (because they are separete applications), which I think is not secure. Or is it ok?
How can I prevent this? I am new to programming...
Thank you
Maintain HTML Form State safely ('Conversation' Tracking)
Keep track of the 'state' of an HTML Form as it is processed by the client and the server.
The typical 'conversation' is:
Send a new form to the client, often for a specific user who has to login.
The client enters data and returns it.
It is validated and may be sent out again.
The data changes are applied.
the client is informed of the result.
It sounds simple. Alas, we need to keep track of the 'state' of the form during the 'conversation'.
We need to record the state in a hidden field. This can open us up to various 'failure modes'.
This answer is one method of reliably keeping track of the 'conversations'.
Including people being 'malicious'. It happens. ;-/
This is a data change form so we don't want it applied to the wrong person.
There are various requirements:
Sensible ones:
prevent a form being processed twice
Ask a user to confirm the data if the form is too old
Malicious ones:
Changing the form to appear to be from a different user
Using an old copy of the form
Changing other hidden data to corrupt the user data
Now, we cannot prevent the client changing the hidden data, or storing it to replay later. etc.
What to do?
We need to ensure that if it is changed then we can detect that it is tampered with and tell the user about it. We do nothing.
If they send us an old stored valid copy then we can detect that as well.
Is there a simple way of doing this? Oh yes! :)
Answers:
Give each form a unique id: makes it easy to determine if we have already seen it.
Give each form a timestamp of when it was first created.
we can then decide the max age we allow to use it.
If it is too old then we just copy the entered data to a new form and ask the user to confirm it. see Captcha :)
When we process the form we store the form id.
The first check before processing a form is to see if we have already processed it
Identifying 'tampering'?
We encrypt it with AES! :) Only the server needs to know the password so there are no client issues.
If it is changed then the decrypt will fail and we just issue a new form to the user with the data input on it. :)
Is it a lot of code? Not really. And it makes forms processing safe.
One advantage is that has the protection for the CSRF attack built in so no separate code needed.
Program Code (FormState Class)
<?php
/**
* every 'data edit' form has one of these - without exeception.
*
* This ensures that the form I sent out came from me.
*
* It has:
* 1) A unique #id
* 2) A date time stamp and a lifetime
*
* Can be automatically generated and checked.
*/
class FormState {
const MAX_FORM_AGE = 600; // seconds
const ENC_PASSWORD = '327136823981d9e57652bba2acfdb1f2';
const ENC_IV = 'f9928260b550dbb2eecb6e10fcf630ba';
protected $state = array();
public function __construct($prevState = '')
{
if (!empty($prevState)) {
$this->reloadState($prevState); // will not be valid if fails
return;
}
$this->setNewForm();
}
/**
* Generate a new unique id and timestanp
*
* #param $name - optional name for the form
*/
public function setNewForm($name = '')
{
$this->state = array();
$this->state['formid'] = sha1(uniqid(true)); // each form has a unique id
$this->state['when'] = time();
if (!empty($name)) {
$this->setAttribute('name', $name);
}
}
/**
* retrieve attribute value
*
* #param $name attribute name to use
* #param $default value to return if attribute does not exist
*
* #return string / number
*/
public function getAttribute($name, $default = null)
{
if (isset($this->state[$name])) {
return $this->state[$name];
} else {
return $default;
}
}
/**
* store attribute value
*
* #param $name attribute name to use
* #param $value value to save
*/
public function setAttribute($name, $value)
{
$this->state[$name] = $value;
}
/**
* get the array
*/
public function getAllAttributes()
{
return $this->state;
}
/**
* the unique form id
*
* #return hex string
*/
public function getFormId()
{
return $this->getAttribute('formid');
}
/**
* Age of the form in seconds
* #return int seconds
*/
public function getAge()
{
if ($this->isValid()) {
return time() - $this->state['when'];
}
return 0;
}
/**
* check the age of the form
*
*#param $ageSeconds is age older than the supplied age
*/
public function isOutOfDate($ageSeconds = self::MAX_FORM_AGE)
{
return $this->getAge() >= $ageSeconds;
}
/**
* was a valid string passed when restoring it
* #return boolean
*/
public function isValid()
{
return is_array($this->state) && !empty($this->state);
}
/** -----------------------------------------------------------------------
* Encode as string - these are encrypted to ensure they are not tampered with
*/
public function asString()
{
$serialized = serialize($this->state);
$encrypted = $this->encrypt_decrypt('encrypt', $serialized);
$result = base64_encode($encrypted);
return $result;
}
/**
* Restore the saved attributes - it must be a valid string
*
* #Param $prevState
* #return array Attributes
*/
public function fromString($prevState)
{
$encrypted = #base64_decode($prevState);
if ($encrypted === false) {
return false;
}
$serialized = $this->encrypt_decrypt('decrypt', $encrypted);
if ($serialized === false) {
return false;
}
$object = #unserialize($serialized);
if ($object === false) {
return false;
}
if (!is_array($object)) {
throw new \Exception(__METHOD__ .' failed to return object: '. $object, 500);
}
return $object;
}
public function __toString()
{
return $this->asString();
}
/**
* Restore the previous state of the form
* will not be valid if not a valid string
*
* #param $prevState an encoded serialized array
* #return bool isValid or not
*/
public function reloadState($prevState)
{
$this->state = array();
$state = $this->fromString($prevState);
if ($state !== false) {
$this->state = $state;
}
return $this->isValid();
}
/**
* simple method to encrypt or decrypt a plain text string
* initialization vector(IV) has to be the same when encrypting and decrypting
*
* #param string $action: can be 'encrypt' or 'decrypt'
* #param string $string: string to encrypt or decrypt
*
* #return string
*/
public function encrypt_decrypt($action, $string)
{
$output = false;
$encrypt_method = "AES-256-CBC";
$secret_key = self::ENC_PASSWORD;
// iv - encrypt method AES-256-CBC expects 16 bytes - else you will get a warning
$secret_iv_len = openssl_cipher_iv_length($encrypt_method);
$secret_iv = substr(self::ENC_IV, 0, $secret_iv_len);
if ( $action == 'encrypt' ) {
$output = openssl_encrypt($string, $encrypt_method, $secret_key, OPENSSL_RAW_DATA, $secret_iv);
} else if( $action == 'decrypt' ) {
$output = openssl_decrypt($string, $encrypt_method, $secret_key, OPENSSL_RAW_DATA, $secret_iv);
}
if ($output === false) {
// throw new \Exception($action .' failed: '. $string, 500);
}
return $output;
}
}
Example Code
Full Example Application Source Code (Q49924789)
Website Using the supplied Source Code
FormState source code
Do we have an existing form?
$isExistingForm = !empty($_POST['formState']);
$selectedAction = 'start-NewForm'; // default action
if ($isExistingForm) { // restore state
$selectedAction = $_POST['selectedAction'];
$formState = new \FormState($_POST['formState']); // it may be invalid
if (!$formState->isValid() && $selectedAction !== 'start-NewForm') {
$selectedAction = "formState-isWrong"; // force user to start a new form
}
} else {
$_POST = array(); // yes, $_POST is just another PHP array
$formState = new \FormState();
}
Start New Form
$formState = new \FormState();
$_POST = array();
$displayMsg = "New formstate created. FormId: ". $formState->getFormId();
Store UserId (Database Id) in the FormState
$formState->setAttribute('userId' $userId);
Check a form being to old?
$secsToBeOutOfDate = 3;
if ($formState->isOutOfDate($secsToBeOutOfDate)) {
$errorMsg = 'Out Of Date Age: '. $secsToBeOutOfDate .'secs'
.', ActualAge: '. $formState->getAge();
}
Reload State from the form hidden field.
$formState = new \FormState('this is rubbish!!');
$errorMsg = "formState: isValid(): ". ($formState->isValid() ? 'True' : 'False');
Check if a form has already been processed.
if (isset($_SESSION['processedForms'][$formState->getFormId()])) {
$errorMsg = 'This form has already been processed. (' . $formState->getFormId() .')';
break;
}
$_SESSION['processedForms'][$formState->getFormId()] = true;
$displayMsg = "Form processed and added to list.";
All,
Background of how problem was detected
My question concerns the performance of a web app, mainly the index page. I noticed the problem when I was giving a demonstration at a local branch of my company that has slow internet (I don't know the exact speeds, or ping rate) judged by the fact that Google took about 10 seconds to load. My index page took ~10-20 times longer to load. I was under the assumption that my app did most of the work on the server side (as php is making all of the database queries...). But this led me to look at the network tool of Chrome and see the latency times of these 4 divs being loaded by ajax (I'll elaborate in a bit). Interestingly, the scripts being called appear to run sequentially, but not necessarily in the order I invoked the ajax calls (sometimes they do, other times they don't).
What are these divs / ajax requests?
Here is a code snippets of a request:
Yii::app()->clientScript->registerScript('leftDiv', '
$( "#left_dash" ).load(
"'.$this->createUrl("/site/page?view=leftDashLoad") .'",
function(){
$("#left_dash p a").click(function() {
$(this).parent().parent().find("div.scroll100").slideUp();
$(this).parent().next().stop(false, false).slideDown();
});
$("p:first-child").next().slideDown();
}
);
' );
Here is the page requested:
$this->widget('widgets.ScrollList',array(
'condition'=>
function($muddJob,$scrollList)
{
$job = $muddJob->Job;; //returns a job or empty array
if(!empty($job) )
{
if( $muddJob->uploadArtwork == null && $muddJob->uploadData == null ) {
array_push($scrollList->_models,$job);
$scrollList->columnValues = array($muddJob->jobDescription,$muddJob->dropDate1);
return true;
}
}
return false;
},
'columns' => array('col1'=>"MuddJob#",'col2'=>"Desc",'col3'=>"Dealer Name"),
'name'=> "Print New Ticket",
'muddJobs' => $currentExchanges->getCurrentMuddExchanges(),
)
);
Imagine that page (the page that ajax has called) having 6 similar declarations that create widgets. The goal is to return html to put back in place of a loading gif on the index page.
Here is the scroll widget:
<?php
Yii::import('widgets.ScrollListBase');
include_once Yii::app()->extensionPath . "/BusinessDay.php";
class ScrollList extends ScrollListBase
{
private $_content;
public $columns = array();
public $columnValues;
private $_listInfo;
public $name;
public $_models = array();
public $condition;
public $muddJobs; //object to pass
public $jobsMailingTodayArray = array();
public function init()
{
//$this->init();
$this->_listInfo = $this->generateListInfo($this->columns);
//$muddJobs = $this->getCurrentMuddExchanges();
$listInfo = $this->newScrollList($this->muddJobs);
$contents = $this->createContent($listInfo,$this->name);
$this->_content = $contents[0];
// $this->_fullTableContent = $contents[1];
//$this->_listInfo = $contents[2];
}
public function run()
{
//if($this->data['isVisible'])
echo $this->_content;
Yii::app()->session["exploded_content_{$this->name}"] = $this->_models;
}
private function newScrollList($muddJobs)
{
$listInfo = $this->_listInfo;
$tempCount = 0;
foreach($muddJobs as $muddJob)
{
$condition = $this->condition;
if($condition($muddJob,$this) && empty($this->jobsMailingTodayArray) ) //if no job exists for the muddExchange...
{
$tempArray = $this->createWidgetLinks($tempCount,$listInfo,$muddJob,$this->columnValues);
$listInfo = $tempArray[0];
$tempCount = $tempArray[1];
}
elseif ( !empty($this->jobsMailingTodayArray ) )
{
foreach ($this->jobsMailingTodayArray as $jobMailingToday) //change to for loop over the length of the jobsMailingToday
{
$tempArray = $this->createWidgetLinks($tempCount,$listInfo,$muddJob,$this->columnValues);
$listInfo = $tempArray[0];
$tempCount = $tempArray[1];
}
$this->jobsMailingTodayArray = array();
}
}
return array($listInfo,$tempCount);
}
}
?>
Here is it's parent:
<?php
class ScrollListBase extends CWidget
{
private $content = "<p>";
private $divDeclaration = "<div class='scroll100'>\n<table class='quickInfoTable'>\n<thead>\n";
private $headTag = "<th>";
private $headTagClose = "</th>\n";
private $theadTagClose = "</thead>\n";
private $bodyTag = "<tbody>\n";
private $listInfo = "<div class='scroll100'>\n<table class='quickInfoTable'>\n<thead>\n<th>Job#</th>\n<th>Package#</th>\n<th>Entry Date</th>\n</thead>\n<tbody>\n";
/**
* Initializes the widget.
*/
public function createContent($listInfo,$name)
{
$largeHref = Yii::app()->request->baseUrl . '/index.php/site/fullTableView';
$this->content .= "<span class='badge' >{$listInfo[1]} </span> <a href='#'>{$name} </a> <a href='$largeHref/Name/{$name}'> <small>(view larger)</small> </a> </p>";
if( $listInfo[1] > 0 )
{
// $this->fullTable .= substr($listInfo[0],22);
// $this->fullTableContent= $this->fullContent .= $this->fullTable . "</tbody>\n</table>\n</div>";
$this->content .= $listInfo[0] . "</tbody>\n</table>\n</div>";
}
return array($this->content);
}
//Helper Methods
/**
*
* #param type $attributeArray. send an accociative array
* #return type = either a job or an empty array
*/
protected function getJobByAttributes($attributeArray)
{
return Jobs::model()->with('MuddExchange')->findByAttributes($attributeArray);
}
protected function createWidgetLinks($tempCount,$listInfo,$muddJob,$columnValues,$url="/MuddExchange/")
{
$tempCount++;
$viewIndex = $muddJob->exchange_id;
$model = $muddJob;
$job = $muddJob->Job;
if ( isset($job ))
{
$model = $job;
$url = "/Jobs/";
$viewIndex = $model->job_id;
}
$link = CHtml::link("$model->jobNumber",array("{$url}{$viewIndex}"));
$listInfo .= "<tr>\n<td>$link</td>\n";
foreach ($columnValues as $columnValue)
{
$listInfo .= "<td>{$columnValue}</td>\n";
}
$listInfo .= "</tr>";
return array($listInfo,$tempCount);
}
protected function getListInfo()
{
return $this->listInfo;
}
/**
* Takes an array of strings to generate the column names for a particular list.
* #param array $heads
* #return string
*
*/
protected function generateListInfo($heads)
{
//<th>Job#</th>\n<th>Package#</th>\n<th>Entry Date</th>\n</thead>\n<tbody>\n";
$htmlScrollStart = $this->divDeclaration;
foreach ($heads as $tableColumn => $name)
{
$htmlScrollStart .= $this->headTag . $name . $this->headTagClose;
}
$htmlScrollStart .= $this->theadTagClose . $this->bodyTag;
return $htmlScrollStart;
}
public function calculateDueDate($jobsMailDate,$job)
{
$package = PackageSchedule::model()->findByAttributes(array('package_id'=>$job->packageID));
$projectedDays = $package->projected_days_before_mail_date;
$dropDate1 = $jobsMailDate->projected_mail_date;
$dropDate = wrapBusinessDay($dropDate1); //use this for actual command...
$toSec = 24*60*60;
$dayInt =0;
$secDropDate = strtotime($dropDate1);
do{
$dayInt +=1;
$daysInSec = ($dayInt) * $toSec ;
$secGuessDueDate = $secDropDate - $daysInSec;
$dueDate = date('Y-m-d',$secGuessDueDate);
$difference = $dropDate->difference($dueDate);
}while( $difference != $projectedDays);
return $dueDate;
}
}
?>
Why I think this behavior is odd
The whole slow internet thing is a beast in and of itself, but I don't think that is in the scope of StackOverflow. I'm more concerned about the loading of these divs. The div that loads last, i.e., takes on average 1.5 to 2 seconds, is an ajax request to a page that creates a single widget. The logic behind it is here:
<?php
include_once Yii::app()->extensionPath . "/CurrentExchanges.php";
$currentExchanges = Yii::app()->session['currentExchanges'];
$this->layout = 'barebones';
$this->widget('widgets.ScrollList',array(
'condition'=>
function($muddJob,$scrollList)
{
if ($muddJob->dropDate1 != null && $muddJob->dropDate1 != '0000-00-00')
{
$job = $muddJob->Job;;
if(!empty($job) && $job->packageID != null) //if job exists for the muddExchange and has a package
{
if($job->uploadArtwork == null )
{
$jobsMailDate = JobsMailDate::model()->findByAttributes(array("job_id"=>$job->job_id,'sequence_num'=>1));
if(!empty($jobsMailDate))
{
$calculatedDueDate = $scrollList->calculateDueDate($jobsMailDate,$job);
if (strtotime($calculatedDueDate) <= strtotime(date("Y-m-d")) )
{
array_push($scrollList->_models , $job);
$scrollList->columnValues = array($muddJob->jobDescription,$muddJob->dropDate1,$jobsMailDate->projected_mail_date);
return true;
}
}
}
}
}
return false;
},
'columns' => array('col1'=>"MuddJob#",'col2'=>"Desc",'col3'=>"Drop Date", 'col4' =>'Projected Drop Date'),
'name'=> "Artwork Due Today",
'muddJobs' => $currentExchanges->getCurrentMuddExchanges(),
)
);
?>
The calculateduedate method makes 2 additional calls to the server.
What I'm failing to understand is why the left div (with the most proccessing to do) is usually the first to return and the artLoad is usually the last to load (by a substantial difference). Here are some times returned by chromes network tool:
leftDashLoad: 475ms
rightDashLoad: 593ms
dataLoad: 825ms
artLoad: 1.41s
dataLoad: 453ms
rightDashLoad: 660ms
leftDashLoad: 919ms
artLoad: 1.51s
rightDashLoad: 559ms
leftDashLoad: 1.17s
dataLoad: 1.65s
artLoad: 2.01s
I just can't fathom why the left/right dashloads return so much faster than the artLoad. The code for artLoad and dataLoad are nearly identical save the actual comparison (the one if statement). If this were truly asynchronous, I'd expect the order to be art/dataLoad, rightDashLoad and leftDashLoad based purely on the amounts of computation done in each page. Perhaps the server isn't multithreading, or there is some weird configuration, but if that were the case, I don't see why the effects of the loading would be hit so hard by slow internet.
If I have overlooked something obvious or failed to use google appropriately, I do apologize. Thanks for any help you can offer!
Language/other tech info
The app was developed using the Yii framework. PHP 5.3. Apache Server. INNODB Tables. Server is hosted in dreamhost.
Update
I've changed the view page so that the ajax calls are now calling a controller action. It seems to have made the loading times more similar (asynchronous?) on my local dev environment, but really slowed it down on the QA environment (hosted on dreamhost...). Here is screen shot of the local network tools info:
dev environment
and the qa site (note, that the databases have about the same amounts of data...)
qa environment
Thoughts? It seems to me that my original problem may be solved, as the return times (on my local dev) look more like I expect them to.
Also, my own ignorance as to how NetBeans debugs was playing a part in this synchronous loading thing as xdebug is using a session. I believe this was forcing the ajax calls to wait their turn.
Thanks to #Rowan for helping me diagnose this strange behavior. PHP was trying to request a session before the session was closed in hopes to prevent a race hazard. There were session requests in my code, and there was a session started by my IDE (NetBeans). Removing all references to sessions in the ajax called pages and having ajax call actions that use renderPartial() proved to return the expected behavior (and much better code IMO). Let me know how to properly thank users in terms of StackOverflow (can I up comments, or what is there available? ). Thank you all!
I am using SimpleBrowser that is a part of SimpleTest PHP framework.
The idea is to imitate user interactions with the website and record returned HTML code into a file for further comparison. But something goes wrong here as empty HTML is sometimes returned.
getTransportError() returns Nothing fetched
It happens in completely random places and I can't use back() function because most pages are submitted forms.
require_once('simpletest/browser.php');
class TesterBrowser extends SimpleBrowser
{
/**
* Test the page against the reference. If reference is missing, is it created
* Uses md5 checksum to check if files are identical
*
* #param string $forcename Optional. Substitude autogenerated filename.
* #param boolean $forceRef Optional. Force file to be saved as the reference
*
* #access public
*
* #return void
*/
public function testPage($forcename = "")
{
//who called me?
//$callers=debug_backtrace();
//$whocalledme = $callers[1]['function'];
//get the current source
$html = $this->getContent();
//generate filename
$filename = empty($forcename) ? preg_replace('/[^\w\-'. ''. ']+/u', '-', $this->getUrl()) : $forcename;
$filename .= ".html";
//is there a gauge?
if(file_exists("ref/".$filename) && filesize(dirname(__FILE__)."/ref/".$filename) > 0)
{
//is there a difference
file_put_contents(dirname(__FILE__)."/actual/".$filename, $html);
if(filesize(dirname(__FILE__)."/actual/".$filename) == 0)
{
return false;
}
if(md5_file(dirname(__FILE__)."/actual/".$filename) != md5_file(dirname(__FILE__)."/ref/".$filename))
{
echo $this->getUrl() . " (" . $filename . ") has changed \r\n";
}
}
else
{
file_put_contents(dirname(__FILE__)."/ref/".$filename, $html);
if(filesize(dirname(__FILE__)."/ref/".$filename) == 0)
{
return false;
}
}
return true;
}
/**
* Output the string to the terminal
*
* #param mixed $string String to output
*
* #access public
*
* #return void
*/
public function output($string)
{
echo date("d-m-Y H:i:s") . " - $string... \r\n";
//update date so that it will be the same on every page
exec('date -s "24 JUN 2013 10:00:00"');
}
/**
* Restore the server date using external NTP server
*
* #access public
*
* #return void
*/
public function restoreDate(){
$this->output("Restoring the date&time from NTP server");
exec("ntpdate 0.uk.pool.ntp.org");
exec("hwclock -systohc");
}
}
And the way tests are performed:
class Tester
{
public $browser = null;
const BASEURL = "http://ticketing/";
function __construct(){
$this->browser = new TesterBrowser();
$this->browser->setConnectionTimeout(180);
//get the list of class method to be run
$methods = array();
foreach(get_class_methods($this) as $var)
{
if(0 === strpos($var, 'test')) //they all start with test
{
$methods[] = $var;
}
}
$methods[] = "cleanUp";
//now we need to run these methods
foreach($methods as $m){
while($this->$m() == false){
$this->browser->output("Empty page, trying again");
sleep(5);
}
}
}
//index page
function testGetIndexPage()
{
$this->browser->output("Getting index page");
$this->browser->get(self::BASEURL);
return $this->browser->testPage();
}
//try to enter wrong password
function testWrongPassword()
{
$this->browser->output("Entering wrong credentials");
$this->browser->setField("username", "wrong");
$this->browser->setField("password", "wrong");
$this->browser->clickSubmitByName("submit");
return $this->browser->testPage("wrong-credentials");
}
//Delete ticket though admin
function testDeleteTicketThroughAdmin()
{
$this->browser->output("Deleting the ticket through admin page");
$this->browser->setField("bulk[]", "375341");
$this->browser->setField("bulkaction", "delete");
$this->browser->clickSubmit("Do Action");
return $this->browser->testPage("deleted-ticket-admin");
}
//Restore the date
function cleanUp()
{
$this->browser->restoreDate();
return true;
}
}
$tester = new Tester();
There are of course much more test performed and this is a stripped version.
I have googled a lot about this problem, there seems to be no adequate documentation whatsoever.
Solved. It was a timeout issue although for some reason no adequate error message is implemented.
$this->browser->setConnectionTimeout(180);
For a survey I am trying to grab google search result from my php page. I grab six results then i click next button to get next page records.but after 9 pages e.g. 64 results it gives following error:
stdClass Object
(
[responseData] =>
[responseDetails] => out of range start
[responseStatus] => 400
)
I just want as much data as possible. i dont mind if it is google search engine or any other search engine. but for accurate result of survey I want large number of resultset. Can anybody knows how can i do that ?
Is it possible to grab results through cron ? Is there any other way ?
ATTENTION
Google tries to prevent scraping and so servers will be blocked and requests will be dropped when they suspect scraping. So you can use this if you occassionally need to get some google results.
Check google-scraper.squabbel.com for a proxy based scraper and more information on googles blocking mechanism. Its also against their policy and thus illigal.
The google api will not allow for more then 64 results, so if you need more you need to scrape the site yourself. As it was a fun project, I have created a class to do this for you.
It requires the free PHP Simple HTML DOM Parser so you need download this code aswell.
it will output an array like
array(100) {
[0]=>
array(3) {
["title"]=>
string(67) "Online Tests - Online aptitude tests for interview, competitive ..."
["href"]=>
string(36) "http://www.indiabix.com/online-test/"
["description"]=>
string(168) "Online aptitude tests for competitive examination, entrance examination and
campus interview. Take various tests and find out how much you score before
you ... "
}
[1]=>
array(3) {
["title"]=>
string(37) "Test your English - Cambridge English"
["href"]=>
string(50) "http://www.cambridgeenglish.org/test-your-english/"
["description"]=>
string(179) "Test Your English. This is a quick, free online test. It will tell you which Cambridge
English exam may be best for you. Click 'Begin Test' and answer each of the ... "
}
//removed for better visibility
}
How to use:
//start the scraper for google.com (english results)
$gs = new GoogleScraper();
//start the scraper for google.nl (dutch results)
//$gs = new GoogleScraper('https://www.google.nl');
//set your search query
$gs->SearchQuery('online exams');
//start loading the pages. You can enter any integer above 0
$gs->LoadPages(10);
//dump the results, but its just an array so you can also do other things with it.
echo '<pre>';
var_dump($gs->GetResults());
echo '</pre>';
?>
And then the GoogleScraper.php
<?php
require_once('simple_html_dom.php');
class GoogleScraper
{
private $_results;
private $_baseUrl;
private $_searchQuery;
private $_resultsPerPage;
/**
* constructor
* I use the constructor to set all the defaults to keep it all in one place
*/
final public function __construct($baseUrl='')
{
$this->_results = array();
$this->_resultsPerPage = 100;
if (empty($baseUrl)) {
$this->_baseUrl = 'https://www.google.com';
} else {
$this->_baseUrl = $baseUrl;
}
}
/**
* cleanup
*/
final public function __destruct()
{
unset($this->_results);
unset($this->_baseUrl);
unset($this->_searchQuery);
}
/**
* Set the query
*/
final public function SearchQuery($searchQuery)
{
if (!(is_string($searchQuery) || is_numeric($searchQuery)))
{
throw new Exception('Invalid query type');
}
$this->_searchQuery = $searchQuery;
}
/**
* Set the number of results per page
*/
final public function ResultsPerPage($resultsPerPage)
{
if (!is_int($resultsPerPage) || $resultsPerPage<10 || $resultsPerPage>100)
{
throw new Exception('Results per page must be value between 10 and 100');
}
$this->_resultsPerPage = $resultsPerPage;
}
/**
* Get the result
*/
final public function GetResults()
{
return $this->_results;
}
/**
* Scrape the search results
*/
final public function LoadPages($pages=1)
{
if (!is_int($pages) || $pages<1)
{
throw new Exception('Invalid number of pages');
}
if (empty($this->_searchQuery))
{
throw new Exception('Missing search query');
}
$url = $this->_baseUrl . '/search?num='.$this->_resultsPerPage.'&q=' . urlencode($this->_searchQuery);
$currentPage = 1;
while($pages--) {
if ($content = $this->LoadUrl($url)) {
/*
Load content in to simple html dom
*/
$html = new simple_html_dom();
$html->load($content);
/*
Find and handle search results
*/
$items = $html->find('div#ires li');
foreach($items as $item) {
/*
Only normal search results have this container. Special results like found images or news dont have it.
*/
$check = $item->find('div.s');
if (count($check)!=1) {
continue;
}
$head = $item->find('h3.r a', 0);
$result['title'] = $head->plaintext;
/*
If we dont have a title, there is no point in continuing
*/
if (empty($result['title'])) {
continue;
}
$result['href'] = $head->href;
/*
Check if we can parse the URL for the actual url
*/
if (!empty($result['href'])) {
$qs = explode('?', $result['href']);
if (!empty($qs[1])) {
parse_str($qs[1], $querystring);
if (!empty($querystring['q'])) {
$result['href'] = $querystring['q'];
}
}
}
/*
Try to find the description
*/
$info = $item->find('span.st', 0);
$result['description'] = $info->plaintext;
/*
Add the results to the total
*/
$this->_results[] = $result;
}
/*
Find next page
*/
$url = $this->_baseUrl . '/search?num='.$this->_resultsPerPage.'&q=' . urlencode($this->_searchQuery) . '$start=' . ($currentPage*$this->_resultsPerPage);
} else {
throw new Exception('Failed to load page');
}
$currentPage++;
}
}
/**
* Load the url
*/
final private function LoadUrl($url)
{
if (!is_string($url))
{
throw new Exception('Invalid url');
}
$options['http'] = array(
'user_agent' => "GoogleScraper",
'timeout' => 0.5
);
$context = stream_context_create($options);
$content = file_get_contents($url, null, $context);
if (!empty($http_response_header))
{
return (substr_count($http_response_header[0], ' 200 OK')>0) ? $content : false;
}
return false;
}
}
?>
Check this PHP Fiddle to see it in action. Because this could be used quite often from this server, there is a chance for 503 errors from google.
You should add a sleep(1) as cooldown between calls, otherwise might get bebanned.
Did you consider going the official way and getting a google API key?
I'm getting the following warning when try to view list_of_holidays.pdf from the remote server:
Warning (2): Cannot modify header information - headers already sent by (output started
at /home/aquinto1/app/views/helpers/flash.php:155) [APP/vendors/tcpdf/tcpdf.php, line 8541]
TCPDF ERROR: Some data has already been output to browser, can't send PDF file
line 155 is the last line in flash.php ie the closing tag for php (?>). Before that it is the code to embedSWF. I don't see anything wrong with that.
However, it is displaying fine on the local server.
I've checked for whitespaces and yet the error is still there.
i'm already using ob_clean before the output.
can someone tell me on what i'm doing wrong. FYI i'm using cakephp with tcpdf.
The following is flash.php
class FlashHelper extends AppHelper {
var $helpers = array('Javascript');
/**
* Used for remembering options from init() to each renderSwf
*
* #var array
*/
var $options = array(
'width' => 100,
'height' => 100
);
/**
* Used by renderSwf to set a flash version requirement
*
* #var string
*/
var $defaultVersionRequirement = '9.0.0';
/**
* Used by renderSwf to only call init if it hasnt been done, either
* manually or automatically by a former renderSwf()
*
* #var boolean
*/
var $initialized = false;
/**
* Optional initializing for setting default parameters and also includes the
* swf library. Should be called once, but if using several groups of flashes,
* MAY be called several times, once before each group.
*
* #example echo $flash->init();
* #example $flash->init(array('width'=>200,'height'=>100);
* #return mixed String if it was not able to add the script to the view, true if it was
*/
function init($options = array()) {
if (!empty($options)) {
$this->options = am($this->options, $options);
}
$this->initialized = true;
$view =& ClassRegistry::getObject('view');
if (is_object($view)) {
$view->addScript($this->Javascript->link('swfobject'));
return true;
} else {
return $this->Javascript->link('swfobject');
}
}
/**
* Wrapper for the SwfObject::embedSWF method in the vendor. This method will write a javascript code
* block that calls that javascript method. If given a dom id as fourth parameter the flash will
* replace that dom object. If false is given, a div will be placed at the point in the
* page that this method is echo'ed. The last parameter is mainly used for sending in extra settings to
* the embedding code, like parameters and attributes. It may also send in flashvars to the flash.
*
* For doucumentation on what options can be sent, look here:
* http://code.google.com/p/swfobject/wiki/documentation
*
* #example echo $flash->renderSwf('counter.swf'); // size set with init();
* #example echo $flash->renderSwf('flash/ad.swf',100,20);
* #example echo $flash->renderSwf('swf/banner.swf',800,200,'banner_ad',array('params'=>array('wmode'=>'opaque')));
* #param string $swfFile Filename (with paths relative to webroot)
* #param int $width if null, will use width set by FlashHelper::init()
* #param int $height if null, will use height set by FlashHelper::init()
* #param mixed $divDomId false or string : dom id
* #param array $options array('flashvars'=>array(),'params'=>array('wmode'=>'opaque'),'attributes'=>array());
* See SwfObject documentation for valid options
* #return string
*/
function renderSwf($swfFile, $width = null, $height = null, $divDomId = false, $options = array()) {
$options = am ($this->options, $options);
if (is_null($width)) {
$width = $options['width'];
}
if (is_null($height)) {
$height = $options['height'];
}
$ret = '';
if (!$this->initialized) {
$init = $this->init($options);
if (is_string($init)) {
$ret = $init;
}
$this->initialized = TRUE;
}
$flashvars = '{}';
$params = '{wmode : "opaque"}';
$attributes = '{}';
if (isset($options['flashvars'])) {
$flashvars = $this->Javascript->object($options['flashvars']);
}
if (isset($options['params'])) {
$params = $this->Javascript->object($options['params']);
}
if (isset($options['attributes'])) {
$attributes = $this->Javascript->object($options['attributes']);
}
if ($divDomId === false) {
$divDomId = uniqid('c_');
$ret .= '<div id="'.$divDomId.'"></div>';
}
if (isset($options['version'])) {
$version = $options['version'];
} else {
$version = $this->defaultVersionRequirement;
}
if (isset($options['install'])) {
$install = $options['install'];
} else {
$install = '';
}
$swfLocation = $this->webroot.$swfFile;
$ret .= $this->Javascript->codeBlock(
'swfobject.embedSWF
("'.$swfLocation.'", "'.$divDomId.'", "'.$width.'", "'.$height.'", "'.$version.'",
"'.$install.'", '.$flashvars.', '.$params.', '.$attributes.');');
return $ret;
}
}
?>
Simply do what the error tells you: Check app/views/helpers/flash.php line 155 and see what it is outputting there and fix it. There must be some code that outputs something.
Could it be one of the return statements?
if (is_object($view)) {
$view->addScript($this->Javascript->link('swfobject'));
return true;
} else {
return $this->Javascript->link('swfobject');
}
$ret .= $this->Javascript->codeBlock(
'swfobject.embedSWF
("'.$swfLocation.'", "'.$divDomId.'", "'.$width.'", "'.$height.'", "'.$version.'",
"'.$install.'", '.$flashvars.', '.$params.', '.$attributes.');');
return $ret;
}
What other code is on the page calling flash?