I am using the Laravel Goutte package to perform some webscraping - the following code works and returns a lot of data, I am trying to filter out only the bit of data I require.
If I load up the browser (whilst injecting jQuery into the page) I am able to get the data I need using jQuery using the following in the console jQuery('ea-proclub-overview')[0]; - I am basically trying to do the equivalent of this command within the Laravel/Goutte instance below.
Using jQuery('ea-proclub-overview')[0].customCrestBaseUrl; in the console I get the exact URL I need - https://fifa21.content.easports.com/fifa/fltOnlineAssets/05772199-716f-417d-9fe0-988fa9899c4d/2021/fifaweb/crests/256x256/l'
Below is my PHP code - I am getting back in the $node variable but I am unsure how to only return the customCrestBaseUrl so it gives me the URL.
$client = new Client();
$client->setServerParameter('HTTP_USER_AGENT', 'Mozilla/5.0 (X11; Linux i686; rv:78.0) Gecko/20100101 Firefox/78.0');
$client->setServerParameter('REFERER', 'https://www.ea.com/');
$url = 'https://www.ea.com/en-gb/games/fifa/pro-clubs/ps5-xbsxs/overview?clubId=2552&platform=ps5';
$crawler = $client->request('GET', $url);
$crawler->filter('ea-proclub-overview')->each(function ($node) {
dd($node);
});
Expected Result
<ea-proclub-overview endpoints="{"settingsEndpoint":"https://proclubs.ea.com/api/fifa/settings","seasonalStatsEndpoint":"https://proclubs.ea.com/api/fifa/clubs/seasonalStats","clubsInfoEndpoint":"https://proclubs.ea.com/api/fifa/clubs/info","matchesEndpoint":"https://proclubs.ea.com/api/fifa/clubs/matches","memberStatEndpoint":"https://proclubs.ea.com/api/fifa/members/stats","memberCareerStatEndpoint":"https://proclubs.ea.com/api/fifa/members/career/stats"}" colors="{"currentDivision":{"startColor":"#FA4358","endColor":"#FA4358"},"nextDivision":{"relegationColor":"#FA4358","pointsColor":"#FA4358"},"pieChart":{"winsColor":"#19A863","lossesColor":"#C4010D","tiesColor":"#282D3B"},"stats":{"wins":{"startColor":"#19A863","endColor":"#94D85D"},"losses":{"startColor":"#C4010D","endColor":"#F80245"},"ties":{"startColor":"#282D3B","endColor":"#282D3B"}}}" match-type="["gameType9","gameType13"]" headers-labels="{"points":"Points","stats":{"wins":{"label":"Wins","description":"Wins"},"losses":{"label":"Losses","description":"Losses"},"ties":{"label":"Draws","description":"Draws"}}}" division-labels="{"title":"Division Ranking","currentDivisionTitle":"Current Division","nextDivisionTitle":"Points To Next Division","seasons":"Season","record":"Record","points":"Points","gamesPlayed":"Games Played","gamesRemaining":"Games Remaining","divisionImgBaseUrl":"https://media.contentapi.ea.com/content/dam/eacom/fifa/pro-clubs/divisioncrest","stats":{"wins":"W","losses":"L","ties":"D"}}" progressbar-labels="{"div":"Div","promotion":"Promotion","relegation":"Relegation","title":"Title"}" members-labels="{"title":"Members","linkText":"View All Members","linkUrl":"members","totalTitle":"Total Members","totalCountsLabel":"Total","memberDetails":{"proOverall":"Overall Rating","ratingAve":"Average Match Rating","gamesPlayed":"Games Played"},"memberPosition":{"defender":"Defender","forward":"Forward","goalkeeper":"Goalkeeper","midfielder":"Midfielder"},"positions":{"defender":"Defenders","forward":"Forwards","goalkeeper":"Goalkeepers","midfielder":"Midfielders"},"defaultMemberAvatar":"https://media.contentapi.ea.com/content/dam/ea/fifa/fifa-21/pro-clubs/common/pro-clubs/avatar.png"}" match-labels="{"title":"Last Match","linkText":"View All Match History","linkUrl":"match-history","altTitle":"No match data was found"}" trophies-labels="{"title":"Trophies","cupsLabel":{"leaguesWon":"Leagues Won","titlesWon":"Titles Won","totalCupsWon":"Total Cups Won"},"cupsImg":{"leaguesWonImgUrl":"https://media.contentapi.ea.com/content/dam/ea/fifa/fifa-21/pro-clubs/common/pro-clubs/league-titles-21.png","titlesWonImgUrl":"https://media.contentapi.ea.com/content/dam/ea/fifa/fifa-21/pro-clubs/common/pro-clubs/all-tiles-21.png","totalCupsWonImgUrl":"https://media.contentapi.ea.com/content/dam/ea/fifa/fifa-21/pro-clubs/common/pro-clubs/cups-won-21.png"}}" history-labels="{"title":"Club History","subTitle":"Overall Record","pts":"Points","division":"Division","historyDetails":{"seasons":"Seasons Played","totalGames":"Total Games","titlesWon":"Titles Won","bestPoints":"Highest Points Total","promotions":"Promotions","relegations":"Relegations"},"stats":{"wins":"Wins","losses":"Losses","ties":"Draws"},"statsShort":{"wins":"W","losses":"L","ties":"D"},"progressBar":{"title":"Best Season Finish","tipLabel":"DIV","startColor":"#9B7801","endColor":"#F9F1A5","divisionBaseUrl":"https://media.contentapi.ea.com/content/dam/eacom/fifa/pro-clubs/divisioncrest"}}" translations="{"4543827":"East Coast US","5723475":"West Coast US","5719381":"Western Europe","4539733":"Eastern Europe","5129557":"Northern Europe","5457237":"Southern Europe","4344147":"British Isles","5456205":"South America","4407629":"Central America","4281153":"Asia","4281683":"Australia / New Zealand"}" crest-base-url="https://fifa21.content.easports.com/fifa/fltOnlineAssets/05772199-716f-417d-9fe0-988fa9899c4d/2021/fifaweb/crests/256x256/l" custom-crest-base-url="https://fifa21.content.easports.com/fifa/fltOnlineAssets/05772199-716f-417d-9fe0-988fa9899c4d/2021/fifaweb/crests/256x256/l" default-crest-url="https://media.contentapi.ea.com/content/dam/ea/fifa/fifa-21/pro-clubs/common/pro-clubs/crest-default.png" loading-image="https://media.contentapi.ea.com/content/dam/eacom/fifa/pro-clubs/loading-animation.png" default-club-name="Disbanded"></ea-proclub-overview>
Actual Result
-- Too much post but it all of the HTML & XML
Below is pastebin of the entire response from the $crawler when dumped out using dd().
https://pastebin.com/qxUTpu9p
According to the documentation:
$customCrestBaseUrl = $crawler
->filter('ea-proclub-overview')
->first()
->extract(['custom-crest-base-url'])
;
I have been searching around for examples but the documentation on:
http://php.net/manual/en/solrclient.query.php
and
https://www.ayalon.ch/en/code-samples/solr-php-client-example
does not give me enough information to alter for my setup.
I am trying to execute a solr query using php from my website.
The search box form method is post and the input name is q.
This is all of the php I have at the moment; I was testing the connection.
<?php
require_once('C:\Program Files\Solr\solr-7.5.0\dist\solr-php-client-master\Apache\Solr\Service.php');
$client = new Apache_Solr_Service( 'localhost', '8983', '/solr/#/sqltest' );
if ( ! $client->ping() ) {
echo 'Solr service not up dude.';
}
else{
echo 'ayee we have a connection here!';
}
//to view the specifics of this setup use
//var_dump($client);
?>
Being new to php, Im really trying to find the proper way to perform the search and display the default json response.
I already have documents indexed to search so my code does not need to add any docs as seen in some of the above tutorials.
When I use this code:
$query = new SolrQuery();
$query->setQuery('lucene');
$query->setStart(0);
$query->setRows(10);
$query->addField('id')->addField('date')->addField('Problem')->addField('Solution');
$query_response = $client->query($query);
$response = $query_response->getResponse();
print_r($response);
I get an error because the class SolrQuery is not found.
I am trying to develop rest full webservice using slim frame work.
There is a registration api to which my input data is
{"firstname":"Rakhi",
"lastname":"Krishnan",
"language":{
"english":["read","write"],
"arabic":["read","speak"],
"hindi":[]
},
"membership":{
"involvement _level":["Atendant Only"]
}
}
I have parsed the input data using
$json = $request->getBody();
$data = json_decode($json, true);
But when I tried to access it gives me
$data['membership']['involvement _level']=array[][];
$data['membership']['involvement _level'][0]="Atendant Only" [][];
I don't know what is wrong in my code.Can any one please help me?
I built a PHP file with the sole purpose of hiding the API keys for Google Search, but part of the file_get_contents() always echo angular.callbacks._0_({ instead of angular.callbacks._0({
This small change makes the rest of the response worthless as Angular throws Uncaught TypeError: angular.callbacks._0_ is not a function. Although the workaround does works flawlessly, I would like to know if someone found the root of this issue or a better solution that is strictly PHP (no curl or any other package.)
search.php
<?php // Created by Deovandski on 2/14/2016
header('Content-type: application/json');
# Setup Base URL and array for Parameters
$host = 'https://www.googleapis.com/customsearch/v1?';
$queries = array();
$queries['cx'] = "XXX";// CSE KEY
$queries['key'] = "XXX"; // API KEY
# Setup possible incoming params
if (isset($_GET['search_term'])) $queries['q'] = $_GET['search_term'];
if (isset($_GET['result_count'])) $queries['result_count'] = $_GET['result_count'];
if (isset($_GET['callback'])) $queries['callback'] = $_GET['callback'];
# Build query and Final URL
$queriesURL = http_build_query($queries) . "\n";
$finalURL = $host.$queriesURL;
echo $finalURL;
/* echo $finalURL output (I only edited the keys out):
https://www.googleapis.com/customsearch/v1?cx=XXX&key=XXX&q=Hatsune+Miku&result_count=10&callback=angular.callbacks._0
*/
// Setup Response
$response = file_get_contents($finalURL);
// workaround
$fixedResponse = str_replace("angular.callbacks._0_", "angular.callbacks._0", $response);
echo $fixedResponse;
?>
This is part of a correct Google API response:
// API callback
angular.callbacks._0({
"kind": "customsearch#search",
"url": {
"type": "application/json",
"template": "https://www.googleapis.com/customsearch/v1?q={searchTerms}&num={count?}&start={startIndex?}&lr={language?}&safe={safe?}&cx={cx?}&cref={cref?}&sort={sort?}&filter={filter?}&gl={gl?}&cr={cr?}&googlehost={googleHost?}&c2coff={disableCnTwTranslation?}&hq={hq?}&hl={hl?}&siteSearch={siteSearch?}&siteSearchFilter={siteSearchFilter?}&exactTerms={exactTerms?}&excludeTerms={excludeTerms?}&linkSite={linkSite?}&orTerms={orTerms?}&relatedSite={relatedSite?}&dateRestrict={dateRestrict?}&lowRange={lowRange?}&highRange={highRange?}&searchType={searchType}&fileType={fileType?}&rights={rights?}&imgSize={imgSize?}&imgType={imgType?}&imgColorType={imgColorType?}&imgDominantColor={imgDominantColor?}&alt=json"
},
I put up a live version of this issue that can be seen on my FTP server. The PHP file can be viewed through this link (AngularJS parameters included on it).
The problem is the escape sequences \n. Which is passed as part of the request. And which is interpreted as space and as part of the callback function name and replaced by the side of the API to underline.
To understand just try this option and look at the result:
$queriesURL = http_build_query($queries) . "\n" . "after";
So just take away a newline.
i am developing an app in php it pulls data from Reuters API i am trying to add the request parameters before making SOAP call , i have the xml which must be generated from my request it must look like this :
<HeadlineMLRequest>
<Filter>
<MetaDataConstraint class="companies" xmlns="http://schemas.reuters.com/ns/2006/04/14/rmds/webservices/news/filter">
<Value>MSFT.O</Value>
</MetaDataConstraint>
</Filter>
</HeadlineMLRequest>
when i build my request parameter object i tried this
protected function getRequest() {
$retval->HeadlineMLRequest->MaxCount = 10;
$retval->HeadlineMLRequest->Filter->MetaDataConstraint->class = "companies";
$retval->HeadlineMLRequest->Filter->MetaDataConstraint->Value = "MSFT.O";
return $retval;
}
but when i echo last xml request i find it like this
<ns1:headlinemlrequest>
<ns1:maxcount>
10
</ns1:maxcount>
<ns1:filter>
<ns2:metadataconstraint class="companies">
<ns2:value>
</ns2:value>
</ns2:metadataconstraint>
</ns1:filter>
if you notice Value is empty although i set it with "MSFT.O" , any help please?
Complex requests with php soap are not well documented and most examples show only basic techniques.
Try the following:
$metaData = '<MetaDataConstraint class="companies" xmlns="http://schemas.reuters.com/ns/2006/04/14/rmds/webservices/news/filter">
<Value>MSFT.O</Value>
</MetaDataConstraint>';
$xmlvar = new SoapVar($metaData, XSD_ANYXML);
$retval->HeadlineMLRequest->Filter->MetaDataConstraint = $xmlVar;