Import XML into MySQL database - php

I'm new to coding, trying to create a database from an XML file which is a list of industries, company names and symbols. I've seen a few examples of a PHP code that imports the data into MySQL which looked something like:
<?php
$url ="http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.industry%20where%20id%20in%20(select%20industry.id%20from%20yahoo.finance.sectors)&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url); //getting url contents
$data = curl_exec ($ch); //execule curl request
curl_close($ch);
$xml = simplexml_load_string($data);
$con=mysql_connect("localhost", "root", ""); //connect to server
mysql_select_db("symbol_list", $con) or die(mysql_error()); //select database
foreach ($xml -> item as $row) {
$industry = $row -> industry;
$company = $row -> name;
$symbol = $row -> symbol;
// perform sql query
$sql = "INSERT INTO 'symbols_xml' ('industry', 'company', 'symbol')"
. "VALUES ('$industry', '$company', '$symbol')";
$result = mysql_query($sql);
if (!$result) {
echo 'MySQL ERROR';
} else {
echo 'SUCCESS';
}
}
?>
However, because the XML is formatted like this:
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="215" yahoo:created="2015-03-02T20:58:57Z" yahoo:lang="en-US">
<results>
<industry id="112" name="Agricultural Chemicals">
<company name="Adarsh Plant Protect Ltd" symbol="ADARSHPL.BO"/>
<company name="African Potash Ltd" symbol="AFPO.L"/>
......
</industry>
<industry id="132" name="Aluminum">
<company name="AEI Corp Ltd" symbol="A18.SI"/>
<company name="Alcoa Inc" symbol="AA.BA"/>
<company name="Alcoa Inc" symbol="AA.MX"/>
......
My PHP code above is unable to recognise the values...
Is there any way to import the attributes (industry name, company name and symbol) so that it looks something like:
#| Industry | Company | Symbol
-|----------- ------------ -------
1| Aluminium | Alcoa Inc. | AA.BA
and so on..
I just recently started learning PHP and databases so please forgive me if this is a noobish question that wasted your time. xD
Thanks.

You were actually not very far.
In order to fetch an attribute, you can do it in two ways :
Access it as if the node was an array $node['attributeName']
Use a method named attributes(), and you could also do : $node->attributes()->attributeName;
Here is your code updated :
<?php
$url ="http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.industry%20where%20id%20in%20(select%20industry.id%20from%20yahoo.finance.sectors)&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url); //getting url contents
$data = curl_exec ($ch); //execule curl request
curl_close($ch);
$xml = simplexml_load_string($data);
$con=mysql_connect("localhost", "root", ""); //connect to server
mysql_select_db("symbol_list", $con) or die(mysql_error()); //select database
foreach ($xml->results->industry as $industryNode)
{
foreach ($industryNode->company as $companyNode)
{
$industry = (string) $industryNode['name'];
$company = (string) $companyNode['name'];
$symbol = (string) $companyNode['symbol'];
// perform sql query
$sql = "INSERT INTO 'symbols_xml' ('industry', 'company', 'symbol')"
. "VALUES ('$industry', '$company', '$symbol')";
$result = mysql_query($sql);
if (!$result)
{
echo 'MySQL ERROR';
}
else
{
echo 'SUCCESS';
}
}
}

Related

How to web scrape date from a website and store that date in database using php and mysql?

I have been looking around on the internet for the ways to get dates of an event and then store that date into the database, but I was not able to find much.
I was able to get the dates from the website, but I don't know how to store it.
I want to get dates only from the website and then store it in the format of Y-m-d. Please if you know any way to do this, tell me.
Link: https://www.brent.gov.uk/events-and-whats-on-calendar/?eventCat=Wembley+Stadium+events
<?php
$curl = curl_init();
$all_data = array();
$url = "https://www.brent.gov.uk/events-and-whats-on-calendar/?eventCat=Wembley+Stadium+events";
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($curl);
$event = array();
preg_match_all('/<h3 style="margin:0px!important;">(.*?)<\/h3>/si',$result,$match);
$event['title'] = $match[1];
print_r($event['title']);
echo $all_data;
?>
don't use regex to parse html, use a proper HTML parser, for example DOMDocument.
a quick inspection of the site reveals that all the dates are in h3 children of the only article element on the page, you can use that to identify them. after extracting their dates, you can use strtotime() to convert it to an unix timestamp, then you can use date() to convert it to to the Y-m-d format, eg
$result = curl_exec($curl);
$domd=#DOMDocument::loadHTML($result);
$dateElements=$domd->getElementsByTagName("article")->item(0)->getElementsByTagName("h3");
foreach($dateElements as $ele){
var_dump(date("Y-m-d",strtotime($ele->textContent)));
}
as for how to store the results in a mysql database, try writing php mysql tutorial -w3schools in google, or read the PDO section here: http://www.phptherightway.com/#pdo_extension
<?php
$db_host = "localhost";
$db_username = "username";
$db_pass = "password";
$db_name = "name";
// Run the actual connection here
$con = mysqli_connect($db_host, $db_username, $db_pass, $db_name);
if ($con->connect_errno) {
die("Failed to connect to MySQL: (" . $con->connect_errno . ") " . $con->connect_error);
}
$curl = curl_init();
//The Website you want to get data from
$url = "https://www.brent.gov.uk/events-and-whats-on-calendar/?eventCat=Wembley+Stadium+events";
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($curl);
libxml_use_internal_errors(true);
$domd=#DOMDocument::loadHTML($result);
//Getting the date from the site
$dateElements=$domd->getElementsByTagName("article")->item(0)->getElementsByTagName("h3");
foreach($dateElements as $ele){
$data = (date("Y-m-d",strtotime($ele->textContent)));
// echo "<br>".$data;
//checking if the date match with database date
$sql = "SELECT * FROM event_table WHERE date = '$data'";
$result = $con->query($sql);
if ($result->num_rows > 0) {
// output data of each row, if date match echo "Data is there";
while($row = $result->fetch_assoc()) {
echo "Data is there";
}
}
//if date is not there then inster it into the database
else {
$results = mysqli_query($con, "INSERT INTO event_table (id, date) VALUES ('',' $data')");
echo "data uploaded";
}
}
?>

Integrating PHP Curl

I have the code below which calls up an MySQLi and presents it in XML form in my browser.
The next stage is that instead of presenting it in my browser I want to send it to another IP address using PHP Curl. Please can someone help me with the extra code needed to do that.
<?php
$mysqli_connection = new MySQLi('localhost', 'root', 'secret', 'edgeserver');
if ($mysqli_connection->connect_error) {
echo "Not connected, error: " . $mysqli_connection->connect_error;
}
$sql = "SELECT SessionLogs.sessionid, SessionLogs.eventid, BetStatus.BetStatus, EventStatus.EventStatus, SessionLogs.activestatusid
FROM SessionLogs INNER JOIN
EventStatus ON SessionLogs.eventstatusid = EventStatus.EventStatusID INNER JOIN
BetStatus ON SessionLogs.betstatusid = BetStatus.BetStatusID
where ActiveStatusID = 1
";
$res = $mysqli_connection->query($sql);
$xml = new XMLWriter();
$xml->openURI("php://output");
$xml->startDocument();
$xml->setIndent(true);
$xml->startElement('Alive');
$xml->writeAttribute('timestamp', date('c'));
if($res === FALSE) {
die(mysqli_error()); // TODO: better error handling
}
while ($row = mysqli_fetch_assoc($res)) {
$xml->startElement("Event");
$xml->writeAttribute('sessionid', $row['sessionid']);
$xml->writeAttribute('eventid', $row['eventid']);
$xml->writeAttribute('BetStatus', $row['BetStatus']);
$xml->writeAttribute('EventStatus', $row['EventStatus']);
$xml->writeAttribute('activestatusid', $row['activestatusid']);
$xml->endElement();
}
$xml->endElement();
$xml->endElement();
header('Content-type: text/xml');
$xml->flush();
?>
Please help. Thanks.
You can send xml data using curl with following code
$input_xml = ''; //XML Data
$url=''; // URL
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS,
"xmlRequest=" . $input_xml);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 300);
$data = curl_exec($ch);
curl_close($ch);
Use $xml->openMemory(); and $xmlString = $xml->outputMemory() to catch the cache of your XMLWriter-Object (Documentation).

How to send and receive a xml to another computer

I have two computers, comp 1 as branch 1 and comp 2 as main branch. in comp 1 i have generated an xml of my database query using php.
`<?php
header ("Content-Type:text/xml");
//database configuration
$config['mysql_host'] = "localhost";
$config['mysql_user'] = "root";
$config['mysql_pass'] = "donnaluz";
$config['db_name'] = "global89_branch1";
$config['table_name'] = "branchsales";
//connect to host
mysql_connect($config['mysql_host'],$config['mysql_user'],$config['mysql_pass']);
//select database
#mysql_select_db($config['db_name']) or die( "Unable to select database");
$xml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>";
$root_element = $config['table_name'];
$xml = "<$root_element>";
/*$title = $doc->createElement("branchsales");
$title = $root->appendChild($title);
$text = $doc->createTextNode("sales");
$text = $title->appendChild($text);
*/
//select all items in table
$sql = "SELECT branch.branchname,branchadmin.username,branchcomp.*,branchsales.*,days.day
FROM branchsales,branch,branchadmin,branchcomp,days
WHERE
branchsales.comp_id = branchcomp.comp_id
AND branchsales.admin_id = branchadmin.admin_id
AND branchsales.branch_id = branch.branch_id
AND branchsales.day_id = days.day_id";
$result = mysql_query($sql);
if (!$result) {
die('Invalid query: ' . mysql_error());
}
if(mysql_num_rows($result)>0)
{
while($result_array = mysql_fetch_assoc($result))
{
$xml .= "<".$config['table_name'].">";
//loop through each key,value pair in row
foreach($result_array as $key => $value)
{
//$key holds the table column name
$xml .= "<$key>";
//embed the SQL data in a CDATA element to avoid XML entity issues
$xml .= "<![CDATA[$value]]>";
//and close the element
$xml .= "</$key>";
}
$xml.="</".$config['table_name'].">";
}
//close the root element
$xml .= "</$root_element>";
//send the xml header to the browser
header ("Content-Type:text/xml");
echo $xml;
?>
` which looks like this
<branchsales>
<branchname>Branch1</branchname>
<username>garfield</username>
<comp_id>1</comp_id>
<admin_id>1</admin_id>
<pcnum>1</pcnum>
<starttime>09:00:00</starttime>
<endtime>10:00:00</endtime>
<totaltime>1:00:00</totaltime>
<compcharge>10.00</compcharge>
<id>1</id>
<branch_id>1</branch_id>
<day_id>5</day_id>
<timeopened>8:00:00</timeopened>
<timeclosed>23:00:00<timeclosed>
blah blah.. so on..
The thing is, I want that generated xml to be sent out to comp 2,
looking like this in a table
|ID|Day |Watcher |Branch | Current Date | Time Opened | Time closed | PC No. | so on...
1 Friday garfield Branch1 29-03-13 8:00:00 23:00:00 1
THIS IS MY SEND CODE, BUT ITS NOT WORKING
<?
php
$file = 'http://localhost/thesis/xmldailyrep.php';
$xml_builder = 'simplexml_load_file($file)';
$ch = curl_init("http://172.16.0.55/dailyrep1.php");
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type:text/xml'));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $xml_builder);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_REFERER, "http://localhost/thesis/xmldailyrep.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$ch_result = curl_exec($ch);
curl_close($ch);
echo $ch_result;
?>
MY RECEIVE CODE IN COMP 2 is this
<?php
/*
* XML Server.
*/
// We use php://input to get the raw $_POST results.
$xml_post = file_get_contents('xmldailyrep.php');
// If we receive data, save it.
if ($xml_post) {
$xml_file = 'received_xml_' . date('Y_m_d-H-i-s') . '.xml';
$fh = fopen($xml_file, 'w') or die();
fwrite($fh, $xml_post);
fclose($fh);
// Return, as we don't want to cause a loop by processing the code below.
return;
}
?>
PLEASE HELP
As far as I know, from the title. I will use frameworks to do this work. Like Apache Camel, Mule ESB. If its going to be a large scale implementation.
If you can tell us the whole story, it could be easier to help you.
-Guru
#gnanagurus
$file = 'http://localhost/thesis/xmldailyrep.php';
$xml_builder = file_get_contents($file);

Inserting Query String Variables in XML using PHP/curl

I'm trying to send some information to our CRM from a form on our site and am getting stuck on inserting the variables into the XML. Here is a simplified version of my code. Notice where I'm trying to insert the $email variable within the XML variable...which is not working.
<?php
$email = $_GET["email"];
$xml = '<xmlrequest>
<details>
<emailaddress>$email</emailaddress>
<mailinglist>8</mailinglist>
<format>html</format>
<confirmed>no</confirmed>
</details>
</xmlrequest>
';
$ch = curl_init('http://mysite.com/xml.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $xml);
$result = #curl_exec($ch);
if ($result === false) {
echo "Error performing request";
} else {
$xml_doc = simplexml_load_string($result);
header( "Location: http://mysite.com/confirmation?email=$email" ) ;
//echo 'Status is ', $xml_doc -> status, '<br/>';
if ($xml_doc -> status == 'SUCCESS') {
echo 'Data is ', $xml_doc -> data, '<br/>';
} else {
echo 'Error is ', $xml_doc -> errormessage, '<br/>';
}
}
?>
If I just type in an email address value for the API works fine. However, I'm clueless on how to pull this in dynamically from a PHP variable. Any help is greatly appreciated!
The string definition is bad
use this
$xml = "<xmlrequest>
<details>
<emailaddress>{$email}</emailaddress>
<mailinglist>8</mailinglist>
<format>html</format>
<confirmed>no</confirmed>
</details>
</xmlrequest>";
or this
$xml = '<xmlrequest>
<details>
<emailaddress>' . $email . '</emailaddress>
<mailinglist>8</mailinglist>
<format>html</format>
<confirmed>no</confirmed>
</details>
</xmlrequest>';
Because this variable probably can be various string I think it's better if you use <![CDATA[]]> section around the email.

parsing multiple urls from the form

I'm trying to make a script, which search the list of urls given in the form for the email adresses. Could anyone advice me how to do it? Is there some alternative to cURL?
I tried to make it with file_get_contents, but the script analyze only the last url given in the form: when I enter for example two urls to the form, the first "print_r("show current_url:". $current_url); is empty and for the second it shows the page(url) content(without pictures).
I asked on different forums, but received no answer. Will really appraciate your help.
Thank you
$urls = explode("\n", $_POST['urls']);
$db = new mysqli('localhost', 'root', 'root', 'urls');
if (mysqli_connect_errno()) {
echo 'Błąd: ';
exit;
}
for ($i=0; $i<count($urls); $i++){
print_r("show link:". $urls[$i]."<br>");
$current_url = file_get_contents($urls[$i]);
print_r("show current_url:". $current_url);
preg_match( "/[\._a-zA-Z0-9-]+#[\._a-zA-Z0-9-]+/i", $current_url, $email);//email
print_r ("show email:".$email[0]);
$query = "INSERT INTO urle set adres = '$email[0]' ";
$result = $db->query($query);
}
if ($query) {
echo $db->affected_rows ."pozycji dodano.";
} else {
echo mysql_errno() . ":" . mysql_error() . "Wystąpił błąd przy dodawaniu urli ";
}
$db->close();
?>
EDIT:
I have tried with curl. var_dump($email); shows: array(0) { }
The script displays now all of the urls given in the form in the browser, but preg_match doesn't work, so it doesn't extract email adresses.
<?php
$urls = explode("\n", $_POST['urls']);
$db = new mysqli('localhost', 'root', 'root', 'linki');
if (mysqli_connect_errno()) {
echo 'Błąd: ';
exit;
}
for ($i=0; $i<count($urls); $i++){
$url = $urls[$i];
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_URL, $url);
$output = curl_exec($ch);
preg_match( "/[\._a-zA-Z0-9-]+#[\._a-zA-Z0-9-]+/i", $output, $email);//email
var_dump($email);
$query = "INSERT INTO urle set adres = '$email[0]' ";
$result = $db->query($query);
curl_close($ch);
}//
if ($result) {
echo $db->affected_rows ."pozycji dodano.";
} else {
echo mysql_errno() . ":" . mysql_error() . "Wystąpił błąd przy dodawaniu urli ";
}
$db->close();
?>
Is there some alternative to cURL?
file_get_contents, which doesn't give you any error messages (unless error_reporting is raised), and which is often blocked unless ini_set("user_agent", ...) was set.
Alternatively HttpRequest on newer PHP installations.
Still curl is not difficult to use. The manual is full of examples.
the first "print_r("show current_url:". $current_url); is empty
Nobody can tell. It's your duty to debug that (especially since you haven't mentioned the affected url in your question). Use curl or httprequest.
Ok, i've fixed it!!!:)
Here is the code:
for ($i=0; $i<count($linki); $i++){
$url = $linki[$i];
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result =curl_exec($ch);
curl_close($ch);
preg_match("/[-a-z0-9\._]+#[-a-z0-9\._]+\.[a-z]{2,4}/", $result, $email);//email
print_r($email);
$zapytanie = "INSERT INTO urle set adres = '$email[0]' ";
$wynik = $db->query($zapytanie);
}

Categories