PHP loop the INSERT MySQL for each result - php

Question

I have already a script which scrapes all the urls of one csv with simple HTML dom.
The output is like this:
CoolerMaster Devastator II Azul
Coolbox DeepTeam - Combo teclado, ratón y alfombrilla
Asus Claymore RED - Teclado gaming
INSERT INTO productos (nombre) VALUES('Asus Claymore RED - Teclado gaming')
Items added to the database!
INSERT INTO productos (nombre) VALUES('Asus Claymore RED - Teclado gaming')
Items added to the database!
INSERT INTO productos (nombre) VALUES('Asus Claymore RED - Teclado gaming')
Items added to the database!
As you can see, the scrape contains 3 different products, but when I try to insert to the MySQL database, it only saves the last product --- but three times.
Here you can see my PHP Code for that:
<?php
require 'libs/simple_html_dom/simple_html_dom.php';
set_time_limit(0);
function scrapUrl($url)
{
$html = new simple_html_dom();
$html->load_file($url);
global $name;
$names = $html->find('h1');
foreach ($names as $name) {
echo $name->innertext;
echo '<br>';
}
$rutaCSV = 'csv/urls1.csv'; // Ruta del csv.
$csv = array_map('str_getcsv', file($rutaCSV));
foreach ($csv as $linea) {
$url = $linea[0];
scrapUrl($url);
}
$servername = "localhost";
$username = "";
$password = "";
$dbname = "";
// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
foreach ($csv as $linea) {
$url = $linea[0];
$sql = "INSERT INTO productos (nombre) VALUES('$name->plaintext')";
print ("<p> $sql </p>");
if ($conn->query($sql) === TRUE) {
echo "Items added to the database!";
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
}
$conn->close();
?>
So, what I need is the MySQL query add:
INSERT INTO productos (nombre) VALUES('CoolerMaster Devastator II Azul')
Items added to the database!
INSERT INTO productos (nombre) VALUES('Coolbox DeepTeam - Combo teclado, ratón y alfombrilla')
Items added to the database!
INSERT INTO productos (nombre) VALUES('Asus Claymore RED - Teclado gaming')
Items added to the database!

Answer 1

You have a bunch of problems in your code.
First, you have function scrapUrl, that takes $url as an argument, but doesn't output anyhting. It's setting global $name variable, but, although it's find several names, it putting only the last one to the $name variable, because it's walking through a series of $names, put it's text into $name, and go for the next one, so, only last item is stored to your $name variable.
I would recommend, that your change your scrapUrl function, so it store names of scrapped products into an array, and return that array.
Second, I'm cannot understand how do you put your data into a csv file, the code, you've privided looks like it shouldn't work properly. Are you sure, that you are writing the right data in a csv file? Maybe here you are just reading data from file - in that case, I'm sorry.
The third: you are reading data from csv, and when moving line by line in the cycle, but the data is going nowhere. To my opinion, you should but $linea[0] into your SQL query, but you are putting $name->plaintext where, when $name is set only once in your scrapUrl, as I've mentioned above.
I would recommend, that you use the right variable in your SQL-query to pass data to it.
Also, it's better to use PDO and prepared statements instead of inserting raw data in your string-literals SQL queries.

Answer 2

Here is your code, just formatted: ( please check it you have a missing } )
function scrapUrl($url)
{
$html = new simple_html_dom();
$html->load_file($url);
global $name; // -- using global is crap - I would avoid that. Pass the object in as an argument of the function eg. scrapUrl($url, $name)
$names = $html->find('h1');
foreach ($names as $name) {
// -- your re-assigning $name overwriting you global on each iteration of this loop
// -- What is the purpose of this? it does nothing but output?
echo $name->innertext;
echo '<br>';
}
// -- missing } where is this function closed at?
$rutaCSV = 'csv/urls1.csv'; // Ruta del csv.
$csv = array_map('str_getcsv', file($rutaCSV));
foreach ($csv as $linea) {
// -- this can be combined with the one with the query
// -- just put the function call in that one and delete this one
$url = $linea[0];
scrapUrl($url); //recursive? depends where you function is closed
// -- whats the purpose of this function, it returns nothing?
}
$servername = "localhost";
$username = "";
$password = "";
$dbname = "";
// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
foreach ($csv as $linea) {
$url = $linea[0]; // -- whats this url used for?
$sql = "INSERT INTO productos (nombre) VALUES('$name->plaintext')";
// -- query is vulnerable to SQL injection? prepared statement
// -- whats $name->plaintext? where is it assigned at?
print ("<p> $sql </p>");
if ($conn->query($sql) === TRUE) {
echo "Items added to the database!";
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
// -- when you loop over the CSV but insert $name->plaintext multiple times
// -- where is that property changed inside this loop, how is it correlated to the csv data
}
$conn->close();
So first off you are missing a closing } Depending where that should be, depends on what else you have wrong.
One of you loops for the CSV can be eliminated ( maybe ), anyway I put bunch of notes in with comments like this // --
Your main issue, or the reason you inserts are the same is these lines
foreach ($csv as $linea) {
$url = $linea[0]; // -- whats this url used for?
$sql = "INSERT INTO productos (nombre) VALUES('$name->plaintext')";
// -- $name->plaintext does not change per iteration of the loop
// -- you are just repeatedly inserting that data
...
See you insert the value of $name->plaintext but this has no correlation to the $csv variable and you are not modifying it. It's no surprise it stays the same.
Ok, now that I picked apart your code ( nothing personal ). Let's see if we can simplify it a bit.
UPDATE This is the best I can do given the above code. I just combined it, fixed some logical errors, trimmed it down and simplified it. It's a common mistake of beginners to over-complicate the task. ( but there is no way for me to test this )
<?php
$servername = "localhost";
$username = "";
$password = "";
$dbname = "";
// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
$rutaCSV = 'csv/urls1.csv'; // Ruta del csv.
$csv = array_map('str_getcsv', file($rutaCSV));
//prepare query outside of the loops
$stmt = $conn->prepare("INSERT INTO productos (nombre)VALUES(?)");
foreach ($csv as $linea) {
//iterate over each csv line
$html = new simple_html_dom();
//load url $linea[0]
$html->load_file($linea[0]);
//find names in the document, and return them
foreach( $html->find('h1') as $name ){
//iterate over each name and bind elements text to the query
$stmt->bind_param('s', $name->plaintext);
if ($stmt->execute()){
echo "Items added to the database!";
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
}
}
There I further simplified it as it doesn't really make sense to have the function scrapUrl(). We're not re-using that code, so it adds a function call and makes the code harder to read by having it.
Even if it doesn't work strait away, I encourage you to compare the original code to what I have. And sort of walk through it in your mind, so you can get an feel for how I removed some of those redundancies etc.
For reference
mysqli prepare: http://php.net/manual/en/mysqli.prepare.php
mysqli bind_param: http://php.net/manual/en/mysqli-stmt.bind-param.php
mysqli execute: http://php.net/manual/en/mysqli-stmt.execute.php
Hope that helps, cheers!

Answer 3

Well, after been thinking about this for quite some time, I've managed to make it work.
I leave the code in case someone else can use it.
<?php
require 'libs/simple_html_dom/simple_html_dom.php';
set_time_limit(0);
function scrapUrl($url)
{
$html = new simple_html_dom();
$html->load_file($url);
global $name;
global $price;
global $manufacturer;
$result = array();
foreach($html->find('h1') as $name){
$result[] = $name->plaintext;
echo $name->plaintext;
echo '<br>';
}
foreach($html->find('h2') as $manufacturer){
$result[] = $manufacturer->plaintext;
echo $manufacturer->plaintext;
echo '<br>';
}
foreach($html->find('.our_price_display') as $price){
$result[] = $price->plaintext;
echo $price->plaintext;
echo '<br>';
}
$servername = "localhost";
$username = "";
$password = "";
$dbname = "";
// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
$price_go=str_replace(",",".",str_replace(" €","",$price->plaintext));
$sql = "INSERT INTO productos (nombre, nombreFabricante, precio) VALUES('$name->plaintext', '$manufacturer->plaintext', $price_go)";
print ("<p> $sql </p>");
if ($conn->query($sql) === TRUE) {
echo "Producto añadido al comparador!";
echo '<br>';
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
$conn->close();
//echo $url;
}
$rutaCSV = 'csv/urls1.csv'; // Ruta del csv.
$csv = array_map('str_getcsv', file($rutaCSV));
//print_r($csv); // Verás que es un array donde cada elemento es array con una de las url.
foreach ($csv as $linea) {
$url = $linea[0];
scrapUrl($url);
}
?>
I'm pretty sure i have some trash in my code, but it works.
I hope it help for someone.
Regards and thanks for all the help.