I have been struggling to create something that can save txt files into a mySQL database. I have managed to create something that saves JSON files but not txt files.
Here is the txt file in question: https://celestrak.org/NORAD/elements/sbas.txt. This txt file contains a few satellites with their data. Each satellite has exactly three lines, no exceptions. So, for example here is one satellite:
AOR-E (EGNOS/PRN 120)
1 24307U 96053A 17257.68868765 -.00000150 00000-0 00000-0 0 9992
2 24307 2.8040 77.2609 0004175 104.1816 44.8421 1.00271450 76939
The first lines tells us the satellite name. The next two lines give us some parameters, which always start with the numbers 1 and 2. This format will not change - the name on line 0 and the two lines after it, which start in 1 or 2.
What I want to be able to do is to create a row for each satellite - with the columns object_name for line 0, tle_line1 for line 1 and tle_line2 for line 2.
I have managed to create something that saves data from a JSON format into the SQL database. Maybe some can be deviated from that?
I am using Laravel and Guzzle for the HTTP requests:
$api = new Client([
'base_uri' => 'https://celestrak.org',
]);
$response = $api->get('jsonlocater');
$data = json_decode($response->getBody()->getContents(), true);
foreach ($data as $attributes) {
$attributes = array_change_key_case($attributes, CASE_LOWER);
Satellites::create($attributes);
}
First of all, I'm not sure what is your response format but using vanilla PHP you may do something like the following to fetch the contents in array:
$url = 'http://celestrak.com/NORAD/elements/sbas.txt';
$lines = file($url, FILE_IGNORE_NEW_LINES);
$arrays = array_map(function($array) {
$columns = ['object_name', 'tle_line1', 'tle_line2'];
return array_combine($columns, array_map('trim', $array));
}, array_chunk($lines, 3));
Now, if you dd($arrays) the result then you'll get something like the following:
From this result, you should be easily able to create entries in your database. Each array in image should be an entry/row in your database table. For example:
\DB::table('table_name')->insert($arrays);
Note that, if you've timestamps (created_at & updated_at) in your table then you've to add those fields in each array when generating the arrays.
thanks for reading!
I have an app that allows people to add, edit and delete items in a CSV. I've encountered a bug where if there are non-unique IDs and you try to edit or delete them, it will edit or delete all of them, as the system parses through the spreadsheet to find the ID - which also corresponds to the object's order when using it so the user must be able to change the ID
The solution I've come up with is quite simple, should the user edit an object and change its ID to one that already exists, then the system will take all of the objects with an ID bigger than or equal to the new ID and increment them all by one.
The following code is my if statement that checks whether the ID already exists
if($exists == "true") //does the $newImageID already exist in the gallery?
{
$table = fopen($fullURL,'r'); //$fullURL is the location of the CSV tested and works
$temp_table_two = fopen($tempURL,'w');
while (!feof($temp_table_two) ) {
$getid = fgetcsv($temp_table_two, 1024);
if($getid[0] >= $newImageID)
{
// $getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
echo $getid[0];
}
}
fclose($table);
fclose($temp_table);
rename($tempURL,$fullURL);
}
This code takes place after fopen and before fclose. In context, $exists is either "true" or "false" (will change to boolean later on), the while loop parses through my $temp_table (a fopen) and if the first column object (the ID) is equal to or bigger than the one in the new ID then it is incremented. This means that the new object gets "slotted in" so to speak and pushes the rest down
Strangely my request is timing out after a long spinner after I execute this code and I have no idea what the problem is
Thanks for all your help in advance
EDIT: I have found the source of the problem is the while loop itself, should I comment everything out as such:
while (!feof($temp_table_two) ) {
$getid = fgetcsv($temp_table_two, 1024);
// if($getid[0] >= $newImageID)
// {
// // $getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
// echo $getid[0];
// }
}
The code still doesn't work yet the only thing left to run is the loop that doesn't do anything
EDIT 2:
Following an answer, I did away with the temp table and just work from the table itself, this if statement is executed BEFORE adding the new data with its ID
if($exists == "true") //does the $newImageID already exist in the gallery?
{
$table = fopen($fullURL,'r+');
while (!feof($table) ) {
$getid = fgetcsv($table, 1024);
if($getid[0] >= $newImageID)
{
echo $getid[0];
$getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
}
}
fclose($table);
}
The code no longer times out, but the items inside $getid[0] are not incremented. I have echoed them and it does echo all of the ID's equal to or bigger than my $newImageID but the $getid[0]++; doesn't seem to be affecting the CSV at all
You are testing if you reach the end of the temp file and that's wrong. You need to check the origin file and also read from it!
while (!feof($table) ) {
$getid = fgetcsv($table, 1024);
Try this:
if ($csv = fopen($temp_table_two, 'r+')) do {
$getid = fgetcsv($csv, 1024);
if($getid[0] >= $newImageID)
{
echo $getid[0]; // $getid[0]++;
}
} while (!feof($csv));
That will prevent your while loop from timing out due to being stuck in an infinite if there is a problem opening the file. feof will return true only if it reaches EOF, it will return false otherwise which will cause it to never be able to break out.
For actually writing your data back to the CSV file, your current code won't work as fgetcsv just gives you an array representation of a CSV line in the file. Writing to that array just changes the array, not back to the file.
For that, see this similar answer: Append data to middle line/row of a csv instead of the last line or row
or
http://php.net/manual/en/function.fputcsv.php
I wanna improve on how to fetch data from an API. In this case I want to fetch every app-id from the Steam API, and list them one per line in a .txt file. Do I need an infinite (or a very high-number) loop (with ++ after every iteration) to fetch everyone? I mean, counting up from id 0 with for example a foreach-loop? I'm thinking it will take ages and sounds very much like bad practice.
How do I get every appid {"appid:" n} from the response of http://api.steampowered.com/ISteamApps/GetAppList/v0001?
<?php
//API-URL
$url = "http://api.steampowered.com/ISteamApps/GetAppList/v0001";
//Fetch content and decode
$game_json = json_decode(curl_get_contents($url), true);
//Define file
$file = 'steam.txt';
//This is where I'm lost. One massive array {"app": []} with lots of {"appid": n}.
//I know how to get one specific targeted line, but how do I get them all?
$line = $game_json['applist']['apps']['app']['appid'][every single line, one at a time]
//Write to file, one id per line.
//Like:
//5
//7
//8
//and so on
file_put_contents($file, $line, FILE_APPEND);
?>
Any pointing just in the right direction will be MUCH appreciated. Thanks!
You don't need to worry about counters with foreach loops, they are designed to go through and work with each item in the object.
$file = "steam.txt";
$game_list = "";
$url = "http://api.steampowered.com/ISteamApps/GetAppList/v0001";
$game_json = file_get_contents($url);
$games = json_decode($game_json);
foreach($games->applist->apps->app as $game) {
// now $game is a single entry, e.g. {"appid":5,"name":"Dedicated server"}
$game_list .= "$game->appid\n";
}
file_put_contents($file, $game_list);
Now you have a text file with 28000 numbers in it. Congratulations?
Each time someone lands in my page list.php?id=xxxxx it requeries some MySQL queries to return this:
$ids = array(..,..,..); // not big array - not longer then 50 number records
$thumbs = array(..,..,..); // not big array - not longer then 50 text records
$artdesc = "some text not very long"; // text field
Because the database from which I make the queries is quite big I would like to cache this results for 24h in maybe a file like: xxxxx.php in a /cache/ directory so i can use it in include("xxxxx.php") if it is present. ( or txt files !? , or any other way )
Because there is very simple data I believe it can be done using a few of PHP lines and no need to use memcached or other professional objects.
Becasuse my PHP is very limited can someone just place the PHP main lines ( or code ) for this task ?
I really would be very thankfull !
Caching a PHP array is pretty easy:
file_put_contents($path, '<?php return '.var_export($my_array,true).';?>');
Then you can read it back out:
if (file_exists($path)) $my_array = include($path);
You might also want to look into ADOdb, which provides caching internally.
Try using serialize;
Suppose you get your data in two arrays $array1 and $array2. Now what you have to do is store these arrays in file. Storing string (the third variable in your question) to file is easy, but to store an array you have to first convert it to string.
$string_of_array1 = serialize( $array1 );
$string_of_array2 = serialize( $array2 );
The next problem is the naming of cache files so that you can easily check if the relevant array is already available in cache. The best way to do this is to create an MD5 hash of your mysql query and use it as cache file name.
$cache_dir = '/path/cache/';
$query1 = 'SELECT many , fields FROM first_table INNER JOIN another_table ...';
$cache1_filename = md5( $query1 );
if( file_exists( $cache_dir . $cache1_filename ) )
{
if( filemtime( $cache_dir . $cache1_filename ) > ( time( ) - 60 * 60 * 24 ) )
{
$array1 = unserialize( file_get_contents( $cache_dir . $cache1_filename ) );
}
}
if( !isset( $array1 ) )
{
$array1 = run_mysql_query( $query1 );
file_put_contents( serialize( $array1 ) );
}
Repeat the above with the other array that should be stored in a separate file with MD5 of the second query used as the name of second cache file.
In the end, you have to decide how long your cache should remain valid. For the same query, there may change records in your mysql table that may make your file system cache outdated. So, you cannot just rely on unique file names for unique queries.
Important:
Old cache files have to be deleted. You may have to write a routine that checks all files of a directory and deletes the files older than n seconds.
Keep the cache dir outside the webroot.
Just write a new file with the name of the $_GET['id'] and contents of the stuff you want cached, and each time check to see if that file exists, else create one. Something like this:
$id = $_GET['id']
if (file_exists('/a/dir/' . $id)) {
$data = file_get_contents('/a/dir/' . $id);
} else {
//do mysql query, set data to result
$handle = fopen('/a/dir/' . $id, 'w+');
fwrite($handle, $data);
fclose($handle);
}
Based on #hamid-sarfraz answer, here is a solution used in PDO extended class, using json_encode/decode instead of serialize :
function get_assoc_c($query, $lifetime = 60*60*24) {
$c_dir = '/path/to/.cache/';
$c_filename = md5($query);
if(file_exists($c_dir . $c_filename)) {
if(filemtime($c_dir . $c_filename) > (time() - $lifetime)) {
return json_decode(file_get_contents($c_dir . $c_filename), true);
}
}
if(!isset($content)) {
if(!file_exists($c_dir))
mkdir($c_dir);
$stmt = $this->query($query);
$content = $stmt->fetchAll(PDO::FETCH_ASSOC);
file_put_contents($c_dir . $c_filename, json_encode($content));
return $content;
}
return false;
}
! Be aware to not use queries with arguments passed as variables (SQL injection).
I have a situation where I have to update a web site on a shared hosting provider. The site has a CMS. Uploading the CMS's files is pretty straightforward using FTP.
I also have to import a big (relative to the confines of a PHP script) database file (Around 2-3 MB uncompressed). Mysql is closed for access from the outside, so I have to upload a file using FTP, and start a PHP script to import it. Sadly, I do not have access to the mysql command line function so I have to parse and query it using native PHP. I also can't use LOAD DATA INFILE. I also can't use any kind of interactive front-end like phpMyAdmin, it needs to run in an automated fashion. I also can't use mysqli_multi_query().
Does anybody know or have a already coded, simple solution that reliably splits such a file into single queries (there could be multi-line statements) and runs the query. I would like to avoid to start fiddling with it myself due to the many gotchas that I'm likely to come across (How to detect whether a field delimiter is part of the data; how to deal with line breaks in memo fields; and so on). There must be a ready made solution for this.
Here is a memory-friendly function that should be able to split a big file in individual queries without needing to open the whole file at once:
function SplitSQL($file, $delimiter = ';')
{
set_time_limit(0);
if (is_file($file) === true)
{
$file = fopen($file, 'r');
if (is_resource($file) === true)
{
$query = array();
while (feof($file) === false)
{
$query[] = fgets($file);
if (preg_match('~' . preg_quote($delimiter, '~') . '\s*$~iS', end($query)) === 1)
{
$query = trim(implode('', $query));
if (mysql_query($query) === false)
{
echo '<h3>ERROR: ' . $query . '</h3>' . "\n";
}
else
{
echo '<h3>SUCCESS: ' . $query . '</h3>' . "\n";
}
while (ob_get_level() > 0)
{
ob_end_flush();
}
flush();
}
if (is_string($query) === true)
{
$query = array();
}
}
return fclose($file);
}
}
return false;
}
I tested it on a big phpMyAdmin SQL dump and it worked just fine.
Some test data:
CREATE TABLE IF NOT EXISTS "test" (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"name" TEXT,
"description" TEXT
);
BEGIN;
INSERT INTO "test" ("name", "description")
VALUES (";;;", "something for you mind; body; soul");
COMMIT;
UPDATE "test"
SET "name" = "; "
WHERE "id" = 1;
And the respective output:
SUCCESS: CREATE TABLE IF NOT EXISTS "test" ( "id" INTEGER PRIMARY KEY AUTOINCREMENT, "name" TEXT, "description" TEXT );
SUCCESS: BEGIN;
SUCCESS: INSERT INTO "test" ("name", "description") VALUES (";;;", "something for you mind; body; soul");
SUCCESS: COMMIT;
SUCCESS: UPDATE "test" SET "name" = "; " WHERE "id" = 1;
Single page PHPMyAdmin - Adminer - Just one PHP script file.
check : http://www.adminer.org/en/
When StackOverflow released their monthly data dump in XML format, I wrote PHP scripts to load it into a MySQL database. I imported about 2.2 gigabytes of XML in a few minutes.
My technique is to prepare() an INSERT statement with parameter placeholders for the column values. Then use XMLReader to loop over the XML elements and execute() my prepared query, plugging in values for the parameters. I chose XMLReader because it's a streaming XML reader; it reads the XML input incrementally instead of requiring to load the whole file into memory.
You could also read a CSV file one line at a time with fgetcsv().
If you're inporting into InnoDB tables, I recommend starting and committing transactions explicitly, to reduce the overhead of autocommit. I commit every 1000 rows, but this is arbitrary.
I'm not going to post the code here (because of StackOverflow's licensing policy), but in pseudocode:
connect to database
open data file
PREPARE parameterizes INSERT statement
begin first transaction
loop, reading lines from data file: {
parse line into individual fields
EXECUTE prepared query, passing data fields as parameters
if ++counter % 1000 == 0,
commit transaction and begin new transaction
}
commit final transaction
Writing this code in PHP is not rocket science, and it runs pretty quickly when one uses prepared statements and explicit transactions. Those features are not available in the outdated mysql PHP extension, but you can use them if you use mysqli or PDO_MySQL.
I also added convenient stuff like error checking, progress reporting, and support for default values when the data file doesn't include one of the fields.
I wrote my code in an abstract PHP class that I subclass for each table I need to load. Each subclass declares the columns it wants to load, and maps them to fields in the XML data file by name (or by position if the data file is CSV).
Can't you install phpMyAdmin, gzip the file (which should make it much smaller) and import it using phpMyAdmin?
EDIT: Well, if you can't use phpMyAdmin, you can use the code from phpMyAdmin. I'm not sure about this particular part, but it's generaly nicely structured.
Export
The first step is getting the input in a sane format for parsing when you export it. From your question
it appears that you have control over the exporting of this data, but not the importing.
~: mysqldump test --opt --skip-extended-insert | grep -v '^--' | grep . > test.sql
This dumps the test database excluding all comment lines and blank lines into test.sql. It also disables
extended inserts, meaning there is one INSERT statement per line. This will help limit the memory usage
during the import, but at a cost of import speed.
Import
The import script is as simple as this:
<?php
$mysqli = new mysqli('localhost', 'hobodave', 'p4ssw3rd', 'test');
$handle = fopen('test.sql', 'rb');
if ($handle) {
while (!feof($handle)) {
// This assumes you don't have a row that is > 1MB (1000000)
// which is unlikely given the size of your DB
// Note that it has a DIRECT effect on your scripts memory
// usage.
$buffer = stream_get_line($handle, 1000000, ";\n");
$mysqli->query($buffer);
}
}
echo "Peak MB: ",memory_get_peak_usage(true)/1024/1024;
This will utilize an absurdly low amount of memory as shown below:
daves-macbookpro:~ hobodave$ du -hs test.sql
15M test.sql
daves-macbookpro:~ hobodave$ time php import.php
Peak MB: 1.75
real 2m55.619s
user 0m4.998s
sys 0m4.588s
What that says is you processed a 15MB mysqldump with a peak RAM usage of 1.75 MB in just under 3 minutes.
Alternate Export
If you have a high enough memory_limit and this is too slow, you can try this using the following export:
~: mysqldump test --opt | grep -v '^--' | grep . > test.sql
This will allow extended inserts, which insert multiple rows in a single query. Here are the statistics for the same datbase:
daves-macbookpro:~ hobodave$ du -hs test.sql
11M test.sql
daves-macbookpro:~ hobodave$ time php import.php
Peak MB: 3.75
real 0m23.878s
user 0m0.110s
sys 0m0.101s
Notice that it uses over 2x the RAM at 3.75 MB, but takes about 1/6th as long. I suggest trying both methods and seeing which suits your needs.
Edit:
I was unable to get a newline to appear literally in any mysqldump output using any of CHAR, VARCHAR, BINARY, VARBINARY, and BLOB field types. If you do have BLOB/BINARY fields though then please use the following just in case:
~: mysqldump5 test --hex-blob --opt | grep -v '^--' | grep . > test.sql
Can you use LOAD DATA INFILE?
If you format your db dump file using SELECT INTO OUTFILE, this should be exactly what you need. No reason to have PHP parse anything.
I ran into the same problem. I solved it using a regular expression:
function splitQueryText($query) {
// the regex needs a trailing semicolon
$query = trim($query);
if (substr($query, -1) != ";")
$query .= ";";
// i spent 3 days figuring out this line
preg_match_all("/(?>[^;']|(''|(?>'([^']|\\')*[^\\\]')))+;/ixU", $query, $matches, PREG_SET_ORDER);
$querySplit = "";
foreach ($matches as $match) {
// get rid of the trailing semicolon
$querySplit[] = substr($match[0], 0, -1);
}
return $querySplit;
}
$queryList = splitQueryText($inputText);
foreach ($queryList as $query) {
$result = mysql_query($query);
}
Already answered: Loading .sql files from within PHP
Also:
http://webxadmin.free.fr/article/import-huge-mysql-dumps-using-php-only-342.php
http://www.phpbuilder.com/board/showthread.php?t=10323180
http://forums.tizag.com/archive/index.php?t-3581.html
Splitting a query cannot be reliably done without parsing. Here is valid SQL that would be impossible to split correctly with a regular expression.
SELECT ";"; SELECT ";\"; a;";
SELECT ";
abc";
I wrote a small SqlFormatter class in PHP that includes a query tokenizer. I added a splitQuery method to it that splits all queries (including the above example) reliably.
https://github.com/jdorn/sql-formatter/blob/master/SqlFormatter.php
You can remove the format and highlight methods if you don't need them.
One downside is that it requires the whole sql string to be in memory, which could be a problem if you're working with huge sql files. I'm sure with a little bit of tinkering, you could make the getNextToken method work on a file pointer instead.
First at all thanks for this topic. This saved a lot of time for me :)
And let me to make little fix for your code.
Sometimes if TRIGGERS or PROCEDURES is in dump file, it is not enough to examine the ; delimiters.
In this case may be DELIMITER [something] in sql code, to say that the statement will not end with ; but [something]. For example a section in xxx.sql:
DELIMITER //
CREATE TRIGGER `mytrigger` BEFORE INSERT ON `mytable`
FOR EACH ROW BEGIN
SET NEW.`create_time` = NOW();
END
//
DELIMITER ;
So first need to have a falg, to detect, that query does not ends with ;
And delete the unqanted query chunks, because the mysql_query does not need delimiter
(the delimiter is the end of string)
so mysql_query need someting like this:
CREATE TRIGGER `mytrigger` BEFORE INSERT ON `mytable`
FOR EACH ROW BEGIN
SET NEW.`create_time` = NOW();
END;
So a little work and here is the fixed code:
function SplitSQL($file, $delimiter = ';')
{
set_time_limit(0);
$matches = array();
$otherDelimiter = false;
if (is_file($file) === true) {
$file = fopen($file, 'r');
if (is_resource($file) === true) {
$query = array();
while (feof($file) === false) {
$query[] = fgets($file);
if (preg_match('~' . preg_quote('delimiter', '~') . '\s*([^\s]+)$~iS', end($query), $matches) === 1){
//DELIMITER DIRECTIVE DETECTED
array_pop($query); //WE DON'T NEED THIS LINE IN SQL QUERY
if( $otherDelimiter = ( $matches[1] != $delimiter )){
}else{
//THIS IS THE DEFAULT DELIMITER, DELETE THE LINE BEFORE THE LAST (THAT SHOULD BE THE NOT DEFAULT DELIMITER) AND WE SHOULD CLOSE THE STATEMENT
array_pop($query);
$query[]=$delimiter;
}
}
if ( !$otherDelimiter && preg_match('~' . preg_quote($delimiter, '~') . '\s*$~iS', end($query)) === 1) {
$query = trim(implode('', $query));
if (mysql_query($query) === false){
echo '<h3>ERROR: ' . $query . '</h3>' . "\n";
}else{
echo '<h3>SUCCESS: ' . $query . '</h3>' . "\n";
}
while (ob_get_level() > 0){
ob_end_flush();
}
flush();
}
if (is_string($query) === true) {
$query = array();
}
}
return fclose($file);
}
}
return false;
}
I hope i could help somebody too.
Have a nice day!
http://www.ozerov.de/bigdump/ was very useful for me in importing 200+ MB sql file.
Note:
SQL file should be already present in the server so that the process can be completed without any issue
You can use phpMyAdmin for importing the file. Even if it is huge, just use UploadDir configuration directory, upload it there and choose it from phpMyAdmin import page. Once file processing will be close to the PHP limits, phpMyAdmin interrupts importing, shows you again import page with predefined values indicating where to continue in the import.
what do you think about:
system("cat xxx.sql | mysql -l username database");