Storing a shapefile into postgresql using PHP - php

I'm trying to develop a PHP script that lets users upload shapefiles to import to a postGIS database.
First of all, for the conversion part, AFAIK we can use shp2pgsql to convert the shapefile to a postgresql table; I was wondering if there is another way of doing the conversion, as I would prefer not to use the exec() command.
I would also appretiate any idea on storing the data in a way that does not require dozens of uniquenamed tables.

There seems to be no other way than using the postgresql's binary to convert the shapefile. Although it is not really a bad choice, I would rather not use exec() if there is a PHP native function, or apache module to do it!
However, it sounds like exec is the only sane option available. So I'm going to use it.
No hard feelings! :)
About the last part, it's a different question and should be asked separately. Although, I'm afraid there is no other way of doing it.
UPDATE example added
$queries = shell_exec("shp2pgsql -s ".SRID." -c $shpfilpath $tblname")
or respond(false, "Error parsing the shapfile.");
pg_query($queries) or respond(false, "Query failed!");
SRID is a constant containing the "SRID"!
$shpfilpath is a path to the desired ShapeFile
$tblname is the desired name for the table

See this blog post about loading shapefiles using the PHP shapefile reader plugin from here. http://www.phpclasses.org/package/1741-PHP-Read-vectorial-data-from-geographic-shape-files.html. The blog post focuses on using PHP on the backend to load data for a Flash app, but you should be able to ignore the flash part and use the PHP portion for your needs.
Once you have the data loaded from the shapefile, you could convert the geometry to a WKT string and use ST_GeomFromText or other PostGIS functions to store in the database.
Regarding the unique columns for a shapefile, I've found that to be the most straightforward way to store ad-hoc shapefile attributes and then retrieve that data. However, you could use a "tuple" system, and convert the attributes to strings, then store them in arbitrarily named columns (col1, col2, col3, etc.) if you don't care about attribute names or types.
If you cared about names and types, you could go one step further and store them as a shapefile "schema" in another table.

Write your shp2pgsql and define its parameters using text editor ie
sublime notepad etc.
Copy, paste and change shapefile name for each
layer.
Save as a batch file .bat.
Pull up command window.
Pull up directory with where yu .bat file is saved.
Hit enter and itll run the code for all your shapefiles and they will be uploaded to the
database you defined in writing your code.
Use qgis, go to postgis window and hit connect.
You are good to go your shapefiles are now ready to go and can be added as layers to your map. Make sure the spatial reference matches what they were prior to running it. Does that make sense? I hope that helped its the quickest way.

Adding this answer just for the benefit of anyone who is looking for the same as the OP and does not want to rely on eval() nor external tools.
As of August 2019, you could use PHP Shapefile, a free and open source PHP library I have been developing and maintaining for a few years that can read and write any ESRI Shapefile and convert it natively from/to WKT and GeoJSON, without any third party dependency.
Using my library, which provides WKT to use with PostGIS ST_GeomFromText() function and an array containing all the data to perform a simple INSERT, makes this task trivial, fast and secure, without the need of evil eval().

Related

Searching text in pdf using php

I am having a big database roughly it has 5 lakh (500K) entries now all those entries also have some document associated with them (i.e. every id has at least pdf file). Now I need a robust method to search for a particular text in those pdf files and if I find it, it should return the respective 'id'
kindly share some fast and optimized ways to search text in a pdf using PHP. Any idea will be appreciated.
note: Changing the pdf to text and then searching is not what I am looking for obviously, it will take a longer time.
In one line I need the best way to search for text in pdf using PHP
If this is a one-time task, there is probably no 'fast' solution.
If this is a recurring task,
Extract the text via some tool. (Sorry, I don't know of a tool.)
Store that text in a database table.
Apply a FULLTEXT index to that table.
Now the searching will be fast.
I myself wrote a website in ReactJS to search for info in PDF files (indexed books), which I indexed using Apache SOLR search engine.
What I did in React is, in essence:
queryValue = "(" + queryValueTerms.join(" OR ") + ")"
let query = "http://localhost:8983/solr/richText/select?q="
let queryElements = []
if(searchValue){
queryElements.push("text:" + queryValue)
}
...
fetch(query)
.then(res => res.json())
.then((result) =>{
setSearchResults(prepareResults(result.response.docs, result.highlighting))
setTotal(result.response.numFound)
setHasContent(result.response.numFound > 0)
})
Which results in a HTTP call:
http://localhost:8983/solr/richText/select?q=text:(chocolate%20OR%20cake)
Since this is ReactJS and just parts of code, it is of little value to you in terms of PHP, but I just wanted to demonstrate what the approach was. I guess you'd be using Curl or whatever.
Indexing itself I did in a separate service, using SolrJ, i.e. I wrote a rather small Java program that utilizes SOLR's own SolrJ library to add PDF files to SOLR index.
If you opt for indexing using Java and SolrJ (was the easiest option for me, and I didn't do Java in years previously), here are some useful resources and examples, which I collected following extensive search for my own purposes:
https://solr.apache.org/guide/8_5/using-solrj.html#using-solrj
I basically copied what's here:
https://lucidworks.com/post/indexing-with-solrj/
and tweaked it for my needs.
Tip: Since I was very rusty with Java, instead of setting classpaths etc, quick solution for me was to just copy ALL libraries from SOLR's solrj folder, to my Java project. And possibly some other libraries. May be ugly, but did the job for me.

Slow searching in php

I'm new in php and mysql. Now i facing a problem is i need search data in a large database, but it take more than 3 minute to search a word, sometime the browser show timeout. I using technique FULLTEXT to do a searching, so any solution to decrease the searching time?
create index for the table field which you will prefer subsequently, even it take some memory space query result should return best results within less time.
This doesn't answer your question directly but is a suggestion:
I had the same problem with full text search so I switched to SOLR:
http://lucene.apache.org/solr/
It's a search server based on the Lucene library written in Java. It's used by some of the largest scale websites:
http://wiki.apache.org/solr/PublicServers
So speed and scalability isn't an issue. You don't need to know Java to implement it however. It offers a REST interface that you can query and even gives the option to return the search results in PHP array format.
Here's the official tutorial:
https://builds.apache.org/job/Solr-trunk/javadoc/doc-files/tutorial.html
SOLR searches through indexed files so you need to get your database contents into xml or json files. You can use the Data Import Handler extension for that:
http://wiki.apache.org/solr/DataImportHandler
To query the REST interface you can simply use get_file_contents() php function or CURL. Or the PHP sdk for SOLR:
http://wiki.apache.org/solr/SolPHP
Depends on how big your database is. Adding an index for the field you are searching is the first thing to do.
I have been into the same problem and adding an index for the field worked great.

Shortest and fastest way to parse php data

I have files I need to convert into a database. These files (I have over 100k) are from an old system (generated from a COBOL script). I am now part of the team that migrate data from this system to the new system.
Now, because we have a lot of files to parse (each files is from 50mb to 100mb) I want to make sure I use the right methods in order to convert them to sql statement.
Most of the files have these following format:
#id<tab>name<tab>address1<tab>address2<tab>city<tab>state<tab>zip<tab>country<tab>#\n
the address2 is optional and can be empty
or
#id<tab>client<tab>taxid<tab>tagid<tab>address1<tab>address2<tab>city<tab>state<tab>zip<tab>country<tab>#\n
these are the 2 most common lines (I'll say around 50%), other than these, all the line looks the same but with different information.
Now, my question is what should I do to open them to be as efficient as possible and parse them correctly?
Honestly, I wouldn't use PHP for this. I'd use awk. With input that's as predictably formatted as this, it'll run faster, and you can output into SQL commands which you can also insert via a command line.
If you have other reasons why you need to use PHP, you probably want to investigate the fgetcsv() function. Output is an array which you can parse into your insert. One of the first user-provided examples takes CSV and inserts it into MySQL. And this function does let you specify your own delimiter, so tab will be fine.
If the id# in the first column is unique in your input data, then you should definitely insert this into a primary key in mysql, to save you from duplicating data if you have to restart your batch.
When I worked on a project where it was necessary to parse huge and complex log files (Apache, firewall, sql), we had a big gain in performance using the function preg_match_all(less than 10% of the time required using explode / trims / formatting).
Huge files (>100Mb) are parsed in 2 or 3 minutes in a core 2 duo (the drawback is that memory consumption is very high since it creates a giant array with all the information ready to be synthesized).
Regular expressions allow you to identify the content of line if you have variations within the same file.
But if your files are simple, try ghoti suggestion (fgetscv), will work fine.
If you're already familiar with PHP then using it is a perfectly fine tool.
If records do not span multiple lines, the best way to do this to guarantee that you won't run out of memory will be to process one line at a time.
I'd also suggest looking at the Standard PHP Library. It has nice directory iterators and file objects that make working with files and directories a bit nicer (in my opinion) than it used to be.
If you can use the CSV features and you use the SPL, make sure to set your options correctly for the tab characters.
You can use trim to remove the # from the first and last fields easily enough after the call to fgetcsv
Just sit and parse.
It's one-time operation and looking for the most efficient way makes no sense.
Just more or less sane way would be enough.
As a matter of fact, most likely you'll waste more overall time looking for the super-extra-best solution. Say, your code will run for a hour. You will spend another hour to find a solution that runs 30% faster. You'll spend 1,7 hours vs. 1.

The simplest way to allow a web user to update a text file using PHP and Javascript?

Problem:
I don't know the simplest way to allow a single web viewer to update data in a text file on a server. (ie. only 1 person will be changing the data.)
Objective:
To make a prototype web application just one person needs to input in the start and end dates of new assignments and locations of staff and the whole company can visualize the information on a GANTT chart, probably using this Jquery libary.
Constraints:
My data is about the equivalent size of 1000 of these javascript list of lists like
*data = [["John Smith" , "assigment" , "1/1/10", "1/1/11", "Peru"],[...],...]*
Employee assignment data must be on an internal server.
I can't use a database (such as SQlite or MySQL).
I can only use PHP, Javascript, and jQuery.
Fact: Javascript cant directly change a data file sitting on the server.
My tentative fuzzy solution:
On client-side: use jQuery getJSON() to pass the data back and forth between dataReadWriter.php.
On server-side: dataReadWriter.php modifies a PHP array as well as writes modified data and reads JSONdata.txt stored in a text file on our internal server.
Given the constraints, it can't be done a lot smarter than what you are suggesting. One thing though, you shouldn't overwrite the only file containing the data, at least switch back and forth between two files, and make sure that your program does not overwrite the other file if one of the files show any signs of being damaged. You can use a PHP session to keep track of which file is the most recent, but better have some in-file timestamps as a fallback.
Is there anything in particular that you worry about?

How to change a string globally in a MySQL database

I have an old drupal site that I'd like to upgrade, but I need to move all the site data files (like jpgs, gifs, etc.) from /files to /sites/default/files.
I'd like to use a PHP script or just a MySQL command to find any instance of /files/* and change it to /sites/default/files/* (without messing up the string in the * part of the name, of course!).
Is this pretty easy to do? Any pointers on a function I could use?
MySQL does have some built-in string replacement functions. How about something like this?
UPDATE table SET field = REPLACE(field,'/files/','/sites/default/files/');
There's other functions you can use for more complex replacements (ie. regular expressions) if you need as well.
I'm pretty sure that it's just a case of changing the 'files' path in the Drupal configuration.
If you're just changing the files table, you can do an UPDATE with SQL, like zombat said. If you have a significant number of other instances of the paths ( IE - full HTML node bodies and the like ) your best bet would be to export the DB to a text file (can do it with mysqldump or the export feature of PHPMyAdmin) and then just update the strings there - either with a suitable text editor, a command-line tool like sed or a bunch of interns.

Categories