I got a task to get to find out ER from json files this is the one record
{ "__v" : 0, "_id" : ObjectId( "52210bf6880b9f0200000003" ), "email" : "", "emailFrequency" : "instant", "feedbackRating" : 0, "field_aboutMe" : "On Fridays I like to eat apples.", "field_contact" : "me#matt.is", "field_intro" : "Sparkboard bug-fixer. Co-founded Hacking Health, Bodo Wellness, and BrainTripping.", "first_time" : false, "hash" : "", "name" : "Matt Huebert", "picture" : "https://www.ssss.com", "ready" : true, "roles" : [ "admin" ], "email" : "", "tags" : [ "Developer", "Designer", "Mentor" ], "updatedAt" : Date( 1384115378947 ), "votesByDomain" : { "sadsard-com" : "525eadf50f13910200000004" } }
after searching i found out it has something to do with MongoDB . I need to parse this data to find out Entities in it . when i check its validation its not a valid json as its been generated by mongo db .
Please any body have any idea ?
Ive tried this as well
$data = file_get_contents('json/users.json');
bson_decode($data);
tried json_decode as well its also not working and returning NULL
I get this error while trying to bson_decode
Fatal error: Uncaught exception 'MongoException' with message 'type 0x5f not supported
Not sure what has dumped this into your json file, possibly from stdout of mongo shell, But this part is the problem:
ObjectId( "52210bf6880b9f0200000003" )
You need to remove the Wrapping "ObjectId(" and closing brace ")" in order to make each line valid JSON. Once that is filtered you are left with valid JSON.
If you know who is providing you with these files, and cannot use the mongodb datasource directly, then ask them to please use mongoexport, which will at least give you valid JSON from the start.
Related
I'm receiving some JSON via a HTTP POST Callback to a PHP page and I'm having an issue parsing out the JSON. Here's an example of what the JSON data that is sent looks like:
[
{
"type" : "message-received",
"time" : "2016-09-14T18:20:16Z",
"description" : "Incoming message received",
"to" : "+12345678902",
"message" : {
"id" : "14762070468292kw2fuqty55yp2b2",
"time" : "2016-09-14T18:20:16Z",
"to" : ["+12345678902"],
"from" : "+12345678901",
"text" : "Hey, check this out!",
"applicationId" : "93de2206-9669-4e07-948d-329f4b722ee2",
"media" : [
"https://messaging.bandwidth.com/api/v2/users/{accountId}/media/14762070468292kw2fuqty55yp2b2/0/bw.png"
],
"owner" : "+12345678902",
"direction" : "in",
"segmentCount" : 1
}
}
]
I'm then processing this as follows:
$eventJSON = file_get_contents('php://input');
$event= json_decode( $eventJSON );
$eventType = $event->type;
but I'm not getting anything so far for my $eventType variable - I think the issue might be that the JSON is an array but I'm not sure how to handle this?
To parse json try
$eventType = $event[0]->type;
Refer :- How do I extract data from JSON with PHP? to know the difference between object properties and array elements
I am using elasticsearch in my project and my requirement pulling a large MySQL data into Elasticsearch using Elasticsearch JDBC River plugin. My need is to sync mysql table to elasticsearch so i'm creating a mapping for jdbc river index.
curl -XPOST http://localhost:9200/city -d '
{
"mappings" : {
"city_type": {
"properties" : {
"domain" : {
"type" : "multi_field",
"fields" : {
"domain" : {
"type" : "string",
"index" : "analyzed"
},
"exact" : {
"type" : "string",
"index" : "not_analyzed"
}
}
},
"sent_date" : {
"type" : "date",
"format" : "dateOptionalTime"
}
}
}
}
}'
After creating the mapping in elasticsearch . i want to load the mysql table data into it. so i'm using the following command.
curl -XPUT 'localhost:9200/river/city/_meta?pretty' -d '{
"type" : "jdbc",
"jdbc" : {
"url" : "jdbc:mysql://localhost:3306/test",
"user" : "root",
"password" : "root",
"sql" : "select id as _id,id as domain from city;",
"strategy":"oneshot"
},
"index" :{
"index" : "city",
"type" : "city_type",
"bulk_size":500
}
}'
These queries are successfully run and after these query when i run the command to find the data in elasticsearch is empty.
http://localhost:9200/river/_search?pretty&q=*
Please check the response of the above query here. Why the data is not showing in the elasticsearch query please help.
River has been deprecated https://github.com/elastic/elasticsearch/issues/10345 by the way.
I would highly recommend jprante jdbc importer which is a java stand-alone allowing to do the operations you are needing. https://github.com/jprante/elasticsearch-jdbc. It is not exactly a river as you have defined one.
Concerning your question, could you please try http://localhost:9200/_search?pretty&q=* ? With your syntax, you are actually looking for data in index river. You should look on all index with the query I wrote or in city index : http://localhost:9200/city/city_type/_search?pretty&q=*
If I were in your shoes, I would use logstash to push the data from MySQL to Elastic. River is deprecated since a long time ago as #Artholl already mentioned.
See https://www.elastic.co/blog/logstash-jdbc-input-plugin
Background Information
I have the following data in my mongo database:
{ "_id" :
ObjectId("581c97b573df465d63af53ae"),
"ph" : "+17771111234",
"fax" : false,
"city" : "abd",
"department" : "",
"description" : "a test"
}
I am now writing a script that will loop through a CSV file that contains data that I need to append to the document. For example, the data might look like this:
+17771111234, 10:15, 12:15, test#yahoo.com
+17771111234, 1:00, 9:00, anothertest#yahoo.com
Ultimately I want to end up with a mongo document that looks like this:
{ "_id" :
ObjectId("581c97b573df465d63af53ae"),
"ph" : "+17771111234",
"fax" : false,
"city" : "abd",
"department" : "",
"description" : "a test",
"contact_locations": [
{
"stime": "10:15",
"etime": "12:15",
"email": "test#yahoo.com"
},
{
"stime": "1:00",
"etime": "9:00",
"email": "anothertest#yahoo.com"
},
]
}
Problem
The code I've written is actually creating new documents instead of appending to the existing ones. And actually, it's not even creating a new document per row in the CSV file... which I haven't debugged enough yet to really understand why.
Code
For each row in the csv file, I'm running the following logic
while(!$csv->eof() && ($row = $csv->fgetcsv()) && $row[0] !== null) {
//code that massages the $row into the way I need it to look.
$data_to_submit = array('contact_locations' => $row);
echo "proving that the record already exists...: <BR>";
$cursor = $contact_collection->find(array('phnum'=>$row[0]));
var_dump(iterator_to_array($cursor));
echo "now attempting to update it....<BR>";
// $cursor = $contact_collection->update(array('phnum'=>$row[0]), $data_to_submit, array('upsert'=>true));
$cursor = $contact_collection->insert(array('phnum'=>$row[0]), $data_to_submit);
echo "AFTER UPDATE <BR><BR>";
$cursor = $contact_collection->find(array('phnum'=>$row[0]));
var_dump(iterator_to_array($cursor));
}
}
Questions
Is there a way to "append" to documents? Or do I need to grab the existing document, save as an array, merge my contact locations array with the main document and then resave?
how can I query to see if the "contact_locations" object already exists inside a document?
Hi yes you can do it !
1st you need to find your document and push the new value you need :
use findAndModify and $addToSet :
$cursor = $contact_collection->findAndModify(
array("ph" => "+17771111234"),
array('$addToSet' =>
array(
"contact_locations" => array(
"stime"=> "10:15",
"etime"=> "12:15",
"email"=> "test#yahoo.com"
)
)
)
);
The best part is $addToSet wont add 2 time the same stuff so you will not have twice the same value :)
Here the docs https://docs.mongodb.com/manual/reference/operator/update/addToSet/
I'm not sure the exact syntax in PHP as I've never done it before but I'm currently doing the same thing in JS with MongoDB and $push is the method you're looking for. Also if I may be a bit nitpicky I recommend changing $contact_collection to $contact_locations as a variable name. Array variable names are usually plural and being more descriptive is always better. Also make sure you find the array in the MongoDB first that you want to append to and that you use the MongoDb "update" command
I'd like to query for all documents in my collection where the field client is an empty string. I've seen plenty of examples of how to check if something's not an empty string but I can't find any examples of how to do the opposite. I've tried db.collection.find({client: ""}) but I get back an empty set.
EDIT: The example entry in the collection looks like
{ "_id" : ObjectId("54eb59699e12a795078b80da"), "reportID" : "1472031", "orgID" : "336", "client" : "", "customerID" : NumberLong(1), "address" : "123 main st", "city" : "Grove City", "state" : "OH", "zip" : "43123", "county" : "Franklin", "gpsLatitude" : null, "gpsLongitude" : null, "dateDue" : ISODate("2012-08-18T00:00:00Z"), "dateDueClient" : ISODate("2012-08-18T00:00:00Z"), "dateComplete" : ISODate("2012-08-18T00:00:00Z"), "dateCompleteEstimate" : NumberLong(0), "contractorSubmissionDate" : ISODate("2012-08-19T00:26:23Z"), "createdOn" : ISODate("2012-08-19T00:21:37Z"), "assignedToContractorOn" : ISODate("2012-08-19T00:21:37Z"), "workTypeID" : "6338", "assignedAdmin" : "7880", "contractorID" : "7880", "categoryID" : "0", "historyLength" : NumberLong(0), "invoiceDate" : ISODate("2012-08-18T00:00:00Z"), "submittedToClient" : ISODate("2012-08-18T00:00:00Z"), "paymentContractor" : NumberLong(0), "paymentClient" : NumberLong(0), "contractorIsPaid" : NumberLong(0), "clientIsPaid" : NumberLong(0), "sentinel" : NumberLong(1), "isFrozen" : NumberLong(0), "numTimesClientReady" : "1", "pcrResponses" : [ ] }
There's a lot of fields but the client one is pretty close to the beginning.
The query you posted should work. Just to verify, I inserted that data as a row in a test collection, and it was returned right back out with db.test.find({client: ''})
I'm thinking you may have a problem somewhere else. I've once or twice been in the wrong database when running queries, or had a typo in the collection name. To verify what my data looks like, I'll often times:
> db.tset.find({client: ''});
[] // What? No results? ...backspacebackspacebackspace
> db.tset.find();
[] // What? No results at all for no query? I must be in the.. OH!
> db.test.find();
[] // What? I thought I had a typo in my collection name. What database am I in?
> db
tseting // OH! I keep making that typo..
> use testing
switched to db testing
> db.test.find({client: ''});
[] // What? Still no results? This is weird...
> db.test.insert({client: ''});
> db.test.find({client: ''});
{ "_id" : ObjectId("54f1b5e5d05052fce4fb6684"), "client" : "" }
// Hmm, ok, so there's nothing wrong with the query. The data came up
// So I'm still just in the wrong place? Really?
// Maybe I woke up on the wrong side of the bed today. I should eat lunch.
Sometimes tunnel vision makes you think the problem lies somewhere that it doesn't. Leave a comment if this is or isn't helpful, maybe we can come up with some more troubleshooting ideas.
I'm trying to loop through some JSON data and pull out specific values. Here is the JSON data and the partially working code.
$jsondata = '
[
{
"id" : "421356",
"trip_update" : {
"trip" : {
"trip_id" : "421356",
"start_time" : "12:05:00",
"start_date" : "20130926",
"route_id" : "15"
},
"stop_time_update" : {
"stop_sequence" :70,
"departure" : {
"delay" : 240,
"time" : 1380215057
},
"stop_id" : "6090"
},
"stop_time_update" : {
"stop_sequence" :71,
"departure" : {
"delay" : 240,
"time" : 1380215075
},
"stop_id" : "6095"
}
}
}]';
$result = json_decode($jsondata);
foreach($result as $value) {
echo "trip_id: ".$value->trip_update->trip->trip_id;
if (gettype($value->trip_update ) == "object") {
foreach($value->trip_update as $item) {
echo " - stop_sequence: ".$item->stop_sequence;
}
}
}
I can get the first level of data under 'trip_update->trip'. But there can be any number of 'stop_time_update' data within 'trip_update' as well. Since this data relates to the trip_update data, I need to loop through it and correlate it.
The end goal is to save this data to a database (not shown in the code), so for clarity, this would be the simplified 2 rows of DB data I would like to save in this example:
trip_id,stop_sequence
421356,70
421356,71
There can be any number of stop_sequences in the source data.
Here is an interactive link to the code for you to edit or mess with:
http://sandbox.onlinephpfunctions.com/code/f21ca8928da7de3e9fb351edb075d0a446906937
You might get better results if you write your own parser or use a stream-parser with callbacks. Here's a PHP implementation of such a parser that works with callbacks. So instead of reading the whole JSON data into memory, the parser will read the data in chunks and notify your "listener-class" whenever a new object starts or a property was read in etc. By doing this, you should get separate callback events for each stop_time_update property instead of just one value in the parsed array.
Very similar to what SAX is for XML.
Hi maybe you can change the name.
function next_update($coincidencias){
$replace=$coincidencias[0].$GLOBALS["json_stop_time_update"]++;
return $replace;
}
$result= preg_replace_callback("/stop_time_update/","next_update",$jsondata);
$result = json_decode($result);
You should rework your JSON - you have multiple keys with the same name, try to do print_r($result) to see what I am talking about - PHP will override the "stop_time_update" key time after time and you will be able to access only the last entry. Instead, you should organize your JSON like that:
[
{
"id" : "421356",
"trip_update" : {
"trip" : {
"trip_id" : "421356",
"start_time" : "12:05:00",
"start_date" : "20130926",
"route_id" : "15"
},
"stop_time_update" : [{
"stop_sequence" :70,
"departure" : {
"delay" : 240,
"time" : 1380215057
},
"stop_id" : "6090"
}, {
"stop_sequence" :71,
"departure" : {
"delay" : 240,
"time" : 1380215075
},
"stop_id" : "6095"
}]
}
}]
then you will be able to iterate through your data like this:
foreach($result[0]->trip_update->stop_time_update as $update)
{
$time = $update->departure->time;
...
}
If you cannot change the data structure, then what probably could help you is a PULL parser - one that does not return parsed data structure, but allows you to use a data stream instead - this way you could iterate over each node. The only one I managed to find is an extension to PHP:
https://github.com/shevron/ext-jsonreader
Check the usage section.
This JSON response is invalid because it contains duplicate keys but JSON doesn't allow duplicate keys.
You should contact the service you're trying to request this response from.
If you have a valid JSON response then you can decode it using the json_decode function which returns an object or an array (depends on the second parameter);
You cannot use a JSON parser for this as it will always overwrite the first element due to the same keys. The only proper solution would be asking whoever creates that "JSON" to fix his code to either use an array or an object with unique keys.
Another option is to write your own decoder function for parse it