Let say, I have 2 collection
first one :-
db.product_main
{
_id:123121,
source_id:"B4456dde1",
title:"test Sample",
price: 250
quantity: 40
}
which consist approx ~10000 objects (Array) and unique field is source_id.
Second :-
db.product_id
{
"_id":58745633,
"product_id":"B4456dde1"
}
which consist of ~500 and only have field "product_id" which is equals to "source_id" of db.product_main
now, i want to intersect two collection so that i only find those which don't exist in db.product_id.
db.product_main.aggregate({any query})
Just use the lookup stage to find the products associated with the 'product_main' collection and then match for empty array (i.e. records where no product_id was found)
db.product_main.aggregate([
{
$lookup: {
from: "product_id",
localField: "source_id",
foreignField: "product_id",
as: "products_available"
}
},
{
$match: {
products_available: {
$size: 0
}
}
}
])
On WRITE operations using aggregate pipeline You can also directly offload statistics update by using $out command and store cached result in product_stats collection (for example).
Later in web/ui/api READ operations just use this cached collection. Of cause, You can create database query methods for cached and non-cached results.
Related
{
"movies": {
"movie1": {
"genre": "comedy",
"name": "As good as it gets",
"lead": "Jack Nicholson"
},
"movie2": {
"genre": "Horror",
"name": "The Shining",
"lead": "Jack Nicholson"
},
"movie3": {
"genre": "comedy",
"name": "The Mask",
"lead": "Jim Carrey"
}
}
}
I am a Firebase newbie. How can I retrieve a result from the data above where genre = 'comedy' AND lead = 'Jack Nicholson'?
What options do I have?
Using Firebase's Query API, you might be tempted to try this:
// !!! THIS WILL NOT WORK !!!
ref
.orderBy('genre')
.startAt('comedy').endAt('comedy')
.orderBy('lead') // !!! THIS LINE WILL RAISE AN ERROR !!!
.startAt('Jack Nicholson').endAt('Jack Nicholson')
.on('value', function(snapshot) {
console.log(snapshot.val());
});
But as #RobDiMarco from Firebase says in the comments:
multiple orderBy() calls will throw an error
So my code above will not work.
I know of three approaches that will work.
1. filter most on the server, do the rest on the client
What you can do is execute one orderBy().startAt()./endAt() on the server, pull down the remaining data and filter that in JavaScript code on your client.
ref
.orderBy('genre')
.equalTo('comedy')
.on('child_added', function(snapshot) {
var movie = snapshot.val();
if (movie.lead == 'Jack Nicholson') {
console.log(movie);
}
});
2. add a property that combines the values that you want to filter on
If that isn't good enough, you should consider modifying/expanding your data to allow your use-case. For example: you could stuff genre+lead into a single property that you just use for this filter.
"movie1": {
"genre": "comedy",
"name": "As good as it gets",
"lead": "Jack Nicholson",
"genre_lead": "comedy_Jack Nicholson"
}, //...
You're essentially building your own multi-column index that way and can query it with:
ref
.orderBy('genre_lead')
.equalTo('comedy_Jack Nicholson')
.on('child_added', function(snapshot) {
var movie = snapshot.val();
console.log(movie);
});
David East has written a library called QueryBase that helps with generating such properties.
You could even do relative/range queries, let's say that you want to allow querying movies by category and year. You'd use this data structure:
"movie1": {
"genre": "comedy",
"name": "As good as it gets",
"lead": "Jack Nicholson",
"genre_year": "comedy_1997"
}, //...
And then query for comedies of the 90s with:
ref
.orderBy('genre_year')
.startAt('comedy_1990')
.endAt('comedy_2000')
.on('child_added', function(snapshot) {
var movie = snapshot.val();
console.log(movie);
});
If you need to filter on more than just the year, make sure to add the other date parts in descending order, e.g. "comedy_1997-12-25". This way the lexicographical ordering that Firebase does on string values will be the same as the chronological ordering.
This combining of values in a property can work with more than two values, but you can only do a range filter on the last value in the composite property.
A very special variant of this is implemented by the GeoFire library for Firebase. This library combines the latitude and longitude of a location into a so-called Geohash, which can then be used to do realtime range queries on Firebase.
3. create a custom index programmatically
Yet another alternative is to do what we've all done before this new Query API was added: create an index in a different node:
"movies"
// the same structure you have today
"by_genre"
"comedy"
"by_lead"
"Jack Nicholson"
"movie1"
"Jim Carrey"
"movie3"
"Horror"
"by_lead"
"Jack Nicholson"
"movie2"
There are probably more approaches. For example, this answer highlights an alternative tree-shaped custom index: https://stackoverflow.com/a/34105063
If none of these options work for you, but you still want to store your data in Firebase, you can also consider using its Cloud Firestore database.
Cloud Firestore can handle multiple equality filters in a single query, but only one range filter. Under the hood it essentially uses the same query model, but it's like it auto-generates the composite properties for you. See Firestore's documentation on compound queries.
I've written a personal library that allows you to order by multiple values, with all the ordering done on the server.
Meet Querybase!
Querybase takes in a Firebase Database Reference and an array of fields you wish to index on. When you create new records it will automatically handle the generation of keys that allow for multiple querying. The caveat is that it only supports straight equivalence (no less than or greater than).
const databaseRef = firebase.database().ref().child('people');
const querybaseRef = querybase.ref(databaseRef, ['name', 'age', 'location']);
// Automatically handles composite keys
querybaseRef.push({
name: 'David',
age: 27,
location: 'SF'
});
// Find records by multiple fields
// returns a Firebase Database ref
const queriedDbRef = querybaseRef
.where({
name: 'David',
age: 27
});
// Listen for realtime updates
queriedDbRef.on('value', snap => console.log(snap));
var ref = new Firebase('https://your.firebaseio.com/');
Query query = ref.orderByChild('genre').equalTo('comedy');
query.addValueEventListener(new ValueEventListener() {
#Override
public void onDataChange(DataSnapshot dataSnapshot) {
for (DataSnapshot movieSnapshot : dataSnapshot.getChildren()) {
Movie movie = dataSnapshot.getValue(Movie.class);
if (movie.getLead().equals('Jack Nicholson')) {
console.log(movieSnapshot.getKey());
}
}
}
#Override
public void onCancelled(FirebaseError firebaseError) {
}
});
Frank's answer is good but Firestore introduced array-contains recently that makes it easier to do AND queries.
You can create a filters field to add you filters. You can add as many values as you need. For example to filter by comedy and Jack Nicholson you can add the value comedy_Jack Nicholson but if you also you want to by comedy and 2014 you can add the value comedy_2014 without creating more fields.
{
"movies": {
"movie1": {
"genre": "comedy",
"name": "As good as it gets",
"lead": "Jack Nicholson",
"year": 2014,
"filters": [
"comedy_Jack Nicholson",
"comedy_2014"
]
}
}
}
For Cloud Firestore
https://firebase.google.com/docs/firestore/query-data/queries#compound_queries
Compound queries
You can chain multiple equality operators (== or array-contains) methods to create more specific queries (logical AND). However, you must create a composite index to combine equality operators with the inequality operators, <, <=, >, and !=.
citiesRef.where('state', '==', 'CO').where('name', '==', 'Denver');
citiesRef.where('state', '==', 'CA').where('population', '<', 1000000);
You can perform range (<, <=, >, >=) or not equals (!=) comparisons only on a single field, and you can include at most one array-contains or array-contains-any clause in a compound query:
Firebase doesn't allow querying with multiple conditions.
However, I did find a way around for this:
We need to download the initial filtered data from the database and store it in an array list.
Query query = databaseReference.orderByChild("genre").equalTo("comedy");
databaseReference.addValueEventListener(new ValueEventListener() {
#Override
public void onDataChange(#NonNull DataSnapshot dataSnapshot) {
ArrayList<Movie> movies = new ArrayList<>();
for (DataSnapshot dataSnapshot1 : dataSnapshot.getChildren()) {
String lead = dataSnapshot1.child("lead").getValue(String.class);
String genre = dataSnapshot1.child("genre").getValue(String.class);
movie = new Movie(lead, genre);
movies.add(movie);
}
filterResults(movies, "Jack Nicholson");
}
}
#Override
public void onCancelled(#NonNull DatabaseError databaseError) {
}
});
Once we obtain the initial filtered data from the database, we need to do further filter in our backend.
public void filterResults(final List<Movie> list, final String genre) {
List<Movie> movies = new ArrayList<>();
movies = list.stream().filter(o -> o.getLead().equals(genre)).collect(Collectors.toList());
System.out.println(movies);
employees.forEach(movie -> System.out.println(movie.getFirstName()));
}
The data from firebase realtime database is as _InternalLinkedHashMap<dynamic, dynamic>.
You can also just convert this it to your map and query very easily.
For example, I have a chat app and I use realtime database to store the uid of the user and the bool value whether the user is online or not. As the picture below.
Now, I have a class RealtimeDatabase and a static method getAllUsersOnineStatus().
static getOnilineUsersUID() {
var dbRef = FirebaseDatabase.instance;
DatabaseReference reference = dbRef.reference().child("Online");
reference.once().then((value) {
Map<String, bool> map = Map<String, bool>.from(value.value);
List users = [];
map.forEach((key, value) {
if (value) {
users.add(key);
}
});
print(users);
});
}
It will print [NOraDTGaQSZbIEszidCujw1AEym2]
I am new to flutter If you know more please update the answer.
ref.orderByChild("lead").startAt("Jack Nicholson").endAt("Jack Nicholson").listner....
This will work.
I'm trying to get the data from my Lists table and include an array with the ID's of the tracks in that list.
This is a sample of the database model, with a relation N:M.
In my List model, I added this method:
public function tracks()
{
return $this->belongsToMany(Track::class, 'List_Tracks', 'id_list', 'id_track');
}
So, in my ListController, I'm doing the following:
$list = List::find($id);
$list->tracks_list = $list->tracks->pluck('track_id');
return $list;
And what I get is as many objects as tracks I have in a same list, for example:
[
{
"id_track": 1,
"name": "Yesterday",
"tracks_list": [
1,
2
]
"pivot": {
"id_list": 1,
"id_track": 1
}
},
{
"id_track": 2,
"name": "Lucy in the sky with diamonds",
"pivot": {
"id_list": 1,
"id_track": 2
}
}
]
But what I want to get is:
{
"id_list": 1,
"name": "The Best of The Beatles",
"tracks_list": [
1,
2
]
}
I think the things I've tried are much more complex than the proper solution.
How would you get the data in this way?
Thanks in advance.
You need to load the relationship first and then write rest of the eloquent query, otherwise you will get only list of ids from the pivot table.
List::with('tracks')->where('id', '=', $id)->first();
// Following method should work too
List::with('tracks')->find($id);
Based on your commentary - to get only array of ids related to the list you don't need to load the relationship and can use only:
List::find($id)->tracks()->pluck('id');
// In case previous method will not work, you can try this one:
List::find($id)->tracks()->pluck('tracks.id');
// before dot should be name of the table, not relationship's.
So if you need only ids of the tracks that are attached to the playlist. I would recommend adding to your List model following method:
// App\Models\List
public function getTrackIdsAttribute()
{
return $this->tracks->pluck('id')->toArray();
}
Then you should be able to simply call List::find($id)->trackIds to get all ids of tracks attached to the given list. If you're not sure why I am defining the method as getTrackIdsAttribute and call only trackIds on the model, take a look at Eloquent: Mutators.
I have a document with this structure:
{"user":{
"nice":{
"funny":"sure"
}
,
"notnice":{
"funny":"maybe"
}
}
}
I know the keys "user","funny" and the value "sure" and "maybe" but I don't know "nice" and "notnice".
How do I do an optimized query to search through many documents.
For example, if I want to search "sure" value knowing the middle keys I do:
$document = $users->findOne([
'$or' => [
['user.nice.funny' => 'sure'],
['user.notnice.funny' => 'sure']
]
]
);
But how do I do the same without knowing "nice" and "notnice".
This should point you in the right direction:
db.collection.aggregate({
$addFields: {
"userTransformed": {
$objectToArray: "$user" // transform "user" field into key-value pair
}
}
}, {
$match: {
"userTransformed.v.funny": "sure" // just filter on the values
}
})
Frankly, this is not going to be fast for lots of documents but there is no other way. Indexes will not be used by this query. If you want to get faster you will need to change your document structure.
I am trying to make a selection based on multiple ids in joins, each of the ids should match another condition of another join.
I need to get "Types" that have all the "Dichotomies" in "Position" 1 or 2. At the moment it gives me results that match one of the Dichotomies passed to the function, but not all of them.
$QB->select("Types","Types,ElementsPositions, Elements, Positions, Dichotomies, Quadras,TypesDescriptions,Relations")
->from($this->get_repository()[0], 'Types');
$QB->leftJoin("Types.ElementsPositions","ElementsPositions", \Doctrine\ORM\Query\Expr\Join::WITH, 'ElementsPositions.Positions = 1 OR ElementsPositions.Positions = 2');
$QB->leftJoin("ElementsPositions.Elements","Elements");
$QB->leftJoin("ElementsPositions.Positions","Positions");
$QB->leftJoin("Elements.Dichotomies","Dichotomies");
$QB->leftJoin("Types.Quadras","Quadras");
$QB->leftJoin("Types.TypesDescriptions","TypesDescriptions");
$QB->leftJoin("Types.Relations","Relations");
if(!empty($where['dichotomies'])){
foreach($where['dichotomies'] as $dichotomy){
$QB->andWhere('Dichotomies.id'.'=:dichotomy');
$QB->setParameter('dichotomy', $dichotomy['id']);
}
}
UPD.
Tables mapping - in JSON:
{
"table-name": "types",
"joins":[
{
"table-name":"elements_positions",
"type":"one-to-many"
},
{
"table-name":"quadras",
"type":"many-to-one"
},
{
"table-name":"types_descriptions",
"type":"one-to-one"
},
{
"table-name":"relations",
"type":"many-to-one"
}
]}
Elements Positions
{
"table-name": "elements_positions",
"joins":[
{
"table-name":"elements",
"type":"many-to-one"
},
{
"table-name":"positions",
"type":"many-to-one"
},
{
"table-name":"types",
"type":"many-to-one"
}
]
}
Elements
{
"table-name": "elements",
"joins":[
{
"table-name":"elements_positions",
"type":"one-to-many"
},
{
"table-name":"quadras",
"type":"many-to-many"
},
{
"table-name":"dichotomies",
"type":"many-to-many"
}
]
}
Positions
"table-name": "positions",
"joins":[
{
"table-name":"elements_positions",
"type":"one-to-many"
}
]
}
Dichotomies:
{
"table-name": "dichotomies",
"joins":[
{
"table-name":"elements",
"type":"many-to-many-inversed"
}
]
}
Your query has a two different problems.
First, multiple parameter values are bound with single parameter. Every next element of $where['dichotomies'] replaces previous value of parameter :dichotomy in the query. The method setParameters() don't really binds values to the prepared statement: it just stores them in QueryBuilder object. So, after the end of foreach-loop all conditions will be use the same value (the last of $where['dichotomies']). To avoid that you need to use different parameter names or numeric indexes.
Second, you add conditions that are contradictory: $QB->andWhere() will produce something like that:
Dichotomies.id = :dichotomy
AND Dichotomies.id = :dichotomy
AND Dichotomies.id = :dichotomy
...
One entity ID obviously cannot be equal to different values simultaneously. So, you need to replace AND by the OR operator.
But better way is to use IN clause:
Calling setParameter() automatically infers which type you are setting as value. This works for integers, arrays of strings/integers, DateTime instances and for managed entities.
Just replace the foreach-loop by the following lines:
$QB->andWhere('Dichotomies.id IN (:dichotomies)');
$QB->setParameters('dichotomies', array_column($where['dichotomies'], 'id'));
The array_column() function returns a list of IDs of all dichotomies. Doctrine generates IN expression and uses that list to generate query placeholders and bind the values.
how do I make an appointment within 30 mongo documents?
Ex:
db.find (). count () = 100 documents //my base
I wanted
db.find ({$ {and 'user.gender': 'Male'}, {'in first 30 records "}) = 5 documents
it is not
db.find ({'user.gender': 'Male'}). limit (30) = 30 documents //I do not want it
You can use the $aggregate in mongodb
for eg
db.collName.aggregate([
{ $limit: 30 },
{ $match: { 'user.gender': 'Male'} }
])
If you want to sort the collection on basics of any field you can use the $sort in the aggregation pipeline.
Also please do remember the fact that "there is no guarantee that your documents are returned in any particular order by a query as long as you don't sort explicitly. Documents in a new collection are usually returned in insertion order, but various things can cause that order to change unexpectedly, so don't rely on it.
When you sort by _id field the items are returned by creation Date. So you can also sort by _id first and then limit your documents.
For eg:
db.collName.aggregate([
{ $sort: { _id: 1 } },
{ $limit: 30 },
{ $match: { 'user.gender': 'Male'} }
])
Explaination:
It will first sort your documents based on the _id field in documents and then limit your documents to 30 and then find a matchon the returned documents based on the query.