My CouchDB database is structured like this:
"custom_details": {
"user_education": [
{
"device_id": "358328030246627",
"college_name": "College",
"college_year": "2014"
},
]
}
"custom_details_1": {
"user_education": [
{
"device_id": "358328030246627",
"college_name": "College",
"college_year": "2014"
},
]
}
I have a lot of arrays within arrays. What I'm trying to do use Elasticsearch to search and find terms, regardless of where it's sitting in an array. Is that possible?
I've been going through the examples on here and haven't quite found what I'm looking for. I've tried using Elastica, the PHP Wrapper, but without fully understanding how to do this with REST, I'm lost. Is it even possible to search for data without knowing the field?
In Lucene, you could create a document instance for each device id:
public void indexRecord(CouchDBRecord rec) {
Document doc = new Document();
doc.add(new Field("device_id", rec.device_id, Store.YES, Index.NOT_ANALYZED));
doc.add(new Field("college_name", rec.college_name, Store.YES, Index.ANALYZED));
doc.add(new Field("college_year", rec.college_year.toString(), Store.YES, Index.NOT_ANALYZED));
this.writer.addDocument(doc);
}
This will allow you to search by keywords in the college name, or by exact device id or year, or some combination thereof.
If you are using Elastica, the whole REST thing is already done for you. If you want to search in all fields, you can define just the search term and it will search in all fields.
If you are having some troubles with Elastica or if some features are missing you need, let me know, as I'm the developer of Elastica.
Related
Quite new to GraphQL and lighthouse library, don't be too harsh.
Since I can't use any models because my data source is an API. I'm trying to create a custom resolver that will pass data to a service who will do everything necessary to retrieve data from the API.
And it constantly returns me this error: "Field \"address\" of type \"[Address!]\" must have a sub selection.",
I believe it's because of the fact I don't use models(just a wild guess)
So far my schema looks like this:
type Query {
address(address: String!): [Address!] #field(resolver: "Address#resolve")
}
type Address {
fullAddress: String!
lowestId: Int!
}
And the mentioned resolver:
public function resolve($rootValue, array $args, GraphQLContext $context, ResolveInfo $resolveInfo): array
{
return array_map(
function ($address): array {
return [
'fullAddress' => $address->getFullAddress()
];
},
$this->service->getAddress($args['address'])
);
}
Thank you in advance!
The error is not even specific to Lighthouse, any GraphQL server will produce a similar error for what you are trying to do. I assume you are trying a query like this:
{
address(address: "foo")
}
Consider the graph in GraphQL: your server describes available data types and relations between them, forming a graph. Each type could have fields that lead to another type, and that type to another type, and so on. Those references can even form cycles. At some points, the graph may end: types such as scalar values mark the leaves of the graph.
Now, how does a server know which part of the graph you want to see and it should resolve? Through a query: a subselection of a part of that graph. That naturally limits how deep the server must go, it can do the minimal amount of work to return the parts of the graph you queried for.
One rule of queries is that you must always end up at leaf nodes. This is where the error message comes into play: the server sees that Address is not a leaf type and thus asks you to specify how deep you want to traverse the graph. A working query could be:
{
address(address: "foo") {
fullAddress
}
}
There are tons of articles, blogs and API docs about REST API resource field expansion but none about how to implement an expansion in aspect of technique and data query in right way.
Simple example for a flat resource response:
GET /api/v1/customers
{
data: [
{
"id": "209e5d80-c459-4d08-b53d-71d11e216a5d",
"contracts": null
},
{
"id": "c641cb83-af29-485d-9be2-97925057e3b2",
"contracts": null
}
],
expandable: ["contract"]
}
Simple example for expanded resource:
GET /api/v1/customers?expand=contract
{
data: [
{
"id": "209e5d80-c459-4d08-b53d-71d11e216a5d",
"contracts": [
{......},
{......},
{......},
]
},
{
"id": "c641cb83-af29-485d-9be2-97925057e3b2",
"contracts": [
{......},
{......},
{......},
]
}
],
expandable: ["contract"]
}
Lets assume we use a api rest controller class which handles the enpoints and a read service (maybe cqs/cqrs based) which uses plain sql for read performance. At which point does the expansion logic start and what is the right way of handling queries without an exponential increase in queries?
Afaik, this is not possible in one or few SQL queries (except the dirty way of GROUP_CONCAT and separation of all data into one field). Should I query all customers and then iterate over all customers and query expanded data for each customer? This would cause an exponential increase of queries.
I was looking for the same thing and I would say that the correct answer is it depends on the infrastructure used.
In a general manner, the implementation should receive the expand or expandable' object in your endpoint and respond acordingly. So, if we have something like your example, when requesting /api/v1/customersviaGETpassing theexpandableobject, you should run another query to the database to get, as in the example, theuser contracts`.
There is not a unique, one size fits all answer for this question, specially if we are not talking about a specific language + framework.
I've been looking into GraphQL as a replacement for some REST APIs of mine, and while I think I've wrapped my head around the basics and like most of what I see so far, there's one important feature that seems to be missing.
Let's say I've got a collection of items like this:
{
"id": "aaa",
"name": "Item 1",
...
}
An application needs a map of all those objects, indexed by ID as such:
{
"allItems": {
"aaa": {
"name": "Item 1",
...
},
"aab": {
"name": "Item 2",
...
}
}
}
Every API I've ever written has been able to give results back in a format like this, but I'm struggling to find a way to do it with GraphQL. I keep running across issue 101, but that deals more with unknown schemas. In my case, I know exactly what all the fields are; this is purely about output format. I know I could simply return all the items in an array and reformat it client-side, but that seems like overkill given that it's never been needed in the past, and would make GraphQL feel like a step backwards. I'm not sure if what I'm trying to do is impossible, or I'm just using all the wrong terminology. Should I keep digging, or is GraphQL just not suited to my needs? If this is possible, what might a query look like to retrieve data like this?
I'm currently working with graphql-php on the server, but I'm open to higher-level conceptual responses.
Unfortunately returning objects with arbitrary and dynamic keys like this is not really a first-class citizen in GraphQL. That is not to say you can't achieve the same thing, but in doing so you will lose many of the benefits of GraphQL.
If you are set on returning an object with id keys instead of returning a collection/list of objects containing the ids and then doing the transformation on the client then you can create a special GraphQLScalarType.
const GraphQLAnyObject = new GraphQLScalarType({
name: 'AnyObject',
description: 'Any JSON object. This type bypasses type checking.',
serialize: value => {
return value;
},
parseValue: value => {
return value;
},
parseLiteral: ast => {
if (ast.kind !== Kind.OBJECT) {
throw new GraphQLError("Query error: Can only parse object but got a: " + ast.kind, [ast]);
}
return ast.value;
}
});
The problem with this approach is that since it is a scalar type you cannot supply a selection set to query it. E.G. if you had a type
type MyType implements Node {
id: ID!
myKeyedCollection: AnyObject
}
Then you would only be able to query it like so
query {
getMyType(id: abc) {
myKeyedCollection # note there is no { ... }
}
}
As others have said, I wouldn't recommend this because you are losing a lot of the benefits of GraphQL but it goes to show that GraphQL can still do pretty much anything REST can.
Hope this helps!
First the actual state: There is a ZF2 application with a form. The form contains some autocomplete fields (implemented on the frontend side with jQuery Autocomplete).
The SQL statements behind it look like this:
SELECT name FROM countries WHERE countries.name LIKE %en%
-> should find "Arg[en]tina", "Arm[en]ia", "B[en]in", "Turkm[en]istan" etc.
or
SELECT name FROM countries WHERE countries.continent_id = 2
-> should find "Afghanistan", "Armenia", "Azerbaijan" etc. (2 = Asia)
or
SELECT name FROM countries WHERE countries.continent_id = 2 AND countries.name LIKE %en%
-> should find "Arm[en]ia", "Turkm[en]istan" etc. (2 = Asia)
Of course, it leads to the problem, that the database gets terrorized by a lot of small autocomplete requests. Caching should help -- and I've already started implementing a Zend\Cache\Storage\Adapter\Apcu-based caching mechanism. But then I saw the next trouble: Using a common cache like APCu I cannot filter the results dynamically. So such a cache seems not to work for a case with autocomplete.
I'm pretty sure, that it's a common issue and there is already a solution for this.
How to realize a caching mechanism in a ZF2 application for the autocomplete functionality?
There is nothing to do with the ZF2 here. This is all about a custom search service design and it's challenges.
Without a proper caching layer and/or full-text search engine, building an autocomplete implementation such this would be suicide for the application. You can easily achieve tens of thousands of unnecessarily repeated queries in a very short time.
In an "ideal" world, a good autocomplete implementation utilizes a full-text search engine under the hood such as Elasticsearch or Apache Solr. And uses their Completion Suggester and Suggester components respectively.
Anyway a simple autocompletion feature still achievable using only an object cache and database. Only you need is a helper method to create a proper "cache key" for the every letter combinations. For example:
function createKeyByQuery($str)
{
return 'autocomplete-prefix-'.(string) $str;
}
and in your suggest() method:
public function suggest($keyword)
{
$key = $this->createKeyByQuery($keyword);
if($this->cache->hasItem($key)) {
return $this->cache->getItem($key);
}
// fetch form the database here
$data = $this->db->query();
$this->cache->setItem($key, $data);
return $data;
}
If the count of your filters are not too much, just make them part of the key too. In this scenario, signature of the suggest method would be:
public function suggest($keyword, array $filters = []);
and key generator needs an update:
function createKeyByQuery($str, array $filters = [])
{
return 'autocomplete-prefix-' . (string) $str . md5(serialize($filters));
}
This solution may not appropriate for the complicated/domain related data because it has a pretty big invalidation challenge. Eg. how you would find the cache keys which holds the "Argentina" in the payload?
Since you're dealing only with the list of countries and continents as filter, it should solve the problem.
For the england keyword and two different filter with total of 10 filter options, there will be 10x2x7 = 140 distinct cache key(s). For a single filter with 5 option, 5x1x7 = 37 distinct keys.
APCu is a good choice for this implementation.
Hope it helps.
Assume you have a GraphQL type and it includes many fields.
How to query all the fields without writing down a long query that includes the names of all the fields?
For example, If I have these fields :
public function fields()
{
return [
'id' => [
'type' => Type::nonNull(Type::string()),
'description' => 'The id of the user'
],
'username' => [
'type' => Type::string(),
'description' => 'The email of user'
],
'count' => [
'type' => Type::int(),
'description' => 'login count for the user'
]
];
}
To query all the fields usually the query is something like this:
FetchUsers{users(id:"2"){id,username,count}}
But I want a way to have the same results without writing all the fields, something like this:
FetchUsers{users(id:"2"){*}}
//or
FetchUsers{users(id:"2")}
Is there a way to do this in GraphQL ??
I'm using Folkloreatelier/laravel-graphql library.
Unfortunately what you'd like to do is not possible. GraphQL requires you to be explicit about specifying which fields you would like returned from your query.
Yes, you can do this using introspection. Make a GraphQL query like (for type UserType)
{
__type(name:"UserType") {
fields {
name
description
}
}
}
and you'll get a response like (actual field names will depend on your actual schema/type definition)
{
"data": {
"__type": {
"fields": [
{
"name": "id",
"description": ""
},
{
"name": "username",
"description": "Required. 150 characters or fewer. Letters, digits, and #/./+/-/_ only."
},
{
"name": "firstName",
"description": ""
},
{
"name": "lastName",
"description": ""
},
{
"name": "email",
"description": ""
},
( etc. etc. ...)
]
}
}
}
You can then read this list of fields in your client and dynamically build a second GraphQL query to get the values of these fields.
This relies on you knowing the name of the type that you want to get the fields for -- if you don't know the type, you could get all the types and fields together using introspection like
{
__schema {
types {
name
fields {
name
description
}
}
}
}
NOTE: This is the over-the-wire GraphQL data -- you're on your own to figure out how to read and write with your actual client. Your GraphQL javascript library may already employ introspection in some capacity. For example, the apollo codegen command uses introspection to generate types.
2022 Update
Since this answer was originally written, it is now a recommended security practice to TURN OFF introspection in production. Reference: Why you should disable GraphQL introspection in production.
For an environment where introspection is off in production, you could use it in development as a way to assist in creating a static query that was used in production; you wouldn't actually be able to create a query dynamically in production.
I guess the only way to do this is by utilizing reusable fragments:
fragment UserFragment on Users {
id
username
count
}
FetchUsers {
users(id: "2") {
...UserFragment
}
}
I faced this same issue when I needed to load location data that I had serialized into the database from the google places API. Generally I would want the whole thing so it works with maps but I didn't want to have to specify all of the fields every time.
I was working in Ruby so I can't give you the PHP implementation but the principle should be the same.
I defined a custom scalar type called JSON which just returns a literal JSON object.
The ruby implementation was like so (using graphql-ruby)
module Graph
module Types
JsonType = GraphQL::ScalarType.define do
name "JSON"
coerce_input -> (x) { x }
coerce_result -> (x) { x }
end
end
end
Then I used it for our objects like so
field :location, Types::JsonType
I would use this very sparingly though, using it only where you know you always need the whole JSON object (as I did in my case). Otherwise it is defeating the object of GraphQL more generally speaking.
GraphQL query format was designed in order to allow:
Both query and result shape be exactly the same.
The server knows exactly the requested fields, thus the client downloads only essential data.
However, according to GraphQL documentation, you may create fragments in order to make selection sets more reusable:
# Only most used selection properties
fragment UserDetails on User {
id,
username
}
Then you could query all user details by:
FetchUsers {
users() {
...UserDetails
}
}
You can also add additional fields alongside your fragment:
FetchUserById($id: ID!) {
users(id: $id) {
...UserDetails
count
}
}
Package graphql-type-json supports custom-scalars type JSON.
Use it can show all the field of your json objects.
Here is the link of the example in ApolloGraphql Server.
https://www.apollographql.com/docs/apollo-server/schema/scalars-enums/#custom-scalars