First login?
Lost password?
 
 
 
General mailing list for discussions and development of PeerLibrary and related software.

List archive Help


Re: [PeerLibrary dev] PeerDB vs. Meteor collection queries


Chronological Thread 
  • From: Mitar < >
  • To:
  • Subject: Re: [PeerLibrary dev] PeerDB vs. Meteor collection queries
  • Date: Wed, 06 Aug 2014 17:24:38 +0100
Hi!

> OK, now I've understood that you have to manually subscribe (on the
> client) and publish (on the server). Imagine I have 2 different queries
> for persons on the same page. Then I subscribe two times to the
> search-persons endpoint and all Persons matching any of the two queries
> end up in the client's Person document class? So I've to filter them
> again on the client?

Hm, this is a bit tricky question. So search-persons is really a special
endpoint (BTW, I am calling things "publish endpoints" because for me it
is much clearer than "publish functions" or just "publish" which is
often used elsewhere in Meteor community). For how it is currently
designed, you can have only one search query active at a given moment
for an entity. But this is because we are using only one field for
search results for each entity. We could change this a bit, but there
was no need for that until now.

So let's discuss some other publish endpoint. But for all other
subscriptions (so one which do not use full-text search through
ElasticSearch) you should subscribe not based on queries but based on
which data you want to have on the client. And this often depends on a
view (page) user is on. For example, you want information about a group,
and all members in the group. So you subscribe on the group and you
subscribe to all members in the group. And then you can do arbitrary
queries against this data.

How complicated queries you allow to be send to the server we decide.
This is what our public API will be later on. The collection of publish
endpoints. We could allow that user's can request a publication of
documents for arbitrary MongoDB query by defining such a publish
endpoint, but we are in fact very conservative and allow only publishing
based on IDs or some very restricted set of fields (the latter,
currently mostly wrapped into search).

So maybe an easier way to think is that multiple subscriptions are
merged on the client. So you could subscribe for documents from author1
and create another subscription for documents from author2. Then you
would have on the client document from author1 and author2 available to
do queries on.

This works even on the field level. You can have one subscription to get
some fields and then another to get another, for the same document. But
it is limited only to top-level fields.

One interesting thing having such federation between server and client
where data is kept in sync and you have same query language both on the
server and client, is that you can run exactly the same code both on the
server and client. This allows things like latency compensation (which
we are not yet using in PeerLibrary), easier code for input validation, etc.

But again, the main complications is then how to define publish
endpoints and how you subscribe to them so that you have all data you
need for queries on the client when you need it, but not more. In the
worst case you could do:

- subscribe with query and get documents for the query to the client
- wait for data to come to the client
- repeat the same query on the client on the documents you have

But this is not really how you do, because it would take quite long.
User would have to wait for the query, for data to get to the client,
and then you would be able to do something. You could, but you do not
really do it like this (maybe we will in some cases). Mostly, what we
currently do is subscribe to the document or documents when you get to
the page (having a nice progress bar, which we do not yet have, while
this data is loading), and then do queries later. In most cases this
works, because at least currently we have mostly one URL one entity or
known list of entities mapping. So for publication page you subscribe to
that publication. And if you need some more data, we sync that in as a
subdocument with PeerDB.

(What I just described is not latency compensation. That is something else.)

(I am hopping that I am not making you even more confused.)

> Btw, I'm not complaining about Meteor not providing enough. I'm just
> trying to see the benefits and downsides of it (compared to techniques I
> know).

:-)

Feel free to write-up what you learn sometime. In general: as you are
learning, thing all the time to write things down. Documentation (wiki
pages) for new contributors are missing a lot. We should expect that
most of our potential contributors would be coming from outside Meteor.
So please, help. This is probably one of the most important things you
are doing now. Learning how to start contributing. So document and think
about how to make things easier for people coming after you.


Mitar

--
http://mitar.tnode.com/
https://twitter.com/mitar_m

Archive powered by MHonArc 2.6.18.

Top of page