Synchronization Vs. Direct access, the case of contacts

Maintaining accurate and useful contact information data is not an easy task, especially when that information is spread over several places. The solution usually proposed for this problem is refered as synchronization. Synchronization is a mechanism involving copies of data between sources in order to reflect the same state everywhere, also known as data replication.

Synchronization brings several problems. It usually tends to cast information patterns into one predifined one. It flattens the information and causes the loss of the “origin” dimension, which may have important semantics on how and where to use it. It usually breaks the link with the source, thus the newly available information from an updated source won’t be distributed until next synchronization. It implies merging choices which result into inconsistencies, loss and duplication of information.

The worst thing with synchronization is that it actually solves nothing. It may be a sufficient solution when considering only local sources like computer or cellphone addressbooks but with the era of Web 2.0, most of the contact information is to be found on the Internet. Contact sources there are whatever services manipulating people, contact of friend objects. This obviously includes social networks. The diversity of those sources make synchronization impossible to consider, their purpose is different hence the provided content shouldn’t be merged, it should be used differently, depending on the context. And why would you actually want to do synchronization? Since all the information is always out there, you only need to access it.

This is why for People, we made the choice not to copy and merge the contact information from the web to a smashed local version of it but instead provide a unified way of accessing it. Using direct access, the only effort to be done is to explicit the grouping of contact information from different sources defining a single person entity. Direct access ensures integrity and reliability of data (it is not manipulated between access and use), allows knowing about the sources thus consider the relevance of it (the full name of a contact is more likely to be accurate on LinkedIn than on Last.fm), and the update of the information is done by the actual owners of it, your contacts. Associating a cache mechanism to avoid network roundtrips and maybe an offline mode to that way of doing things should benefit the user better than forcing him to understand and suffer the mist of the broken synchronization scenarios.

Where People fits in GNOME

Right after our GUADEC presentation, Ali Sabil and I met with Owen Taylor, Marina Zhurakhinskaya (both from Red Hat’s Online Desktop project), Robert McQueen (Telepathy) and Travis Reitter (Soylent). The goal of that quick meeting was to define how to integrate all those pieces of technologies together. Even if People aims to define a data model and a D-Bus API to be used on all the free desktops, we are also part of the GNOME community, thus particularly eager to see People integration with GNOME technologies happening.

It’s been always difficult to make people understand what People is. Moreover, our project is quite unknown. We are not present on GNOME’s main communication channel (Planet GNOME) and that’s something you feel also at GUADEC when the conference room isn’t very populated. In that blog post, I’ll try to explain what is People and what it will be for GNOME.

The best explanation I can give to begin with is maybe that People is actually like “libsoylent”, which has been announced few time ago. The main goal is the same: provide first-class people objects to applications. Our approach is different: we designed People from the backend to application level, which allows us to consider whatever contact source, while libsoylent, based on what is done in Soylent, only uses Evolution Data Server and Telepathy as sources (for now). I think that our development is more advanced and our solution wider, hence even Soylent should use People.

On the next diagram, you can find several components that are part of the GNOME ecosystem. It is based on the GUADEC after-talk meeting result and is not meant to be exhaustive.

• Green boxes are what we call contact sources. Any application, service or data source handling and exposing objects referred as contacts, friends, persons, people is a potential contact source. Those contact sources are plugged into the People framework through a backend system. That interaction is represented as dashed arrows on the diagram. Each contact source provides partial information about some contacts to People.
• The blue box is the framework. It contains the backend system, a mechanism to gather partial contact informations into high level contact object (often called meta-contacts) and a D-Bus daemon to expose those high level contacts to applications. At some point, we’d also like to expose backend directly to allow synchronization solutions to use them.
• Yellow boxes are a few possible applications that could use the framework.

People in GNOME

Concerning Empathy, the current contact list could be turned into a people list, where each item wouldn’t be an instant messaging contact anymore but just a known person, that you may reach using instant messaging. That list would contain some friends you have in Facebook and you could actually subscribe to them on Jabber directly from Empathy, as People would provide to Empathy the information saying that your friend has a jabber account.

This is what People does, it provides contact information to application, and then applications should know how to use that information. People won’t tell you to launch Evolution to write an e-mail, it’ll just tell you that you can reach a certain contact through e-mail. We then need sort of an “activity launcher” and I think that’s what Soylent should be. Using the people list provided by the Empathy widget set, Soylent should allow you to start activities with the people you know. At some point, maybe Empathy’s client and Soylent are meant to be the same application.

Our demonstration at GUADEC was an application containing a list of high level contacts built from local (sqlite database) and distant sources (lastfm and friendfeed web services). The list exposed contact information such as the full name of the contact, a picture, an e-mail address, a phone number, an icon per social network he’s susbscribed to, all those information being provided by People. At some point, Ali brought his mobile phone next to the laptop and his phone’s addressbook automatically became a contact source (over bluetooth) for the application, adding contacts from the phone in the list. This was supposed to show how flexible the backend system is and how cool are the things that People enables .

Yesterday we set up www.people-project.org which is currently a wiki that will be useful to collaborate on stuff that are not code. We are also releasing 0.0.5 “Smelly hotel lobby” (tribute to our hotel in Istanbul, where people still smoke in public places and People don’t) today, as dictated by our sprint schedule. We are currently developing more backends (Online Desktop, Telepathy, EDS, …) and trying to improve what we have for now. The backlog is huge but considering all the possible cool applications, will is around.

People in brief

As of now, my previous post about People is the only explanation of what People is, and more precisely what it is supposed to be. Not to mention it is heavy and a bit messy. This one will clearly and shortly expose what you can expect from People, in the next two months. The following goals are roughly the one we plan to complete in time for our presentation at GUADEC on July:

• Let the user explicit meta contacts as an aggregation of informations from different contact sources.
• Request address books from a D-Bus service, the content of those address books being meta contacts defined as above.
• Use Google Contacts, Microsoft Live, Evolution Data Server and your phone as a contact source.

And this is pretty much it. Felipe joined us on developing backends, which is great and Ali has been working on improving the server side D-Bus support for Vala, developing the address book service. On my side, after working on the library on which will rely the address book service, I have to continue on the backends and start to think seriously about the demos.

People: a contact management framework

Today, people subscribe to social networks, use instant messaging, subscribe to podcasts and blog feeds, use electronic mail, and communication devices to exchange information but more importantly to keep in contact with their contacts and their entourage. Tomorrow people may use different means to achieve this same goal, but in the end it will always be about contacting people, and maybe by then smarter solutions would be found. But in the mean time, we are are stuck with the proliferation of independent and disconnected applications and appliances, each one tearing unique logical entities away to make them fit into limited models. In simpler words, the way applications treat contacts brings a bad user experience.

Unfortunately, we cannot really do much about it, but we can still try to attenuate the effects on the Free Desktop by providing a unified vision of who is a person to the user. The solution we propose is named “People“.

Many people agree that, in the desktop, a “people framework” is needed. Things get more complicated when it comes to define the scope of it. The People project intends to provide an unified way to access and manipulate contacts for the desktop applications. The goal is not, at first, to gather the pieces and simulate unity, but more to bring the tools allowing to do it in a smart way, among other things. In People, we consider that each contact source is incomplete and provides just a restricted vision on contacts (the way they are represented and the way we can act on them). With that vision, contact sources can be as numerous and lightweight as needed to cover every place where the notion of person appears: LDAP, Google Contacts, Facebook, MIT Public Key Server, Telepathy, EDS, phone address book…

Among the possible top level components that People aim to bring, a service providing meta-contacts (gathering all the little pieces of contacts and bringing back the notion of unique persons) is fundamental, as well as a synchronization solution (to update or enhance whatever contact source from another one). Another idea would be to provide a service managing ephemeral contact-related information (like the presence status of a person) to be shared among applications. An obvious and really great use of People would be an address book management application.

The idea behind the People Project is people sanity for the desktop: when some start to talk about amazing people integration in GNOME, the first step is to get consistency around the notion of a person. As the semantic of a person can’t be defined as a standard, we have to allow and exploit all the possible representations of it.

There are tons of possible applications to be explored: meta-contacts in Empathy, Gimmie or Soylent, integration with Seahorse, personal presence handling, activity framework, contact relationships, multiplayer games… People also fits the definition of what the address book component should be in the Online Desktop.

People is architectured around two libraries: a low level one to build backends and a higher one on top of which will lay a D-Bus interface. That interface must not be tainted by People as other implementations could come up. The libraries are developed using the Vala programming language. We want to allow backends to implement custom interfaces when it makes sense so we are not heading to a less common denominator syndrome. As an example, a relationship interface could be implemented by backends supporting FOAF. We have neat feature ideas that we take care to bring with several hot spots in mind, amongst which stand memory usage, network bandwidth usage (use on mobile devices) and i18n (name representation, automatic phone number formatting, …).

The idea that gave birth to People came up during a discussion I had one year ago with Felipe Contreras. It took some time for us to realize the potential of it. Few months ago, Ali Sabil got interested and passed it as a university project to get some time to hack on it, which brought some helpful hands as well. Lately, the development of People has been more and more active. We are constantly questioning our work in an iterative process to get the better implementation we can provide and each added feature is tested. Our effort aims to provide a working and validating implementation as soon as possible (with the release of a first usable version by June). There is currently a resource request for the People project at Freedesktop.org.

We hope to give a talk at GUADEC to get more people interested!

FOSDEM and other cool stuff

So, lots of good things are happening to me these times. Well, I guess.

Studies are soon-to-be-finished and I actually began my final year internship that ends in June, when I’ll graduate. For one week now, I have been a trainee at Nokia, Helsinki (Finland). This is quite a big move for me, since I never really left France until now. I’m really enjoying that new environment, this is challenging to adapt, on a personal side. I’m working in the OSSO Multimedia team at Nokia and I got surprised of how my arrival went well, people are really cool and I’m enjoying to be there, that’s a great opportunity to learn from them and actually discover the “big company” world, even using open source technologies. I’m sure that experience will bring me a lot and I’m not waiting for less on that.

So it’s Sunday night and I’m in Brussels (Belgium), spending some cool time with friends in the hotel hall. That was my first FOSDEM and I enjoyed it. Well, the classic thing about FOSDEM is to say that it was too short, and it was, but I can add my very own observations as well. The top advantage of such conferences is to gather and share some words with cool and respected people that you already know, like the Collabora crew and the GNOME community or meet with guys you aspire to share with, like the Vala developers. It’s also still refreshing to spend time with friends that are usually just too far away from you. FOSDEM is just maybe too subject-wide and not a lot of the talks grabbed my attention. As a comparison, GUADEC talks were a far lot more oriented to my fields of interest.

That conference was also the opportunity for Ali and I to share a bit about the project we’re working on, People. People is an unified address book management framework and I’ll blog about it very soon.

Tomorrow morning I fly back to Helsinki and try to begin well my second week there. The real difficult thing is to keep myself alive (i.e. feeding and basic other things like that) but I enjoy the freedom I have there and all the free time I can use (to hack a lot on People, eventually).

pymsn article : the crack-powered address book

This article attempts to provide an overview of the contact management system shipped with the pymsn library. In the following I assume you are already familiar with the basic pymsn API that allows you to connect to the server and attach the various event handlers.

Contact management is handled by the AddressBook object which manipulates AddressBookStorage, PendingContact, Group and Contact objects. I will try to describe each of these objects API and related event interfaces.

When you develop a client using pymsn, you generally start by instanciatiating the Client class, and then instanciating the event handlers you created. The Client instance holds an instance of the AddressBook that can be accessed through the address_book property:

client.address_book

The AddressBook instance provides a set of methods allowing you to interact with you address book stored remotely on the MSN servers, in fact each method call result in one or more SOAP requests (ouch). The results of those actions are then reflected on pymsn internal representation of data and the related methods from the AddressBookEventInterface are called to notify you that the changes have been done (contact added, group deleted, …). So all you have to do is to inherit from that event interface and fill the methods with your own needs then instanciate your instance.

Now let’s have a closer look on all those entities…

AddressBook

The AddressBook is the central object in contact management. It owns four important properties :

• contacts is an AddressBookStorage instance. It’s a collection containing Contact objects (actually representing the address book contact list). This object is very useful to request a subset of specific contacts using chained method calls.
• state is an AddressBookState value which describes the synchronization state of the address book. The address book is said to be synchronized when the local AddressBook data reflects excactly what’s stored on the server. When using the Client class, the given AddressBook instance is automatically synchronized at startup (else you would have to call the sync method).
• groups is a dictionary of Group objects indexed by their ids. It contains all the groups that are defined for that address book.
• pending_contacts is a set of PendingContact. Those objects are simple ones containing information on the contacts that asked to be added to the address book. Those pending invitations are to be accepted or declined.

API description

The API is short, simple, and method actions are self explained by their name and the name of their parameters, however when can here explain some bloody details.

Invitations

accept_contact_invitation(self, pending_contact, add_to_contact_list=True)

Method used to accept a pending invitation. It takes a PendingContact object from the pending_contacts set. You can specify if you want to add the new contact to your contact list (default behavior) or not. The pending contact will then be automatically removed from pending_contacts.

decline_contact_invitation(self, pending_contact)

Method used to decline a pending invitation. It takes a PendingContact object from the pending_contacts set. The contact will be blocked and won’t appear in you contact list. The pending contact will then be automatically removed from pending_contacts.

Contacts

add_messenger_contact(self, account, invite_display_name='', invite_message='')

Method used to add a “messenger contact” to the contact list. The “messenger contact” means a MSN or Yahoo contact (as opposed to “mail” or “mobile” contacts which are pending soon-to-be-added pymsn features). The main argument given to this method is the account (a valid email address). Giving a Yahoo domain email address, pymsn will try to add it both as a MSN account and a Yahoo account. You can also specify a display name and a message that may be displayed by the client used by the contact receiving the invitation.

delete_contact(self, contact)

Method used to delete a contact from the contact list. The given argument must be a Contact object from the contacts collection. The contact will then be automatically removed from contacts.

block_contact(self, contact)

Method used to block a contact. The given argument must be a Contact object from the contacts collection.

unblock_contact(self, contact)

Method used to unblock a contact. The given argument must be a Contact object from the contacts collection.

Groups

add_group(self, group_name)

Method used to add a group to the address book.

delete_group(self, group)

Method used to delete a group from the address book. The given argument must be a Group object from the groups dictionary.

rename_group(self, group, new_name)

Method used to rename a group. The first argument must be a Group object from the groups dictionary. The second one is a string.

add_contact_to_group(self, group, contact)

Method used to add a contact to a group. The first argument must be a Group object from the groups dictionary. The second one must be a Contact object from the contacts collection.

delete_contact_from_group(self, group, contact)

Method used to delete a contact from a group. The first argument must be a Group object from the groups dictionary. The second one must be a Contact object from the contacts collection.

AddressBookStorage

An AddressBookStorage is a set that contains Contact objects. This storage object provides several way to request for contacts. Queries are method calls that can be separated in two families : method beginning with “search_by” return sub address books (which are AddressBookStorage too) based on a field selection and method beginning with “group_by” group contacts together in a dictionary, based on a field value. The class provides some methods to query on not trivial Contact attributes :

search_by_memberships(self, memberships)

This method returns an AddressBookStorage object that contains the Contact objects which match the given memberships parameter. That parameter is a bitwise OR of Membership lists. There are lists for blocked contacts, allowed contacts, contacts that got you in their contact list, etc.

search_by_groups(self, *groups)

This method returns an AddressBookStorage object that contains the Contact objects which are part of the given groups. Each group parameter must be a Group instance.

group_by_group(self)

This method returns a dictionary in which each key is a Group and its linked data an AddressBookStorage object containing the Contact objects which are part of the key group.

In addition, pymsn makes use of the python __getattr__ method to provide custom query methods. Those are based on the next two methods that perform queries on the Contact object attributes :

search_by(self, field, value)

This method returns an AddressBookStorage containing the Contact objects for which the field attribute matched the given value.

group_by(self, field)

This method returns a dictionary which associates values of field to the related Contact objects.

Since the “search_by” methods return AddressBookStorage objects, you can execute complex request chaining method calls. For instance, the next instruction will select all the Yahoo Messenger contacts from the address book and then group them in a dictionary using their presence as keys :

address_book.contacts.search_by_network_id(pymsn.profile.NetworkID.EXTERNAL).group_by_presence()

Contact, PendingContact and Group

Those objects are the ones manipulated by the address book but the best thing to do is to refer to the API documentation to learn more about them so they won’t be described here.

AddressBookEventInterface

As with the other pymsn event interfaces, you’ll have to implement that interface and attach it to the Client. We won’t describe each method of the interface : almost each of the AddressBook methods fires an event method when its task is accomplished (the given arguments are then the concerned Contact or Group objects). We can however note that an event method exist to acknowledge you when there’s a new pending contact and that the errors that could occur performing the task (for instance when trying to add a contact who is already in the address book) are redirected to the client error handler.

Conclusion

Pymsn provides a powerful contact management through the address book. The API is still young and could certainly be improved : feel free to submit patched or bug reports. Feature requests are welcome too, we already have some pending ideas to be implemented : contact informations update (concerning mail addresses, birthdate and that kind of stuff), possibility to automatically add a contact to groups when adding him to the address book, etc.

If anybody find it useful, that blog post will become a page somewhere and will be updated to each API change.