NiceCover: A Serverless Webapp for Crowdsourcing Data Extraction and Knowledge Generation on Top of Scientific Portals

The title is a mouthful, I know. But you need that long to describe the idea.

There is a story behind it. I have first-hand experience in how major scientific portals (won't tell you which one exactly) upgrade their portals. Not good, my friends, not good ... are the processes and routines these people use.

Anyway, NiceCover is literally a nice cover for the scientific portals. The target is:

(1) Build a social collaboration layer on top of scientific portals.
(2) Write serverless webapps.
(3) Make sure you are within terms of use, but go wild otherwise.

These slides report on the first step.

There will be more soon.

Social Applications that Live in Clouds and Help People Link Their Work

Social Application are not Social Networks, especially not those that are popular today. Social apps are simple applications which do work socially -- normally by facilitating social collaboration. Note that there are not such apps today.

You can see my previous posts about social collaboration here and here. In fact, one app that welcomes social collaboration is still running in the wild ... wide open to the public .. with not a single client other than my own in these few weeks. Let's put social collaboration itself aside.

Consider this app:

It lives in the cloud because it is hosted by, say, Google Drive (or Dropbox, or some other) and keeps its output in the same place in the cloud it came from. In order to make it work you need two things:

(1) Solution to the Stringex Problem on which I wrote before, preferably a working software client.

(2) People willing to participate to form the three in the figure in the bottom-up fashion. Note that there are Miners and Mappers.


The problem is that (2) creates security problems. Specifically, people might be afraid to open up their data to public. This fear is not completely ungrounded. Dropbox API, for example, opens your public spaces wide open which means that anyone with the keys can overwrite, erase, add, or do anything with your data. Which is why most such apps limit their scope to the Dropbox account of the user him/herself. You basically write to your own account. On the other hand, Social Apps NEED to share their public spaces.

Still can be done. As long as you provide a separate PRIVATE space which no one can touch and develop a SMART SYNC. The smartness of the sync should be judged the same as Wikipedia does it -- it should be much easier to revert malicious changes than to make them.

Open problem which I am currently trying to close...

ArcGIS or GoogleMapsAPI?

The problem earlier expressed in this blog and solved in this webapp keeps nagging me.

Hypothetical: I am sitting in say, a Toyota car and use its navigator to find how to get from A to B. I can enter anything I want as A or B -- addresses, telephone numbers, names of shops (which in Japan include region/city/etc names). Navigator then calculates the route and shows it to me.

Now, say I need to build a robot that performs this unit action but does it many many times. For 100 locations I would need ~5K route requests. As I explained earlier, I need it to build a graph. You can loosely call it a dataset. Since such datasets do not exist in nature -- they are neither maps now traditional GIS formats (vector or otherwise) -- the datasets have to be created from scratch. Note that what ArcGIS calls a network is not really a graph -- it is more like a set of loosely-coupled pair-wise routes.


Now, consider this:

(1) Toyota Navigators, or actually their DVD ROMs are in proprietary format. Basically you cannot use it.
(2) ArcGIS can actually find addresses and you can use ArcScripts to find routes between places (see this example), but it is only possible in a paid ArcGIS Desktop version of software. However, if you have the paid version, you can load is with various data and use it offline.
(3) GoogleMapsAPI does all that in one package. It provides the scripts (API) that find you the route between two locations. It is equally strong to a car navigator in terms of being able to find places based on irregular string identifiers. When the API returns the route, it is split in laps (roads?) and each is provided with coordinates of intersections. Each road also has a name. The problem is the daily quota in the free access to the API.


The reason why this problem is nagging me is this. ArcGIS is a paid product -- it is actually not cheap. There is also ArcGIS Japan which probably has reliable maps of Japan. I can also overcome the Google Maps API quota problem by paying $9 a month to be able to issue 100K requests a day -- I can basically build a large dataset a day with this quota and I can stop paying when I am done (monthly contracts).

But. Not that I am cheap .... but I was really hoping that I could build an online community of collaborators which need the same kinds of datasets as I do and which I could work with to possibly build a library of custom datasets. The pilot webapp that builds such a dataset is kept wide open at this link for several days now but the only collaborators I can see are my own clients.

Will wait/try some more ... just for the fun of it. If not, will have to resort to the paid version of Google Maps API.

Google Maps as GIS, Road Graphs, and Social Collaboration

There is an active discussion on whether Google Maps is a GIS or not: http://highearthorbit.com/is-googlemaps-gis/

For me personally, GooogleMapsAPI wins this contest by a huge margin. The biggest win for Google's take at GIS is that it uses GPS coordinates -- specifically, you normally find the hash with { ih, kb, ...} keys ... I do not actually remember the keys but they are there and they actually represent the ... what was it? .... longitude and latitude of a coordinate. My knowledge of the meaning underlying the coordinates themselves is lame. I know it. I do not apologies for it. Simply because for me the physical meaning is not important, as long as I can plot them on a 2D or 3D visualization. In fact, I once plotted a worldwide AS-level network topology randomly assigning X and Y only to realize that the world was on its side. You could actually tell because the ISPs formed pretty god outlines of continents.

Now, all I need to know about GISes is the datatypes. By far the most popular datatype in traditional GIS is SHAPE (.shp). Sometimes these files are referred to as SHAPEFILES. You can read all about the format here, but I would recommend to see the diagram from here:

Sorry for the Japanese, but I could not find English diagram with the same level of visual clarity. The wikipedia page definitely does not tell you that right away. The .SHP format is very simple:

(1) It is a vector format.
(2) It can be in 2D-mesh or 3D-mesh forms, where the 2D and 3D do not stand for coordinate dimensions (which is why Google search on the format will only confuse you), but rather precision of the coordinate itself (see figure above). Simply put, 2D-mesh gives you 10x10km dots and 3D-mesh gives you 1x1km dots.

I once officially requested Tokyo road map from some official association, got their CD ... only to find out that the data was in 2D-mesh format. This meant that the center of Tokyo on the plot looked like a single dot. Literally!

That was the point at which I decided to just use Google Maps as my main GIS system. YES, this improved precision of my datasets, but NO, it did not solve all my problems -- the big problem further on.


I recently had to use Google Maps as GIS to build road maps (graph structures) collecting certain places. The problem I stumbled upon is collaboration. Specifically, collaboration in building datasets -- not using them. Majority of GIS portals I know (included Google Maps Engine) are there for 2 purposes:

(1) to use a GIS dataset together
(2) to build a dataset based on the interface provided by the portal.

The simple purpose I am trying to pursue is left unfulfilled by both the above. Which is actually strange because quotas on requests exist for any existing API -- GoogleMapsAPI or ArcGIS, although it could be in slightly varied forms -- ArcGIS is paid from the start and has rather big quotas while GoogleMapsAPI has very restrictive quotas in its free form.

Why quotas? This is simple. Google as well as all other big players have to ration access to their Big Data. Imagine all free users having unlimited quotas to all Google APIs?! Specifically, a free GoogleMapsAPI client (per IP) has the daily quota of 2500 requests. This is key because ... say, if I have 300 locations and I need to build an undirectional (A-B = B-A) graph among them, I need to make > 35K API requests. About a month for one client. About a day for 30 clients. About an hour if the crowd piles up and helps me with it ... on demand. The simple idea behind online collaboration on this topic is that I would borrow daily quotas from users who ... in their majority ... do not use much of their daily quotas anyway. Note that Google's Terms of Use do not prohibit such a use of its API.

For example:

http://t.co/3SmOedz2l0

is a simple serverless web application which uses client's daily quota to suck routes from GoogleMapsAPI and store them in my Dropbox folder. All clients write to the same folder. Collisions are resolved by the web application. This is a very simple collaboration.

Note that its purpose is different from that of default modes of ArcGIS or GoogleMaps (API or Engine) and cannot be created by the either of them. I am surprised to find myself in the position that public collaboration on a (useful?) dataset is completely impossible just because it departs -- a little bit -- from the default use.

Let me go one step deeper into this rant. I tried posting a request on Twitter with useful hashtags like #googlemaps #googlemapsapi #datasets, etc. in three languages, only to find out later that my message did not show up in none of the hashtag streams. I do not know what the criteria are used by Twitter when it aggregates its hashtag streams, but it looks like posts saying "I really like Google Maps" are more worthy the inclusion into the stream than a meaningful post. In the end, my posts ended up viewed only by the few (3?) people I have as my direct followers. Not a very social person, obviously.

Now I am posting it here. Let's see if I can actually find a way to broadcast this message beyond the small social box in which I live.