Web Services & Interoperability

THDL Toolbox > Web Services & Interoperability

Web Services & Interoperability

Contributor(s): David Germano, Jed Verity.

Web Services 101: The Problem of Interoperability, i. e. Sharing Data & Behaviors Across Web sites

I. The Basic Problem

The Web is made of many different Web sites which consist of data (content); metadata (semantic information about the content); and tools (applications which enable users to create, reivse and utilize the data in conjunction with its metadata). Each Web site has its own distinctive sets of data, presentational styles or look and feel, and specific sets of functions or behaviors for consulting and/or analyzing the data. Many Web sites are content to just be self-contained Web sites to which users must go if they want to access their data and interactive functions.

However, there is a great interest in a more fluid vision and practice, in which data is shared as “services” offered across Web sites. In this way, a Web site’s content can be utilized within other web sites as contextualized within their own settings. Thus, for example, a dictionary from one project might be utilized as a service within a Web site created by a totally different project. Rathe than simply referring the former web site’s users over to the dictonary’s own web site, the users are able to stay within the former web site, but seamlessly use the dictionary’s content in relationship to that web site’s contents. In this way, web sites are able to fluidly weave together a variety of data streams created by themselves and originating externally, while their users have a consistent and integrated experience without having to struggle with different web sites and discordant looks and feels.

However, there are significant problems to fully actualizing such seamless interoperabilty across web sites and applications, such that this constitutes yet another Holy Grail sought after across the emergent digital landscape. One of the easiest problems to solve is that of credit and attribution. While a Web site aspires to consistency and integration, data providers want to ensure that their data is clearly credited and attributed to themselves, even as they appear within different web sites. This is commonly resolved by a standard attribution line, which says something like “Courtesy of SOURCE X”, and may specify the URL of the source’s native web site, its own ID for the resource in question, and potentially a link to see the resource in its original setting.

A more serious problem is how data gets shared across web sites in an automated fashion so that the data is integrated in the external web site. The standard way this happens is creating a web services API, also referred to as “Web API” or just “web services”. These involve exposing the data in a standard exchange format such as XML or JSON, such that other web sites can import that data and then embed and style it as they like into their own web sites. API (advanced programming interface) basically signifies a standardized vocaublary for accessing data by which another program or application can automatically interact with the source application to retreive data and understand the data’s structure. The creation of such web services is relatively straight forward. It basicaly involves providing data as a service, rather than any specific behaviors or tools associated with that data. Simple examples include an image with a caption, place, and date, including the URL for the actual image and the URL for its original full cataloging record; or a structured biographical description, which has some twenty distinct fields – name, ethnicity, profession, description, and so forth. In this way, web sites can query other web resources external to it, and get back either specific resources, or search results, from which users then choose a given resource and view it in more detail.

A far more difficult problem, however, is the sharing of behaviors across Web sites. In other words, an entire application’s functionality is shared, namely “objects” which consist of interrelated data and behaviors. This is far more complex that simply providing data. Firstly, there is the “same source” or “same origin” problem, namely that if an aplication and a web site are not from the same server, there are security problems that prevent the straightforward of the application and its functions into the web site ). Secondly, this involves the application controlling the data’s presentation, which, however, can then appear quite discordant in terms of look and feel within an external web site.

II. Goals

Our goal within the Tibetan and Himalayan Library (THL) is to become a publisher of web sites and services. Firstly, THL is publishing a variety of web sites which provide consistent and self-contained experiences for users, and yet use a common underlying set of applications and servers. The consistency and self-contained character of a given web sites entails that for the user all data and functions are visually integrated with consistent look and feel, while the user is never forced to go to any external web site, including anything like an “audio-video archive” or other repository. Rather, those repositories provide the data and metadata to the web site in question so that it is visually integrated. That means that THL applications have to function as services internally to provide not only data, but also characteristic behaviors associated with that data. In addition, the various web sites published by THL have to strike a balance between comon design principles that enable users to fluidly move across various THL-published web sites in terms of navigation and behavior, and visual variations that enable each web site to have its own distinctive character.

Secondly, THL aspires to publish data streams and their behaviors as services to completley external web sites, so that THL data can be seamlessly integrated within a broad variety of web sites. In doing so, we want to provide our applications’ functionality, and yet allow the external web sites to adjust the visual presentation and styling so that the data is visually integrated in their web site as much as possible.

The actualization of these aspirations is, as they say, a non-trivial challenge.

NOTES TO EDIT IN

json meant to bring out data, and then you create whoel things around it. not good for bringing all the style

You can create full documents iwth just java script. if the plug in is taking the data and saying let’s create a whole web page out of it, it can be done. That is exactly what XMl and XSLT is good at – taking basicaly raw data and applying and building a whole page out of the raw data with a relatively easy way to provide new styles. you would still need java script for functionality if you are doing ajax through the different applications. but constructing a document XSLT could do. otherwise you do this whole java script trees of creating tables, and building whole more or less html documents or a big part of it. but then it is harder to have the data and presenation separate, and becomes much less managable that way.

take a smaller application and mock up with json. would probably be fine with THL, but getting it outside and having someone deploy it and fitting it to their styels is more of a concern. maybe part of plugin is a default stylesheet you can edit to make it fit.

The standard way is through web serivces-api. You have a script that has defined parameters like here is your main search term, here i how you specify how many results you want, etc., and then that is fed back in XML and it is up to you who are making the api call to take that xml data and do whatever you want with it. it is thus very similiar to json in that the web services delivers whatever data you want. but is almost always just in this data framework, rather than brining a whole page from one application to antoher. rather we take data from that repository, and plug it into our own repostiory

concern in how easy stylesheet functioning will look. we could hard code it so that it gets presented in a generic, readable fromat. that is easy. but what is hard is to add in all the html and styling stuff within the java script code. better to searpate as much as possible.

most people just give access to the data, which is the important thing, and you can do whatever you want with it. part of this is security concerns that people have with JSON compared to XML. for us is fine sicne we say we are the server, and we are givignt he client what they ened to contact the server. but you have to really trust the server. thus many sites don’t do json and instead do xml/http request stuff which is more secure.

jed thinks not so common that you build whole pages out of java script if you have other options.

concern – if you change the format of the data being output via json, you should not have to go in and do really internsive work on the plugin side,t he receiving side, to make it work.s o concern about making as unbreakable as possible, as easily customizable as possible. requires some creative templating with transcripts that you can keep the data layer and presentation layer distinct which makes it easier to updat enad cutomize

to make this work just on a different THL web site, how would we do it? Make modular so can deliver to various parts of THL without redoing it. But can be a mistake to do serious developmen to make it appear you are not leaving the context. you are dealing witha whole page of functionality that is no way integrate to the sitate you are except to make it appear you have not left, makes it more sense to have baner and navigation brought over to the application, rather than hav applciation brought over to what is toehrwise a static banner and Nvigation.

Vs. sera monastery pulling into the flow of an application native to sera monastery, like photos from mms going into an essay.

Knowledge Map linke dapplications

Himalayan Dictioanries

Tibetan Dictionary

XML Repositories – Essays, Catalogs & Tibetan texts

Place Dictionary

Never leave the web site! Only exception is clicking on the THL Site Map or General LINks.

Click on Knowledge Maps

Use Lightbox popups for videos, slide shows, etc.

Use by Non-THL Web sites

RoR is good at returning json, Javascript objects, rather than XML. So if person programming website is good wtih java script, we could make THL Java script library. a jquery plugin that had all the funtions people needed to do with THL. What one could do is use this plugin and then essentially you are implementing everything locally to get around these problems. Gives you the maximum control since not hindred by hinky inframe and ajax securiety issues. get data you need from jason queries, and use java script to get what you want. This is what Ed did for the work on the non-THL page. That is our best bet to get a robust solution that is long term.

Worth creating a really good java script library that everyone maintains in common. And we just have some functions we can call and the person implementing it, creatng the web page, doesn’t need to be a java script expert. Just like using jquery is dead easy.

If you have a javascript application, you can give to toher partners and they can invoke jsut as easily on their web sites without getting deeply into our databases,

Jed – have same security issues you have with ajax.

if main concern is iframe looks wierd since is in totally different contet that is ported over, what if have iframe that is pipped in with styleshet specification sot ahh tthe data is context aware within the iframe, though you will stil have some problems.

otherwise you are reubilding the iframe functionality with what you are talking about with using the iframe. do everythingt he iframe is supposed to do but building from scartch. maybe make it more seamles with content by sending dyamc argument that incldues the context inforamtion so that it nkows how to present itself dynamically.

3. outside frame and inside frame divorced. wh do you care thought if porting in a whole noterh application.

Identify 2-3 small kinds of services

1. display a image

2. embed a map

3. etc.

then work your way up from there.

Ingesting Json is easy. Are existing Javascript libraries that would allow you to ingest Json and manipulate as you want. The part that is difficult is packaging up a function that has the THL look and feel and behaviors is difficult. Json is jus transmission protocol for your data package.

You need to package up the behavior, not just the data. That is the problem. The data is coming from the web service, but the behavior is the thing that is problematic. Json is just the wrapping paper in your data package. You can transmit the data package via XML or Json. Json preferable sometims since is native Java script. if ou are going to ingetst it into a javascript library, then save a step and ingest a java script objet.

RoR application send XML or Json packages out to another Web page. But when it arrives at the other web page, you can program the web pages. If PHP web page, you couldl write a pHP progrm eery time you want to processe this data coming from the network. But you have certain predefined bheaviors you want to do again and again. So instead you could use ajax, which is good at – if ii have a html page, and a div1 tht is my main content, ajax is good at saying erase whatever is in that div1 and fill it up with whatever i got fro the web service. but if i wanted to do that with a ajax query from the web page to the web page, or app to app, no problem. but if i try to run that ajax query from the web page to the app, it wil nt worik. you get around that by having local copies of the java script on ever machine so that hte bheavior is only ever running locally, and it is only slotting in the data that it is getting over the network.

a lot of the rails app activity for users is java script – but not all. if non-java script, wold have to make java script to emulate.

rails app is grabbing stuff out of the database. java script library will never want to grab stuff out of database, but rather grab stuff off the network through web services.

Knowledge map builder needs to create a web service that sends a repsentation of all teh stuff on that web page when called, and you need a jva script library that can tkae that package of data and make it appear in the browser the way you want it to appear. are seaprtedly layers in applciaton – data fromm presnetation.

open sources connections is local firm good with java script. erik pugh.

external link: http://www.opensourceconnections.com/

Issue is flat data vs. something more intersting and rich. Advanced functionality vs. a flat web page.

DHC orientation towards being services aware. a big essay doesn’t care about services aware. but if dictionary, then a really useful web service, or map, tha ta lot of people could make use of.

json or xml would be easy for Place Dictionary now.

for gis mapping, simliar set of consdierations and solutions. we provide wms map serices. if you go to a given url formatted in tehr ight way, you can get back an open stadndard way of providing geospatial data. and are open source java script libraries that know how to ingest wms web services. that is what THL will do – each THL page will have the java scrip libraries adamn provdies to render these.

the presenation layer of mapping software is java script based. the geospatial engein and map making stuf is not. but user expereinc and controls is all java script

pecha publisher.

Specific changes

andres – tnraslation on tibetan grammaticl features proper nounts ; himalayan languages down

andrse move the knowledge mpa toc to the right

phase 2 – make editing funcdtions of json output to work. now jsut add link to editorial page.

page to collaet allt he workflow system.

chekc over site map – no wnolege map

ritual works, but dog gives error message

issue of selecting text and invoking search. like reading essay and you want to hglight text and have up come search results.

vs. applications standard search and results showing up in various web sites.

vs. editing

dictionary link back to knowelge map

THDL Toolbox > Web Services & Interoperability