Why We Need a User Interface Resource Server for Intelligent Environments

Gottfried Zimmermann
Eberhard Karls Universität Tübingen, Germany

Benjamin Wassermann
Eberhard Karls Universität Tübingen, Germany

This paper will be presented at: 4th International Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’09), and will be published in: Augusto, J.C. (chief editor). Ambient Intelligence and Smart Environments. IOS Press, Amsterdam, The Netherlands.

Table of Contents:


For intelligent environments to become broadly adopted, they need to employ advanced user interface technologies. Such user interfaces have to provide support for personalization, accessibility, context-awareness, openness for 3rd-party contributions, agent-based user interfaces and management of user interfaces. In this paper, we introduce the concept of a user interface resource server, and how it supports each of these requirements. We describe related existing systems and how far they go in meeting these requirements, including the global resource server provided by dot UI. Finally, we identify future work.


User interface, resource server, Universal Remote Console (URC), artificial intelligence, ambient intelligence, pluggable user interfaces, user profile


Imagine the following scenario: You are on business travel, checking into a hotel in a foreign city. When you enter your hotel room, the air conditioning is automatically set to your preferred daytime room temperature. The TV shows a welcome screen. You pull your smart phone out of your pocket, and use it to switch to your favorite news channel. This is easy since the smart phone shows the same familiar user interface that you use for your home TV, even though the hotel TV is a different brand and model. Of course, all labels are shown in your native language. While waiting for the news to start, with a few touches on your smart phone you program the TV to wake you up at 6am with your favorite song in the next morning. No fussing around with complicated and unknown alarm clocks. Just take your personal user interface with you, wherever you go.

This is just a glimpse of what intelligent environments should be able to do for you. You might expect them to be even "smarter", for example automatically switch on the coffee machine in the hotel room 15 minutes prior to the wake-up call – to have coffee ready when you wake up.

In this paper, we focus on user interfaces in intelligent environments, and what features they need to bring to bear, in order to serve the needs of the users. Even though we strive for intelligent systems that can make decisions on behalf of the user, there will always be the need for user interaction at some point in the process. Ultimately, the user wants to be in control, either for confirmation of the default behavior, or for triggering an exception from it.

We define a user interface resource server as a platform for the storage, processing and retrieval of information related to user interfaces for service environments. This platform is globally accessible from any system that wants to make use of it, anytime and from everywhere. Note that we assume that user interaction devices have some kind of connectivity with a resource server, typically through the Internet, possibly involving gateways.

We argue that such a user interface resource server is needed for intelligent environments. In the following sections, we will first present selected requirements of user interfaces in intelligent environments. These requirements are based on common knowledge and on our own experiences from projects in the area of intelligent environments and smart homes. We will also describe how a user interface resource server would help to meet these requirements. Then, we will look at a selection of systems and technologies, and how far they go in meeting these requirements. We will finally look at our resource server implementation (based on the Universal Remote Console framework) in this regard. In conclusion, we will summarize achievements and where areas of open issues remain.

1. Requirements of User Interfaces in Intelligent Environments

1.1. Personalization

Intelligent environments consist of multiple devices and services that are connected in a mostly heterogeneous approach. Commonly, every device/service has their own one user interface, and everybody is expected to use it. Even small components can become quite complex, and current user interfaces are reflecting much of this complexity, thus contributing to displeasure of the user. Intelligent environments will only be accepted and adopted if they make it easy for users to achieve their goals.

Personalized user interfaces are deemed to help alleviate this situation. They escape the one-size-fits-all approach that doesn't make sense for user interfaces. Different people have different needs and preferences when they interact with their environments. The needs and preferences of different users are very diverse and cannot be accommodated in total by a simple layering of functionality in the user interface. A new breed of personal user interfaces is needed that allows for variation of style, scope, presentation modalities and content. These user interfaces should be based on some kind of abstract user interface model, but tailored to a specific person and the context of use (see section 1.3).

A user profile contains information about a user's interaction needs and preferences, and has to be persistently accessible from anywhere. Different approaches and models exist for user profiles in service environments. Currently, ETSI has a work item on personalization and user profile management [1] which is expected to result in a European standard in 2010. Of course, special care has to be taken with regard to security and privacy issues.

A user interface resource server supports personalization by providing a persistent platform for user profiles, and profile-specific user interface components such as labels, icons, skins, etc. Users can "carry their personal user interface with them", even when they are on the move and using non-personal interaction devices such as a touch-screen kiosk on the airport, or an advanced universal remote control in a hotel room.

In a more sophisticated approach, the resource server itself could generate personalized (custom-tailored) user interfaces, based on an abstract user interface and user profile data.

1.2. Accessibility

Accessibility is a special case of personalization, addressing specific needs and preferences of users due to permanent or temporary impairments. For years, service providers have been struggling with the diverse needs of users with disabilities, trying to make one single user interface that would suit the needs of all users based on Universal Design principles.

We need an advanced approach of Universal Design involving the delivery of a set of alternative user interfaces with each addressing a different type of user. These alternative user interfaces can then be made available through a resource server. Such an approach would go a long way toward overcoming current accessibility problems caused by device and service manufacturers not having the expertise to design user interfaces for special user groups with disabilities. The resource server approach lets 3rd parties (e.g. assistive technology vendors) contribute specialized versions of user interfaces that are tailored to the special needs of a particular user group, e.g. people who are blind, dyslexic, etc. (see also section 1.4). Also, this way alternative user interfaces can be added after the product (hardware) has already been shipped.

1.3. Context-Awareness

Aside from a user's profile, a user interface should also be tailored to the context of use, i.e. the specific interaction device used and other situational parameters. For the same user, the context can change for different interaction instances, or even during the course of a dialog.

For generating a context-aware user interface, we need to know the exact parameters of the interaction device used, such as screen resolution, pixel density, capabilities for touch control, voice output, etc. Rather than having the device convey the set of parameters every time it communicates with the service environment, it can point to a public device profile document containing these parameters. This approach is facilitated in the User Agent Profile standard [2]. A resource server can store common device profiles and make them available for intelligent service environments.

Another need for adaptation arises from varying parameters in possible use situations, such as noise level, ambient light, and temporary impairments due to a person's current activity. For example, a person who is driving a car needs voice interaction, and a tourist wearing gloves needs bigger buttons on the touch-screen of the train ticket machine.

1.4. Openness for 3rd-Party Contributions

The pluggable user interface approach allows for substitution of user interfaces for devices and services [3]. This is facilitated by an architecture that decouples the user interface (front-end) from the functionality of the device/service (back-end), such as realized in XForms [4] (for Web forms) and in the Universal Remote Console (URC) framework [5][6]. If there is a public connection point for user interfaces to access the back-end (this is called a "user interface socket" in URC), any 3rd party can build a user interface that fits to the connection point of the back-end device/service.

The openness for 3rd-party user interfaces creates a competition on user interfaces in the market. In fact, it creates a whole new market. This is good news for the user since competition means more selection and higher quality of products (user interfaces). It is also good for producers since it allows for new business models centered around user interfaces. (The ring-tone trading for cell phones is a little taste of what this new market could afford in terms of revenues.) It should be noted that the envisioned openness for user interfaces does not only apply to remote user interfaces, but also to built-in user interfaces that can load user interface code dynamically (skins, icons, menus, etc.).

Openness for 3rd-party contributions is a strong argument for a common user interface resource server. With such a resource server, manufacturers of devices and services, manufacturers of interaction devices, and 3rd parties can all join in a competitive market of user interfaces for intelligent environments, for the benefit of the user.

1.5. Support for Agent-Based User Interfaces

The idea of agent-based user interfaces is that intelligent agents communicate with components of a service environment on behalf of the user. The user is relieved from the tedious work of interacting with individual devices and services one by one, and instead relies on the agent to pursue their goals.

For this to become true, agents need to have implicit or explicit knowledge about specific domains, and about the user and their preferred behavior. Rules can be used to reflect knowledge about particular domains and their devices and services. Also, rules can describe behavior for personal automation (for example heating up the coffee machine 15 minutes prior to wake-up time). Rules represent explicit knowledge and can be shared across users (possible involving parameterization).

Task models [7] are another method of making domain knowledge explicit. They describe how high-level tasks can be achieved by using lower-level tasks such as provided by devices and services. Task models are an excellent back-bone for dynamically building seamless user interfaces that cover the functionality of a whole set of devices/services in an environment rather than having one user interface per device/service.

Explicit knowledge on domains and users is required by intelligent agents to communicate to service environments and to interact with the user. This knowledge can be managed by a resource server that stores rules, task models and voice interaction parameters. Thus it would be available for a person's personal controller device (even if they buy a new one), for public terminals, for "temporary home environments" such as hotel and hospital rooms, and for the interactive voice response system at the help desk call center.

1.6. Support for Management of User Interfaces

If the user were to pick their choice out of tens or hundreds of user interfaces, they would be overwhelmed. We need a better mechanism of managing pluggable user interfaces for users. We imagine three possible approaches, each requiring some kind of a resource server:

  1. A resource server infers one "best-match" pluggable user interface, based on data from the use context (user profile, device profile, situational context) and properties of user interfaces.
  2. The user subscribes to a family of user interfaces that are consistently designed across common devices/services. These user interfaces would have something in common, e.g. the same provider, the same layout style, or accessibility for a particular user group. A resource server delivers the one user interface out of the subscribed family that fits a particular device/service that the user wishes to control.
  3. The user assigns a person responsible for creating and/or managing their personal user interfaces. This could be a family member, a friend or a professional installer. The assigned manager can then remotely manage the pluggable user interfaces for their customers via a resource server.

Another management aspect is the provision of user interface updates for devices and appliances. With the resource server managing the user interfaces of a device, user interface updates can be deployed easily and without risking breaking the device.

The Mozilla Foundation [8] maintains a registry of Firefox add-ons, including so-called "themes". Once downloaded by the user, a theme determines the appearance of the browser, including icons. Although restricted to the visual appearance of an application ("skinning"), this is an example of an existing ecosystem for 3rd party user interface resources.

XForms [4] allows web forms to reference externally provided metadata (such as labels) via a URL in the 'src' attribute. This mechanism can be used to retrieve localized and/or contextualized user interface components from a Web server acting as a kind of resource server (although out of scope for the XForms specification).

The ServiceUI API [9] defines classes and interfaces that enable service UIs to be associated with Jini services. It provides a basic mechanism for "pluggable user interfaces" in the Jini world, in which a Jini lookup service would fulfill the role of a simple user interface resource server.

ICrafter [10] manages user interfaces for interactive workspaces in a ubiquitous computing environment. It can either use custom (pre-defined) or automatically generated user interfaces, based on service descriptions, interaction device profiles and contextual parameters. However, ICrafter does not provide personalized user interfaces, since it doesn't know about user profiles. Furthermore, it is a closed system without an explicit resource server, relying on proprietary protocols, and therefore cannot accommodate 3rd-party contributions in a dynamic fashion.

The "personal home server" [11] is similar, but includes support for personalization. The automatically generated user interface for a home appliance is based on a user profile which is confined to the personal home server for privacy and security reasons. Implemented inside a PDA, watch or mobile phone, it is carried along by the user as they encounter environments beyond the home. Having no concept for a user interface resource server, this system again lacks the openness for 3rd-party contributions.

Other pervasive computing systems, including XWeb [12] and Speakeasy [13], provide proprietary mechanisms for dynamically selecting and/or assembling user interfaces for service environments, with some support for personalization and context-awareness, but do not cater for 3rd-party contributions, in general.

3. URC Resource Server Implementation

The Universal Remote Console (URC) framework [5][6] introduces the concept of a resource server for the provision of user interface components in the context of pluggable user interfaces, and resource sheets for the specification of personalized and contextualized user interface resources. Specific implementation guidelines by the URC Consortium complement this framework, in particular with regard to resource server interface and extensible set of resource properties [14][15]. An open-source toolkit for the URC framework is available [16]. Several European projects are using the URC technology, with i2home [17] providing technological support, and an implementation using task models.

Figure 1: Screen shot of resource server page

Fig. 1. Screenshot of dot UI's resource server page for managing resources (https://res.dotui.com). In the section "Add New Resource Or Upload New Version of Existing", a resource provider can upload a user interface resource file, select its URC-framework specific type (e.g. resource sheet, icon, etc.) and specify a URI as its globally unique name (as an option the system can generate a name in the form of a URN). In the section "Existing Resources", all resources that the user owns are listed, with links for each to add a new version, to edit its properties and access rights, to retrieve a version or to remove a resource. Text description.

A global resource server for the URC ecosystem has been implemented (see screenshot in fig. 1), based on HTTPS, and is provided by dot UI Inc. [18] as a service. It is currently being used by selected projects in a pilot testing phase.

Resource servers for user interfaces can be installed locally or globally. The radius of local implementations is restricted to a home network, or some other local network. Local resource servers can provide support for some of the requirements, but miss to facilitate openness for 3rd-party contributions. We argue that, in order to reap the full potentials of advanced user interfaces, there needs to be a global resource server as an open platform for user interface contributions from all stakeholders. Local resource servers can be useful, but should be part of a larger user interface platform providing synchronization mechanisms between the components involved.

There is still a question remaining on whether there should be one single global resource server or multiple instances. We slightly lean toward a single instance acting as a global root directory service that can point to subordinate resource servers for content delivery. However, we can also imagine a decentralized platform consisting of multiple components based on peer-to-peer technology. This issue needs further investigation, potentially reusing existing Web service technologies for the purpose of service discovery and orchestration.

4. Future Work

While the URC framework generally supports all requirements mentioned in this paper, current implementations (including the resource server) do not make use of the full spectrum. In particular, the core metadata vocabulary on user interface resources is still quite small, and the mechanisms for finding best-match user interfaces do not currently involve advanced data on user profiles and use contexts (beyond simple device profile matching).

In the future, we plan to extend the metadata vocabulary for user interface resources, and develop more sophisticated algorithms for match-making between different contexts of use and user interfaces and their components. We will look into artificial intelligence based methods such as rule based systems, machine learning and statistical approaches, to fill this gap. Also, some types of user interfaces may be generated automatically based on an abstract user interface description. (We are currently investigating the generation of simple DHTML based user interfaces for digital media control, running on the iPhone.)

We realize that the URC ecosystem is still in its infancy. Technical reports will evolve from draft status to approved publications, forming the glue between the different components of the ecosystem. This will be driven by a community process within the URC Consortium (http://myurc.org).

5. Conclusion

Based on user interface requirements for intelligent environments, we have justified the need for a resource server for advanced user interface implementations in this area. Such a resource server would go a long way in supporting personalization, accessibility, context-awareness, openness for 3rd-party contributions, support for agent-based user interfaces and support for management of user interfaces.

However, in our survey we have found that the concept of a user interface resource server – as we envision it – has not been considered widely, and that existing implementations realize some aspects only. The public resource server of dot UI Inc. advances the field of user interfaces for intelligent environments, supported by emerging implementation guidelines for components within the URC ecosystem.


Part of this work was funded by EU 6th Framework Program under grant FP6-033502 (i2home); and by the US Dept of Education, NIDRR, under Grant H133E030012 (RERC on IT Access). The opinions herein are those of the authors and not necessarily those of the funding agencies.


[1] European Telecommunications Standards Institute (ETSI). Specialist Task Force 342: Personalization and User Profile Management Standardization. http://portal.etsi.org/STFs/STF_HomePages/STF342/STF342.asp

[2] Wireless Application Protocol Forum. User Agent Profile Specification (UAProf). Oct. 2001. http://www.openmobilealliance.org/tech/affiliates/wap/wap-248-uaprof-20011020-a.pdf

[3] G. Zimmermann. Open User Interface Standards - Towards Coherent, Task-Oriented and Scalable User Interfaces in the Home Environments. In: Proceedings of 3rd IET International Conference on Intelligent Environments (IE07), Sep. 24-25, 2007, Ulm University, Germany, p. 36ff. The IET, 2007.

[4] W3C. XForms 1.0 (Third Edition). W3C Recommendation 29 October 2007. http://www.w3.org/TR/2007/REC-xforms-20071029/

[5] ISO/IEC 24752:2008. Information technology – User interfaces – Universal Remote Console. International standard in 5 parts. ISO/IEC, 2008.

[6] URC Consortium. URC Technical Primer 1.0. Draft Technical Report 2008-11-24. http://myurc.org/TR/urc-tech-primer1.0-20081124/

[7] C. Rich. Building Task-Based User Interfaces with ANSI/CEA-2018. Submitted to IEEE Computer, 2009.

[8] The Mozilla Foundation. http://www.mozilla.org/

[9] The ServiceUI API. http://www.artima.com/jini/serviceui/doc11/index.html

[10] S.R. Ponnekanti, B. Lee, A. Fox, P. Hanrahan, T. Winograd. ICrafter: A Service Framework for Ubiquitous Computing Environments. In: Ubicomp 2001: Ubiquitous Computing, Lecture Notes in Computer Science, Vol. 2201/2001, pp. 56-75. Springer Berlin / Heidelberg, 2001.

[11] T. Nakajima, I. Satoh. A software infrastructure for supporting spontaneous and personalized interaction in home computing environments. Pers Ubiquit Comput (2006) 10: 379–391.

[12] D.R. Olsen, S. Jefferies, T. Nielsen, W. Moyes, P. Fredrickson. Cross-modal Interaction using XWeb. In: Proc. 13th Annual Symposium on User Interface Software and Technology 2000.

[13] M.W. Newman, S. Izadi, W.K. Edwards, J.Z. Sedivy, T.F. Smith. User Interfaces When and Where They are Needed: An Infrastructure for Recombinant Computing. In: Proc. 15th annual ACM symposium on User interface software and technology, pp. 171-180. ACM New York, 2002.

[14] URC Consortium. Resource Server HTTP Interface 1.0. Draft Technical Report 2008-08-14. http://myurc.org/TR/res-serv-http1.0-20080814/

[15] URC Consortium. Resource Property Vocabulary 1.0. Draft Technical Report 2008-11-24. http://myurc.org/TR/res-prop-vocab1.0-20081124/

[16] URC Consortium. Tools and Prototype Implementations for the URC Framework. http://myurc.org/tools/

[17] i2home. Intuitive Interaction for Everyone with Home Appliances based on Industry Standards. http://www.i2home.org/

[18] dot UI. http://www.dotui.com/