What if my top frequently used Apps are transformed into conversational user interface (Part 3/5)

Gaming world has its own innovative and visually artistic experience that couldn’t be found elsewhere. When a dialogue pops up, it brings all settings and crews ready in their standby positions in order. Similarly the elements shown in the user interface also carry a certain message contributed back to layers of relationships between people and their hidden intentions.

From the article of Luke Wroblewski personal blog, one of my favorites and eminent digital product leaders in the world, has beautifully and well addressed the point that chat user interface interaction empowered by artifitial intelligence nowadays might be able to evolve into more user-friendly chat UI utility visual elements, such as collapes & expand, integrated actions and object experiences powered by large language models, which are some of the fundamental patterns we use all the time.

Collapse & Expand (Source: Expanding Conversational User Interfaces from LUKEW)

Integrated Actions (Source: Expanding Conversational User Interfaces from LUKEW)

I believe all readers can learn tons of insights and inspirations from Luke’s essays, where I will come back to explore more design possibilities in Part 4 extended from the concept above.

This wave of conversation-oriented interaction does aggressively open a variety of applications, once self-owned database has been poured into the training set and generate answers to the questions people ask. Like gasoline vehicles if they all have been plugged in online to collect all the mileage on road, those cars might out-perform competitors in terms of the amount of data and design iterations.

The Apps installed in my Pixel phone have been carried out with a long history of my personal usage logs and data, which naturally provide a valuable off the shelf input of the predecessor of my own personal AI assistant. Apart from the Apps that are used for work or particular purpose (such as those for entertainment, education, transaction or GPS navigation while driving), other Apps that I use a lot include LINE, Facebook, Linkedin, Spotify and Chrome, etc, which I usually spend dozens of minutes to hours per day in average.

Those Apps serve and act distinctively in terms of content and goals provided, while people probably have completely different expectations from those Apps (or we can call it information portals) even if we all download the same one. For example, Instagram might be an outfit inspiration to one person, but more like a restaurant recommendation guide to another.

To illustrate this differences reflected back to my design explorations, I make a simple and small assumption and distill the scenarios from my own experience: we all might have a goal to accomplish, whether the goal is big or small, deliberately or unknowingly, and those giant Apps could take advantage of this conversational UI pattern from the LLM (Large Language Model) to provide unique user experience so we could jump right in and start interacting.

empower messenger app with llms

Chatroom, Voice & Video Calls, Payment and OpenChat (Source: line.me/zh-hant/)

Take LINE for example. LINE App becomes one of the must-have messaging Apps in Taiwan, where almost over 90% of the population has LINE installed in their phones. Like WeChat in China, or WhatsApp in India or Brazil, people use it to send text messages or share videos, and make phone call or conduct payment in this single one platform everyday. From the circle of friends and family, to social groups of community, LINE’s services coverage expand more and more ubiquitous that people cannot live without.

Under the Chats tab of global navigation, there are couple of categories user can choose from, like Friends, Groups, Official Accounts and OpenChats. In each chatroom, user can reply particular old message, and also the system will automatically store photos & videos, links and files that are sent in the messages thread between you and your friends. In other words, LINE knows every information you provide in the chatrooms.

However, rather than figuring out what answer each LLM-based model could deliver, we should figure out what kind of questions we could ask first. Let’s take a look of what these conversational AI models are capable of before we start chatting or prompting (Take desktop version as example).

Dataset training amount and examples of questions to ask on each AI language model platform

Each platform has its own unique personality and relationship with people who use it. While we might be hesitate to type the first question to ask, or start thinking what we know we want to ask, on the landing page it shows you some examples you can start from, as well as connect what you might feel beneficial from the pairs of question and answer, to what those powerful platforms are capable of.

People usually search or browse, two essential behaviors people do for a whole bunch of contents and materials in App or online. As you can see, when you enter those LLMs platforms, apart from the basic menu and tips, it is nearly all empty on the screen, while the input box is waiting for your first interaction. But people might treat messaging App in different way, after all it is text messages of you and your friends and family that drag you over here to close the distance essentially. As being a messenger App that people use everyday, what would people might expect from LINE if it could be as intelligent and competitive as Bard or ChatGPT?

Take one of the real cases of mine to be a target of this new exploratory design. When I plan to kick off the house renovation, couple of homework need to be done and prepared in advance. Merely in the LINE App, a few chatroom groups have been created just for this matter, where each group represents exclusive relationships and goals between me and other people.

Messages and Official Accounts involved for the house rennovation planning

Chatrooms are like a dedicated channel that grabs people into one place and let them talk to each other instantly at anytime. People can ask specific questions, share files or links, or even can have a video call to catch up. In other words, whom you talk to or which topic you are involved decide what chatrooms will be created and associated with multi-media files in that single group. In my discussion and preparation of house make over plan, there are roughly 3 chatrooms with friends, 9 files, 30 photos and over 80 messages and couple of official accounts I join since this initiative has kicked off.

We definitely can keep searching and browsing the contents or certain piece of information arbitrarily or intentionally. However by making use of intelligent models released out there, what kind of new way of interaction could be ahead of these messages layers and gain a new life for people to get the information quickly and easily would be an interesting and a big question. Based on the examples of questions suggested in those LLMs brought up earlier, the questions that people might want to be asking in LINE for this house renovation plan could be:

  • Can you give me the summary and next to-do lists based on all my previous conversations with design contractor I contact with?

  • Can you retrieve any cost estimates or quotations provided by the contractors during the discussions?

  • Can you suggest relevant resources based on the content of my messages?

  • Draw me a new floor plan based on my favorite styles in the latest discussion.

  • Can you identify any potential conflicts or contradictions in the discussions that need clarification or resolution?

  • Can you rename the photos that I provide for reference and highlight the point I like, and send it to the contractor?

  • Can you categorize or organize the chat content based on different aspects of the renovation, such as kitchen, bathroom, flooring, etc.?

  • Are there any specific design recommendations or suggestions provided by the contractors that I may have missed or overlooked?

  • Can you provide me with a consolidated list of the proposed materials, products, or brands discussed during the conversations?

  • Are there any relevant permits, licenses, or legal requirements that were mentioned in the conversations, and can they be easily accessed?

The list could be growing on and on and continue to prompt more questions or instructions based on the topic of interest. Even though jumping out of these groups of chatrooms to allow the system to summarize, extract, create follow-on questions, make exploratory questions, or generate answers are also compelling ways to explore what the App could offer with its available corpus of data and information. Chances are that one day, we all have our own intelligent agents where they talk to each other, and stop by the authority that requires human beings confirmed or involved. Messages are not meant to induce more messages; they just get the job done.

If you look broader, you can always ask more additional questions that are not limited to particular chatrooms, people or information. You can compare the pros and cons of each contractors in regard to their proposals and cost, to fulfill your needs and schedule. Back to the design, let’s see how this new conceptual design might look like:

New conceptual design flow (Image source: https://www.muji.net/ie/kinoie/)

In the new design, the Chats tab has been re-shaped by topics, which represents more solid subjectivity of what context has been involved in one place. For example, user is able to drag several different chatrooms together, to make one subject more complete in terms of data corpus traning if it will. In each chatroom, user can always trigger chatbot AI to ask questions based on the messages in the chatroom. The beauty of this conceptual design relies on what you can get by throwing the questions or asking for further demands. There are always rooms for additional functions like, asking follow-up questions, tweaking the output, sharing the answer to the chatroom, or even giving me surprise!

LINE acts like its own ecosystem, where each chatroom live next to each other and is responsible for the messages or other media in it. LINE App got its own advantage of generating AI talking with you, so does other Apps. Therefore the next question might will be, what if thoses Apps come into ONE AI in my Pixel? Will it be more powerful? WIll the conversations be more thoughtful or full-packed? Will the boundary be broken? Those are what we are going to cover and explore deeper next in Part 4.

In case you miss it, here is the full index of my essays below:

Stay tuned.

Party with Super Mario Bros. on Switch via dialogue before jumping right into AI (Part 2/5)

In the Part 1, the table of “What if” ideas show how I think the Pixel products could be differently interacting with me among the scenarios I encounter in real life. Somehow in essence that my Pixel has been translated into my personal smart butler like Jarvis, where Pixel helps me handle all kinds of big-and-small things. Jarvis can talk, but since my Pixel has 6.4 inches in screen size, so currently we “talk” with each other through the graphical user interface by tapping on the screen. Until one day, I looked up and was facing toward my TV set, while playing the video game on Switch.

Text-based conversational and instructional interfaces are usually what Switch has been interacting with players. Switch does have sound effect and voice over throughout the game, but while been surrounded with laugher and playful sound in the living room, text on the screen with a pause can instantly catch people’s attention, like the Captain Toad saying down below, “Welcome, adventurers […].“

Welcome screen (Source: interfaceingame.com)

There are a whole lot of elements in a game, such as characters, level selection, menu, overlay, progress, and scoreboard, you name it. Dialogue is merely one of them. In order for players to conquer each one of the quests, those words in the dialogue reveals a facet of humanity, and guide the players to accomplish goals. I am not a gamer, and game-specific conversational experiences design probably are different from what we see on ChatGPT either, but we still can learn something unique and valuable out of it. Below are illustrated couple of observations that we can highlight in the game dialogue.

Text display orientation

Several styles of text alignment are shown below. In the Switch games, left or middle, or even right-aligned text format are all possible, where it depends on the culture and the language characteristics. But in the conversational user interface with avatar, character or profile photo, the text is usually closer and attached to the elements that it refers to or sources from. And since the response is generative letter by letter, which creates an atmosphere of real person on the other side that the player actually talks to.

Super Mario Party (Source: interfaceingame.com)

Text on the screen is usually the instruction that we all follow throughout the game. Therefore the timing, the context and the bounded screen size of that moment all convey critical messages of how well the experience would be felt delighted or disappointed. Needless to say that the people or hidden individual, whom we interact with behind the activity, always play the key role.

Queues placement and weight

People tend to unconsciously interpret the queues that they receive and construct reasonable meaning of the information displayed on the screen. In the game world, effectively building the sense of 2D or 3D environment could help players more immersively navigate and collaborate with each other. When a clear or vague goal in mind, making right or quick decision have significantly associated with the consistency by aligning the mindset with the action to be taken next. This would be even obvious when a variety of information formats thrown out at once in front of user.

In the screenshot of Mario Party Superstars down below, you can see that the four corners at top left and right, and the bottom left and right are showing the four individual players and their scores and rankings in order of the position where those characters are standing from the left to right. Each player can be able to quickly identify their own position as well as the place where they can quickly spot their eye focus on and toward the specific direction.

Mario Party Superstars Review - The Fault In Our Stars (Source: Gamespot)

And indeed the placement of the user interface elements usually depends on the game attributes and responsiveness. If it is a cooking or racing game, or the main stage is supposed to be at the center of the screen, the figures or additional information might be clustered and moved into left or right side of the edges accordingly (or thoughtfully provide accessible infographic style of information), so that players can quickly scan it and get the point they need. A few examples are pull out below to give you a brief idea of this concept, where you can see some examples of players’ avatar displayed differently on screen.

You can also watch the clip below and identify similar pieces of information (avatar photos, scores, rankings, assets, etc) are located in various places from one game to another.

Screenshots of Mario Party Superstars (Source: Mario Party Superstars Review)

Mario Party Superstars Review (Source: GameSpot Youtube)

text color and contrast

Texts itself communicate many kinds of messages to reader with subtle differentiated intentions on screen. Especially in the game world, where the visual details are all around from the dialogue conversation to the objects mixed with scattered distractions. One of the most highly immersive games, The Legend of Zelda: Breath of the Wild (BOTW), fully illustrates the environment of conversational experience and the landscapes the player engages with. I personally haven’t played this game yet on Switch, but from one of the game’s screenshots below, it can be easily recognized that the visual hierarchy of how the texts in the dialogue try to convey.

The color-coded texts and the contrast quickly catch player’s eyes and allow her to tell the differences between those highlighted objects and others. One of the interesting parts of this game’s dialogue is that the styles of the dialogue text varies from one to another in terms of how the contexts at that moment try to suggest. Take Kaneli for example. Kaneli is one of the charaters and the chief of Rito Village. When he talks, he talks in two manners: internal and external. He talks to himself internally, and also talks to other characters out loud, where the text transparency differs in two manners in these two dialogues. You can compare these two side by side below, and also can go to the article that the outstanding UX and product designer Carlos Morla well addresses this juicy point in the game design.

The Legend Of Zelda: Breath Of The Wild - Kaneli, Sha Warvo & Snowy Bird Search - Episode 92 (Source: Olizandri video on Youtube)

And of course, the game also includes several other kinds of dialogue designs that imply different contexts like instruction, narration, and explanation throughout the game, where it creates the emotions and feelings that naturally fuse the conversation into the story that lets you even unaware of all this magic.

Gaming is a world inherent with unique view and philosophy where it builds its own bricks and blocks, and the rule. In order to be caught up and live within so many impressive scenes and story, players would unavoidably bring the real physical world experience back to the virtual landscapes. We definitely could expect that the conversational interface, like what ChatGPT demonstrates to the world, be applied to the realms of digital or physical communication methods as one of the potential candidates of multi-channel interactive mechanisms.

In the next section we will go back to the Apps on my Pixel, and explore any possibilities or potentials when in applying conversational way of interaction in some of my frequently used Apps. It will be more like a discovery journey, a realistic case and scenario down to the ground. If you haven’t visit the first part, feel free to stop by in here. The full index is provided below:

Stay tuned.