Microsoft's Bing AI Chatbot Will Let Users Post Images, Ask Questions About Them

Microsoft Corp. is getting rid of the waitlist to try its new OpenAI-based Bing search and chat, and adding features like the ability to request and post images in an effort to sustain its renewed momentum in the market.

Users can search for an image, such as a crocheted teddy bear, for example, and ask Bing, “How do I make this?” The AI tool then suggests “how to crochet” websites and other kinds of craft projects to explore. The visual search feature will be available in the coming weeks to months, Yusuf Mehdi, a Microsoft vice president, said in an interview. It uses OpenAI’s GPT-4 system, which enables so-called multimodal features that blend text and images. While OpenAI announced that feature in March, it has offered few examples so far.

Other new Bing features are coming sooner, such as the ability to use third-party services like OpenTable and Wolfram Alpha through Bing chat and get answers that include videos and charts, Mehdi said. At a Manhattan demo, Bing representatives showed off some features in hands-on demonstrations, showing how a user can type in, say, “Find me a dinner reservation for two in New York City tonight,” and get a link to the reservation service OpenTable.

Microsoft upended the search market in February when it unveiled a complete overhaul of Bing, incorporating OpenAI’s ChatGPT, and offering the possibility of renewed competition in a market dominated for years by Alphabet Inc.’s Google. According to Mehdi, the new software has hosted half a billion chat sessions in the past 90 days. The move pushed Google to introduce its own Bard AI tools, efforts it largely kept within its own labs until Microsoft threatened the company’s highly profitable ad business.

“Chat really is the thing that is resonating with people and is in fact transforming search,” Mehdi said. About 70% of people trying out the chat features are using them for search-related tasks, and Microsoft finds those customers perform a lot more searches.

Microsoft Chief Executive Officer Satya Nadella said last month that Bing app installations have quadrupled since the launch of the AI-powered product, and that Bing increased its market share in the US market in the quarter, without offering specifics. The efforts are part of a larger company shift at Microsoft to leverage the company’s expanded investment in OpenAI and weave AI into all major products.

The new visual search features will pose an expanded challenge to the Microsoft Responsible AI team that has been trying to cope with content that is biased, disturbing or inappropriate as hundreds of millions of users try the new product. Microsoft says it has made progress but is still working on issues that also include misinformation and incorrect or incomplete answers. Mehdi said one of the reasons the visual features aren’t available yet is the team is working to ensure safety.

“There’s a lot of work to review technology and plans against,” Mehdi said. “Is it harmful? Is it transparent? Is it inclusive? Does it not have bias? We’re going to do the same thing on the images, and adding multimodal adds another degree of complexity.”

Microsoft unveiled the new features at a press event in New York where Bing’s latest capabilities were demonstrated via a juice bar where the chatbot generated drink recipes and customized souvenir cards for attendees.

Bing will also get an enhanced ability to summarize long documents. And the company is improving the chat history function — Microsoft plans for users’ search histories to be easily accessible soon in a sidebar running along its Edge browser. Users will be able to go back to previous conversations and start where they left off.