A Few Factual Errors Were Made in Microsoft's Bing AI Launch Demonstration Last Week

More than 1 million individuals signed up to try Microsoft's tool in the first 48 hours during last week's chatbot hoopla, the company reported, as Microsoft and Google competed to showcase the earliest iterations of AI-powered search.

‍

Microsoft CEO Satya Nadella stated to Trade Algo that the technology was "maybe the industrial revolution brought to knowledge work" since it can spit out full responses that appear to have been authored by a human.

‍

The AI, though, leaves a lot to be desired in terms of precision.

‍

In a demonstration for journalists, Microsoft's Bing search engine's ChatGPT-like technology examined Gap and Lululemon's financial reports. The chatbot underreported some figures when comparing its responses to the actual reports. Others seem to be fabrications.

‍

"During their demo, Bing AI provided some utterly incorrect replies. But nobody paid attention," independent search researcher Dmitri Brereton stated in a Monday Substack article. As an alternative, "everyone boarded the Bing hype train."

‍

Brereton noted potential factual problems in the Microsoft demo, in addition to the financial mistakes, in its comments regarding the vacuum cleaner specifications and travel arrangements to Mexico. When he looked more closely to write a comparison of the AI announcements from Google and Microsoft, he said he wasn't originally looking for faults.

‍

Experts in AI refer to this tendency of tools built using big language models to just make things up as "hallucinations," or phenomena. Last week, Google released a rival AI program that also contained factual inaccuracies, though viewers promptly pointed them out.

‍

Following the explosion of ChatGPT, which OpenAI released to the general public in November, both businesses are scrambling to add new varieties of generative AI into search engines and are eager to demonstrate their accomplishments. Microsoft has contributed billions to OpenAI, while rival businesses like Stability AI and Hugging Face have also had their valuations soar to the billion-dollar mark in private fundraising rounds.

‍

While Google has been hesitant to integrate AI-generated answers into search engines due to reputational risk and safety concerns, Microsoft highlighted the possibilities of making the technology available to a portion of the public in the near future in its statement last week.

‍

In my opinion, it's crucial to avoid being in a lab, Nadella remarked. "You need to safely remove these things."

‍

There were some issues when it came time to demonstrate Bing AI's response to a question on company earnings.

‍

When visiting Gap's investor relations website in November, Microsoft marketing executive Yusuf Mehdi asked Bing AI to highlight the "important takeaways" from the retailer's release of its third-quarter earnings.

‍

Really cool. We've saved a ton of time," Mehdi said.

‍

The following errors were found in the summary:

‍

• Gross margin as stated by Gap was 37.4%. The adjusted gross margin, however, was 38.7% after Yeezy-related expenses were taken out.

• The gap operating margin, which differs from the company-reported 5.9%, was actually 4.6%.

• Instead of $0.42 as stated in the report, adjusted diluted earnings per share were $0.71. The amount Gap disclosed includes a $0.33 adjusted income tax advantage.

• There is no prediction for operating margin or EPS. In the third-quarter report, Gap stated that "net sales could be down mid-single digits year-over-year in the fourth quarter," which would indicate a decrease in revenue for the entire year as opposed to an "increase in the low double digits."

• Microsoft claimed to be aware of the problems and to anticipate mistakes from the Bing AI.

‍

A Microsoft representative told Trade Algo, "We're aware of this report and have evaluated its findings in our attempts to improve this experience. The feedback is essential so that we can learn and aid in the improvement of the models because we acknowledge that there is still work to be done and that the system may make mistakes during this preview period.

‍

Microsoft then requested that Bing AI compare the earnings reports of Gap and Lululemon. Mehdi requested that Bing create a table with the data from the two reports.

‍

He exclaimed, "Look how amazing this is. "I can acquire an answer to this query in a similar manner at a single table. Consider how long that would have taken in the alternative.

‍

The table contains a number of mistakes, beginning with margins.

‍

• Instead of 58.7%, Lululemon's gross margin was 55.9%.
• Instead of 20.7%, the company's operating margin was 19%.
• In addition to adjusted EPS of $1.62, Lululemon announced diluted EPS of $2. Bing reported diluted earnings per share of $1.65.
• Instead of $1.4 billion, Gap had $679 million in cash and cash equivalents.
• Instead of $1.9 billion, Gap had $3.04 billion in inventory.