Analysis at Microsoft 2023: A yr of groundbreaking AI advances and discoveries


It isn’t typically that researchers on the reducing fringe of know-how see one thing that blows their minds. However that’s precisely what occurred in 2023, when AI specialists started interacting with GPT-4, a big language mannequin (LLM) created by researchers at OpenAI that was skilled at unprecedented scale. 

“I noticed some mind-blowing capabilities that I assumed I wouldn’t see for a few years,” stated Ece Kamar, associate analysis supervisor at Microsoft, throughout a podcast recorded in April.

All year long, speedy advances in AI got here to dominate the public dialog (opens in new tab), as know-how leaders and ultimately most of the people voiced a mixture of surprise and skepticism after experimenting with GPT-4 and associated functions. May we be seeing sparks of synthetic normal intelligence (opens in new tab)—informally outlined as AI methods that “show broad capabilities of intelligence, together with reasoning, planning, and the flexibility to be taught from expertise (opens in new tab)”? 

Whereas the reply to that query isn’t but clear, we have now definitely entered the period of AI, and it’s bringing profound adjustments to the best way we work and dwell. In 2023, AI emerged from the lab and delivered on a regular basis improvements that anybody can use. Tens of millions of individuals now interact with AI-based providers like ChatGPT. Copilots (opens in new tab)—AI that helps with complicated duties starting from search to safety—are being woven into enterprise software program and providers.

Underpinning all of this innovation is years of analysis, together with the work of a whole bunch of world-class researchers at Microsoft, aided by scientists, engineers, and specialists throughout many associated fields. In 2023, AI’s transition from analysis to actuality started to speed up, creating extra tangible outcomes than ever earlier than. This put up appears again on the progress of the previous yr, highlighting a sampling of the analysis and techniques that may assist even higher progress in 2024.

Strengthening the foundations of AI

AI with optimistic societal impression is the sum of a number of integral transferring components, together with the AI fashions, the appliance of those fashions, and the infrastructure and requirements supporting their growth and the event of the bigger methods they underpin. Microsoft is redefining the cutting-edge throughout these areas with enhancements to mannequin effectivity, efficiency, and functionality; the introduction of latest frameworks and prompting methods that enhance the usability of fashions; and greatest practices that contribute to sustainable and accountable AI. 

Advancing fashions 

  • Researchers launched Retentive Networks (RetNet), a substitute for the dominant transformer structure in language modeling. RetNet helps coaching parallelism and powerful efficiency whereas making vital positive aspects in inference effectivity. 
  • To contribute to extra computationally environment friendly and sustainable language fashions, researchers offered a 1-bit transformer structure referred to as BitNet.
  • Microsoft expanded its Phi household of small language fashions with the 2.7 billion-parameter Phi-2, which raises the bar in reasoning and language understanding amongst base fashions with as much as 13 billion parameters. Phi-2 additionally met or exceeded the efficiency of fashions 25 occasions its measurement on complicated benchmarks.
  • The discharge of the language fashions Orca (13 billion parameters) and, a number of months later, Orca 2 (7 billion and 13 billion parameters) demonstrates how improved coaching strategies, similar to artificial knowledge creation, can elevate small mannequin reasoning to a stage on par with bigger fashions.
  • For AI experiences that extra carefully replicate how individuals create throughout mediums, Composable Diffusion (CoDi) takes as enter a mixture of modalities, similar to textual content, audio, and picture, and produces multimodal output, similar to video with synchronized audio.
  • To higher mannequin human reasoning and pace up response time, the brand new method Skeleton-of-Thought has LLMs break duties down into two components—creating a top level view of a response and offering particulars on every level in parallel.

Advancing strategies for mannequin utilization

  • AutoGen is an open-source framework for simplifying the orchestration, optimization, and automation of LLM workflows to allow and streamline the creation of LLM-based functions.
  • Medprompt, a composition of prompting methods, demonstrates that with considerate and superior prompting alone, normal basis fashions can outperform specialised fashions, providing a extra environment friendly and accessible various to fine-tuning on expert-curated knowledge.
  • The useful resource assortment promptbase provides prompting methods and instruments designed to assist optimize basis mannequin efficiency, together with Medprompt, which has been prolonged for utility exterior of medication.
  • Aimed toward addressing points related to prolonged inputs, similar to elevated response latency, LLMLingua is a prompt-compression methodology that leverages small language fashions to take away pointless tokens.

Growing and sharing greatest practices 

Accelerating scientific exploration and discovery

Microsoft makes use of AI and different superior applied sciences to speed up and rework scientific discovery, empowering researchers worldwide with modern instruments. Throughout international Microsoft analysis labs, specialists in machine studying, quantum physics, molecular biology, and plenty of different disciplines are tackling urgent challenges within the pure and life sciences.

  • Due to the complexities arising from a number of variables and the inherently chaotic nature of climate, Microsoft is utilizing machine studying to reinforce the accuracy of subseasonal forecasts.
  • Distributional Graphormer (DIG) is a deep studying framework for predicting protein buildings with higher accuracy, a elementary downside in molecular science. This advance might assist ship breakthroughs in important analysis areas like supplies science and drug discovery.
  • Leveraging evolutionary-scale protein knowledge, the general-purpose diffusion framework EvoDiff helps design novel proteins extra effectively, which may help within the growth of business enzymes, together with for therapeutics.
  • MOFDiff, a coarse-grained diffusion mannequin, helps scientists refine the design of latest metal-organic frameworks (MOFs) for the low-cost removing of carbon dioxide from air and different dilute fuel streams. This innovation might play a significant position in slowing local weather change.
  • This episode of the Microsoft Analysis Podcast sequence Collaborators explores analysis into renewable power storage methods, particularly stream batteries, and discusses how machine studying may help to determine compounds superb for storing waterpower and advancing carbon seize.
  • MatterGen is a diffusion mannequin particularly designed to deal with the central problem in supplies science by effectively producing novel, secure supplies with desired properties, similar to excessive conductivity for lithium-ion batteries.
  • Deep studying is poised to revolutionize the pure sciences, enhancing modeling and prediction of pure occurrences, ushering in a brand new period of scientific exploration, and resulting in vital advances in sectors starting from drug growth to renewable power. DeepSpeed4Science, a brand new Microsoft initiative, goals to construct distinctive capabilities via AI system know-how improvements to assist area specialists unlock immediately’s largest science mysteries. 
  • Christopher Bishop, Microsoft technical fellow and director of the AI4Science staff, lately printed Deep Studying: Foundations and Ideas, a guide that “provides a complete introduction to the concepts that underpin deep studying.” Bishop mentioned the motivation and course of behind the guide, in addition to deep studying’s impression on the pure sciences, within the AI Frontiers podcast sequence.

Maximizing the person and societal advantages of AI

As AI fashions develop in functionality so, too, do alternatives to empower individuals to attain extra, as demonstrated by Microsoft work in such domains as well being and training this yr. The corporate’s dedication to optimistic human impression requires that AI know-how be equitable and accessible.

Past AI: Main know-how innovation

Whereas AI rightly garners a lot consideration within the present analysis panorama, researchers at Microsoft are nonetheless making loads of progress throughout a spectrum of technical focus areas.

  • Challenge Silica, a cloud-based storage system underpinned by quartz glass, is designed to offer sustainable and sturdy archival storage that’s theoretically able to lasting hundreds of years.
  • Challenge Analog Iterative Machine (AIM) goals to resolve tough optimization issues—essential throughout industries similar to finance, logistics, transportation, power, healthcare, and manufacturing—in a well timed, energy-efficient, and cost-effective method. Its designers consider Challenge AIM might outperform even essentially the most highly effective digital computer systems.
  • Microsoft researchers proved that 3D telemedicine (3DTM), utilizing HoloportationTM communication know-how, might assist enhance healthcare supply, even throughout continents, in a novel collaboration with docs and governments in Scotland and Ghana.
  • In one other collaboration that goals to assist enhance precision drugs, Microsoft labored with trade and educational colleagues to launch Terra, a safe, centralized, cloud-based platform for biomedical analysis on Microsoft Azure.
  • On the {hardware} entrance, Microsoft researchers are exploring sensor-enhanced headphones, outfitting them with controls that use head orientation and hand gestures to allow context-aware privateness, gestural audio-visual management, and animated avatars derived from pure physique language.

Collaborating throughout academia, industries, and disciplines

Cross-company and cross-disciplinary collaboration has all the time performed an necessary position in analysis and much more in order AI continues to quickly advance. Massive fashions driving the progress are parts of bigger methods that may ship the worth of AI to individuals. Growing these methods and the frameworks for figuring out their roles in individuals’s lives and society requires the information and expertise of those that perceive the context by which they’ll function—area specialists, lecturers, the people utilizing these methods, and others.

Partaking and supporting the bigger analysis group

All year long, Microsoft continued to interact with the broader analysis group on AI and past. The corporate’s sponsorship of and participation in key conferences not solely showcased its dedication to the appliance of AI in various technological domains but additionally underscored its unwavering assist for cutting-edge developments and collaborative group involvement.

Useful programming

  • Microsoft was a proud sponsor of ICFP 2023, with analysis contributions protecting a spread of practical programming subjects, together with reminiscence optimization, language design, and software-development methods.

Human-computer interplay

  • At CHI 2023, Microsoft researchers and their collaborators demonstrated the myriad and various methods individuals use computing immediately and can sooner or later. 

Massive language fashions and ML

  • Microsoft was a sponsor of ACL 2023, showcasing papers starting from equity in language fashions to pure language technology and past.
  • Microsoft additionally sponsored NeurIPS 2023, publishing over 100 papers and conducting workshops on language fashions, deep studying methods, and extra ideas, strategies, and functions addressing urgent points within the area.
  • With its sponsorship of and contribution to ICML 2023, Microsoft showcased its funding in advancing the sphere of machine studying.
  • Microsoft sponsored ML4H (opens in new tab) and took part in AfriCHI (opens in new tab) and EMNLP (opens in new tab), a number one convention in pure language processing and AI, highlighting its dedication to exploring how LLMs may be utilized to healthcare and different very important domains.

Programs and superior networking

Listeners’ selection: Notable podcasts for 2023

Thanks for studying

Microsoft achieved extraordinary milestones in 2023 and can proceed pushing the boundaries of innovation to assist form a future the place know-how serves humanity in outstanding methods. To remain abreast of the newest updates, subscribe to the Microsoft Analysis Publication (opens in new tab) and the Microsoft Analysis Podcast (opens in new tab). You can even comply with us on Fb (opens in new tab), Instagram (opens in new tab), LinkedIn (opens in new tab), X (opens in new tab), and YouTube (opens in new tab). 

Writers, Editors, and Producers
Kristina Dodge
Kate Forster
Jessica Gartner
Alyssa Hughes
Gretchen Huizinga
Brenda Potts
Chris Stetkiewicz
Larry West

Managing Editor
Amber Tingle

Challenge Supervisor
Amanda Melfi

Microsoft Analysis International Design Lead
Neeltje Berger

Graphic Designers
Adam Blythe
Harley Weber

Microsoft Analysis Artistic Studio Lead
Matt Corwine



Related Articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest Articles