Home Big Data Meta quietly releases Llama 2 Lengthy AI mannequin

Meta quietly releases Llama 2 Lengthy AI mannequin

0
Meta quietly releases Llama 2 Lengthy AI mannequin

[ad_1]

VentureBeat presents: AI Unleashed – An unique government occasion for enterprise knowledge leaders. Community and be taught with business friends. Be taught Extra


Meta Platforms confirmed off a bevy of latest AI options for its consumer-facing companies Fb, Instagram and WhatsApp at its annual Meta Join convention in Menlo Park, California, this week.

However the largest information from Mark Zuckerberg’s firm could have really come within the type of a pc science paper revealed with out fanfare by Meta researchers on the open entry and non-peer reviewed web site arXiv.org.

The paper introduces Llama 2 Lengthy, a brand new AI mannequin based mostly on Meta’s open supply Llama 2 launched in the summertime, however that has undergone “continuous pretraining from Llama 2 with longer coaching sequences and on a dataset the place lengthy texts are upsampled,” in keeping with the researcher-authors of the paper.

Because of this, Meta’s newly elongated AI mannequin outperforms a number of the main competitors in producing responses to lengthy (greater character depend) person prompts, together with OpenAI’s GPT-3.5 Turbo with 16,000-character context window, in addition to Claude 2 with its 100,000-character context window.

Occasion

AI Unleashed

An unique invite-only night of insights and networking, designed for senior enterprise executives overseeing knowledge stacks and methods.

 


Be taught Extra

How LLama 2 Lengthy got here to be

Meta researchers took the unique Llama 2 obtainable in its completely different coaching parameter sizes — the values of knowledge and data the algorithm can change by itself because it learns, which within the case of Llama 2 are available 7 billion, 13 billion, 34 billion, and 70 billion variants — and included extra longer textual content knowledge sources than the unique Llama 2 coaching dataset. One other 400 billion tokens-worth, to be actual.

Then, the researchers stored the unique Llama 2’s structure the identical, and solely made a “crucial modification to the positional encoding that’s essential for the mannequin to attend longer.”

That modification was to the Rotary Positional Embedding (RoPE) encoding, a way of programming the transformer mannequin underlying LLMs akin to Llama 2 (and LLama 2 Lengthy), which primarily maps their token embeddings (the numbers used to signify phrases, ideas, and concepts) onto a 3D graph that exhibits their positions relative to different tokens, even when rotated. This permits a mannequin to supply correct and useful responses, with much less info (and thus, much less computing storage taken up) than different approaches.

The Meta researchers “decreased the rotation angle” of its RoPE encoding from Llama 2 to Llama 2 Lengthy, which enabled them to make sure extra “distant tokens,” these occurring extra not often or with fewer different relationships to different items of knowledge, had been nonetheless included within the mannequin’s information base.

Utilizing reinforcement studying from human suggestions (RLHF), a standard AI mannequin coaching methodology the place AI is rewarded for proper solutions with human oversight to verify it, and artificial knowledge generated by Llama 2 chat itself, the researchers had been in a position to enhance its efficiency in widespread LLM duties together with coding, math, language understanding, widespread sense reasoning, and answering a human person’s prompted questions.

Little marvel the open supply AI neighborhood

With such spectacular outcomes relative to each Llama 2 common and Anthropic’s Claude 2 and OpenAI’s GPT-3.5 Turbo, it’s little marvel the open-source AI neighborhood on Reddit and Twitter and Hacker Information have been expressing their admiration and pleasure about Llama 2 for the reason that paper’s launch earlier this week — it’s a giant validation of Meta’s “open supply” strategy towards generative AI, and signifies that open supply can compete with the closed supply, “pay to play” fashions provided by well-funded startups.

VentureBeat’s mission is to be a digital city sq. for technical decision-makers to realize information about transformative enterprise expertise and transact. Uncover our Briefings.



[ad_2]