ديجيتال فابلت

الوسم: Meta AI

  • Meta Faces Criticism for Llama 4 Benchmark Manipulation Again

    Meta Faces Criticism for Llama 4 Benchmark Manipulation Again

    Meta has recently introduced its Llama 4 series of AI models, making waves by outperforming GPT-4o and Gemini 2.0 Pro in the Chatbot Arena (previously known as LMSYS). The company boasts that its Llama 4 Maverick model, which uses a mixture of experts (MoE) to activate just 17 billion parameters out of a total of 400 billion across 128 experts, scored an impressive ELO of 1,417 on the Chatbot Arena benchmark.

    This achievement caught the attention of the AI community, especially since the smaller MoE model surpassed significantly larger language models like GPT-4.5 and Grok 3. The unexpected results prompted many AI enthusiasts to independently evaluate the model. However, it turned out that Llama 4 Maverick’s real-world performance did not live up to Meta’s benchmark claims, particularly in coding tasks.

    On 1Point3Acres, a widely used forum for North American Chinese users, a post from someone claiming to be a former Meta employee stirred controversy. This post, which has since been translated into English on Reddit, alleged that Meta’s leadership may have combined various test sets during the post-training phase to artificially boost the benchmark scores and fulfill internal objectives.

    The departing employee expressed their disapproval of this practice and subsequently resigned. They also requested that their name be removed from the Llama 4 technical report. Furthermore, the employee asserted that the recent resignation of Meta’s AI research head, Joelle Pineau, is closely tied to the alleged manipulations surrounding the Llama 4 benchmarks.

    In light of these allegations, Ahmad Al-Dahle, who leads Meta’s Generative AI division, responded with a post on X, strongly refuting claims that Llama 4 had been post-trained on test sets. Al-Dahle stated:

    We’ve also heard claims that we trained on test sets — that’s simply not true, and we would never do that. Our best understanding is that the variable quality people are seeing results from is due to the need for stabilizing implementations.

    He acknowledged the varying performance of Llama 4 across different platforms, urging the AI community to allow a few days for the implementation to stabilize.

    LMSYS Addresses Allegations of Llama 4 Benchmark Manipulation

    In response to the rising concerns from the AI community, LMSYS — the organization behind the Chatbot Arena leaderboard — released a statement aimed at enhancing transparency. LMSYS clarified that the model submitted to Chatbot Arena was “Llama-4-Maverick-03-26-Experimental,” a custom variant tailored for human preferences.

    LMSYS admitted that “style and model response tone were significant factors” which may have inadvertently benefited the custom Llama 4 Maverick model. They acknowledged that this crucial information was not communicated clearly by Meta. Additionally, LMSYS noted that “Meta’s interpretation of our policy did not align with our expectations from model providers.”

    To be fair, Meta did mention in its official Llama 4 blog that an “experimental chat version” achieved an ELO of 1,417 on Chatbot Arena, but they did not provide additional details.

    To promote transparency further, LMSYS included the Hugging Face version of Llama 4 Maverick in Chatbot Arena and has also released more than 2,000 comparative battle results for public scrutiny. These results cover prompts, model responses, and user preferences.

    Upon reviewing the battle results, it was surprising to find that users frequently preferred Llama 4’s answers, which often contained inaccuracies and verbosity. This raises significant questions about the reliability of community-driven benchmarks like Chatbot Arena.

    This is not the first instance of Meta facing accusations of manipulating benchmarks through data contamination, which involves mixing benchmark datasets into the training corpus. Earlier this year, Susan Zhang — a former Meta AI researcher who now works at Google DeepMind — revealed a study in reaction to a post by Yann LeCun, Meta AI’s chief scientist.

    The study indicated that over 50% of test samples from major benchmarks were included in Meta’s Llama 1 pretraining data. The paper reported significant contamination in key benchmarks like Big Bench Hard, HumanEval, HellaSwag, MMLU, PiQA, and TriviaQA.

    Now, with the new allegations surrounding Llama 4’s benchmarks, Zhang sharply remarked that Meta should at least credit their “previous work” from Llama 1 for this “unique approach.” Her comment suggests that such benchmark manipulation is not incidental but rather a deliberate strategy by Meta aimed at artificially inflating performance metrics.

  • Meta AI Glasses Leak Tips One-Eyed Screen Android Soul High Ask

    Meta AI Glasses Leak Tips One-Eyed Screen Android Soul High Ask

    Meta has achieved remarkable success with its Stories smart glasses developed in partnership with Ray-Ban. This innovative wearable, equipped with embedded cameras for capturing social media content and featuring an integrated AI assistant, has gained enough traction for Meta to even offer high-end fashion versions aimed at a luxury market.

    However, one thing that has been notably absent is an interactive display. Moving forward, Meta seems set on adding a screen to these stylish smart glasses, enhancing their usability significantly. Unfortunately, this added functionality is expected to come at a high cost. According to Bloomberg, potential buyers should brace themselves for a price tag starting at $1,000, potentially reaching $1,300 to $1,400.

    “Employees at Meta estimate that the device, currently code-named Hypernova, will retail for over $1,000,” the report states. The smart glasses are anticipated to launch by the end of 2025.

    Front view of Meta Aria Gen 2 smart glasses.
    Meta Aria Gen 2 smart glasses. Meta

    For context, most VR-ready smart glasses on the market typically retail around the $500 mark. Popular models include the Viture Pro, Xreal One, and Rokid Max, all of which provide high-resolution displays along with impressive spatial computing capabilities and appealing designs.

    A stylish pirate eye-patch?

    The current trend among VR smart glasses involves the inclusion of a display for each eye, usually located behind the lenses. Some models even feature adjustable focus for those who wear prescription glasses. However, Meta’s forthcoming smart glasses might adopt a more minimalistic design by only offering a display for the right eye.

    According to Bloomberg, this approach will result in a monocular design, meaning the display will be placed in the lower right corner, allowing for clear vision through the left lens.

    Disassembled view of the Meta Gen 2 Aria smart glasses.
    Meta Aria Gen 2 smart glasses. Meta

    “This means that information will only be visible in the right eye, with optimal clarity when the wearer glances downward,” the report explains. This design choice allows users to maintain visibility of their surroundings through the left lens.

    The Hypernova smart glasses will enable users to capture photos, watch videos, navigate maps, and receive notifications from apps like WhatsApp. The user interface will respond to gestures along the stems, incorporating swipes and taps for control.

    Notably, Meta plans to include a wristband that allows users to interact with the interface using hand gestures, such as pinch-to-zoom and wrist rotations. This accessory is currently being developed under the codename “Ceres” and is expected to be part of the final product.

    Powered by Qualcomm technology, these glasses are designed to run a specially adapted version of Android. Interestingly, Samsung is also planning to introduce Android-based smart glasses around the same time in 2025. Meanwhile, Meta continues to innovate, working on its ambitious Orion holographic glasses alongside the research-focused Aria Gen 2 AR glasses.

  • WhatsApp Unveils Meta AI Widget for Easy Home Screen Access

    WhatsApp Unveils Meta AI Widget for Easy Home Screen Access

    WhatsApp is currently developing a new feature that will enable users to access Meta AI directly from their home screens, eliminating the need to open the app each time.

    This feature was uncovered in the most recent beta version of WhatsApp (2.25.1.27) for Android.

    The forthcoming Meta AI widget aims to enhance user convenience and streamline access. Once it becomes available, users will be able to start chats with Meta AI straight from their home screens, which means they won’t have to launch WhatsApp and navigate through it to engage in a conversation.

    One of the standout aspects of this new widget is its ability to facilitate quick photo capturing and sharing with Meta AI. Users will have the opportunity to request the AI’s help with a variety of tasks, including photo editing, content analysis, or answering questions related to the images.

    Photo Credit: wabetainfo

    However, this convenient widget will only be accessible to those who already have AI-driven chat capabilities. Presently, this functionality is restricted to specific countries and selected users, meaning that not everyone will be able to use the widget upon its release.

    The widget will appear alongside other WhatsApp widgets, and users will be given the option to add it to their home screens. WhatsApp’s goal is to facilitate smoother interactions with Meta AI, enabling users to quickly start tasks like photo editing or content exploration without needing to open the app each time.

    In addition, WhatsApp is also working on a feature that will allow users to create and personalize their own AI chatbots right within the app. This new tool will give users the ability to define the personality and objectives of their AI companions, customizing their behavior to meet specific needs, such as providing motivation or assistance in organizing tasks.

    The Meta AI widget is still in development and is expected to launch in the coming weeks. While WhatsApp has not provided an exact timeline for its release, the feature is anticipated to enhance the overall user experience by making AI tools more accessible and responsive.