Gist: Exploring the quiet shift from cloud-dependent AI towards locally-run language models and what this means for education, privacy, and technological agency
There's a particular satisfaction in running something locally that was once the exclusive domain of distant servers and corporate infrastructure. As I write this, a 7-billion parameter language model sits quietly on my laptop, ready to assist with text generation, code completion, or complex reasoning tasks—all without sending a single byte of data across the internet.
This isn't merely a technical curiosity. The emergence of sophisticated offline large language models (LLMs) represents what I would argue is one of the most significant shifts in how we conceptualise and interact with artificial intelligence. It's a movement that intersects directly with questions of digital sovereignty, educational equity, and the democratisation of advanced computational tools.
From Centralisation to Distribution: A Paradigm Shift
The dominant narrative around AI has been one of centralisation. We've become accustomed to AI as a service—something that happens ‘elsewhere,’ mediated by APIs, subject to usage limits, and fundamentally controlled by others. This model has served us well in many respects, enabling rapid deployment and continuous improvement of AI capabilities.
Yet this centralisation comes with considerable trade-offs. Unlike proprietary models developed by companies like OpenAI and Google, open source LLMs are licensed to be freely used, modified, and distributed by anyone. They offer transparency and flexibility, which can be potentially useful for research, development, and customisation in various applications.
The implications extend beyond mere technical considerations. When we examine the learning environments we're creating, particularly in educational contexts, the question of who controls the tools becomes paramount. Users have full control over the data processed by these models, eliminating concerns of third-party access or data mishandling. Organisations can deploy open source LLMs on their private infrastructure, ensuring sensitive information remains in-house and complies with data protection requirements.
For educators working with personal identifiable information (PII) such as student data, researchers handling sensitive information, or organisations operating under strict data governance requirements, this represents a fundamental shift in what's possible.
Open Source V Closed Open Source
A little explainer about Open source and closed source represent two fundamentally different approaches to software development and distribution, each with distinct implications for how technology is created, shared, and controlled.
Open Source Software
Open source software is characterised by the availability of its source code—the underlying instructions that make the program function. This code is freely accessible to anyone who wishes to view, modify, or distribute it. The open source movement is governed by various licences, such as the GNU General Public License (GPL) or the MIT License, which establish the terms under which the software can be used and shared.
The collaborative nature of open source development means that programmers worldwide can contribute improvements, fix bugs, and add new features. This distributed approach often leads to rapid innovation and robust security, as many eyes examining the code can identify vulnerabilities more quickly than a closed development team might. Notable examples include the Linux operating system, the Firefox web browser, and the Apache web server software.
Closed Source Software
Closed source software, also referred to as proprietary software, keeps its source code private and protected. Only the original developers or the company that owns the software can access, modify, or distribute the underlying code. Users receive only the compiled version of the program and must accept the software ‘as is,’ without the ability to inspect or alter its functionality.
This model allows companies to maintain complete control over their intellectual property and business model. They can charge licensing fees, control how the software is used, and protect trade secrets embedded in the code. Microsoft Windows, Adobe Photoshop, and most commercial video games exemplify this approach.
Key Distinctions
The philosophical divide between these models extends beyond mere technical considerations. Open source advocates argue that freely available code promotes innovation, transparency, and democratic access to technology. Users can verify that software behaves as claimed and isn't collecting data inappropriately or containing hidden vulnerabilities.
Conversely, closed source proponents contend that proprietary development enables companies to invest heavily in research and development, knowing they can recoup costs through sales. This model can lead to polished, user-friendly products with dedicated support services.
From a practical standpoint, open source software is typically free to obtain and use, though organisations may pay for professional support services. Closed source software usually requires purchasing licences, but often comes with guarantees of support and regular updates.
Contemporary Relevance
Today's technology landscape increasingly features hybrid models. Many companies release some components as open source whilst keeping core technologies proprietary. Cloud computing has further complicated these distinctions, as users may never see the underlying software at all, regardless of whether it's open or closed source.
The debate between these models remains relevant as societies grapple with questions of digital sovereignty, privacy, and the concentration of technological power. Understanding these distinctions helps users make informed choices about the tools they use and the digital ecosystems they support.
The Tooling Renaissance: Making the Complex Accessible
What strikes me most about the current moment is how the barriers to entry are collapsing. Tools like LM Studio have changed the deployment of sophisticated language models from a specialised technical endeavour into something approaching consumer software, but not quite. LM Studio is a desktop app for developing and experimenting with LLMs locally on your computer, abstracting away the complexity whilst maintaining the full capabilities of state-of-the-art models. LM Studio is closed open-source
The approach taken by AnythingLLM is particularly compelling from an educational technology perspective. The all-in-one AI application for everyone. Chat with docs, use AI Agents, and more - full locally and offline. What distinguishes AnythingLLM is its commitment to privacy by design: AnythingLLM comes with sensible and locally running defaults for your LLM, embedder, vector database, and storage and agents. Nothing is shared unless you allow it. AnythingLLM is open source, free, & MIT licensed.
For those seeking maximum flexibility, LocalAI offers perhaps the most comprehensive approach. LocalAI is a free, Open Source OpenAI alternative. Run LLMs, generate images, audio and more locally with consumer grade hardware, whilst maintaining API compatibility that allows existing applications to work seamlessly. The platform's modular ecosystem extends beyond text generation to encompass multimodal capabilities–image generation, audio processing, and autonomous agents–all running entirely on local infrastructure. LocalAI is pen source, free, & MIT licensed.
The Model Landscape: Open Source Excellence
The quality and diversity of available open source models has reached an interesting inflection point. Meta's LLaMA 3 family represents perhaps the most significant development in democratising access to capable language models. Available in two sizes: 8 billion (8B) and 70 billion (70B) parameters, these models are optimised for conversational interactions whilst maintaining efficiency through architectural innovations like Grouped-Query Attention. Grouped-query attention (GQA) is a technique used in transformer models to improve the efficiency of the attention mechanism, particularly for large language models (LLMs).
The context handling capabilities are particularly noteworthy for educational applications. Version 3.2 upgraded this to 128K tokens, enabling extended document analysis, sustained conversations, and complex reasoning tasks that would have been impossible with earlier generations of locally-run models.
Google's Gemma 2 demonstrates how major technology companies are embracing open source distribution. Available in 9B and 27B parameters, providing options for various computational needs and performance requirements, with efficiency gains that are genuinely impressive: the 27B model delivers performance similar to models more than twice its size.
Perhaps most fascinating from an architectural perspective is Mistral-8x22b, which showcases the potential of Mixture-of-Experts models. A sparse Mixture-of-Experts (SMoE) model that leverages 39 billion active parameters out of a total 141 billion, it demonstrates how architectural innovation can deliver enterprise-grade capabilities on consumer hardware. Mixture of experts (MoE) is a machine learning approach that divides an artificial intelligence (AI) model into separate sub-networks (or ‘experts’), each specialising in a subset of the input data, to jointly perform a task.
Educational Implications: Rethinking AI in Learning
From a pedagogical standpoint, the shift towards offline LLMs opens fascinating possibilities. The ability to create contained, customisable learning environments–free from external dependencies and data privacy concerns–fundamentally alters what we can achieve in educational technology.
Consider the implications for smaller educational institutions or researchers in regions with limited internet infrastructure. A local deployment of sophisticated language models can provide capabilities that were previously available only to institutions with substantial cloud computing budgets or reliable high-speed internet connections.
There's also the question of digital literacy and understanding. When students can examine, modify, and experiment with AI systems locally, it transforms AI from something mysterious and external into something they can understand and control. This transparency becomes particularly crucial as we consider how to prepare learners for a world where Digital, including AI, literacy is increasingly essential.
The Complexity of Choice: Trade-offs and Considerations
Yet this shift towards local deployment isn't without its complexities. The hardware requirements, whilst more accessible than ever, still represent a significant consideration. Modern language models benefit enormously from substantial RAM and dedicated graphics processing units, creating potential equity issues between those who can afford high-specification hardware and those who cannot.
There's also the question of model currency and improvement. Cloud-based services continuously refine their models, learning from vast interaction datasets and incorporating new capabilities. Offline models represent a snapshot in time—powerful, but static until manually updated.
The maintenance burden shifts fundamentally. Whilst tools like LM Studio and LocalAI abstract much of the technical complexity, users must still manage storage requirements (models can consume tens of gigabytes), updates, and performance optimisation. This represents both an opportunity for deeper technical understanding and a potential barrier to adoption.
Understanding the Costs
Costs come can be presented under three distinct headings: Environment, Financial, and Data Quality.
Environmental Impact
Large language models impose significant environmental costs through their substantial energy requirements. The computational power needed to run these systems, particularly larger models, results in considerable energy consumption and an associated carbon footprint. Additionally, the powerful hardware generates substantial heat, necessitating robust cooling systems that further increase overall energy consumption.
Financial Considerations
The financial implications of implementing local LLMs are multifaceted. Initial setup requires significant capital investment in high-performance hardware, including powerful GPUs, CPUs, and storage systems. Ongoing operational costs include substantial electricity consumption for both computing and cooling, typically ranging from £40 to £160 monthly, depending on usage patterns and local energy rates.
Long-term financial commitments extend beyond operational costs. Regular maintenance, software updates, and periodic hardware upgrades contribute to the total cost of ownership. Furthermore, organisations must consider opportunity costs, as managing local LLMs requires specialised personnel who could otherwise contribute to core business activities.
When comparing deployment options, cloud services may appear costly initially but offer flexibility and scalability advantages. However, organisations with consistent, high-volume usage may find local deployment more economical over time, particularly when factoring in potential tax benefits from hardware ownership.
Data Quality Challenges
LLMs face inherent limitations in data scope, as they are trained on specific datasets that may not encompass real-time information or data outside their training parameters. Maintaining current information presents ongoing challenges, as updating models with fresh data requires complex, resource-intensive processes.
Whilst local LLMs provide greater data control, they demand careful management to ensure robust data security and privacy protocols. Additionally, models can inherit biases from their training data, potentially leading to unfair or inaccurate outputs that require ongoing monitoring and mitigation.
Key Trade-offs
Organisations must navigate several critical trade-offs when implementing LLMs. The balance between cost and control sees local deployment offering greater data oversight and potentially lower long-term costs for high-volume users, whilst requiring substantial initial investment and ongoing maintenance commitments.
Privacy versus scalability presents another consideration, with local deployment potentially offering enhanced privacy protection but potentially limited scalability compared to cloud-based solutions. Finally, the performance-cost relationship requires careful evaluation, as smaller, more economical models may not deliver the same performance or accuracy as larger, more resource-intensive alternatives.
Towards Digital Sovereignty: The Ethical Dimension
The movement towards offline LLMs intersects with broader questions about technological sovereignty and the distribution of AI capabilities. On one hand, it democratises access to powerful tools, reducing dependence on large technology corporations and their terms of service. On the other, it potentially enables misuse by removing the content filtering and usage monitoring that centralised services can provide.
This tension between capability and responsibility becomes particularly acute in educational contexts. The transparency that comes with open source models allows for unprecedented scrutiny of AI behaviour, but it also requires educators and learners to take greater responsibility for understanding and managing the outputs of these systems.
Looking Forward: The Implications of Distributed Intelligence
As we consider the trajectory of this technology, several trends seem particularly significant. The continued improvement in model efficiency suggests that the performance gap between local and cloud-based models will continue to narrow. Edge computing capabilities are advancing rapidly, and the integration of AI-specific hardware into consumer devices seems increasingly likely.
Perhaps most intriguingly, we may be witnessing the emergence of a parallel AI ecosystem—one built on principles of transparency, user control, and data sovereignty. This isn't necessarily superior to cloud-based alternatives, but it offers a fundamentally different value proposition.
For those of us working at the intersection of technology and education, offline LLMs represent both an opportunity and a responsibility. The opportunity lies in creating more equitable, private, and customisable learning experiences. The responsibility involves ensuring that as we embrace these powerful tools, we maintain focus on the pedagogical principles and human relationships that make education meaningful.
The localisation of intelligence continues, one installation at a time. The question that remains is how thoughtfully we'll navigate this transition and what kind of educational futures we'll build with these newly accessible tools.
What has been your experience with local AI deployment? How do you see these tools reshaping educational practice? I'm particularly interested in hearing from educators and researchers who have experimented with offline models in their work.
We absolutely agree with this and have built a service for deliver exactly that on AWS Bedrock infrastructure.