I’ve been looking into self-hosting LLMs or stable diffusion models using something like LocalAI and / or Ollama and LibreChat.

Some questions to get a nice discussion going:

  • Any of you have experience with this?
  • What are your motivations?
  • What are you using in terms of hardware?
  • Considerations regarding energy efficiency and associated costs?
  • What about renting a GPU? Privacy implications?
  • rutrum@lm.paradisus.dayEnglish
    1·
    1 month ago

    Ive been playing with the nixified.ai project, which packages two web interfaces for LLMs and image generation. Im also looking into Tabby.ml for code assistant as well. I haven’t gotten deep, but these all look like promising options for utilitizing a server’s hardware but offering the functionality across the network.