Home Artists Posts Import Register
Join the new SimpleX Chat Group!

Downloads

Content

Hey everyone! I've created a 1-click auto installer for the Oobabooga Text-Generation-Webui, a tool to run locally Large Language Models on your computer! These file automates the entire install process with just 1-CLICK!

Here's how it works:

1. Download ONE OF THE the batch file (or .sh file) from this Patreon post (.bat for Windows/WSL and .sh for Mac/Linux) and put it in an empty folder somewhere on your computer and preferably choose a folder name without any spaces in it.
2. Double click the batch file to run it. Select "1" for Windows install & then sit back and relax! The file will automatically:

- Install Git IF you don't have it installed already

- Clone the Text-gen webui repository from GitHub

- Launch Text-gen webui installation process

- Install miniconda and other dependencies

Once the installation has started it will ask you a few questions that you will need to answer depending on your situation.

What is your GPU?

A) NVIDIA

B) AMD (Linux/MacOS only. Requires ROCm SDR 5.6 on Linux)

C) Apple M Series

D) Intel Arc (IPEX)

N) None [II want to run models in CPU mode)

Input>>>  If you have a GPU, you need to choose what brand it is. Usually for most people, it's Nvidia. But choose what correspond to your system.  HOWEVER if you don't have a GPU and you want to run it on your CPU instead, then choose "N" for None.

--------

Would you like to use CUDA 11.8 instead of 12.1? This is only necessary for older GPUs like Kepler.

If unsure, say "N".

Input>>> Usually for most of you who have a GPU, you should answer "N" BUT if your GPU is really old (like 10+ years like 600 or 700 series) then choose "Y"


Then wait for everything to be installed!

And as always, supporting me on Patreon allows me to keep creating helpful resources like this for the AI art community. Thank you for your support - now go have some fun! 

Files

Comments

Leonardo Lopes

what model is best to use with a 3060?

cool1

It depends on 1) which models will fit in the VRAM (different 3060s have different VRAM amounts I think so check what yours has and check whether the model you are checking fits in it) 2) what you want to use it for and whether that AI model was trained for it. But the Mistral-7B models (do a search on huggingface.co) are small models that should fit in that GPU's VRAM (around 4GB I think). 3) whether it will work on what you want to use eg. the oobabooga web ui - you could check on the model page details on the huggingface site

hanZ

Mac-Version is not working for me - throws syntax errors, mentions invalid identifiers. Maybe I've started it the wrong way? (ran it via the terminal)

cool1

when I added the bit for it to load Llava and the bit with "--multimodal-pipeline llava-13b" at some point it said "UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in [the folder name]". I'm running windows 10. Is there anything that could be changed to help there to make it more effecient in how it stores cached stuff - if its needed? It also gave some warning message about using AutoGPT and there being some problem with CUDA and that it may be slow (this is without running your new installer - but with an existing install of oobabooga) but I'll check to see how it runs. The UI also says "ExLlama_HF is recommended over AutoGPTQ for models derived from LLaMA" so I don't know if it might be better with that over AutoGPT

Aitrepreneur

I'm not a mac sure so I'm not sure where the issue could but make sure that you first make the script executable by running chmod +x TEXT-GEN_INSTALLER-MAC-LINUX.sh and then you run the script ./TEXT-GEN_INSTALLER-MAC-LINUX.sh

ComeOverHere

i ran into an error while downloading. Here's the error: fatal: unable to access 'https://github.com/...': Failed to connect to github.com port 443 after 21067 ms: Couldn't connect to server Any ideas why?

Michael

why not just install pinokio

Steven Markley

same here! not sure if i'm doing anything wrong but no matter what it seems to hit me with syntax errors when trying to run it this same way. This is what I'm showing on my end after running - : command not foundoads/TEXT-GEN_INSTALLER-MAC-LINUX.sh: line 2: Select your installation type: 1. Linux 2. macOS : command not foundoads/TEXT-GEN_INSTALLER-MAC-LINUX.sh: line 6: echo : command not foundoads/TEXT-GEN_INSTALLER-MAC-LINUX.sh: line 7: Type 1 or 2 and press Enter: 2 ': not a valid identifierEXT-GEN_INSTALLER-MAC-LINUX.sh: line 8: read: `choice : command not foundoads/TEXT-GEN_INSTALLER-MAC-LINUX.sh: line 9: /Users/steven/Downloads/TEXT-GEN_INSTALLER-MAC-LINUX.sh: line 11: syntax error n'ar unexpected token `{ /Users/steven/Downloads/TEXT-GEN_INSTALLER-MAC-LINUX.sh: line 11: `clone_and_ins'all() {

Aitrepreneur

I saw someone having that Llava error too but has long it doesn't inpact its use it's fine, yes Exllama is better and faster and AutoGPTQ since it's made to run Llama based models

Mark

Hi Altrepreneu is there are way to get this to work with AMD RocM warn("The installed version of bitsandbytes was compiled without GPU support.) im running Ubuntu and have double checked that my gpu is Rocm compatible , Cheers

chris torrence

"Failed building wheel for sentence-transformers" and "ERROR: could not build wheels for sentence -transformers, which is required to install pyproject . toml-based projects" I tried on one of my old PCs and it has no GPU so I chose N.

Anas AL Chayah

i am getting the error that This script relies on Miniconda which can not be silently installed under a path with spaces. what should i do

Anas AL Chayah

sorry i got the memo i was watching your video while installing i removed the space now its working

Augusto Baldim

How do I get some tech support? I've been getting an error everytime I download a model

Aitrepreneur

yes it should, but it requires ROCm SDK 5.4.2 or 5.4.3 to be installed. Some systems may also need: sudo apt-get install libstdc++-12-dev Edit the "one_click.py" script using a text editor and un-comment and modify the lines near the top of the script according to your setup. In particular, modify the os.environ["ROCM_PATH"] = '/opt/rocm' line to point to your ROCm installation.

Aitrepreneur

Saw someone in the forum having this issue too, could be a bug. You can try launching the file called "cmd_windows.bat" which will bring a command prompt window and inside you can copy and paste this command then press enter: python -m pip install git+https://github.com/UKPLab/sentence-transformers@v2.2.2 then relaunch the webui by running the start_windows.bat file

cool1

I've somehow messed my textgen web ui up. I already had it installed and it was working fine and running fast (seemed to use the GPU with GPTQ models and loading 13B models with 8K context with ExLLama worked okay). Somehow after installing the Stable Diffusion web ui (automatic1111) and trying (unsucessfully) to install and run the KOHYA-SS for Lora training, then trying updating textgen webui to try the Llava model (as shown in the video), with loading that model with the settings and editing the cmd_flags.txt to add "--multimodal-pipeline llava-13b" after that it was giving me errors/warnings about Torch not being compiled for GPU and trying to reload past models that worked, using things that worked before (eg. ExLlama) doesn't work. I tried clicking the "update_windows.bat" to update things but it still isn't all working. Messages I get are things like: Torch not compiled with CUDA enabled ERROR:Could not find repositories/exllama. Please ensure that exllama (https://github.com/turboderp/exllama) is cloned inside repositories/ and is up to date. when I click "update windows" one of the output lines says "TORCH: 2.1.0+cpu" - is that something wrong - is that saying it's CPU only (not GPU)? Do I need to remove folders and re-install textgen again (after backing up the models & characters)? What most likely made it go wrong? And does it really matter that I have installed a slightly higher version of Python than one of the warnings recommended? If I used the new one click install would that make sure it used its own version of everything (python etc) so there wouldn't be conflicts with anything?

Aitrepreneur

Wow, yeah hard to say, not sure why thing suddenly gone wrong, try reinstalling the webui? There shouldn't be any issues running all of the webuis, I do it myself

cool1

Thanks. I'll try that. I'll move or rename the current folder so I have a copy of the current models etc. then try the new one click isntaller. Ideally they could make it so things could be installed without affecting the models folder etc. and they could make sure everything has it's own version of everything (eg. python or torch etc.). edit. After using your install option (with copying the models and characters back from the other folder) it's now wokring okay thanks and the stable diffusion gui is too (I tried them both at the same time but that slowed things too much with 8K context). And the Llava thing worked too (though it isn't always totally accurate about what's in an image). I'l try a re-install of the KOHYA later too to try to get that working.

vanNiko

Hey! Everytime i try to generate a response this error comes up: RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Phillip Gimmi

which model types should i use?

Enchanted Pixie

Heya , so I did the install and it works a charm but there is no model included . Where do I get the models and what do you suggest ?

Aitrepreneur

what's your GPU? For the smallest model you try the one I showcased in my video called mistral 7b

Kristopher Hodge

I seemed to have installed it fine, but ran into an issue when trying to chat. When I chat I see it appear for 1 sec. then my chat disappears and the model will try to respond but shows nothing. I have a Nvidia 1080 TI and using tried both TheBloke_LLaMA2-13B-Tiefighter-AWQ and TheBloke_Mistral-7B-v0.1-AWQ, I followed the install video "INSTALL UNCENSORED TextGen Ai WebUI LOCALLY in 1 CLICK" on youtube.

vanNiko

So i tried a few models now (these ones ->) TheBloke_Llama-2-7B-Chat-AWQ TheBloke_Llama-2-7B-Chat-GPTQ TheBloke_manticore-13b-chat-pyg-AWQ trion-m-7b.Q5_K_M.gguf and the problem only comes up with the AWQ models. The GPTQ and GGUF work just fine.

Fash Demented

I think we're having the same problem. I'm on a 1080 Gtx. Nothing is generating. When i check the script its showing the below error RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Robert Bersch

Hmm everything worked fine install wise till I tried to load a model... using model llava-13b-v0-4bit-128g. Downloaded ok but get this error trying to load. File "C:\Users\secto\Desktop\ChatAI\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 2992, in from_pretrained raise EnvironmentError( OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory models\wojtab_llava-13b-v0-4bit-128g. Edited: Installed TheBloke/claude2-alpaca-7B-AWQ and it started doing stuff.. Seems more random nonsense then AI.

Aitrepreneur

for the llava model do not forget to modify the CMD_FLAGS.text file with this command: --multimodal-pipeline llava-13b otherwise that's not gonna work, just like I showed in my video. For the other models, you can try using gptq models instead and see it that works better, by loading them with either autogptq or exllama (depending on the model you've chosen)

Robert Bersch

Yeah that seemed to make it work. That and changing to AutoGPTQ. That model seems to not work well with my system, 4070Rtx with 12 gig and 64gig on mobo. But my cpu is AMD Ryzen 9 3900X so that may be the problem. TheBloke_Mistral_7B_Dolphin2.1_LIMA0.5-AWQ works pretty well though.

Retired Android

Yeah. Same boat here with 1070. Unfortunately I have not been able to run anything on pytorch 2.1 with CUDA 12.1 on this card (and this is not limited to text models). One option would be to downgrade to CUDA 11.8. HOWEVER I found info in some git issue that AWQ model loading requires now CUDA 12.1 and indeed it fails to load. So that's it. It seems that the problem is either Pytorch that silently stopped providing binary kernels for 10xx cards or bitsandbytes not providing it. However, I was not able to figure this out and I spent too many time today on this. ;) (GGFU models works fine though)

Chad Fell

No matter which model I use, I get this response in the chat t0. Assist in setting achievable goals related to personal growth and self-improvement. Help identify specific areas where growth is desired and create actionable steps towards achieving those goals. Provide motivation and encouragement throughout the process. Offer accountability by checking in periodically and discussing progress made.

tdog

My install gets stuck at installing miniconda. I think I already have it from running A1111 But the installer freezes there. Can you help me get this up and running?

Aitrepreneur

I see, I indeed saw some other people say that for some reason awq don't work as well as gptq. Well in your case just use GPTQ models, if they are based on llama, use exllama loader for faster token generation

Aitrepreneur

with your GPU I may recommend this model, just copy and paste that line in the model download field and click download: TheBloke/Nethena-MLewd-Xwin-23B-GPTQ:gptq-4bit-32g-actorder_True then choose the exllama loader and click on load

Aitrepreneur

You can try turning off your antivirus/firewall for a few minutes see if that makes it better maybe they are blocking the connection, but usually this depends on your machine and sometimes internet connection too

pib boris

Hello, i can t install with the link...i did on my laptop. Now i can used it on my first pc and i can't install it on my desk one. Can i have some answers ?

pib boris

Le problème c'est qu'il n'y a même pas de code erreur la l'invite de commande admin s'arrête et se ferme au bout de 10 secondes d'installation...?

hanZ

Ran perfectly on my windows PC. D*mn - this is an even deeper rabbit hole than image generation. I'm pretty experienced with image generation, but with text generation I'm feeling like a complete newbie regarding AI/ML. 4bit vs. 8bit, all those loaders, hundreds of base models etc.pp. And I've haven't even started with training LORAs for those.

Lexx 1100

I have never been able to get your one click programs to work! they sure as heck not one click. and your video's don't seem to help either.

Peruvian Dota

I would love resources to finetune models with the cloud. For now with regulat GPU's I have not been able to find good applications for the models I have tested. Meta models 13b produce great content but super slow even with a 4090.

totally_not_a_weirdo

I've tried a couple different models, but they only generate gibberish. Are there not Parameters>Generation settings that have to be made? What are they?

Aitrepreneur

that's probably an issue with the models, default parameters should be good enough as it is, what models have you tried?

hanZ

It's clear what video we need next from AItrepreneur: Local install Lora-Training for text-generation AI with Axolotl or else

Robert Hentschel

Type 1 or 2 and press Enter: 1 Git is not installed. Downloading and installing Git for Windows... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 58.3M 100 58.3M 0 0 8656k 0 0:00:06 0:00:06 --:--:-- 9666k Failed to download Git installer.

Aitrepreneur

make sure your antivirus and or firewall is not blocking the download, just disable them for a few minutes before launching the bat file

Robert Hentschel

The download worked fine, I could see the file in the folder. I think the install failed because a previous version was present. I went ahead and installed manually, it runs now. However, the models are returning gibberish for my questions. Perhaps a troubleshooting video for commons issues would be helpful. Nobody is making videos quite like yours. Much appreciated.

DeadFlame

For some reason it won't let me click on the URL after it's done installing. What do I do about that?

Anthony

neither windows or mac works, same error.

Nicholas Canada

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. openai-whisper 20230918 requires tiktoken==0.3.3, but you have tiktoken 0.5.1 which is incompatible. Successfully installed tiktoken-0.5.1

Nicholas Canada

I can't figure out how to launch the UI please help 😔🙏

Brent Johnson

Thank you Aitrepreneur for the tutorial! Btw, If anyone wants the MUCH higher quality V2 voices for Elevenlabs with ability to adjust style setting, I posted the code and notes here. https://github.com/oobabooga/text-generation-webui-extensions/issues/51

TechnoCore

Hello, one-clicked worked fine, got a model, made a character. All working well, so thanks! Just some follow up questions... Is there a video or where can I find out about all the details of Oobabooga environment? Like what Chat-Style I should use for what? What should I set on the parameters page? What do all the extension do? Etc... Thanks!

Aitrepreneur

I might do an extended video explaining some thing further if people want it but there isn't a lot you need to know, if you want you can read the documentation to learn more: https://github.com/oobabooga/text-generation-webui/wiki But you shouldn't really touch anything in the parameters page, the default presets are already good enough

TechnoCore

"But you shouldn't really touch anything in the parameters page, the default presets are already good enough" mostly what I needed to know! ha ha, thanks!

Mark Mynhier

I click on the bat file, hit 1 for Windows, the CMD box disappears and then nothing. I have Git installed. What else could be the problem?

Aitrepreneur

You can try this simpler installer, no need to press anything just double click it: https://we.tl/t-CiPTkrEUTE

rick esping

Nothing happens when I try opening the down loaded files.

Aitrepreneur

Probably getting blocked by your antivirus and or firewall, disable them for a few minutes before relaunching the installer

KdaK

How are you. After installing link, I saw these error statements, not sure what I should do -------- Downloads\text-generation-webui\installer_files\env\Lib\site-packages\gradio\components\dropdown.py:231: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: 128 or set allow_custom_value=True. and ---- Downloads\text-generation-webui\modules\models.py", line 306, in AutoAWQ_loader from awq import AutoAWQForCausalLM ModuleNotFoundError: No module named 'awq'

Martin Čarný

Hi! At first it failed to install Git, so i installed it manually. Then this error poped up "fatal: failed to load library 'libcurl-4.dll' Failed to clone the oobabooga text-generation-webui repository." Defender and firewall are disabled.

Darkwan75

This is probably going to sound dumb but I can't find where the installer put the .bat files to launch the programs that were installed. I'm not sure what directory they get installed to in windows.

FruitSnack

I really enjoy the idea behind it and the possibilities, I try doing it manual months ago with your older video and was able to installed but it was running slow. I saw your new video and choose to subscribe, great content, I was able to install and run the interface, them I downloaded yarm mistral model you recommended to see if I would have better luck this time, I try to loaded and I get and error while fusing layers, unable to load the model, I wonder if I am missing something or did something wrong? I have a laptop with a AMD Ryzen 7 5800H, 16ram and a Nvidia Geforce RTX 3050Ti, not sure if my laptop is not powerful enough to run the AI. while installing the interface I put the option of Nvidia, not sure if my mistake was there, I am out of ideas

Sean Ashcraft

I'm getting the following error when I run the installer; I'm very new to all of this and have no idea what to do to correct it. "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. openai-whisper 20230918 requires tiktoken==0.3.3, but you have tiktoken 0.5.1 which is incompatible." I think (am hoping) this is the cause of other problems I'm having post-installation, like not being able to connect to Tavern.AI.

Mentos

It didnt install miniconda. Miniconda isnt found. Its a pain

Night Wolf

Hey, I got "Warning you haven't downloaded a model yet. Once the web ui launched, head over to 'model' tab and download one." I'm in the model tab, the drop down for model only says "none"... I have NO IDEA where to get a model. What do I click?

Peejie

I was able to fix it on my Windows machine with Nvidia GPU. Not exactly one-click, but it might be helpful! I believe that my issue was that I already had a PATH set for another Linux environment. Installed miniconda manually (https://docs.conda.io/projects/miniconda/en/latest/) When installing, I chose NOT to add it to PATH and NOT to set it as my default Python 3.11. After installing, I ran Anaconda 3 (Or Anaconda PowerShell Prompt) I then navigated (using CD) to the directory I wanted to install. Once there, I ran the following commands - ((text on the line after # is not to be entered)): #Create new Conda Environment conda create -n textgen python=3.11 conda activate textgen Install PyTorch conda install -y -c "nvidia/label/cuda-12.1.0" cuda-runtime Install WebUI git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui pip install -r requirements.txt _______________________ Solved this by giving ChatGPT AiTrepeneur's .bat script along with the readme text for the Text Generation. I then gave it the error message I was getting and the directory I wished to install it in. Hope this helps!

Colin

My generation is going extremely slow. "Output generated in 230.27 seconds (0.20 tokens/s, 46 tokens)" It strangely only happens with your girlfriend model. I have a 4070 Ti 12GB, what could be the cause for this?

SINHORA MOSES

I tried to install a model and I got along list of this error look a like: 2023-12-03 02:15:51 ERROR:Failed to load the model. Traceback (most recent call last): File "C:\Users\moses\Downloads\text-generation-webui\modules\ui_model_menu.py", line 209, in load_model_wrapper shared.model, shared.tokenizer = load_model(shared.model_name, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\moses\Downloads\text-generation-webui\modules\models.py", line 85, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\moses\Downloads\text-generation-webui\modules\models.py", line 250, in llamacpp_loader model, tokenizer = LlamaCppModel.from_pretrained(model_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\moses\Downloads\text-generation-webui\modules\llamacpp_model.py", line 91, in from_pretrained result.model = Llama(**params) ^^^^^^^^^^^^^^^ File "C:\Users\moses\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp\llama.py", line 933, in __init__ self.scores: npt.NDArray[np.single] = np.ndarray( ^^^^^^^^^^^ numpy.core._exceptions._ArrayMemoryError: Unable to allocate 15.6 GiB for an array with shape (131072, 32000) and data type float32 Exception ignored in: Traceback (most recent call last): File "C:\Users\moses\Downloads\text-generation-webui\modules\llamacpp_model.py", line 49, in __del__ del self.model ^^^^^^^^^^ AttributeError: 'LlamaCppModel' object has no attribute 'model' N.B: IF THERE IS ANYTHING YOU CAN DETECT THAT I MESSED UP PLEASE LET ME KNOW: i CHOSE NONE AT THE BEGINING cause I DID NOT KNOW IF I WAS TO CHOOSE CPU OR GPU. 16GB RAM. LAPTOP.

David Soler

How can i change the language of the chats to spanish? I have Mistral model

Aitrepreneur

Only with the gf model? That's weird, it shouldn't matter what character you chose, the speed should stay the same, it's usually more a model issue, what model are you using?

Aitrepreneur

Just start talking in spanish, but I'm not sure that Mistral has been trained on Spanish language though

Colin

The 7B AWQ model you showed in the video. The weird thing is, it gets worse and worse as the chat goes on. At the beginning of the chat it was fine, could be finished in 5 seconds at times. But after 10 messages it could take up to 300 seconds for one response.

Will Simon

my generation is SUPER slow. I've got a RTX 3060, Ryzen 5 5600G, and 64 gb ram. Doing art gen works fine. usually can do 100 steps with SDXL in about 1:17. but this text stuff is SLOW. I gotta have something configured wrong right?

Aitrepreneur

yeah actually some people (including myself) have some issues running some AWQ models for some reason, which is why right now I just recommend using GPTQ models. You can try using this pygmalion 2 model: just copy and paste this line in the model field: TheBloke/Pygmalion-2-7B-GPTQ:gptq-4bit-32g-actorder_True Then once it's downloaded, select the exllama loader and click load

Aitrepreneur

It's usually either the drivers or the model, what model are you using? When it comes to the nvidia drivers, are they up to date and have you disabled system memory fallback? https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion This will greatly improve the speed for both SD and text-gen

Will Simon

NousResearch_Nous-Hermes-Llama2-13b Going for disabling system mem fallback now.

Will Simon

Damn. Same issue. I put the Prefer No Sysmem Fallback for windows start.bat. closed and reopened the bat file. loaded model (cause I haven't set it to auto load still). Asked it how to make meth, and it's still slow af.

Will Simon

I really want a FULLY uncensored model, so after some reddit crawling found this Nous-Hermes LLama2 13b to fit the bill, if you do the "Sure Thing!" instruction. But maybe it's the model? Do you know of a good uncensored model that might run better?

Aitrepreneur

hum..but..yeah, that's because it's not a quantized model and it's not even good to begin with xD try this one: just copy and paste that line in the model field, then click download TheBloke/LLaMA2-13B-Tiefighter-GPTQ:gptq-4bit-32g-actorder_True once it's done, select the exllama loader then try again, no need to use sure thing with that model

KHU

Hello. I'm able to install this but I'm getting this. When I try to interact in the chat, I'm unable to see the responses. https://imgur.com/a/BRcB7Et

Malte Steinlandt

Hello. First sorry for my English, When i start the .bat file its Downloading the files from Github but after that i get this message: This script relies on Miniconda which can not be silently installed under a path with spaces. Need some help

Aitrepreneur

what model have you loaded? What error does it give you in the command prompts window?

Aitrepreneur

yes as I explained in the video and as the text implies you need to make sure that there is no spaces in the folder path. So if you have something like: C:\Users\yourusername\AI\AI INSTALL\ then you need to delete the space (in this case AI INSTALL) or replace it with something else (like this for example: AI_INSTALL)

Colin

Strange. Thanks, I'll give this model a try.

KHU

Um... no sure what to show you but this is what I'm seeing at the cmd window right now... https://imgur.com/a/80aYBIY

KHU

ps. kinda forgot I downloaded the ONE-Click version you provide. I'll try reinstalling with that one and see if it works out.

KHU

this is the model I loaded... TheBloke/NeuralHermes-2.5-Mistral-7B-AWQ ps. tried re-installing through One-clikc... it didn't work out either. https://imgur.com/a/LMxwxbw

Valedictorian

Hey! So it says that CUDA has run out of memory. In simple terms, what does this mean, and how do I avoid/allocated more memory? Thank you in advance!

Aitrepreneur

what's your GPU? It might not be compatible with AWQ if it's older than the 2000 series. Try this model: just copy and paste this line in the model field: TheBloke/Pygmalion-2-7B-GPTQ:gptq-4bit-32g-actorder_True then download it, once it's done, chose the exllama loader and click load.

Aitrepreneur

That means your GPU doesn't have enough memory to run the operation/run the model. What GPU do you have and what chat model are you running?

Valedictorian

I'm running a RTX3070 on a mystra/tiefighter (as the video stipulated me to use). I have 16gb of ram installed, not sure if thats applicable to the situation. Cheers!

Aitrepreneur

Hmm weird this should be enough memory for that... try this one: copy and paste this in the model field: TheBloke/LLaMA2-13B-Tiefighter-GPTQ:gptq-4bit-32g-actorder_True then download it, once it's done, select the exllama loader and load the model then try chatting, this should work fine with your GPU

KHU

I'm have a GTX 1060 6G VRAM... Is that the cause? I'll this model you recommended thx!

KHU

seems there's a version 2 of exllama. Do I pick that or the default one?

KHU

Ok this one worked! Thank you so much 😭 Btw... I also have another issue but it's trying to install the Voice. I used the One-click version you provide... I'm having on the installation... I'll post it on the related thread.

John Tackman

if you run out of memory you simply need a smaller task or a smaller model, it's about your VRAM size on your graphics card

troulertrian

So, I have a newbie question. When I used the one click install it installed everything in my C drive, which is an hdd. Can I just paste the folder to my ssd at D without wrecking stuff?

bruh

"2023-12-09 13:46:12 ERROR:No model is loaded! Select one in the Model tab." is what I see in the cmd bar when I try to send it a message, but whenever I go to the model that the only model option i have is "none"

Aitrepreneur

after a model is downloaded, click on the small blue refresh button to refresh the list

bruh

I went ahead and downloaded TheBloke/LLaMA2-13B-Tiefighter-AWQ before you responded but it takes so long for responses to be generated. Is there any way around this? I have a desktop 3060ti

Aitrepreneur

you only have 8Gb of vram that's why, you won't be able to run anything above 7B with good speed. You can try this model: just copy and paste this line in the model field, download it and choose the exllama loader TheBloke/Pygmalion-2-7B-GPTQ:gptq-4bit-32g-actorder_True

Jason

keep giving miniconda is missing. i tried to download miniconda separately but still doesn't work.

Zyfire

I keep getting ERROR: Failed building wheel for tts, not sure what to do

Jason

i posted it in your tech support channel on discord

Aitrepreneur

try downloading this: visualstudio.microsoft.com/visual-cpp-build-tools follow this image guide: media.discordapp.net/attachments/1025498368631377970/1180633461858516992/image.png?ex=657e2198&is=656bac98&hm=14fd9acdb50677653c811affdb532ef5545c4d88c795d7a07dd8327f5ef07866&=&format=webp&quality=lossless&width=2124&height=1176 let me know if that works better, don't forget to restart your pc, some people had to do that too before it worked you can also open the cmd_windows.bat file and paste this line right here: conda install -c conda-forge python-crfsuite then press enter

Serino Alejandro

I my case it just stuck downloading at 23%, and then CMD autmatically gets closed, any help?

Hurt Porter III

i was able to download and i got the web address but I needed to restart my computer and now i dont know how to get everything back up and running again. do I need to reinstall everytime? Im not sure how to turn oogabooga back on

Aitrepreneur

please don't create a new message on the page, just send me a dm with the error log, your pc specs and a quick explanation on what went wrong, thanks.

Aitrepreneur

could be an antivirus/firewall blocking the connection from running further, either disable them before running the installer or create an exception for the file

Tom Braider

Thank you so much for all the work you've done! 2 little points which I wish I had known before the install: 1) in windows, for your batch file to work, git has to be installed and the git.exe in the "path". (one has to logout then login again for the environment variable to updated, apparently) 2) All the files of the installer bat file will be put in the directory where one runs the install bat file from! (I'm kind of used to being asked where to install into). I ran out of space on the small partition I was using. ANYWAY, the video really helped and it's working! It's a bit of a transition from "Faraday.dev" but there are enough similarities that I didn't feel too lost. Can't wait to go through some of the other ones now I have the text-generation-webui installed. I couldn't have done it without you!

littlerunning frog

hi, I get an error, please can you help please. ----- copying build\lib\lm_eval\tasks\model_written_evals\persona\believes-it-is-not-being-watched-by-humans.yaml -> build\bdist.win-amd64\wheel\.\lm_eval\tasks\model_written_evals\persona copying build\lib\lm_eval\tasks\model_written_evals\persona\believes-it-knows-better-than-humans-about-how-the-system-should-behave.yaml -> build\bdist.win-amd64\wheel\.\lm_eval\tasks\model_written_evals\persona error: could not create 'build\bdist.win-amd64\wheel\.\lm_eval\tasks\model_written_evals\persona\believes-it-knows-better-than-humans-about-how-the-system-should-behave.yaml': No such file or directory [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for lm-eval Failed to build lm-eval ERROR: Could not build wheels for lm-eval, which is required to install pyproject.toml-based projects Command '"E:\AI\Oobabooga\text-generation-webui\installer_files\conda\condabin\conda.bat" activate "E:\AI\Oobabooga\text-generation-webui\installer_files\env" >nul && python -m pip install -r temp_requirements.txt --upgrade' failed with exit status code '1'. Exiting now. Try running the start/update script again. Press any key to continue . . .

littlerunning frog

hi, I know this sounds bit odd but I get the same error by running the update windows bat file :(( please help and happy Christmas

Aitrepreneur

very often this kind of error happen when you install it in a folder path that is too long or that is located under the C windows folder. So try installing it in a short folder on your desktop or something like: C:\Users\ohmni\Desktop\AI

littlerunning frog

hi, I tried your suggestion path and the system ask me to provide administration and run but got the same error, I also tried to put the file to another location E:\AI system didn't ask for administration this time but still get the same error. I tried various different path/folder but not get any lucky, am I missing some thing?

littlerunning frog

I also find another error during the installation --- Using cached distlib-0.3.8-py2.py3-none-any.whl (468 kB) Building wheels for collected packages: lm-eval Building wheel for lm-eval (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for lm-eval (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [3345 lines of output] running bdist_wheel running build running build_py creating build creating build\lib creating build\lib\lm_eval

Andy Merz

"Miniconda hook not found" whats the solution?

Aitrepreneur

first don't run the bat file in admin rights as I said, then for this error apparently you can fix this by typing regedit in the windows search bar. This should give you the Registry Editor. In the registry editor navigate to: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled Double click on "LongPathsEnabled" and change value from 0 to 1, then click OK.

Aitrepreneur

usually this error happens because of where you installed the webui, try installing it in a folder without spaces and without special characters

Andy Merz

Thank you, your advice got me pass that issue. Now I am getting "ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\Users\\andy_\\OneDrive\\Desktop\\LLM\\text-generation-webui\\installer_files\\pip-install-7nifgqmh\\lm-eval_85a62f66ec974b38aa5fba1225dcd99d\\lm_eval/tasks/model_written_evals/persona/willingness-to-intentionally-make-mistakes-to-achieve-higher-final-performance.yaml' HINT: This error might have occurred since this system does not have Windows Long Path support enabled. You can find information on how to enable this at https://pip.pypa.io/warnings/enable-long-paths" which recall when trying before you had instructions for newbies somewhere maybe.

Andy Merz

Gold star for me I found out how to extend path length and now have Oobabooga working! Yay go me!

Dr. Brian Burton

Where do I get the elevenlabs_tts extension?

Tonic69

I have been looking for this extension as well, it did come come preloaded like the others.

Aitrepreneur

yeah...unfortunately oobabooga decided to remove completely the elevenlabs option from the webui since now we have the free xttsv2 option, I don't agree with it personally. If you want to use it, either revert back to an older version of the text gen or download and install sillytavern (just follow my latest video and patreon posts) it should still have the elevenlabs option available

matryoshka12

why when i download the model it doesnt start? the 7.25GB file stuck in 14.7MB

What to watch high

Can you do a video on Lora training in Ooba? I don't see anyone using them so is it just not useful?

RMR

One question the text generation from TheBloke_Yarn-Mistral-7B-128k-AWQ is slow and takes 1-2 minutes to generate text. is there a way to quicken it up or use a different model?

jordo

Is there a way to move the installation to a different drive after installing?

S i

It all been working out great till I had a problem downloading the model, Whenever I tried to Load the model this message appears: Traceback (most recent call last): File "C:\Users\bluxf\OneDrive\Desktop\text-generation-webui\modules\ui_model_menu.py", line 213, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\bluxf\OneDrive\Desktop\text-generation-webui\modules\models.py", line 87, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\bluxf\OneDrive\Desktop\text-generation-webui\modules\models.py", line 310, in AutoAWQ_loader from awq import AutoAWQForCausalLM ModuleNotFoundError: No module named 'awq'

Alex

hello I need help. i already have GIT installed but after i press 1 when opening the text installer, it tries to redownload GIT. it gets to 90% and the command prompt page closes.

Aitrepreneur

what's your GPU? I recommend maybe using a different model like this one: TheBloke/Pygmalion-2-7B-GPTQ:gptq-4bit-32g-actorder_True copy and paste that in the model filed, dl it and then use the exllama loader

Aitrepreneur

I recommend using GPTQ models like this one for example: TheBloke/Pygmalion-2-7B-GPTQ:gptq-4bit-32g-actorder_True copy and paste that in the model filed, dl it and then use the exllama loader

bananaboy

Hi thanks for everything you do here man it's awesome to learn about it. I'm just a bit stuck... Probably something easy so I'm sorry. But I've installed SillyTavern, and now trying to get this Text-Generation-Webui to work, but when I click on "start_windows.bat" the command prompt window pops up and seems to start loading it, but then nothing happens. The last thing it says is "To create a public link, set `share=True` in `launch()`."

Aitrepreneur

it should give you a url, just put that url in your browser, you can watch my install video if you need more info: https://youtu.be/C-7jGYOGvy4

bananaboy

omg thank you I'm new to this sort of thing as you can probably tell lol. Thanks a lot I wasn't expecting a reply so soon too :)

Aitrepreneur

no worries I'm on vacation right now so I might be too slow to answer so question right now but I'll answer them as soon as I can ;)

RMR

NVIDIA GeForce RTX 3060 TI Thanks I'll try that one.

Kennedy Vieira Lopes

This is what appears to me when loading the model: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB. GPU 0 has a total capacty of 6.00 GiB of which 0 bytes is free. Of the allocated memory 8.59 GiB is allocated by PyTorch, and 309.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF What do I do?

Aitrepreneur

this error means you don't have enough Vram to run the operation/model, what model were you trying to run?

Aitrepreneur

you can try this one: TheBloke/Pygmalion-2-7B-GPTQ:gptq-4bit-32g-actorder_True copy and paste that line in the model field, dl it and use the exllama loader

Kennedy Vieira Lopes

I worked! I can't believe it, thank you very much, I love your work, I send a hug to you here in Brazil!!

Aaron1611

tried adding voice but elevenlabs_tts was not in the extension list.

Aitrepreneur

unfortunately oobabooga decided to remove completely the elevenlabs extension from the webui since now we have the free xttsv2 option. If you want to use it, either revert back to an older version of the text gen or download and install sillytavern (just follow my latest video and patreon posts)

Jacob Dow

So, I've installed Text-Generation-Webui Oobabooga with the installer, adjusted the settings as per the walkthrough video with selecting the presets, etc. and I have downloaded 2 separate models as well as the Sandra character. When I try loading any of the 2 models in the webui, I get the following error: "Traceback (most recent call last): File "C:\AI\text-generation-webui\modules\ui_model_menu.py", line 213, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\text-generation-webui\modules\models.py", line 87, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\text-generation-webui\modules\models.py", line 247, in llamacpp_loader model_file = list(Path(f'{shared.args.model_dir}/{model_name}').glob('*.gguf'))[0] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^ IndexError: list index out of range" Any help would be greatly appreciated.

Scruffy Scruffington

So if i have AMD on windows i shouldn't select the amd option and just go with cpu?

Gage

Hey! So, I ran into an issue. After running the bat. installer (I'm using Windows) the prompt says: Installing Miniconda to C:\Windows\System32\text-generation-webui\installer_files\conda Miniconda version: The system cannot find the path specified. Miniconda not found. Press any key to continue . . . The thing is, I checked the path and found the miniconda_installer.exe file, but it won't open. It says "error launching installer."

Aitrepreneur

hmm you might have an antivirus that blocked the file during download, try installing miniconda manually: https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe and don't forget to check the box, "add miniconda to path"

Ma No

I sent you a dm but having some issues installing this with the bat file. ValueError: When localhost is not accessible, a shareable link must be created. Please set share=True or check your proxy settings to allow access to localhost. Press any key to continue . . .

Glitch Gremlin

i installed it from the Installer here, but i can't get the program to come on or anything... am i missing something? i thought i simply installed it from the file here and that'd be it?

Glitch Gremlin

it was blank if i remember correctly.. ill try again in a bit and let you know exactly what it said, though.

BladeMech22

The installer didn't work. I asked for help but that comment is now gone. ERROR: Wheel 'hqq' located at C:\users\j1\appdata\local\pip\cache\wheels\5b\37\da\86e3082cb5c41972d221b79a79259433f68c29285bf35a7448\hqq-0.1.2-py3-none-any.whl is invalid. Command '"C:\AI\OOBABOOGA\text-generation-webui\installer_files\conda\condabin\conda.bat" activate "C:\AI\OOBABOOGA\text-generation-webui\installer_files\env" >nul && python -m pip install -r temp_requirements.txt --upgrade' failed with exit status code '1'.

Bradford Klipan

Hello. After I use the installer, and then download and load the model, I get the following error when I try to chat with the AI: "RuntimeError: CUDA error: no kernel image is available for execution on the device". I installed for CUDA version 12.1, but the CUDA version on my system is 12.2. What can I do? To provide more info, my system has 2 x GTX 1070TI.

Lawrence Wong

Hi Hows it going? Been following the silly tavern guide which lead me to dwonload the ooga booga webui from a previous model. It works well up until I try to load my model, which just gives me [Connection : Error]

Lawrence Wong

I think it might just be my laptop. I'm using the one suggested in the video ( I believe it was Yarn-minstrel GGUC 4k M

Mostafa Amir

Hello, on my Linux build with a RX 7900 XTX GPU with 24gb VRAM, and ROCM 5.6, i'm still failing to load the model shown in the video and I get these errors, any help?: shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ from awq import AutoAWQForCausalLM ModuleNotFoundError: No module named 'awq'

St Lask

Heya im getting IndexError: string index out of range trying to use, it wouldnt load the list and displayed that. Tried to load the list for Yarn-Mistral-7B-128k-GGUF as i got the CUDA memory error from the awq and got the error

Aitrepreneur

what's your GPU? Have you tried the gptq version instead? What did you put inside the model field?

Keiron Gulrajani

I downloaded an uncensored model but the text generation still gives me messages about it not being able to answer specific questions even when I choose the uncensored model; is there a setting somewhere that needs to be set for it to be truly uncensored?

St Lask

RTX 2040, I managed to get it to work but now im having issues connecting to ST. The given URL the CMD gives doesnt connect.

Aitrepreneur

I don't know if that one is uncensored I haven't tried it, try this one instead though: https://huggingface.co/TheBloke/Xwin-MLewd-7B-V0.2-GPTQ put that in the model field: TheBloke/Xwin-MLewd-7B-V0.2-GPTQ:gptq-4bit-32g-actorder_True dl it and use the exllama loader

St Lask

CMD is saying ValueError: No tokenizer is loaded. URL given by CMD: http://127.0.0.1:5000

Extro

I'm also having a issue, tried running mixtral -8x7B-Instruct-v0.1-AWQ, I dont get any response in silly tavern.

Aitrepreneur

lol well that's because it's the most powerful local llm right now and it requires something like 80Gb+ of Vram to run😅 So basically no one with a consumer grade gpu can run this model, it's made to be run on multiple professional grade GPU. You need to use a normal model :D

Extro

OH, well then, any recommendations for an uncensored model? I have a 3090 as well

Aitrepreneur

I like this one: https://huggingface.co/TheBloke/LLaMA2-13B-Psyfighter2-GPTQ put this inside the model field: TheBloke/LLaMA2-13B-Psyfighter2-GPTQ:gptq-4bit-32g-actorder_True dl it and load it with exllama loader

Keiron Gulrajani

I think I sorted it, I loaded the model and then created a new chat and it worked, I think before I didn't create a new chat after loading, I assumed loading a model created a new chat with the new model but it doesn't appear to be the case

Extro

I get trace backs whenever I try to load it, not sure why Update; Figured it out, it was cause I ended up needing to put the second string you have there in the first model field, didn't realize it's only for the cpu packs

Riccardo Mui

Question - I was able to successfully install using your installer, however when attempting to download a model via huggingface using the model tab I end up with traceback errors. However I am still able to choose the model within the model tab. When trying to get a response from the chat, the character types and responds with a blank reply. I do not see any errors. I am attempting to use TheBloke_Yarn_Mistral-7B-128k-AWQ model on a 1080ti Nvidia card. I was curious if you've seen this before?

Aitrepreneur

it's for the gpu, as I said put the whole line in the field to dl the specific model version

Aitrepreneur

yes try another model and format because of your GPU. Try this instead: put this inside the model field: TheBloke/Xwin-MLewd-7B-V0.2-GPTQ:gptq-4bit-32g-actorder_True dl it and load it with exllama loader

Justin

Hey man, I was able to install it and run it just fine but when I tried downloading a model, it says "string index out of range". It also mentions line 242 on the ui_model_menu.py file and line 47 in the download-model.py file. Help? I downloaded silly tavern right before this, idk if that matters.

Justin

nvidia 4070super, and i tried downloading llama13B psyfighters 2

Aitrepreneur

try putting this in the model dl field: TheBloke/LLaMA2-13B-Psyfighter2-GPTQ:gptq-4bit-32g-actorder_True dl it and then use the exllama loader

Justin

Lol I restarted it and it worked for some reason lol. Started playing around with it a little and its actually mindblowing how it all works so well. Thanks for the help anyways. and the installer, really helpful

james

maybe its just me but when i tried to install it i get a error of miniconda not found and when i manually install it, and run update or start windows it still says miniconda not found.

Aitrepreneur

probably because you had a previous bad install of miniconda that was not added to path, uninstall miniconda, then relaunch the installer in anoter folder

Cr Fultz

Ok, well I thought I was following your instructions, but, something has gone haywire somewhere.........I have a 4070ti with 16GB so Im thinking i might be ok with gpu mem but it will not load the model. is there anything you can see that i might need to do to correct whatever step i might be doing wrong? Running on local URL: http://127.0.0.1:7860 To create a public link, set `share=True` in `launch()`. 06:43:00-713643 INFO Loading mistralai_Mixtral-8x7B-Instruct-v0.1 Replacing layers...: 100%|█████████████████████████████████████████████████████████████| 32/32 [00:04<00:00, 6.85it/s] 06:43:10-261499 ERROR Failed to load the model. Traceback (most recent call last): File "C:\Users\Chuck\Desktop\textai\text-generation-webui\modules\ui_model_menu.py", line 213, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Chuck\Desktop\textai\text-generation-webui\modules\models.py", line 87, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Chuck\Desktop\textai\text-generation-webui\modules\models.py", line 311, in AutoAWQ_loader model = AutoAWQForCausalLM.from_quantized( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Chuck\Desktop\textai\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\auto.py", line 56, in from_quantized return AWQ_CAUSAL_LM_MODEL_MAP[model_type].from_quantized( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Chuck\Desktop\textai\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\base.py", line 208, in from_quantized load_checkpoint_and_dispatch( File "C:\Users\Chuck\Desktop\textai\text-generation-webui\installer_files\env\Lib\site-packages\accelerate\big_modeling.py", line 545, in load_checkpoint_and_dispatch load_checkpoint_in_model( File "C:\Users\Chuck\Desktop\textai\text-generation-webui\installer_files\env\Lib\site-packages\accelerate\utils\modeling.py", line 1494, in load_checkpoint_in_model set_module_tensor_to_device( File "C:\Users\Chuck\Desktop\textai\text-generation-webui\installer_files\env\Lib\site-packages\accelerate\utils\modeling.py", line 276, in set_module_tensor_to_device raise ValueError(f"{module} does not have a parameter or a buffer named {tensor_name}.") ValueError: WQLinear_GEMM(in_features=4096, out_features=14336, bias=False, w_bit=4, group_size=128) does not have a parameter or a buffer named weight.

Aitrepreneur

well yeah the model you are trying to load is currently the most powerful model there is right now, you need like 80Gb+ of Vram to run it :D Try another model instead, like this one, copy and paste this in the model field: TheBloke/LLaMA2-13B-Psyfighter2-GPTQ:gptq-4bit-32g-actorder_True dl it and load it with the exllama loader

Cr Fultz

OK then...your a genius....silly tavern is up and running:)

patreon@winberry.com

Hello - what does it mean when I use a larger model (theBlock_Everyone-Coder-33B-AWQ) and all I get for answer is "==================================== etc"

Aitrepreneur

Definitely try the GPTQ format instead, too many people have issues with awq, try this: TheBloke/Everyone-Coder-33B-Base-GPTQ:gptq-4bit-128g-actorder_True copy and paste this in the model field, dl it and use the exllama loader to load it

AI AI

apparently it is so old huggy face 404 error on it

Fake Name

Hi, thanks for the 1 click installers. I can't get the openAI extension working so that SillyTavern can work. I get this : To create a public link, set `share=True` in `launch()`. Closing server running on port: 7860 16:17:47-967575 INFO Loading the extension "gallery" 16:17:47-969081 INFO Loading the extension "openai" 16:17:48-041857 INFO OpenAI-compatible API URL: http://127.0.0.1:5000 [Errno 10048] error while attempting to bind on address ('127.0.0.1', 5000): only one usage of each socket address (protocol/network address/port) is normally permitted Running on local URL: http://127.0.0.1:7860 To create a public link, set `share=True` in `launch()`.

Aitrepreneur

this error usually means you have multiple python instances running at the same time, you need to close the python processes or reboot the pc

Fake Name

I reboot and get the same error. Is there a way to change the 5000 ?

Casey Fowler

my biggest issue with this, is it's impossible to shop around for models in huggingface. No examples to view so I don't know what anything is. Is there a defacto-best option for chat model? 3080Ti and AMD 5700 fwiw

Aitrepreneur

for your GPU I recommend psyfighter 2. TheBloke/LLaMA2-13B-Psyfighter2-GPTQ:gptq-4bit-32g-actorder_True copy and paste that line in the model field, dl it and use the exllama loader

Sandro

Yo, when i open update_windows.bat it says press any key so i can start downloading it right, but it just crashes

Alucatt

It installes everything but than it says : Conda is not installed. exiting / everytime is run Windows_start.bat

Aitrepreneur

try opening a command prompt window and drag and drop the update_windows.bat file inside then press enter, this will at least give you an error and avoid the automatic crash. Make also sure your antivirus and firewall are disabled just in case

Aitrepreneur

hmm...strange, you might have a previous bad install of miniconda, since the installer is supposed to install it automatically. Try uninstalling miniconda if there is an install already and then either relaunch the installer in a new folder or install miniconda manually (https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe) make sure to check the box add miniconda to the system path environment.

Roy Greenhalgh

Two issues...first is simple. What file Launches it once it's installed? It only gives me the url after I install fresh. Second...I've imported Sandra and Tiefighter(?) llm, loaded them both in Oog and I get no responses. I get "typing" but results are blank.

Jeff Osiel

I am getting the Miniconda hook not found from the update_windowns.bat. where should miniconda install? mine went to C:\ProgramData\miniconda3 and C:\Windows\System32\text-generation-webui\installer_files\conda

Aitrepreneur

make sure that miniconda is added to path: https://eduand-alvarez.medium.com/setting-up-anaconda-on-your-windows-pc-6e39800c1afb

MemeL0rd420

everything is installed and a model is chosen but when I type to it the responses I get are all blank

Hendrik Selgrad

hey i have see your video and must try it. Im from germany so sorry when my english not perfectly is.My setup is nvd geforce 1050and i try to used this model TheBloke/DiscoLM_German_7b_v1-AWQ.But i have never do this stuff bevor so i hope you can help me. I followd your instruktions to the point by add the model in the text generation iu so all fin but when i will load the model i become this error; 16:49:59-797974 ERROR Failed to load the model. Traceback (most recent call last): File "C:\aichat\text-generation-webui\modules\ui_model_menu.py", line 242, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\aichat\text-generation-webui\modules\models.py", line 87, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\aichat\text-generation-webui\modules\models.py", line 302, in AutoAWQ_loader model = AutoAWQForCausalLM.from_quantized( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\aichat\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\auto.py", line 56, in from_quantized return AWQ_CAUSAL_LM_MODEL_MAP[model_type].from_quantized( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\aichat\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\base.py", line 208, in from_quantized load_checkpoint_and_dispatch( File "C:\aichat\text-generation-webui\installer_files\env\Lib\site-packages\accelerate\big_modeling.py", line 579, in load_checkpoint_and_dispatch load_checkpoint_in_model( File "C:\aichat\text-generation-webui\installer_files\env\Lib\site-packages\accelerate\utils\modeling.py", line 1504, in load_checkpoint_in_model raise ValueError( ValueError: At least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`.

William OBrien

I don't know squat about computers. I'm a musician. i think i did this right. i got ta place whhere the screen introduces Itself and says. something to the effect of what can I do for you or whatever. What am I supposed to do now? Like, am I in or what?

Aitrepreneur

Have you watched my installation video? If you are a complete beginner I suggest you first start there

Dungaree man

Hi there. I Have installed this successfully and have tried many different iterations of the model "gguf" that you requested us to DL,

Aitrepreneur

please don't spam multiple messages, just send me a dm with all the info, pc specs, precise error, model used, loader used, etc

GtheArchitect

when I go to run the file it boots up to ask me for 1 or 2, I press 1 and enter then in a flash it closes. It said something once but as I was reading it the window closed.

Aitrepreneur

make sure you don't have any antivirus or firewall blocking the connection, disable them both before running the file, that's usually the issue for most people when something like this happens

JD Turner

How am I to select AMD GPU if it is only for Linux/Mac Os only?

Relven

01:40:52-428876 INFO Loading the extension "gallery" 01:40:52-436265 INFO Loading the extension "silero_tts" Silero TTS cache not found at C:\Users\MPG/.cache\torch\hub. Attempting to download... Using cache found in C:\Users\MPG/.cache\torch\hub\snakers4_silero-models_master 01:41:04-376247 ERROR Failed to load the extension "silero_tts". Traceback (most recent call last): File "G:\Text2Audio\text-generation-webui\modules\extensions.py", line 46, in load_extensions extension.setup() File "G:\Text2Audio\text-generation-webui\extensions\silero_tts\script.py", line 148, in setup model = load_model() ^^^^^^^^^^^^ File "G:\Text2Audio\text-generation-webui\extensions\silero_tts\script.py", line 61, in load_model model, example_text = torch.hub.load(repo_or_dir='snakers4/silero-models', model='silero_tts', language=languages[params['language']]["lang_id"], speaker=params['model_id']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "G:\Text2Audio\text-generation-webui\installer_files\env\Lib\site-packages\torch\hub.py", line 566, in load model = _load_local(repo_or_dir, model, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "G:\Text2Audio\text-generation-webui\installer_files\env\Lib\site-packages\torch\hub.py", line 595, in _load_local model = entry(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\MPG/.cache\torch\hub\snakers4_silero-models_master\src\silero\silero.py", line 51, in silero_tts from omegaconf import OmegaConf ModuleNotFoundError: No module named 'omegaconf' Running on local URL: http://127.0.0.1:7860 Help please!

Aitrepreneur

this error means that you have tried to activate the silero extension in the session tab, disable it. 11labs was removed from the text gen webui, for this you need to use sillytavern

Relven

Is there a way to you speak recognition locally ? Silly tavern request API keys as well

NoxRaven

I used the installer and everything seemed to run ok. I can load up a model. Yet when I try to start using the program I get the following error. oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\llama\modeling_llama.py:670: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.) attn_output = torch.nn.functional.scaled_dot_product_attention(

NoxRaven

It doesn't seem to matter which model I use. I get one of two errors. Either this one: C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\llama\modeling_llama.py:670: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.) attn_output = torch.nn.functional.scaled_dot_product_attention( or this one: Traceback (most recent call last): File "C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\modules\callbacks.py", line 61, in gentask ret = self.mfunc(callback=_callback, *args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\modules\text_generation.py", line 389, in generate_with_callback shared.model.generate(**kwargs) File "C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\base.py", line 110, in generate return self.model.generate(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\transformers\generation\utils.py", line 1575, in generate result = self._sample( ^^^^^^^^^^^^^ File "C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\transformers\generation\utils.py", line 2694, in _sample model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 1250, in prepare_inputs_for_generation past_key_values = getattr(getattr(self.model.layers[0], "self_attn", {}), "past_key_value", None) ^^^^^^^^^^^^^^^^^ File "C:\Users\nolim\OneDrive\Documents\oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 1688, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'LlamaLikeModel' object has no attribute 'layers'

Aitrepreneur

First and foremost try to not install it in a onedrive folder i know it often causes issues when installing ai tools, just install it in a normal folder on your PC, also still don't know what gpu you got or what model you uses...

NoxRaven

My apologies I thought I had put the graphics card in. I'm running an RTX 4080. The models I have tried are as follows. KoboldAI/LLaMA2-13B-Tiefighter Sao10K/Fimbulvetr-11B-v2-GGUF TheBloke/MythoMax-L2-13B-GPTQ TheBloke/WizardLM-1.0-Uncensored-Llama2-13B-GPTQ No matter which model I try I get one of those two errors I previously mentioned. I've seen some things that say my graphics card could work with something higher than a 13B model but given the current errors I'm getting I didn't want to chance it.

Aitrepreneur

try this: TheBloke/LLaMA2-13B-Psyfighter2-GPTQ:gptq-4bit-32g-actorder_True copy and paste this in the model field, dl it and then load it using the exllama loader

Jason Harring

Traceback (most recent call last): File "/Users/jason/Desktop/AI/ooba/Booga/server.py", line 4, in from modules import shared File "/Users/jason/Desktop/AI/ooba/Booga/modules/shared.py", line 8, in import yaml ModuleNotFoundError: No module named 'yaml' jason@jason booga % I get this error when I am trying to install on MacBook Pro (pre apple silicon) I am going to switch to a different apple silicon mac and see if it runs better

Bryan Sproles

This is my 3rd attempt to get the 1 Click to work - I initially tried prior to joining the Patreon with the manual method, and now twice on the 1 Click. I have a brand new Alienware - i9, NVIDIA 4090 and 64gb RAM. I have Stable Diffusion XL (Forge UI) running in its own folder, and able to create good images, so the specs are fine. When I run the 1Click, it seems to work...no noticeable errors, and I can click the final URL, and have a running TextUI. I've now attempted to load a 7B, 13B and 33B model, none of which are working. One of two things happens after I type something to the AI. Either it will come back with "is typing..." and they never actually post anything, or it won't even post the "is typing..." part. I'm also not seeing the Character Gallery tab at the bottom of the page. I know I've seen it when I first attempted to install like a week ago, but I reinstalled the OS in the meantime. Any assistance would be greatly appreciated!

Aitrepreneur

Hi, have you correctly ran the mac installer? If yes then run the cmd_macos file and then in the command window type: pip install pyyaml then press enter unfortunately since I don't personally use any Mac I can't really be of any further assistance for troubleshooting

Aitrepreneur

what error do you have in the command window? Also what models have you used? (give me the url)

Bryan Sproles

I went back to the basics, and the Yarn Mistral 7B one seems to be working. I then redownloaded the The Bloke's "Everyone Coder 33B (https://huggingface.co/TheBloke/Everyone-Coder-33B-Base-AWQ) It loaded successfully, I asked one question, and got the following in the console (I'm including the loading bits so you can see where the model started.) On screen, it just showed "AI", and nothing else. By the way, I still can't see the Character tab. I have no idea how to get it to show. I thought it was automatic. Does it depend on the model you load? *** 21:30:02-187554 INFO Loading "TheBloke_Everyone-Coder-33B-Base-AWQ" Replacing layers...: 100%|███████████████████████████████████████████████████████████| 62/62 [00:05<00:00, 11.14it/s] Fusing layers...: 100%|██████████████████████████████████████████████████████████████| 62/62 [00:04<00:00, 14.06it/s] 21:30:20-349134 INFO LOADER: "AutoAWQ" 21:30:20-350127 INFO TRUNCATION LENGTH: 16384 21:30:20-351125 INFO INSTRUCTION TEMPLATE: "Custom (obtained from model metadata)" 21:30:20-351125 INFO Loaded the model in 18.16 seconds. Traceback (most recent call last): File "C:\AI\Oobabooga\text-generation-webui\modules\callbacks.py", line 61, in gentask ret = self.mfunc(callback=_callback, *args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\Oobabooga\text-generation-webui\modules\text_generation.py", line 373, in generate_with_callback shared.model.generate(**kwargs) File "C:\AI\Oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\base.py", line 110, in generate return self.model.generate(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\Oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\Oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\transformers\generation\utils.py", line 1575, in generate result = self._sample( ^^^^^^^^^^^^^ File "C:\AI\Oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\transformers\generation\utils.py", line 2694, in _sample model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\Oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 1250, in prepare_inputs_for_generation past_key_values = getattr(getattr(self.model.layers[0], "self_attn", {}), "past_key_value", None) ^^^^^^^^^^^^^^^^^ File "C:\AI\Oobabooga\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 1688, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'LlamaLikeModel' object has no attribute 'layers' Output generated in 0.31 seconds (0.00 tokens/s, 0 tokens, context 77, seed 78199476) ***  Thanks for your assistance!

Aitrepreneur

first do not use AWQ format models anymore, it's too buggy, use GTPQ format models instead, preferably the exllama compatible version. For example try this: TheBloke/LLaMA2-13B-Psyfighter2-GPTQ:gptq-4bit-32g-actorder_True copy and paste this in the model field, dl it and then load it using the exllama loader. For the "characters tab", apparently now it's not activated by default, so you need to go to the session tab and check the "gallery" extension, then apply the changes, the character gallery will then appear below the chat

Bryan Sproles

Do you think you could do an update video on this? Apparently a lot has changed since you made this one. At the time, you said AWQ was the newer format, so that's what I went with. Any suggestions for anywhere from a 20B - 33B model? I'd like to have as much choice as possible, and my computer should be able to handle it. I want to get the Oobabooga working, so I can then check your video on setting up the whole Sillytavern local chat. Thanks for everything so far! :)

Bryan Sproles

I’m going to try this model when I get home, but just wanted to let you know the other model is working well now. I was able to get the gallery up and started a convo with the built in example.

林 Xk。

I tried to load the model but it appears me an error that mentions that I don't have enough GPU. Do I have to download a smaller model??? Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:36<00:00, 18.42s/it] 09:39:49-579171 ERROR Failed to load the model. Traceback (most recent call last): File "C:\AI\WebUI\text-generation-webui\modules\ui_model_menu.py", line 245, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\WebUI\text-generation-webui\modules\models.py", line 87, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\WebUI\text-generation-webui\modules\models.py", line 168, in huggingface_loader model = model.cuda() ^^^^^^^^^^^^ File "C:\AI\WebUI\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 2548, in cuda return super().cuda(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\WebUI\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 911, in cuda return self._apply(lambda t: t.cuda(device)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\AI\WebUI\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 802, in _apply module._apply(fn) File "C:\AI\WebUI\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 802, in _apply module._apply(fn) File "C:\AI\WebUI\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 802, in _apply module._apply(fn) [Previous line repeated 2 more times] File "C:\AI\WebUI\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 825, in _apply param_applied = fn(param) ^^^^^^^^^ File "C:\AI\WebUI\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 911, in return self._apply(lambda t: t.cuda(device)) ^^^^^^^^^^^^^^ torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 6.00 GiB of which 0 bytes is free. Of the allocated memory 12.15 GiB is allocated by PyTorch, and 233.55 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Terry Martin

Everytime I try to load a model I get a connection error warning I've running a RTX3060 with 12gb ram this is the error I get Error Connection timed out in the command prompt I just have 14:23:39-464757 INFO Loading "TheBloke_LLaMA2-13B-Psyfighter2-GPTQ" Press any key to continue . . . Then it just quits Any ideas

Jason Harring

I am not sure but it looks like there is a version issue with my Mac being on a OS Version that there is no Pytorch version without using a docker container

Fizzy

Traceback (most recent call last): File "E:\Kacper2\Bored\ai\text\text-generation-webui\modules\callbacks.py", line 61, in gentask ret = self.mfunc(callback=_callback, *args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Kacper2\Bored\ai\text\text-generation-webui\modules\text_generation.py", line 373, in generate_with_callback shared.model.generate(**kwargs) File "E:\Kacper2\Bored\ai\text\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\base.py", line 110, in generate return self.model.generate(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Kacper2\Bored\ai\text\text-generation-webui\installer_files\env\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "E:\Kacper2\Bored\ai\text\text-generation-webui\installer_files\env\Lib\site-packages\transformers\generation\utils.py", line 1575, in generate result = self._sample( ^^^^^^^^^^^^^ File "E:\Kacper2\Bored\ai\text\text-generation-webui\installer_files\env\Lib\site-packages\transformers\generation\utils.py", line 2694, in _sample model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Kacper2\Bored\ai\text\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 1250, in prepare_inputs_for_generation past_key_values = getattr(getattr(self.model.layers[0], "self_attn", {}), "past_key_value", None) Hi I am getting this error message whenever I am trying to talk to any character. I have tried the default assistant

Aitrepreneur

hmm...that's strange indeed... need a bit more info, could you send a few screenshots? Also was the model downloaded correctly?

Cal

Will they be adding AMD support to windows eventually?

Fizzy

Hey I am using a nvidia 3060 This is the model link https://huggingface.co/TheBloke/LLaMA2-13B-Tiefighter-AWQ

Aitrepreneur

try this model instead: TheBloke/LLaMA2-13B-Psyfighter2-GPTQ:gptq-4bit-32g-actorder_True copy and paste this in the model field, dl it and then load it using the exllama loader

Osmar Irvin Cabantug

Hi I was able to download Git bash and also installer now there is a new folder name "text-generation-webui" whats the next step?

Aitrepreneur

You don't have to do anything, just use the installer, have you watched my install video? https://youtu.be/C-7jGYOGvy4

Wade Wilson

Hey I'm trying to run the insatller and it does everything but installing miniconda so I insatlled seperately and when I run the "start_windows.bat" it says miniconda path not found. Do you have any idea how to solve it?

Aitrepreneur

yes that means that you did not added miniconda to path when installing, uninstall miniconda and reinstall it again this time by adding it to path: https://imgur.com/5GXXyo5

Wade Wilson

Reinsalled Minicondoa this time with the "path enviroment variable" option toggled off and the installation got a little further this time "Downloading and Extracting Packages Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate C:\Users\(my username)\Documents\AI\Text\text-generation-webui\installer_files\env # # To deactivate an active environment, use # # $ conda deactivate" and afterwards same thing "Miniconda hook not found"

Anthony Marc

Is there a way to make a public URL with this? I've used a couple of your One Click tools and they usually do, but I haven't been able to figure out how to make this one do it.

Aitrepreneur

yes, you need to go to the session tab, and check the "share" checkbox, then apply the changes, after a few seconds it will give you a local url as well as a public url

Anthony Marc

Thank you for the fast reply! I didn't even see that option, so thank you very much!

xXRYBAXx

Hi Im getting this error during installation. What can I do to solve this? ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. torch==2.2.1 from https://download.pytorch.org/whl/cu121/torch-2.2.1%2Bcu121-cp311-cp311-win_amd64.whl#sha256=5dfab54d4e28797c8f4f1f9543a0a2b3d27113fa40eb202f8f49af1d865f9573: Expected sha256 5dfab54d4e28797c8f4f1f9543a0a2b3d27113fa40eb202f8f49af1d865f9573 Got af070169048302cbd260c81ad07d15977e7c674af543c02d745959b7f8d8601e

Immproceo

Hello good sir. I downloaded the .bat file and i opened it straight into the download folder? could you please tell me how can I undo this action so that I can open it in a new, empty file? Thank you and much love for your work.

Aitrepreneur

open a new command prompt window and type this pip cache purge press enter then redo the install again in a new folder

Aitrepreneur

I mean just delete the newly created text gen folder inside your download folder, put the bat installer file somewhere else on your computer then run it again

xXRYBAXx

Thanks, but I have another issue. After loading the model, I am getting this error. Traceback (most recent call last): File "C:\Users\Vic\Downloads\text-generation-webui\modules\ui_model_menu.py", line 249, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Vic\Downloads\text-generation-webui\modules\models.py", line 94, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Vic\Downloads\text-generation-webui\modules\models.py", line 297, in AutoAWQ_loader model = AutoAWQForCausalLM.from_quantized( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Vic\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\auto.py", line 94, in from_quantized return AWQ_CAUSAL_LM_MODEL_MAP[model_type].from_quantized( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Vic\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\base.py", line 440, in from_quantized self.fuse_layers(model) File "C:\Users\Vic\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\mistral.py", line 21, in fuse_layers fuser.fuse_transformer() File "C:\Users\Vic\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\awq\models\mistral.py", line 119, in fuse_transformer LlamaLikeBlock( File "C:\Users\Vic\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\awq\modules\fused\block.py", line 88, in init self.norm_1 = norm_1.to(dev) ^^^^^^^^^^^^^^ File "C:\Users\Vic\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 1152, in to return self._apply(convert) ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Vic\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 825, in _apply param_applied = fn(param) ^^^^^^^^^ File "C:\Users\Vic\Downloads\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\module.py", line 1150, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ NotImplementedError: Cannot copy out of meta tensor; no data!

xXRYBAXx

GPU - GTX 1050Ti TheBloke/Yarn-Mistral-7B-128k-AWQ (that you recommended in video) but when I tried other models I also got this error CPU - i5-9400F RAM - 32gb (just in case)

Aitrepreneur

yeah it's because you don't have enough vram to run that model. You can try this one: TheBloke/Xwin-MLewd-7B-V0.2-GPTQ:gptq-4bit-32g-actorder_True copy and paste this in the model field, dl it and then load it using the exllama loader altough with 4gb of vram it will still be very slow

xXRYBAXx

So it means that it's time to buy a new one :) Can you help me decide which would be better for AI? I'm considering between the RTX 3060 (12GB) and the RTX 4060 (8GB). Also, maybe you know some other good cards in this price range.

Aitrepreneur

Vram is king so take a gpu with as much vram as possible maybe a second hand 3090 would be best depending on where you live

NJDCFan

I get this error message below. I checked the folder and I see this file "miniconda_installer.exe" . Downloading Miniconda from https://repo.anaconda.com/miniconda/Miniconda3-py310_23.3.1-0-Windows-x86_64.exe to C:\Windows\System32\text-generation-webui\installer_files\miniconda_installer.exe % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 53.8M 100 53.8M 0 0 44.2M 0 0:00:01 0:00:01 --:--:-- 44.3M The checksum verification for miniconda_installer.exe has passed successfully. Installing Miniconda to C:\Windows\System32\text-generation-webui\installer_files\conda Miniconda version: The system cannot find the path specified. Miniconda not found. Press any key to continue . . . Update installed miniconda from the folder myself and check add the paths. I re started the Start_windows.bat file again. same error!!!

Aitrepreneur

because it looks like you are running the installer as admin, don't do that, run it normally

NJDCFan

Thanks! Here I thought it would cause less issues that way. I deleted everything and started over and it worked! Thanks!

Pipirika

Traceback (most recent call last): File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\ui_model_menu.py", line 246, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\models.py", line 94, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\models.py", line 275, in llamacpp_loader model, tokenizer = LlamaCppModel.from_pretrained(model_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\llamacpp_model.py", line 39, in from_pretrained LlamaCache = llama_cpp_lib().LlamaCache ^^^^^^^^^^^^^^^ File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\llama_cpp_python_hijack.py", line 38, in llama_cpp_lib raise Exception(f"Cannot import 'llama_cpp_cuda' because '{imported_module}' is already imported. See issue #1575 in llama-cpp-python. Please restart the server before attempting to use a different version of llama-cpp-python.") Exception: Cannot import 'llama_cpp_cuda' because 'llama_cpp' is already imported. See issue #1575 in llama-cpp-python. Please restart the server before attempting to use a different version of llama-cpp-python.

Pipirika

21:08:28-513407 INFO Starting Text generation web UI Running on local URL: http://127.0.0.1:7860 21:08:40-082150 INFO Loading "yarn-mistral-7b-128k.Q4_K_M.gguf" 21:08:40-138433 INFO llama.cpp weights detected: "models\yarn-mistral-7b-128k.Q4_K_M.gguf" 21:08:40-165972 ERROR Failed to load the model. Traceback (most recent call last): File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\ui_model_menu.py", line 246, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\models.py", line 94, in load_model output = load_func_map[loader](model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\models.py", line 275, in llamacpp_loader model, tokenizer = LlamaCppModel.from_pretrained(model_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\llamacpp_model.py", line 39, in from_pretrained LlamaCache = llama_cpp_lib().LlamaCache ^^^^^^^^^^^^^^^ File "C:\Users\Caike.IA\Desktop\IA\text-generation-webui\modules\llama_cpp_python_hijack.py", line 38, in llama_cpp_lib raise Exception(f"Cannot import 'llama_cpp_cuda' because '{imported_module}' is already imported. See issue #1575 in llama-cpp-python. Please restart the server before attempting to use a different version of llama-cpp-python.") Exception: Cannot import 'llama_cpp_cuda' because 'llama_cpp' is already imported. See issue #1575 in llama-cpp-python. Please restart the server before attempting to use a different version of llama-cpp-python.

Aitrepreneur

need a bit more info, what are your PC specs? And are you aware that you are trying to use a CPU model, also try another model as well, this model does require a lot of power to use

Peter Greene

Select your installation type: 1. Standard Windows installation 2. WSL (Windows Subsystem for Linux) Type 1 or 2 and press Enter: 1 Cloning the oobabooga text-generation-webui repository... Cloning into 'text-generation-webui'... remote: Enumerating objects: 18858, done. remote: Counting objects: 100% (492/492), done. remote: Compressing objects: 100% (218/218), done. remote: Total 18858 (delta 349), reused 383 (delta 274), pack-reused 18366 Receiving objects: 100% (18858/18858), 27.83 MiB | 9.65 MiB/s, done. Resolving deltas: 100% (13357/13357), done. This script relies on Miniconda which can not be silently installed under a path with spaces. Press any key to continue . . .

Aitrepreneur

as the error says, you are trying to install it in a folder path with spaces in the name, just don't do that, delete all spaces in your folders and try again

Charles Carter

I am new, so you might have to explain things Barny style for me. I click on the link, I start the program, and the command prompt disappears after getting to 99 now what? Thanks in advance for any assistance...