Importing Yi-9B to Ollama
The log of importing Yi-9B LLM model to Ollama library.

Get The Yi-9B For Your Ollama

Here is the model:

Or just run it from terminal:

ollama run shinyzhu/yayi:9b --verbose

And you can run Yi-9B-200K too:

ollama run shinyzhu/yayi:9b-200k --verbose

Why I Do This

If you like to run LLM models locally with Ollama like I do.


You discovered a better model but it’s not published in the Ollama library.

Then you can import the model to Ollama by following the import doc.

This post is the log of the importing steps on my machine. Let me show you.


First of all. You’d check what type of the model is. Here shows the Yi-9B model importing. It’s published as safetensors on HuggingFace. So we’ll follow the Importing (PyTorch & Safetensors) guide.

According to the doc. You need a dev machine to do the python and make jobs.

Here I use a vm in my NAS which has i3-N300 CPU and total 16G RAM. The vm configuration is 4 CPU cores, 8 GB RAM and 64GB storage. I just hoped it work. But the importing job is just some building and converting tasks. It ran at an acceptable rate.

The OS is Ubuntu 22.04. You need to install these packages first:

sudo apt install python3.10-venv make gcc g++

And install Git LFS.

sudo ./

Or you can install them when you are stuck anytime. Actually that’s what I did.

Now we are ready to go.

Setup Ollama & Llama.cpp

I’d show the shell output here.

shiny@ubuntuinr2:~/llmbuild$ git clone ollama
Cloning into 'ollama'...
remote: Enumerating objects: 12866, done.
remote: Counting objects: 100% (1143/1143), done.
remote: Compressing objects: 100% (545/545), done.
remote: Total 12866 (delta 614), reused 1014 (delta 530), pack-reused 11723
Receiving objects: 100% (12866/12866), 7.64 MiB | 5.20 MiB/s, done.
Resolving deltas: 100% (8001/8001), done.

shiny@ubuntuinr2:~/llmbuild$ cd ollama/
shiny@ubuntuinr2:~/llmbuild/ollama$ git submodule init
Submodule 'llama.cpp' ( registered for path 'llm/llama.cpp'

shiny@ubuntuinr2:~/llmbuild/ollama$ git submodule update llm/llama.cpp
remote: Enumerating objects: 14257, done.
remote: Counting objects: 100% (14257/14257), done.
remote: Compressing objects: 100% (3902/3902), done.
remote: Total 13898 (delta 10363), reused 13382 (delta 9882), pack-reused 0
Receiving objects: 100% (13898/13898), 11.01 MiB | 5.65 MiB/s, done.
Resolving deltas: 100% (10363/10363), completed with 287 local objects.
 * branch            37e7854c104301c5b5323ccc40e07699f3a62c3e -> FETCH_HEAD
Submodule path 'llm/llama.cpp': checked out '37e7854c104301c5b5323ccc40e07699f3a62c3e'

shiny@ubuntuinr2:~/llmbuild/ollama$ python3 -m venv llm/llama.cpp/.venv
shiny@ubuntuinr2:~/llmbuild/ollama$ source llm/llama.cpp/.venv/bin/activate
(.venv) shiny@ubuntuinr2:~/llmbuild/ollama$ pip install -r llm/llama.cpp/requirements.txt

Successfully installed MarkupSafe-2.1.5 certifi-2024.2.2 charset-normalizer-3.3.2 einops-0.7.0 filelock-3.13.3 fsspec-2024.3.1 gguf-0.6.0 huggingface-hub-0.22.2 idna-3.6 jinja2-3.1.3 mpmath-1.3.0 networkx-3.3 numpy-1.24.4 nvidia-cublas-cu12- nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12- nvidia-cufft-cu12- nvidia-curand-cu12- nvidia-cusolver-cu12- nvidia-cusparse-cu12- nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.1.105 packaging-24.0 protobuf-4.25.3 pyyaml-6.0.1 regex-2023.12.25 requests-2.31.0 safetensors-0.4.2 sentencepiece-0.1.99 sympy-1.12 tokenizers-0.15.2 torch-2.1.2 tqdm-4.66.2 transformers-4.39.3 triton-2.1.0 typing-extensions-4.11.0 urllib3-2.2.1

(.venv) shiny@ubuntuinr2:~/llmbuild/ollama$ make -C llm/llama.cpp quantize
make: Entering directory '/home/shiny/llmbuild/ollama/llm/llama.cpp'
I ccache not found. Consider installing it for faster compilation.
I llama.cpp build info: 
I UNAME_S:   Linux
I UNAME_P:   x86_64
I UNAME_M:   x86_64
I CFLAGS:    -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion 
I CXXFLAGS:  -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG 
I NVCCFLAGS: -std=c++11 -O3 
I CC:        cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
I CXX:       g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

g++ -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -c common/build-info.cpp -o build-info.o
cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml.c -o ggml.o
g++ -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -c llama.cpp -o llama.o
cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml-alloc.c -o ggml-alloc.o
cc  -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion    -c ggml-backend.c -o ggml-backend.o
cc -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -std=c11   -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -march=native -mtune=native -Wdouble-promotion     -c ggml-quants.c -o ggml-quants.o
g++ -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -c unicode.cpp -o unicode.o
g++ -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -c unicode-data.cpp -o unicode-data.o
g++ -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  -c examples/quantize/quantize.cpp -o examples/quantize/quantize.o
g++ -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread  -march=native -mtune=native -Wno-array-bounds -Wno-format-truncation -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_GNU_SOURCE -DNDEBUG  build-info.o ggml.o llama.o ggml-alloc.o ggml-backend.o ggml-quants.o unicode.o unicode-data.o examples/quantize/quantize.o -o quantize  
make: Leaving directory '/home/shiny/llmbuild/ollama/llm/llama.cpp'

See? You’d follow the Ollama doc. Haha.

Clone Model: Yi-9B

I failed to clone from HuggingFace. Changed to

(.venv) shiny@ubuntuinr2:/data/llmbuild/ollama$ git clone model
Cloning into 'model'...
remote: Enumerating objects: 105, done.
remote: Counting objects: 100% (105/105), done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 105 (delta 56), reused 90 (delta 44), pack-reused 0
Receiving objects: 100% (105/105), 1.04 MiB | 5.79 MiB/s, done.
Resolving deltas: 100% (56/56), done.
Filtering content: 100% (3/3), 4.44 GiB | 9.70 MiB/s, done.

Convert & Quantize

(.venv) shiny@ubuntuinr2:/data/llmbuild/ollama$ python llm/llama.cpp/ ./model --outtype f16 --outfile converted.bin
Loading model file model/model-00001-of-00002.safetensors
Loading model file model/model-00001-of-00002.safetensors
Loading model file model/model-00002-of-00002.safetensors
params = Params(n_vocab=64000, n_embd=4096, n_layer=48, n_ctx=4096, n_ff=11008, n_head=32, n_head_kv=4, n_experts=None, n_experts_used=None, f_norm_eps=1e-06, rope_scaling_type=None, f_rope_freq_base=10000, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=<GGMLFileType.MostlyF16: 1>, path_model=PosixPath('model'))
Loaded vocab file PosixPath('model/tokenizer.model'), type 'spm'
Vocab info: <SentencePieceVocab with 64000 base tokens and 0 added tokens>
Special vocab info: <SpecialVocab with 0 merges, special tokens {'bos': 1, 'eos': 2, 'unk': 0, 'pad': 0}, add special tokens {'bos': False, 'eos': False}>
Permuting layer 0

Wrote converted.bin
(.venv) shiny@ubuntuinr2:/data/llmbuild/ollama$ llm/llama.cpp/quantize converted.bin quantized.bin q4_0
main: build = 2581 (37e7854)
main: built with  for unknown
main: quantizing 'converted.bin' to 'quantized.bin' as Q4_0
llama_model_loader: loaded meta data with 23 key-value pairs and 435 tensors from converted.bin (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama

[ 435/ 435]                   output_norm.weight - [ 4096,     1,     1,     1], type =    f32, size =    0.016 MB
llama_model_quantize_internal: model size  = 16841.52 MB
llama_model_quantize_internal: quant size  =  4802.22 MB

main: quantize time = 313916.31 ms
main:    total time = 313916.31 ms

Then you’ll get a file quantized.bin. Transfer it to local machine (a MacBook Pro in my case) if you like.

Create Your Model

Create a Modelfile alongside the quantized.bin.

FROM quantized.bin
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
{{ .Prompt }}<|im_end|>
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
You are a helpful and powerful assistant. Respond to user's input carefully.

Then run ollama to create a model:

yi9b % ollama create yayi:9b -f Modelfile 
transferring model data 
creating model layer 
creating template layer 
creating parameters layer 
creating config layer 
using already created layer sha256:0def2b0dd002d747de88ba0b8f396d4d4cb4fcc4daa71ab34bd68834f09b4734 
using already created layer sha256:a47b02e00552cd7022ea700b1abf8c572bb26c9bc8c1a37e01b566f2344df5dc 
using already created layer sha256:f02dd72bb2423204352eabc5637b44d79d17f109fdb510a7c51455892aa2d216 
writing layer sha256:85263e68fb902f60df147b8cf5984fdc76e84fb5783b4a85591ef7fb0a498e6f 
writing manifest 

Using The Model

Now you can use the yayi:9b model like any other models you have.


[Updated] Update The Model

After about 2 weeks, there are many new commits of the model and Ollama as well. So I decide to re create the model. Here is what I do.

Update Ollama and llama.cpp git repo with these commands:

shiny@ubuntuinr2:/data/llmbuild/ollama$ git pull
shiny@ubuntuinr2:/data/llmbuild/ollama$ git submodule update --remote --merge

Remove “old” files.

shiny@ubuntuinr2:/data/llmbuild/ollama$ rm -f converted.bin
shiny@ubuntuinr2:/data/llmbuild/ollama$ rm -f quantized.bin

Update model repo:

shiny@ubuntuinr2:/data/llmbuild/ollama$ cd model/
shiny@ubuntuinr2:/data/llmbuild/ollama/model$ git pull
shiny@ubuntuinr2:/data/llmbuild/ollama$ cd ..

Then you are ready to build it again.

shiny@ubuntuinr2:/data/llmbuild/ollama$ source llm/llama.cpp/.venv/bin/activate
(.venv) shiny@ubuntuinr2:/data/llmbuild/ollama$ python llm/llama.cpp/ ./model --outtype f16 --outfile converted.bin
(.venv) shiny@ubuntuinr2:/data/llmbuild/ollama$ llm/llama.cpp/quantize converted.bin quantized.bin q4_0

Create the model with Ollama and push to library, and TADA!

Fun Facts

Here are some funny points I learned.

The compression of Git LFS is SUPER!

If you have a look at the sizes of the tensors. They are about 18 GB in total with 2 files!

-rw-rw-r--  1 shiny shiny  9.3G Apr  7 09:49 model-00001-of-00002.safetensors
-rw-rw-r--  1 shiny shiny  7.2G Apr  7 09:47 model-00002-of-00002.safetensors

But. Did you see this:

Filtering content: 100% (3/3), 4.44 GiB | 9.70 MiB/s, done.

What? Just 4.44 GB? and DONE?

I believe it benefited from the LFS.

The outputs are really HUGE

Sizes matter again.

shiny@ubuntuinr2:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         63G   59G  444M 100% /data

OK. It’s full of LLM model. :D

shiny@ubuntuinr2:/data/llmbuild/ollama$ du -h -d 1
33G     ./model
4.8G    ./llm
59G     .
4.7G    llm/llama.cpp/.venv

It’s ok for a build machine.

-rw-rw-r--  1 shiny shiny  17G Apr  7 10:06 converted.bin
-rw-rw-r--  1 shiny shiny 4.7G Apr  7 10:11 quantized.bin

The final model size is 5 GB.

Python venv Error?

If you changed your path like moved to another volume of the .venv , you need to re create it.

Questions and Further Doings

See what’s wrong with the responses:


Here is the first edition of the Modelfile which I followed the doc.

FROM quantized.bin
TEMPLATE "[INST] {{ .Prompt }} [/INST]"

So I changed to the template that the official Yi models use.

I also added a system prompt to it.

Question here:

How can I make a base model like Yi-9B to be a chat model with less efforts?

It seems doesn’t work correctly. e.g. this.

Question here:

How can I measure or explain how the model works?


Any feedback would be greatly appreciated.

Don’t forget trying it out:

Last modified on April 8, 2024