Skip to content

Commit

Permalink
update LICENSE and README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
ArchieMeng committed Apr 23, 2021
1 parent 98d67b5 commit b41571e
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 111 deletions.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
The MIT License (MIT)

Copyright (c) 2020 nihui
Copyright (c) 2021 ArchieMeng

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
146 changes: 36 additions & 110 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,37 @@
# RIFE ncnn Vulkan
# RIFE ncnn Vulkan Python

![CI](https://github.com/nihui/rife-ncnn-vulkan/workflows/CI/badge.svg)
## Introduction
[rife-ncnn-vulkan](https://github.com/nihui/rife-ncnn-vulkan) is nihui's ncnn implementation of Real-World Super-Resolution via Kernel Estimation and Noise Injection super resolution.

ncnn implementation of RIFE, Real-Time Intermediate Flow Estimation for Video Frame Interpolation.
rife-ncnn-vulkan-python wraps [rife-ncnn-vulkan project](https://github.com/nihui/rife-ncnn-vulkan) by SWIG to make it easier to integrate rife-ncnn-vulkan with existing python projects.

rife-ncnn-vulkan uses [ncnn project](https://github.com/Tencent/ncnn) as the universal neural network inference framework.
## Downloads

## [Download](https://github.com/nihui/rife-ncnn-vulkan/releases)
Linux/Windos/Mac X86_64 build releases are available now. However, for Linux you may need to compile it yourself if you
are using python version other than 3.8 . (The compiled python C extension library is bound to python version now)

Download Windows/Linux/MacOS Executable for Intel/AMD/Nvidia GPU
## Build

**https://github.com/nihui/rife-ncnn-vulkan/releases**
First, you have to install python, python development package (Python native development libs in Visual Studio), vulkan SDK and SWIG on your platform. And then:

This package includes all the binaries and models required. It is portable, so no CUDA or PyTorch runtime environment is needed :)
### Linux
```shell
git clone https://github.com/ArchieMeng/rife-ncnn-vulkan-python.git
cd rife-ncnn-vulkan-python
git submodule update --init --recursive
cmake -B build src
cd build
make
```

### Windows
I used Visual Studio 2019 and msvc v142 to build this project for Windows.

Install visual studio and open the project directory, and build. Job done.

The only problem on Windows is that, you cannot use [CMake for Windows](https://cmake.org/download/) to generate the Visual Studio solution file and build it. This will make the lib crash on loading.

The only way is [use Visual Studio to open the project as directory](https://www.microfocus.com/documentation/visual-cobol/vc50/VS2019/GUID-BE1C48AA-DB22-4F38-9644-E9B48658EF36.html), and build it from Visual Studio.

## About RIFE

Expand All @@ -25,112 +44,26 @@ Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchan
https://rife-vfi.github.io

https://arxiv.org/abs/2011.06294

## Usages

Input two frame images, output one interpolated frame image.

### Example Commands

```shell
./rife-ncnn-vulkan -0 0.jpg -1 1.jpg -o 01.jpg
./rife-ncnn-vulkan -i input_frames/ -o output_frames/
```

Example below runs on CPU, Discrete GPU, and Integrated GPU all at the same time. Uses 2 threads for image decoding, 4 threads for one CPU worker, 4 threads for another CPU worker, 2 threads for discrete GPU, 1 thread for integrated GPU, and 4 threads for image encoding.
```shell
./rife-ncnn-vulkan -i input_frames/ -o output_frames/ -g -1,-1,0,1 -j 2:4,4,2,1:4
```

### Video Interpolation with FFmpeg

```shell
mkdir input_frames
mkdir output_frames

# find the source fps and format with ffprobe, for example 24fps, AAC
ffprobe input.mp4
### Example Program

# extract audio
ffmpeg -i input.mp4 -vn -acodec copy audio.m4a
```Python
from PIL import Image
from rife_ncnn_vulkan import RIFE

# decode all frames
ffmpeg -i input.mp4 input_frames/frame_%08d.png

# interpolate 2x frame count
./rife-ncnn-vulkan -i input_frames -o output_frames

# encode interpolated frames in 48fps with audio
ffmpeg -framerate 48 -i output_frames/%08d.png -i audio.m4a -c:a copy -crf 20 -c:v libx264 -pix_fmt yuv420p output.mp4
```

### Full Usages

```console
Usage: rife-ncnn-vulkan -0 infile -1 infile1 -o outfile [options]...
rife-ncnn-vulkan -i indir -o outdir [options]...

-h show this help
-v verbose output
-0 input0-path input image0 path (jpg/png/webp)
-1 input1-path input image1 path (jpg/png/webp)
-i input-path input image directory (jpg/png/webp)
-o output-path output image path (jpg/png/webp) or directory
-m model-path rife model path (default=rife-HD)
-g gpu-id gpu device to use (-1=cpu, default=auto) can be 0,1,2 for multi-gpu
-j load:proc:save thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
-x enable tta mode
-u enable UHD mode
-f pattern-format output image filename pattern format (%08d.jpg/png/webp, default=ext/%08d.png)
im0, im1 = Image.open("0.png"), Image.open("1.png")
rife = RIFE(0)
im = rife.process(im0, im1)
im.save("interframe.png")
```

- `input0-path`, `input1-path` and `output-path` accept file path
- `input-path` and `output-path` accept file directory
- `load:proc:save` = thread count for the three stages (image decoding + rife interpolation + image encoding), using larger values may increase GPU usage and consume more GPU memory. You can tune this configuration with "4:4:4" for many small-size images, and "2:2:2" for large-size images. The default setting usually works fine for most situations. If you find that your GPU is hungry, try increasing thread count to achieve faster processing.
- `pattern-format` = the filename pattern and format of the image to be output, png is better supported, however webp generally yields smaller file sizes, both are losslessly encoded

If you encounter a crash or error, try upgrading your GPU driver:

- Intel: https://downloadcenter.intel.com/product/80939/Graphics-Drivers
- AMD: https://www.amd.com/en/support
- NVIDIA: https://www.nvidia.com/Download/index.aspx

## Build from Source

1. Download and setup the Vulkan SDK from https://vulkan.lunarg.com/
- For Linux distributions, you can either get the essential build requirements from package manager
```shell
dnf install vulkan-headers vulkan-loader-devel
```
```shell
apt-get install libvulkan-dev
```
```shell
pacman -S vulkan-headers vulkan-icd-loader
```

2. Clone this project with all submodules

```shell
git clone https://github.com/nihui/rife-ncnn-vulkan.git
cd rife-ncnn-vulkan
git submodule update --init --recursive
```

3. Build with CMake
- You can pass -DUSE_STATIC_MOLTENVK=ON option to avoid linking the vulkan loader library on MacOS

```shell
mkdir build
cd build
cmake ../src
cmake --build . -j 4
```

### TODO

* test-time temporal augmentation aka TTA-t

### Model

| model | upstream version |
Expand All @@ -152,18 +85,10 @@ cmake --build . -j 4

### Interpolate with rife rife-anime model

```shell
rife-ncnn-vulkan.exe -m models/rife-anime -0 0.png -1 1.png -o out.png
```

![rife](images/out.png)

### Interpolate with rife rife-anime model + TTA-s

```shell
rife-ncnn-vulkan.exe -m models/rife-anime -x -0 0.png -1 1.png -o out.png
```

![rife](images/outx.png)

## Original RIFE Project
Expand All @@ -176,3 +101,4 @@ rife-ncnn-vulkan.exe -m models/rife-anime -x -0 0.png -1 1.png -o out.png
- https://github.com/webmproject/libwebp for encoding and decoding Webp images on ALL PLATFORMS
- https://github.com/nothings/stb for decoding and encoding image on Linux / MacOS
- https://github.com/tronkko/dirent for listing files in directory on Windows
- https://github.com/nihui/rife-ncnn-vulkan the original rife-ncnn-vulkan project

0 comments on commit b41571e

Please sign in to comment.