Test and Benchmark

Tests are an essential part of Feluda as they make sure all the core components like server, store and queue are working properly.

Note

Before running the tests, make sure you have Feluda setup and relevant operators installed. Follow the instructions listed here to setup Feluda locally.

Test

Important

All the test files can be located at the src/tests folder. All the tests should be run from the src folder i.e. /app folder in the docker conatiner.

Index and Search Tests as Client

The index and search tests as client are needed to make sure if a media file (image and video) are being stored into Elasticsearch properly and can be searched as well.

To run the tests:

# start the server
python server.py
# next in a new terminal
python -m unittest tests.endpoint.test_index_api_as_client

This test will store (index) media files into Elasticsearch. Now we run the test to search for these files

# start the server
python server.py
# next in a new terminal
python -m unittest tests.endpoint.test_search_api_as_client

Elasticsearch Tests

• These tests will make sure data is being indexed and stored in Elasticsearch
• The server doesn't have to be started to run these tests. Make sure you are in the src folder

ES Vec Test

python -m unittest tests.core.store.test_es_vec

ES Video Test

python -m unittest tests.core.store.test_video_es_vec

ES Audio Test

python -m unittest tests.core.store.test_audio_es_vec

Operator Tests

Image Vector Resnet

• The tests checks if the vector dimensions of an image are being extracted properly.

cd core/operators
python -m unittest test_image_vec_rep_resnet.py

Video Vector Resnet

• The tests checks if the vector dimensions of a video are being extracted properly.

cd core/operators
python -m unittest test_vid_vec_rep_resnet.py

Run all the tests inside a folder

• To run all the tests inside the tests folder
• Make sure you are in the /app folder in the docker container

# start the server
python server.py
# keep the server running, and in a new terminal
nose2 tests

Benchmark

Important

All the benchmark test files can be located at the src/benchmark folder. All the files should be run from the src folder. The benchmark tests help us assess the throughput of Feluda on AWS EC2 containers.

Video Benchmark

• The video benchmarking has a separate Dockerfile hence will require a different docker build.
• Then exec into the docker container to run the benchmark tests.

# build the new docker container
cd src
docker build --build-arg UID=1000 --build-arg GID=1000 -t testvidvec:1 -f benchmark/vidvec/Dockerfile.vid_vec_rep_resnet . 
docker run testvidvec:1
# now, keep the above running and in a new terminal
docker exec --user python -it testvidvec:1 /bin/sh
./benchmark/benchmark-video.sh

The benchmark-video.sh file will run different bench-marking tests to assess the throughput of Feluda video operators.

To run an individual benchmark file

For instance, if you just want to run the src/benchmark/vidvec/video_vec_operator_time.py file. That could be done this way:-

cd src
./benchmark/vidvec/video_vec_operator_time.sh