Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

router test failing with TimeoutExpired from mb command #240

Open
dry923 opened this issue Sep 10, 2021 · 0 comments
Open

router test failing with TimeoutExpired from mb command #240

dry923 opened this issue Sep 10, 2021 · 0 comments

Comments

@dry923
Copy link
Member

dry923 commented Sep 10, 2021

When running the router test it has been periodically failing with the following error:

...
Indexing documents in router-test-results
10-09-2021T18:56:46 Sleeping for 6s before next test
10-09-2021T18:56:52 Generating config for termination passthrough with 200 clients 0 keep alive requests and path /1024.html
10-09-2021T18:56:53 Copying mb config http-scale-passthrough.json to pod http-scale-client-6fc5db9645-9lpld
10-09-2021T18:56:55 Executing sample 1/1 using termination passthrough with 200 clients and 0 keepalive requests
Executing 'mb -i /tmp/http-scale-passthrough.json -d 1 -o /tmp/results.csv'
Traceback (most recent call last):
  File "/usr/lib64/python3.6/subprocess.py", line 425, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib64/python3.6/subprocess.py", line 863, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib64/python3.6/subprocess.py", line 1535, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib64/python3.6/subprocess.py", line 891, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command 'mb -i /tmp/http-scale-passthrough.json -d 1 -o /tmp/results.csv' timed out after 5 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workload/workload.py", line 92, in <module>
    exit(main())
  File "/workload/workload.py", line 66, in main
    result_codes, p95_latency, p99_latency, avg_latency = run_mb(args.mb_config, args.runtime, args.output)
  File "/workload/workload.py", line 35, in run_mb
    timeout=int(runtime) * 5)
  File "/usr/lib64/python3.6/subprocess.py", line 430, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command 'mb -i /tmp/http-scale-passthrough.json -d 1 -o /tmp/results.csv' timed out after 5 seconds
command terminated with exit code 1

Example env.sh

# General
export KUBECONFIG=/root/gcp/gcp_kube_3
export UUID=$(uuidgen)

# ES configuration
export ES_SERVER=ES_SERVER
export ES_INDEX=${ES_INDEX:-router-test-results}
export ES_SERVER_BASELINE=ES_SERVER_BASELINE

# Gold comparison
COMPARE_WITH_GOLD="false"
ES_GOLD=${ES_GOLD:-${ES_SERVER}}
GOLD_SDN=${GOLD_SDN:-openshiftsdn}
GOLD_OCP_VERSION=4.8

# Environment setup
NUM_NODES=$(oc get node -l node-role.kubernetes.io/worker --no-headers | grep -cw Ready)
ENGINE=${ENGINE:-podman}
KUBE_BURNER_RELEASE_URL=${KUBE_BURNER_RELEASE_URL:-https://github.com/cloud-bulldozer/kube-burner/releases/download/v0.11/kube-burner-0.11-Linux-x86_64.tar.gz}
KUBE_BURNER_IMAGE=quay.io/cloud-bulldozer/kube-burner:latest
TERMINATIONS=${TERMINATIONS:-"http edge passthrough reencrypt mix"}
INFRA_TEMPLATE=http-perf.yml.tmpl
INFRA_CONFIG=http-perf.yml
export SERVICE_TYPE=${SERVICE_TYPE:-NodePort}
export NUMBER_OF_ROUTERS=${NUMBER_OF_ROUTERS:-1}
#export NUMBER_OF_ROUTERS=${NUMBER_OF_ROUTERS:-2}
export HOST_NETWORK=${HOST_NETWORK:-true}
export NODE_SELECTOR=${NODE_SELECTOR:-'{node-role.kubernetes.io/workload: }'}

# Benchmark configuration
#RUNTIME=${RUNTIME:-60}
#TLS_REUSE=${TLS_REUSE:-true}
#URL_PATH=${URL_PATH:-/1024.html}
#SAMPLES=${SAMPLES:-2}
#QUIET_PERIOD=${QUIET_PERIOD:-60s}
#KEEPALIVE_REQUESTS=${KEEPALIVE_REQUESTS:-"0 1 50"}
# Benchmark configuration
RUNTIME=${RUNTIME:-1}
TLS_REUSE=${TLS_REUSE:-true}
URL_PATH=${URL_PATH:-/1024.html}
SAMPLES=${SAMPLES:-1}
QUIET_PERIOD=${QUIET_PERIOD:-6s}
KEEPALIVE_REQUESTS=${KEEPALIVE_REQUESTS:-"0"}

# Comparison and csv generation
THROUGHPUT_TOLERANCE=${THROUGHPUT_TOLERANCE:-5}
LATENCY_TOLERANCE=${LATENCY_TOLERANCE:-5}
PREFIX=${PREFIX:-$(oc get clusterversion version -o jsonpath="{.status.desired.version}")}
LARGE_SCALE_THRESHOLD=${LARGE_SCALE_THRESHOLD:-24}
METADATA_COLLECTION=${METADATA_COLLECTION:-true}
SMALL_SCALE_BASELINE_UUID="29d520a2-039a-4a1e-b139-83fe2e63fda1"
LARGE_SCALE_BASELINE_UUID="9df8255d-2038-42ed-869d-f748f671da07"
GSHEET_KEY_LOCATION=/root/gcp/gsheet
EMAIL_ID_FOR_RESULTS_SHEET="[email protected]"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant