Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes Error "Request Entity Too Large" #585

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ class WaveContainerRecord {
/**
* The conda file associated with the request
*/
final String condaFile
String condaFile
pditommaso marked this conversation as resolved.
Show resolved Hide resolved

/**
* The container arch platform
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ class SurrealPersistenceService implements PersistenceService {

@Override
void saveBuild(WaveBuildRecord build) {
build.condaFile = truncateLargeCondaFile(build.condaFile)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think can also happen for the docker file in principle

Copy link
Member Author

@munishchouhan munishchouhan Aug 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok will add it too
I think when condafile is present continerfile will be our template so it wont exceed size, but when we only have dockerfile, size can be bigger

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

surrealDb.insertBuildAsync(getAuthorization(), build).subscribe({ result->
log.trace "Build request with id '$build.buildId' saved record: ${result}"
}, {error->
Expand Down Expand Up @@ -148,6 +149,7 @@ class SurrealPersistenceService implements PersistenceService {

@Override
void saveContainerRequest(String token, WaveContainerRecord data) {
data.condaFile = truncateLargeCondaFile(data.condaFile)
surrealDb.insertContainerRequestAsync(authorization, token, data).subscribe({ result->
log.trace "Container request with token '$token' saved record: ${result}"
}, {error->
Expand Down Expand Up @@ -234,4 +236,17 @@ class SurrealPersistenceService implements PersistenceService {
return result
}

/**
* Surreal allows 16KB as maximum size for a payload
* This method will truncate the payload if it exceeds 14 KB
* Truncate the payload if it exceeds the maximum size
*/
protected static truncateLargeCondaFile(String condafile) {
int maxSize = 14 * 1024 //14 KB for the file and 2 KB for the rest of the fields
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make a config option for this as usual

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if( condafile && condafile.length() > maxSize )
return condafile.substring(0, maxSize) + "\n[content truncated]"
else
return condafile

}
}
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
package io.seqera.wave.service.persistence.impl

import spock.lang.Specification
import spock.lang.Unroll

import java.nio.file.Path
import java.time.Duration
Expand Down Expand Up @@ -293,4 +294,17 @@ class SurrealPersistenceServiceTest extends Specification implements SurrealDBTe
result2 == scanRecord2
}

@Unroll
void "truncateLargeCondaFile should return correct size conda file"() {
expect:
SurrealPersistenceService.truncateLargeCondaFile(CONDA_FILE) == RESULT

where:
CONDA_FILE | RESULT
null | null
"" | ""
"a" * (14 * 1024) | "a" * (14 * 1024)
"a" * (15 * 1024) | "a" * (14 * 1024)
}

}
Loading