-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PRO-937: Add create
and insert
table endpoints to sdk
#114
Changes from 12 commits
3853295
f97dc83
fd95ef8
dd6d30d
6a3a1c5
2f14b9a
3338e36
eaa0fdb
8c7f8e6
6130465
d5fe607
e3f6c98
0c1c14d
7fdbfa5
4e11311
642cc49
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,6 +19,7 @@ | |
) | ||
from dune_client.api.execution import ExecutionAPI | ||
from dune_client.api.query import QueryAPI | ||
from dune_client.api.table import TableAPI | ||
from dune_client.models import ( | ||
ResultsResponse, | ||
DuneError, | ||
|
@@ -36,7 +37,7 @@ | |
POLL_FREQUENCY_SECONDS = 1 | ||
|
||
|
||
class ExtendedAPI(ExecutionAPI, QueryAPI): | ||
bh2smith marked this conversation as resolved.
Show resolved
Hide resolved
|
||
class ExtendedAPI(ExecutionAPI, QueryAPI, TableAPI): | ||
""" | ||
Provides higher level helper methods for faster | ||
and easier development on top of the base ExecutionAPI. | ||
|
@@ -316,40 +317,6 @@ def download_csv( | |
), | ||
) | ||
|
||
############################ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think some people might still be depending on this. We might need to keep it for backwards compatibility There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Experience for users would not change |
||
# Plus Subscription Features | ||
############################ | ||
def upload_csv( | ||
self, | ||
table_name: str, | ||
data: str, | ||
description: str = "", | ||
is_private: bool = False, | ||
) -> bool: | ||
""" | ||
https://docs.dune.com/api-reference/tables/endpoint/upload | ||
The write API allows you to upload any .csv file into Dune. The only limitations are: | ||
|
||
- File has to be < 200 MB | ||
- Column names in the table can't start with a special character or digits. | ||
- Private uploads require a Plus subscription. | ||
|
||
Below are the specifics of how to work with the API. | ||
""" | ||
response_json = self._post( | ||
route="/table/upload/csv", | ||
params={ | ||
"table_name": table_name, | ||
"description": description, | ||
"data": data, | ||
"is_private": is_private, | ||
}, | ||
) | ||
try: | ||
return bool(response_json["success"]) | ||
except KeyError as err: | ||
raise DuneError(response_json, "UploadCsvResponse", err) from err | ||
|
||
############################################################################################## | ||
# Plus Features: these features use APIs that are only available on paid subscription plans | ||
############################################################################################## | ||
|
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,98 @@ | ||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
Table API endpoints enables users to | ||||||||||||||||||||||||||||||||||||||||
create and insert data into Dune. | ||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
from __future__ import annotations | ||||||||||||||||||||||||||||||||||||||||
from typing import List, Dict, Any, IO | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
from dune_client.api.base import BaseRouter | ||||||||||||||||||||||||||||||||||||||||
from dune_client.models import DuneError | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
class TableAPI(BaseRouter): | ||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
Implementation of Table endpoints - Plus subscription only | ||||||||||||||||||||||||||||||||||||||||
https://docs.dune.com/api-reference/tables/ | ||||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note: There is no landing page for the Tables API. This url redirects to https://docs.dune.com/api-reference/tables/endpoint/create and there only appears to be one direct link in the docs page (inside a comment on uploadCSV) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, you are correct. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Great callout! It should probably be an "Overview" or similar like with the other endpoint groups. |
||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
def upload_csv( | ||||||||||||||||||||||||||||||||||||||||
self, | ||||||||||||||||||||||||||||||||||||||||
table_name: str, | ||||||||||||||||||||||||||||||||||||||||
data: str, | ||||||||||||||||||||||||||||||||||||||||
description: str = "", | ||||||||||||||||||||||||||||||||||||||||
is_private: bool = False, | ||||||||||||||||||||||||||||||||||||||||
) -> bool: | ||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
https://docs.dune.com/api-reference/tables/endpoint/upload | ||||||||||||||||||||||||||||||||||||||||
This endpoint allows you to upload any .csv file into Dune. The only limitations are: | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
- File has to be < 200 MB | ||||||||||||||||||||||||||||||||||||||||
- Column names in the table can't start with a special character or digits. | ||||||||||||||||||||||||||||||||||||||||
- Private uploads require a Plus subscription. | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
Below are the specifics of how to work with the API. | ||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
response_json = self._post( | ||||||||||||||||||||||||||||||||||||||||
route="/table/upload/csv", | ||||||||||||||||||||||||||||||||||||||||
params={ | ||||||||||||||||||||||||||||||||||||||||
"table_name": table_name, | ||||||||||||||||||||||||||||||||||||||||
"description": description, | ||||||||||||||||||||||||||||||||||||||||
"data": data, | ||||||||||||||||||||||||||||||||||||||||
"is_private": is_private, | ||||||||||||||||||||||||||||||||||||||||
}, | ||||||||||||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||||||||||||
try: | ||||||||||||||||||||||||||||||||||||||||
return bool(response_json["success"]) | ||||||||||||||||||||||||||||||||||||||||
except KeyError as err: | ||||||||||||||||||||||||||||||||||||||||
raise DuneError(response_json, "UploadCsvResponse", err) from err | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
def create_table( | ||||||||||||||||||||||||||||||||||||||||
self, | ||||||||||||||||||||||||||||||||||||||||
namespace: str, | ||||||||||||||||||||||||||||||||||||||||
table_name: str, | ||||||||||||||||||||||||||||||||||||||||
schema: List[Dict[str, str]], | ||||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. At some point we probably want to create an enum of the possible types, but not yet! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you happen to have a list of all the possible types? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yup they're defined here |
||||||||||||||||||||||||||||||||||||||||
description: str = "", | ||||||||||||||||||||||||||||||||||||||||
is_private: bool = False, | ||||||||||||||||||||||||||||||||||||||||
) -> Any: | ||||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That is what There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If so, should this change now? 😄 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am trying to say that
But we should already have a response returned from the API, right ?
The fact that If not an easy change, let's keep it like that. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree that it was probably not the best decision. dune-client/dune_client/api/base.py Lines 139 to 150 in e3f6c98
Should have just returned the And as such, I don't think is worth it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I take this back, this was not as simple as I thought it would be. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks 🖖 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry folks, this was my fault dune-client/dune_client/api/execution.py Lines 66 to 72 in b9db190
Here would have been the same thing. Since I am currently implementing these API routes in the Node project, I will submit a corresponding PR for this one. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here is the related PR: #117 |
||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
https://docs.dune.com/api-reference/tables/endpoint/create | ||||||||||||||||||||||||||||||||||||||||
The create table endpoint allows you to create an empty table with a specifc schema in Dune. | ||||||||||||||||||||||||||||||||||||||||
waddahAldrobi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
The only limitations are: | ||||||||||||||||||||||||||||||||||||||||
- If a table already exists with the same name, the request will fail. | ||||||||||||||||||||||||||||||||||||||||
- Column names in the table can’t start with a special character or a digit. | ||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
return self._post( | ||||||||||||||||||||||||||||||||||||||||
route="/table/create", | ||||||||||||||||||||||||||||||||||||||||
params={ | ||||||||||||||||||||||||||||||||||||||||
"namespace": namespace, | ||||||||||||||||||||||||||||||||||||||||
"table_name": table_name, | ||||||||||||||||||||||||||||||||||||||||
"schema": schema, | ||||||||||||||||||||||||||||||||||||||||
"description": description, | ||||||||||||||||||||||||||||||||||||||||
"is_private": is_private, | ||||||||||||||||||||||||||||||||||||||||
}, | ||||||||||||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
def insert_table( | ||||||||||||||||||||||||||||||||||||||||
self, | ||||||||||||||||||||||||||||||||||||||||
namespace: str, | ||||||||||||||||||||||||||||||||||||||||
table_name: str, | ||||||||||||||||||||||||||||||||||||||||
data: IO[bytes], | ||||||||||||||||||||||||||||||||||||||||
content_type: str, | ||||||||||||||||||||||||||||||||||||||||
) -> Any: | ||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
https://docs.dune.com/api-reference/tables/endpoint/insert | ||||||||||||||||||||||||||||||||||||||||
The insert table endpoint allows you to insert data into an existing table in Dune. | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
The only limitations are: | ||||||||||||||||||||||||||||||||||||||||
- The file has to be in json or csv format | ||||||||||||||||||||||||||||||||||||||||
- The file has to have the same schema as the table | ||||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||
return self._post( | ||||||||||||||||||||||||||||||||||||||||
route=f"/table/{namespace}/{table_name}/insert", | ||||||||||||||||||||||||||||||||||||||||
headers={"Content-Type": content_type}, | ||||||||||||||||||||||||||||||||||||||||
data=data, | ||||||||||||||||||||||||||||||||||||||||
) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
date,dgs10 | ||
2020-12-01T23:33:00,10 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{"date":"2020-12-01T23:33:00","dgs10":10} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mutable default arguments is indeed dangerous in Python. So please remove the pylint thing and change: This will only be initialized once, and any changes to it will be durable. See https://docs.python-guide.org/writing/gotchas/
So we need to change it to
Optional[Dict[str, str]] = None
and then handle it in the body of the methodThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was done in response to this.
#114 (comment)
But I can change it back
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please do!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done!