Database Adapter - TIMES PanEU

[Ludee]

• reeem_db_setup_times_paneu.sql
• reeem_adapter_times_paneu.py
• reeem_io.py
• reeem_db_plot_TIMESPanEU.ipynb
• Feedback from modeller

modeller: @pkork

[Ludee]

Recent problems:

• Inconsistent file names (Pilot)
• Inconsistent years (2010 & 2015)
  -> 2010 Input & 2015 Output
• Inconsistent use of columns "field" & "category" & "source"
• EU28 not importable

[Ludee]

• Rename SP-folder from TIMES PanEU to TIMESPanEU

• Adjusted file name (missing underscore after date)
  2017-11-15-StorageInnov_TIMESPanEU_FrameworkV1_DataV1_Output.xlsx
  2017-11-15_StorageInnov_TIMESPanEU_FrameworkV1_DataV1_Output.xlsx

[Ludee]

• File 2017-11-15_StorageInnov_TIMESPanEU_FrameworkV1_DataV1_Input.xlsx
  Has major problems in columns Field, Category, Aggregation, Source!

[Ludee]

table	pathway	version	count
Input	Base	DataV1	77392
Input	Base	DataV2	81424
Input	HighRES	DataV1	78176
Input	Pilot	DataV1	23416
Input	StorageInnov	DataV1	77392
Output	Base	DataV1	420672
Output	Base	DataV2	437424
Output	Base(withRen.Target)	DataV1	421984
Output	HighRES	DataV1	421984
Output	Pilot	DataV1	421992
Output	Pilot2	DataV1	421976
Output	StorageInnov	DataV1	421984

[Ludee]

I found some inconsistencies in the numbering of the indicators.
Every indicator should have a unique nid.

I think this is due to a bug in the database adapter and appears on the Base pathway of the output.

• Check all nid for input and output to locate the bug (only some regions!)
• Remove Bug
• Restart upload

[4lm]

Hi @pkork,

the sources info in the metadata of the input model data of TIMES PanEU is incomplete, please follow this instruction to provide the infos.

[pkork]

In the excel, i provided I see that they are complete. The sources are either IER research or WP2 roadmaps. Do you refer to the information in the column P actually or you are asking additional work to do? I'm sorry it is not clear for me:(

[Ludee]

In the TIMES input data, there is a column named "sources" you are refering to. @4lm is refering to the "metadata string" for the entire table. The metadata uses JSON syntax (key-value).
Have a look at this example.

The link directs to the corresponding lines 62-64 for the input table:

"sources": [
{"name": "", "description": "", "url": "", "license": "", "copyright": ""},
{"name": "", "description": "", "url": "", "license": "", "copyright": ""} ],

You need to fill out these 5 keys! Use "none" if na.
Adding "IER research" as your source is possible, but IMHO this is scientific BS.
But I know it quite difficult to track every value and source.

[4lm]

Hi @pkork, hi @Ludee,

I wanted to get an understanding of all sources for the input data, so that I can assist adding them to the metadata JSON string. I had a look at the REEEM database and in the table "reeem_times_paneu_input" I aggregated all the sources for all pathways, frameworks and data versions using this SQL query:

SELECT source, COUNT(*) AS amount, ROUND((COUNT(*) * 100.0 / (SELECT COUNT(*) FROM model_draft.reeem_times_paneu_input)), 2) AS percentage 
FROM model_draft.reeem_times_paneu_input
GROUP BY source;

As a result I get 2 sources:

source	amount	percentage
NULL	32586	2.30
WP2 Roadmap F1_TI1_P1	30240	2.13
IER research	1354142	95.57

Please note: The NULL values are from the Pilot pathway and the EU28 aggregations.

@pkork, for this two sources we now need a closer description, source-url, license and copyright terms. Please follow the general instruction to do so: #55 (comment)

[4lm]

Hi @pkork, any news concerning the source infos for the metadata?

[4lm]

Hi @pkork, also the license info in the metadata for the input data is stil set on tba: https://github.com/ReeemProject/reeem_db/blob/master/database_setup/reeem_db_setup_times_paneu.sql#L65-L71

[4lm]

Hi @pkork,

me again :)

There is a file in the Sharepoint Model_Data folder, named:

2019-05-06_All Pathways_TIMESPanEU_Input.xlsx

Two things:

• In "All Pathways" is a whitespace, which is not allowed my the naming convention and also breaks the import script. I would remove the whitespace, so that it is in camelcase "AllPathways"
• Data Version is missing, because it's for all pathways, but we need a mandatory version, also for the script and also that all data in the database follows a common minimum set of follows. That's why I would set the data version to "DataV0" (zero, because it's the aggregation of all input data with multiple data versions).

Are you OK with this changes?

@Ludee, what do you think? Is this viable (especially my second "DataV0" suggestion)?

Edit: I also would set the missing framework value to "FrameworkV0"

EditEdit: @Ludee, I just see, that we used in another model "PathwayNA", so I assume "NA" stands for not available, we also could use this here. So, "DataNA" and "FrameworkNA" instead of "DataV0" and "FrameworkV0" - what do you think? I find both solutions OK, but we already used "NA" in such a case, so I would stick with this solution ...

[pkork]

I’m ok with the changes and I will come back to other points within next week. Regards, Pinar From: Alexis Michaltsis [mailto:[email protected]] Sent: Wednesday, May 29, 2019 5:04 PM To: ReeemProject/reeem_db <[email protected]> Cc: Pinar Korkmaz <[email protected]>; Mention <[email protected]> Subject: Re: [ReeemProject/reeem_db] Database Adapter - TIMES PanEU (#4) Hi @pkork<https://github.com/pkork>, me again :) There is a file in the Sharepoint Model_Data folder, named: 2019-05-06_All Pathways_TIMESPanEU_Input.xlsx Two things: * In "All Pathways" is a whitespace, which is not allowed my the naming convention and also breaks the import script. I would remove the whitespace, so that it is in camelcase "AllPathways" * Data Version is missing, because it's for all pathways, but we need a mandatory version, also for the script and also that all data in the database follows a common minimum set of follows. That's why I would set the data version to "DataV0" (zero, because it's the aggregation of all input data with multiple input versions). Are you OK with this changes? @Ludee<https://github.com/Ludee>, what do you think? Is this viable (especially my second "DataV0" suggestion)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#4?email_source=notifications&email_token=AIBB3WM6FY72VYRYJSUNYYDPX2LOZA5CNFSM4EB6INL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWPUDNQ#issuecomment-496976310>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AIBB3WKK3CECCXIQ6DCK4WLPX2LOZANCNFSM4EB6INLQ>.

[4lm]

IMPORTANT: Request for OEP-Publishing

Hi @JuliaIER and @pkork,

you licensed reeem_times_paneu_output as CC-BY-4.0 and therefore suitable for publishing on the Open Energy Platform (OEP). We now pre-staged the data on OEP and are ready to hit the button :) Nevertheless we want to ask for your final OK!

If you want to have a look at the data, you find it in the REEEM DB in model_draft.reeem_times_paneu_output

Thanks in advance!

[4lm]

Hi @JuliaIER and @pkork,

a little reminder, we still need the input sources and license info of TIMES for the input metadata.

Please provide us now with the metadata of your data, just post it in a comment in this issue thread!

SOURCES: "REEEM Times PanEU Input"

"sources": [
        {"name": "none", "description": "none", "url": "none", "license": "none", "copyright": "none"},
        {"name": "none", "description": "none", "url": "none", "license": "none", "copyright": "none"} ],

LICENSE: "REEEM Times PanEU Input"

    "license":
        {"id": "none",
        "name": "none",
        "version": "none",
        "url": "none",
        "instruction": "none",
        "copyright": "none"},

Thanks!

CC: @Ludee

[4lm]

IMPORTANT: Request for OEP-Publishing

Hi @JuliaIER and @pkork,

you licensed reeem_times_paneu_output as CC-BY-4.0 and therefore suitable for publishing on the Open Energy Platform (OEP). We now pre-staged the data on OEP and are ready to hit the button :) Nevertheless we want to ask for your final OK!

If you want to have a look at the data, you find it in the REEEM DB in model_draft.reeem_times_paneu_output

Thanks in advance!

Hi @JuliaIER and @pkork, what about this request? What is the status here?

CC: @Ludee