Database Adapter - MESSAGE

[Ludee]

• reeem_db_setup_message.sql
• reeem_adapter_message.py
• reeem_io.py
• reeem_db_plot_MESSAGE.ipynb
• Feedback from modeller

modeller:

[4lm]

Hi @vidasl,

@Ludee forwarded me your email, I will try to give you some guidance ... in the next hour I will add some info to this issue ...

[4lm]

@vidasl,

I created a team LEI: https://github.com/orgs/ReeemProject/teams and invited you as team member.

After you accepted the team invite, I can add you as assignee to this very issue: #11

This will help you to stay informed about all issues concerning MESSAGE.

[4lm]

@vidasl,

I checked, you provided input and output model data of the MESSAGE model. In our internal cheat the data delivery is marked with "feedback". I will ask @Ludee if he needs more infos or if we are good to go.

If we are good to go, we will write an adapter to get the data from the excel sheets into the REEEM database.

[4lm]

@vidasl,

what we further need is the infos for the metadata of the MESSAGE model, especially for:

• titles [1]
• sources: follow this link for instruction
• licences: follow this link for instruction

[1] please check if this input title AND this output title infos are accurate and if not, provide us with the correct info for the titles. best way to inform us either per comment in this issue or directly via email to me @4lm (Alexis Michaltsis): [email protected] (please also add @Ludee (Ludwig) in CC)

[4lm]

@vidasl,

I checked, you provided input and output model data of the MESSAGE model. In our internal cheat the data delivery is marked with "feedback". I will ask @Ludee if he needs more infos or if we are good to go.

If we are good to go, we will write an adapter to get the data from the excel sheets into the REEEM database.

Hi @vidasl,

I spoke to @Ludee (Ludwig), there were some issues with the data delivery. Please look for an email with the subject: "AW: AW: [REEEM] Energy security case study" for more instruction, send on the 4th of February by @Ludee.

[vidasl]

I have uploaded new versions of data structures to sharepoint taking into account @Ludee comments. If it is OK, I could porceed with the metadata.

[vidasl]

Hi @Ludee , @4lm,
were you able to check the new versions of data structures?

[4lm]

Hi @vidasl,

funny, I'm on it right in this moment :)

So, I have some points:

• (INPUT|OUTPUT) file naming does not comply with naming convention: Since we haven't imported the old versions yet, you could (i) overwrite the old versions OR (ii) simply write DataV2 instead of DataV1 OR (iii) if its a revision something like DataV1R1.
• (OUTPUT) output file lacks IDs, we need unique IDs (ascending, starting with 1) for every row in the ID column of a table
• (INPUT|OUTPUT) field description is missing (especially important since input indicators are kind of cryptic)
• (INPUT|OUTPUT) category is missing (especially important since input indicators are kind of cryptic)
• (INPUT|OUTPUT) aggregation info is missing
• (INPUT) source is missing - every row should have a short source info, which then is linkable to the source info of the metadata info.

Only after above changes are made we can write adapter, cleanup and tagging programs which are used for processing the data in the REEEM database.

@Ludee. Do you have further points?

Edit: In the indicators column you use subheadlines, which will be lost after database import, so it would be much better to include the subheadlines into the concerning indicator fields, e.g.:

BAD:

Electricity Import from <-- Subheadline
Sweden
Estonia
Russia
Norway

GOOD:

Electricity Import from Sweden
Electricity Import from Estonia
Electricity Import from Russia
Electricity Import from Norway

[4lm]

Hi @vidasl,

another point. In my opinion, you can already start with providing the infos for the metadata, because data delivery and processing is weakly linked with the metadata creation process, so please also continue with my request from above: #11 (comment)

Edit: especially source and license infos are missing. Look for example at EnergyPro's metadata for comparsion: https://github.com/ReeemProject/reeem_db/blob/master/database_setup/reeem_db_setup_energypro.sql#L62-L109

[vidasl]

New data structures have been uploaded

[4lm]

Hi @vidasl,

thanks, we will have a look at it and if OK, will proceed with writing an adapter for the data import into the REEEM DB ... please provide us now with the metadata of your data delivery, just post it in a comment in this issue thread. This is the two metadata strings we have so far, everywhere you read "none", we need infos (except in the "contributors" and "fields" sections):

REEEM MESSAGE Input:

{
    "title": "REEEM MESSAGE Input",
    "description": "Input parameters: Service Demands, Fuel Prices, Potentials",
    "language": [ "eng" ],
    "spatial": 
        {"location": "none",
        "extent": "Europe",
        "resolution": "Country"},
    "temporal": 
        {"reference_date": "2015",
        "start": "2015",
        "end": "2060",
        "resolution": "1 year"},
    "sources": [
        {"name": "none", "description": "none", "url": "none", "license": "none", "copyright": "none"},
        {"name": "none", "description": "none", "url": "none", "license": "none", "copyright": "none"} ],
    "license": 
        {"id": "none",
        "name": "none",
        "version": "none",
        "url": "none",
        "instruction": "none",
        "copyright": "none"},
    "contributors": [
        {"name": "Ludee", "email": "none", "date": "2019-01-22", "comment": "Create table and metadata"} ],
    "resources": [
        {"name": "model_draft.reeem_message_input",
        "format": "PostgreSQL",
        "fields": [
            {"name": "id", "description": "Unique identifier", "unit": "none"},
            {"name": "nid", "description": "Row id", "unit": "none"},
            {"name": "dfid", "description": "Internal dataframe id", "unit": "none"},
            {"name": "pathway", "description": "REEEM pathway", "unit": "none"},
            {"name": "framework", "description": "REEEM framework", "unit": "none"},
            {"name": "version", "description": "REEEM version", "unit": "none"},
            {"name": "schema", "description": "1. classification", "unit": "none"},
            {"name": "category", "description": "2. classification", "unit": "none"},
            {"name": "tags", "description": "Free classification", "unit": "none"},
            {"name": "field", "description": "Old classification", "unit": "none"},
            {"name": "region", "description": "Country or region", "unit": "none"},
            {"name": "year", "description": "Year", "unit": "none"},
            {"name": "indicator", "description": "Parameter name", "unit": "none"},
            {"name": "value", "description": "Parameter value", "unit": "unit"},
            {"name": "unit", "description": "Parameter unit", "unit": "none"},
            {"name": "aggregation", "description": "True if aggregated", "unit": "none"},
            {"name": "updated", "description": "Timestamp", "unit": "none"},
            {"name": "source", "description": "Data source", "unit": "none"} ] } ],
    "metadata_version": "1.3"}

REEEM MESSAGE Output:

{
    "title": "REEEM MESSAGE Output",
    "description": "Output parameters: ",
    "language": [ "eng" ],
    "spatial": 
        {"location": "none",
        "extent": "Europe (EU28+NO+CH)",
        "resolution": "Country"},
    "temporal": 
        {"reference_date": "2015",
        "start": "2015",
        "end": "2050",
        "resolution": "1 year"},
    "sources": [
        {"name": "MESSAGE", "description": "Model documentation", "url": "none", "license": "none", "copyright": "none"} ],
    "license": 
        {"id": "none",
        "name": "none",
        "version": "none",
        "url": "none",
        "instruction": "none",
        "copyright": "none"},
    "contributors": [
        {"name": "Ludee", "email": "none", "date": "2019-01-22", "comment": "Create table and metadata"} ],
    "resources": [
        {"name": "model_draft.reeem_message_output",
        "format": "PostgreSQL",
        "fields": [
            {"name": "id", "description": "Unique identifier", "unit": "none"},
            {"name": "nid", "description": "Row id", "unit": "none"},
            {"name": "dfid", "description": "Internal dataframe id", "unit": "none"},
            {"name": "pathway", "description": "REEEM pathway", "unit": "none"},
            {"name": "framework", "description": "REEEM framework", "unit": "none"},
            {"name": "version", "description": "REEEM version", "unit": "none"},
            {"name": "schema", "description": "1. classification", "unit": "none"},
            {"name": "category", "description": "2. classification", "unit": "none"},
            {"name": "tags", "description": "Free classification", "unit": "none"},
            {"name": "field", "description": "Old classification", "unit": "none"},
            {"name": "region", "description": "Country or region", "unit": "none"},
            {"name": "year", "description": "Year", "unit": "none"},
            {"name": "indicator", "description": "Parameter name", "unit": "none"},
            {"name": "value", "description": "Parameter value", "unit": "unit"},
            {"name": "unit", "description": "Parameter unit", "unit": "none"},
            {"name": "aggregation", "description": "True if aggregated", "unit": "none"},
            {"name": "updated", "description": "Timestamp", "unit": "none"} ] } ],
    "metadata_version": "1.3"}

[4lm]

@vidasl, a little reminder for: #11 (comment) - it's urgent now!

CC: @Ludee

PS: Most important are sources (for input and output data) and under which licenses you distribute your output data.

[vidasl]

hi @4lm
I have put the data as well as metadata info to sharepoint ("final data" folder).

[4lm]

Hi @vidasl,

thanks, I will try to import the data and add the metadata ...

[vidasl]

Hi @4lm ,
thank you!

[4lm]

IMPORTANT: Request for OEP-Publishing

Hi @vidasl,

you licensed reeem_message_input and reeem_message_output as CC-BY-4.0 and therefore suitable for publishing on the Open Energy Platform (OEP). We pre-staged the data and we are ready to hit the button :) Nevertheless we want to ask for your final OK!

If you want to have a look at the data, you find it in the REEEM database in:

• model_draft.reeem_message_input
• model_draft.reeem_message_output

Thanks in advance!

CC: @Ludee

[vidasl]

OK for all the data in "Final data" folder.

[4lm]

Hi @vidasl,

there is also V2 data in the database, from this two files:

• 2019-01-28_Base_MESSAGE_FrameworkNA_DataV2_Output.xlsx
• 2019-01-28_Base_MESSAGE_FrameworkNA_DataV2_Input.xlsx

Shall I keep and also publish this data on OEP or delete it? Technically it's no problem to have multiple data versions. So it's more of a licensing question ...

CC: @Ludee

[vidasl]

Hi @4lm ,
These files were basically used to agree on the data structure. So, please delete them and publish only data in "Final data" folder.