xdmod issue #31911 Trying to figure out the hierarchy file(s)

[rob-baron]

Robert Bartlett Baron, reported 7 months ago
I'm currently trying to figure out how to map the various pieces of information of personnel to projects.

Starting with the hierarchy files:

1. hierarchy.json

{
"top_level_label": "Institution",
"top_level_info": "Top Level Institution that has an MOU with MGHPCC regarding NERC services and billing",
"middle_level_label": "School/Center",
"middle_level_info": "Top divisional level of the institution, like College of Engineering, Medical School",
"bottom_level_label": "Discipline",
"bottom_level_info": "Reserch Discipline of the PI at the Institution that is responsible for the project"
}

hierarchy.csv:

"BU","Boston University",""
"HU","Harvard University",""
"HARIRI-BU", "Rafik B. Hariri Institute for Computing and Computational Science & Engineering", "BU"
"Phys-BU","Physical Science","CAS"
"Phys-HU","Physical Science","FAS"
"CS-CoE-BU", "BU Computer Science", "BU"
"PS-CS-CoE-BU", "Physical Science", "CS-CoE-BU"
"PS-Phys-HU", "Physical Science", "Phys-HU"
"PS-Phys-BU", "Physical Science", "Phys-BU"

--> Question - does the first column need to be unique? Is there a better way to specific the 3 different "Physical Science" entries?

defining the groups:

"MOCA", "HARIRI-BU"
"MOC", "HARIRI-BU"
"Phys-Gr-BU", "Phys-BU"
"Phys-Gr-HU", "Phys-HU"

Defining names:

"MOCA", "", "Mass Open Cloud Alliance"
"MOC", "", "Mass Open Cloud"
"Phys-Gr-BU", "BU Physics Group"
"Phys-Gr-HU", "HU Physics Group"
"robbaron", "Robert", "Baron"
"[email protected]", "A", "M"
"[email protected]", "Peter", "Desnoyers"

I have a list of projects and people:

bu528-ceph-prefetch, [email protected] (PI)
bu528-ceph-prefetch, [email protected]
bu528-ceph-prefetch, [email protected]

moc-infrastructure, robbaron (PI)
moc-infrastructure, [email protected]
moc-infrastructure, [email protected]

So how do I associate people to projects whether or not they create something in the actual project, and how do I specify who is the PI on a given project?

Gregary Dean , said 7 months ago
Ticket: https://help.xdmod.org/support/tickets/31911

Hi Robert,

Yes, the first column does need to be unique. As for your other question, are the projects you want to specify the PI for OpenStack projects? If so, there is a section in the cloud documentation, https://open.xdmod.org/10.0/cloud.html, titled "Adding PI Information" that has instructions for assigning a PI to an OpenStack project.
​
​Thanks,
Greg Dean
Robert Bartlett Baron , said 7 months ago
Well, that is part of it.

Can I also add people with roles, such as a business administrator to a project - ie, the contact person that should recieve an invoice for services rendered? This may be beyond the current scope of xdmod.

Where is it that I map PIs to projects that have a HPC project?

Thanks,

Rob.
Gregary Dean , said 7 months ago
Ticket: https://help.xdmod.org/support/tickets/31911

Hi Rob,

XDMoD does not have functionality to add people with other roles to a project. As for mapping PI's to projects for HPC jobs, the Jobs realm does not have projects like the cloud realm does. My suggestion would be to use the lowest level of your hierarchy as your project and then you can map your PI's to the bottom level hierarchy. The documentation for this is about halfway down the Hierarchy documentation page, https://open.xdmod.org/10.0/hierarchy.html. The XDMoD documentation uses the term user group instead of PI on this page but those terms are synonymous for this functionality. I'm not sure how well this fits with your how you were planning to layout your hierarchies but it is probably the best way to get what you are after for HPC jobs.

XDMoD defaults to using the group name field from job logs for the PI but you can configure it to use another field for the PI information if you wish. The documentation for that is in the resources.json section of the configuration guide, https://open.xdmod.org/10.0/configuration.html#resourcesjson.

-greg

Robert Bartlett Baron , said 6 months ago
Ok, I'm not quite certain I understand this fully.

On the cloud side, I am able to associated the the cloud project to a PI though I don't quite see how I am to associate the PI to a cloud project with the hierarchy based on the instructions in https://open.xdmod.org/10.0/configuration.html#resourcesjson.

Robert Bartlett Baron , said 6 months ago
Ok, to clarify. I am able to get groups listed to match up with groups for HPC data, but not for Cloud data.

I have also gotten the PI associated with the cloud via the file, but I also want to get the PI associated with the groups or the 3 level hierarchy as defined in the hierarchy.json and hierarchy.csv file. Are there instructions to do this?
Gregary Dean , said 6 months ago
Ticket: https://help.xdmod.org/support/tickets/31911

Hi Rob,

Did you create a groups-to-hierarchy.csv file and ingest it? This is the file that maps a PI to the bottom level of the hierarchy. If so, did you include the PI's for your cloud data in that file? It sounds to me like you might have the PI's for your HPC data in that file, but are missing the PI's for your cloud data.

The hierarchy guide, https://open.xdmod.org/10.0/hierarchy.html, has the command to ingest this data and the format.

Thanks,
Greg
Robert Bartlett Baron , said 6 months ago
Greg,

I had a cache issue, it started working without any modification.

I have taken your suggestion of using the lowest level of the 3 level hierarchy for the PI. However this causes PI to be listed in 2 places in the interface. Is there a way to rename the 2 PI to groups or projects?
PNG
Screenshot 2...
(181 KB)
Robert Bartlett Baron , said 5 months ago
Greg,

Do you have any suggestions?

Rob.
Gregary Dean , said 5 months ago
Ticket: https://help.xdmod.org/support/tickets/31911

Hi Rob,

The text shown for the bottom level hierarchy element's name is taken from the 'bottom_level_label` field in the hierarchy.json file. Changing that field from PI to whatever you want will make it so you do not have PI showing up twice.

-greg
Robert Bartlett Baron , said 5 months ago
Greg,

I have the bottom level of the hierarchy as the PI, what I want to change is the PI label that groups by the group in HPC.
Gregary Dean , said 5 months ago
Ticket: https://help.xdmod.org/support/tickets/31911

Hi Rob,

Do your jobs logs contain both the PI and group you want to group by?

The reason I ask is because a better implementation may be to use the group as the bottom level of your hierarchy and have the PI's show up in the PI group by. Is there a reason your group information can't be the bottom level of your hierarchy? You can specify the field to use from your job logs for your HPC PI data in the resources.json file (https://open.xdmod.org/10.0/configuration.html#resourcesjson). You can then set the group as the bottom level of your hierarchy. To match up the PI to group you would use the group-to-hierarchy.csv file with the PI as the first column and the group as the second column.

After doing the above editing the names of the hierarchy group by's can be done in the hierarchy.json file.

The hierarchy guide gives more instructions, https://open.xdmod.org/10.0/hierarchy.html.

If you do not want to do the above I can get you some steps to change the name of the PI group by but it might be a bit more involved and files you'll likely need to change are ones we generally try to discourage people from changing.

Thanks,
Greg
Robert Bartlett Baron , said 5 months ago
Greg,

Part of my thoughts is that we are using the hierarchy files for both cloud and HPC resources. Currently we are modeling OpenShift as an HPC resource, so having the project at the group level makes intuitive sense since the cloud side we can map project name onto project name in the PI to project csv file.

Conversely, making the PI name at the group level and placing the project being the lowest level of the hierarchy is possible. Unfortunately, this doesn't enforce 1 PI per project, though it does tie the institution to the project. Also since field of science is in the middle layer, it doesn't make the most amount of sense saying that the field of science is the PI's field of science and not the project's field of science. Shall we just agree that this is an interesting design choice that doesn't quite match my design sensibilities.

And yes, I do realize that the group to hierarchy is a mapping table which is a many to many relationship which allows for a PI/group to have joint appointments between multiple institutions

Although I am working on crafting the files with the PI at the group level, can you give me instructions as to what I would have to change to modify the group level text to indicate that it is either a group or a project?

Also what are the plans for xdmod in the future involving the hiearchy? Are you going to make changing the label on the PI/Group to be variable so that people will be able to change it to what they want in the future?

Rob.

Gregary Dean , said 4 months ago
Ticket: https://help.xdmod.org/support/tickets/31911

Hi Rob,

I'm sorry for the delay in responding. It is easier to change the name of the group by than I thought. In '/etc/xdmod/datawarehouse.d/ref/group-by-common.json' there is a section that starts with the key 'pi'. This should be on line 158. In that section, there is a name field that should be on line 183. That is the name that is displayed in XDMoD. That is what you will need to change to get the display value you want. There is also a description field on the line above that you may want to change as well. After saving the filel you should see the change in the web interface. If not, run the following command, touch /etc/xdmod/datawarehouse.json. XDMoD caches these json configuration information and touching that file should cause XDMoD to invalidate the cache and read from these json configuration files.
​
This is the section you are looking for in the group-by-common.json file. It should start on line 158.
​
"pi": {
"attribute_table_schema": "modw",
"attribute_to_aggregate_table_key_map": [
{
"person_id": "principalinvestigator_person_id"
}
],
"attribute_values_query": {
"joins": [
{
"name": "piperson"
}
],
"orderby": [
"order_id"
],
"records": {
"id": "person_id",
"name": "long_name",
"order_id": "order_id",
"short_name": "short_name"
}
},
"category": "Administrative",
"description_html": "The principal investigator of a project.",
"name": "PI"
},

Thanks,
Greg
Robert Bartlett Baron , said 3 months ago
ok, this worked, so let's close this issue for now.

Thanks.
This ticket has been Closed | 3 months ago Reply