Kuali Rice Roadmap
  1. Kuali Rice Roadmap
  2. KRRM-56

Expand support for XML import and export of data

    Details

    • Type: Rice Enhancement Rice Enhancement
    • Status: Open Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Component/s: Unknown
    • Labels:
      None
    • Rice Theme:
      Service Orientation
    • Priority Score:
      18
    • Functional Justification :
      Hide
      using sql to migrate roles and permissions from one environment to another is difficult to coordinate due to the sequence numbers on the KRIM tables. Ideally roles could be tested in a test environment then individually migrated to the production environment once tested and approved. with the tools in place today either all roles and permissions have to be migrated or roles and permissions have to be rebuilt in each environment..
      Show
      using sql to migrate roles and permissions from one environment to another is difficult to coordinate due to the sequence numbers on the KRIM tables. Ideally roles could be tested in a test environment then individually migrated to the production environment once tested and approved. with the tools in place today either all roles and permissions have to be migrated or roles and permissions have to be rebuilt in each environment..
    • Impact if not Implemented:
      Hide
      Creating and migrating data from one Rice instance to another will be a cumbersome task and adoption/implementations more difficult. General tasks will continue to be an extremely technical one instead of one that could be distributed beyond a development team.
      Show
      Creating and migrating data from one Rice instance to another will be a cumbersome task and adoption/implementations more difficult. General tasks will continue to be an extremely technical one instead of one that could be distributed beyond a development team.
    • Priority - KFS:
      Critical
    • Priority - KC:
      High
    • Priority - KS:
      Critical
    • Priority - Rice:
      Critical
    • Theme:
      Ease of Implementation
    • Application Impact:
      Low
    • Effort Estimate:
      Medium ~ 500 hrs

      Description

      There is currently the concept of XML import/export built into the KEW module. This needs to be pulled out into Rice as a core service so that it can be used in more than just the KEW application.

      These xml capabilities are extremely useful for migrating data between test environments and also into production. Specifically, we don't currently have XML import/export support for KIM data (permissions and roles being the most valuable in this space) which is going to make implementation for institutions more difficult since they will have to manage SQL for migrating data between environments which can be extremely cumbersome.

        Issue Links

          Activity

          Hide
          Eric Westfall added a comment -

          Very important: part of this work should include setting up the XML schemas so that they can be properly handled by XML authoring tools (this is not currently the case with the custom schema names we are using). See linked issue for more details.

          Show
          Eric Westfall added a comment - Very important: part of this work should include setting up the XML schemas so that they can be properly handled by XML authoring tools (this is not currently the case with the custom schema names we are using). See linked issue for more details.
          Hide
          Kymber Horn added a comment -

          KFS is also wondering why this is unspecified, we don't remember talking about this.

          Show
          Kymber Horn added a comment - KFS is also wondering why this is unspecified, we don't remember talking about this.
          Hide
          Cath Fairlie (Inactive) added a comment -

          We need this now.

          Show
          Cath Fairlie (Inactive) added a comment - We need this now.
          Hide
          added a comment -

          I would like to see support in this feature include administrative screens to handle potential key conflicts and duplication issues that might arise when importing data. Especially for KIM related data in which similar or duplicate roles and permissions (or identities?) might exist in various source systems. This then would make the feature useful on a real-time or as needed basis as opposed to just at upgrade or implementation situations.

          Show
          added a comment - I would like to see support in this feature include administrative screens to handle potential key conflicts and duplication issues that might arise when importing data. Especially for KIM related data in which similar or duplicate roles and permissions (or identities?) might exist in various source systems. This then would make the feature useful on a real-time or as needed basis as opposed to just at upgrade or implementation situations.
          Hide
          Eric Westfall added a comment -

          I want to affirm the requirements for this functionality. Is the following list the complete set that is expected to be delivered for this?

          1) Add support for XML import/export for the Kuali Identity Management module.
          2) Rewrite our XML schemas such that they are defined in such a way which works well with XML authoring tools (essentially, schema urls that resolve to actual hosted artifacts on rice.kuali.org)
          3) Address memory-related issues with existing import/export mechanism on large XML files or export data sets (essentially, introduce xml pull parsing techniques for stream-driven processing).
          4) A user interface which allows the person performing the export to assemble the export data set from different exporters (right now, export is performed by doing a search on a lookup, so it's not possible to say "export everything within this namespace" or "export all workflow and eDocLite configuration related to this particular document type".

          Other features for consideration:

          1) A remotable API for performing import/export of data. Currently, this can only be done via the user interface.
          2) The ability to decentralize XML ingestion by adding support on the backend for securing XML import operations based on the data being imported. For example, being able to configure a permission in KIM that says only people is a specific role can update KIM configuration related to a particular namespace (in certain situtions, this could take advantage of permissions already defined in KIM, like "Grant Permission" or "Assign Role").
          3) User interface functionality that can be used at the time of import to resolve potential issues with the import (duplicates, etc.). Right now, the person attempting to import would simply get an error and would have to resolve it manually. This could also include a screen which provides a preview of the import before it actually happens. Allowing the user to verify that the correct data is being imported.

          Other questions:

          1) Are changes to the XML format which wouldn't be compatible with older versions of the XML be permitted or will we be required to support "legacy" xml formats?

          Show
          Eric Westfall added a comment - I want to affirm the requirements for this functionality. Is the following list the complete set that is expected to be delivered for this? 1) Add support for XML import/export for the Kuali Identity Management module. 2) Rewrite our XML schemas such that they are defined in such a way which works well with XML authoring tools (essentially, schema urls that resolve to actual hosted artifacts on rice.kuali.org) 3) Address memory-related issues with existing import/export mechanism on large XML files or export data sets (essentially, introduce xml pull parsing techniques for stream-driven processing). 4) A user interface which allows the person performing the export to assemble the export data set from different exporters (right now, export is performed by doing a search on a lookup, so it's not possible to say "export everything within this namespace" or "export all workflow and eDocLite configuration related to this particular document type". Other features for consideration: 1) A remotable API for performing import/export of data. Currently, this can only be done via the user interface. 2) The ability to decentralize XML ingestion by adding support on the backend for securing XML import operations based on the data being imported. For example, being able to configure a permission in KIM that says only people is a specific role can update KIM configuration related to a particular namespace (in certain situtions, this could take advantage of permissions already defined in KIM, like "Grant Permission" or "Assign Role"). 3) User interface functionality that can be used at the time of import to resolve potential issues with the import (duplicates, etc.). Right now, the person attempting to import would simply get an error and would have to resolve it manually. This could also include a screen which provides a preview of the import before it actually happens. Allowing the user to verify that the correct data is being imported. Other questions: 1) Are changes to the XML format which wouldn't be compatible with older versions of the XML be permitted or will we be required to support "legacy" xml formats?
          Hide
          Sarah Christen (Inactive) added a comment -

          This list of requirements looks good to me. In the other features to consider, I think number 3 is really important

          3.) User interface functionality that can be used at the time of import to resolve potential issues with the import (duplicates, etc.). Right now, the person attempting to import would simply get an error and would have to resolve it manually. This could also include a screen which provides a preview of the import before it actually happens. Allowing the user to verify that the correct data is being imported.

          1.)I personally think the user interface for import/export is fine, I'm not sure how we might use the remotable API but I will follow up with other here to be sure.

          I can see the ability to decentralize this function being used here, at Cornell, if we could tie the import export permissions to namespace.

          Show
          Sarah Christen (Inactive) added a comment - This list of requirements looks good to me. In the other features to consider, I think number 3 is really important 3.) User interface functionality that can be used at the time of import to resolve potential issues with the import (duplicates, etc.). Right now, the person attempting to import would simply get an error and would have to resolve it manually. This could also include a screen which provides a preview of the import before it actually happens. Allowing the user to verify that the correct data is being imported. 1.)I personally think the user interface for import/export is fine, I'm not sure how we might use the remotable API but I will follow up with other here to be sure. I can see the ability to decentralize this function being used here, at Cornell, if we could tie the import export permissions to namespace.
          Hide
          Sandra Agee (Inactive) added a comment - - edited

          Additional Information:

          Business Scenario: The ability to migrate permission roles into each environment. i.e. testing...production.

          Currently, in the security db if tested one role type have to migrate all or nothing or perform a manual build.

          Show
          Sandra Agee (Inactive) added a comment - - edited Additional Information: Business Scenario: The ability to migrate permission roles into each environment. i.e. testing...production. Currently, in the security db if tested one role type have to migrate all or nothing or perform a manual build.
          Hide
          Eric Westfall added a comment -

          Created 5 sub-tasks on this jira based on the comments. Moved the other two items out to separate roadmap items KRRM-114 and KRRM-115.

          Show
          Eric Westfall added a comment - Created 5 sub-tasks on this jira based on the comments. Moved the other two items out to separate roadmap items KRRM-114 and KRRM-115 .
          Hide
          Eric Westfall added a comment -

          Just as a general note, the majority of the ~500 hours is probably represented by the two GUI-related subtasks as those are potentially the most complex and also the ones which we know the least about in terms of detail at the moment.

          Show
          Eric Westfall added a comment - Just as a general note, the majority of the ~500 hours is probably represented by the two GUI-related subtasks as those are potentially the most complex and also the ones which we know the least about in terms of detail at the moment.

            People

            • Assignee:
              Sarah Christen (Inactive)
              Reporter:
              Eric Westfall
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Structure Helper Panel