Kuali Rice Development
  1. Kuali Rice Development
  2. KULRICE-8849

removePrincipalFromRole and getRoleMemberPrincipalId methods very slow

    Details

    • Type: Improvement Improvement
    • Status: Closed Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.3
    • Component/s: Development, Performance
    • Security Level: Public (Public: Anyone can view)
    • Labels:
      None
    • Similar issues:
      KULRICE-9448Performance issues with jumpTo and focusOn scripts for very large pages (ie large collections)
      KULRICE-4070XmlHelper.appendXml(Node, String) can be really slow. (i.e. takes minutes)
      KULRICE-13070Pages load really slow after rice upgrade
      KULRICE-8836Person maintenance screen loading very slowly when the person has a lot of roles
      KULRICE-9835removePrincipalFromRole uses attribute id instead of attribute name in qualifier
      KULRICE-12991Application startup time slow after Spring 4.0.x upgrade
      KULRICE-11559superuseractions.tag slows down all pages
      KULRICE-12988KC app startup time really slow with the latest rice 2.5 revision
      KULRICE-8847Role lookups slow when there are over 200 roles
      KULRICE-8929KRAD rederning is slow due to Freemarker's FMParser initializing LookaheadSuccess multiple times
    • Rice Module:
      KIM
    • Application Requirement:
      KC
    • KAI Review Status:
      Not Required
    • KTI Review Status:
      Not Required

      Description

      removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset. Please let me know if you have trouble loading the data, a MySQL "source fileName" should do it. If you need a different format, please let me know.

      Using the test data set, a unit test to remove principals from a role with 85000 members took 11 seconds and the method getRoleMemberPrincipalId took about 1 minute. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix. The method assignPrincipalToRole may also require a similar fix since that is a potential place for performance enhancements also. Most of these issues with Role performance seem to be related to the RoleBo class which loads all the role members for the role. In some places, RoleBoLite has been used to improve performance but since these methods require the membership information, it might not be an option.

      1. _KimOjbSpringBeans_xml.patch
        0.8 kB
        Gayathri Athreya
      2. KCINFR-713.sql.gz
        3.41 MB
        Gayathri Athreya
      3. KCINFR-713-oracle.sql.gz
        3.71 MB
        Gayathri Athreya
      4. ProposalPermissionsAddUserToRoleTest.java
        3 kB
        Gayathri Athreya
      5. RoleDaoOjb.java
        46 kB
        Gayathri Athreya

        Issue Links

          Activity

          Gayathri Athreya made changes -
          Field Original Value New Value
          Link This issue fixes KCINFR-713 [ KCINFR-713 ]
          Gayathri Athreya made changes -
          Description removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took close to a minute and the method getRoleMemberPrincipalId took about 11 seconds. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix.
          removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took close to a minute and the method getRoleMemberPrincipalId took about 11 seconds. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix.
          Gayathri Athreya made changes -
          Description removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took close to a minute and the method getRoleMemberPrincipalId took about 11 seconds. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix.
          removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took 11 seconds and the method getRoleMemberPrincipalId took about 1 minute. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix.
          Gayathri Athreya made changes -
          Description removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took 11 seconds and the method getRoleMemberPrincipalId took about 1 minute. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix.
          removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset. Please let me know if you have trouble loading the data, a MySQL "source fileName" should do it. If you need a different format, please let me know.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took 11 seconds and the method getRoleMemberPrincipalId took about 1 minute. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix.
          Gayathri Athreya made changes -
          Description removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset. Please let me know if you have trouble loading the data, a MySQL "source fileName" should do it. If you need a different format, please let me know.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took 11 seconds and the method getRoleMemberPrincipalId took about 1 minute. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix.
          removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset. Please let me know if you have trouble loading the data, a MySQL "source fileName" should do it. If you need a different format, please let me know.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took 11 seconds and the method getRoleMemberPrincipalId took about 1 minute. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix. The method assignPrincipalToRole may also require a similar fix since that is a potential place for performance enhancements also.
          Gayathri Athreya made changes -
          Description removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset. Please let me know if you have trouble loading the data, a MySQL "source fileName" should do it. If you need a different format, please let me know.

          Using the test dataset, a unit test to remove principals from a role with 85000 members took 11 seconds and the method getRoleMemberPrincipalId took about 1 minute. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix. The method assignPrincipalToRole may also require a similar fix since that is a potential place for performance enhancements also.
          removePrincipalFromRole and getRoleMemberPrincipalId methods perform poorly when there are about 85,000 members assigned to the role. Attaching simulated datasets for testing but this requires a KC application running because of the KIM_TYP_IDs and the attribute definition used in the dataset. Please let me know if you have trouble loading the data, a MySQL "source fileName" should do it. If you need a different format, please let me know.

          Using the test data set, a unit test to remove principals from a role with 85000 members took 11 seconds and the method getRoleMemberPrincipalId took about 1 minute. Higher execution times were observed while using actual data. Please look at the associated KCINFR-713 jira for the issue reported at IU and Eric's comment on the fix. The method assignPrincipalToRole may also require a similar fix since that is a potential place for performance enhancements also. Most of these issues with Role performance seem to be related to the RoleBo class which loads all the role members for the role. In some places, RoleBoLite has been used to improve performance but since these methods require the membership information, it might not be an option.
          Peter Giles (Inactive) made changes -
          Assignee Sona Sona [ sonam ]
          Hide
          Sona Sona (Inactive) added a comment -

          Hi Gayathri,
          I got KC running locally but I have it setup with Oracle as the db. I wanted to check with you if you could get me the dataset as an oracle dump? Thanks.

          Show
          Sona Sona (Inactive) added a comment - Hi Gayathri, I got KC running locally but I have it setup with Oracle as the db. I wanted to check with you if you could get me the dataset as an oracle dump? Thanks.
          Hide
          Gayathri Athreya added a comment - - edited

          Hi Sona,
          Can I get you a .sql file so you can do a @fileName.sql ? Or CSV? I can modify my script to get you whatever format you want, just let me know.

          Show
          Gayathri Athreya added a comment - - edited Hi Sona, Can I get you a .sql file so you can do a @fileName.sql ? Or CSV? I can modify my script to get you whatever format you want, just let me know.
          Gayathri Athreya made changes -
          Attachment KCINFR-713-oracle.sql.gz [ 64696 ]
          Hide
          Gayathri Athreya added a comment -

          Hi, I've attached the oracle version that you should be able to import using @KCINFR-713-oracle.sql . I could not test it since I run MySQL so let me know if you find errors.

          Show
          Gayathri Athreya added a comment - Hi, I've attached the oracle version that you should be able to import using @KCINFR-713-oracle.sql . I could not test it since I run MySQL so let me know if you find errors.
          Sona Sona (Inactive) made changes -
          Link This issue relates to KULRICE-8836 [ KULRICE-8836 ]
          Grant Trudel (Inactive) made changes -
          Documentation Review Status Pending Review [ 14643 ] Not Required [ 14642 ]
          Hide
          Sona Sona (Inactive) added a comment -

          Added fixes for getRoleMemberPrincipalId and the time is down to ~10 sec from 60 sec. Also added a fix for removePrincipalFromRole and assignPrincipalToRole and the time is down to ms. There is one more thing that can be done is to look at the actual query that is being fired and get some column indices set up if we can. This involves setting up P6Spy with KC which is not working for me right now. So, I am resolving this issue and creating another jira to research if some column indices can be added.

          Show
          Sona Sona (Inactive) added a comment - Added fixes for getRoleMemberPrincipalId and the time is down to ~10 sec from 60 sec. Also added a fix for removePrincipalFromRole and assignPrincipalToRole and the time is down to ms. There is one more thing that can be done is to look at the actual query that is being fired and get some column indices set up if we can. This involves setting up P6Spy with KC which is not working for me right now. So, I am resolving this issue and creating another jira to research if some column indices can be added.
          Sona Sona (Inactive) made changes -
          Link This issue is related to KULRICE-8959 [ KULRICE-8959 ]
          Sona Sona (Inactive) made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Gayathri Athreya added a comment -

          Do I need any other patches to test this than the ones I used for KULRICE-8847?

          Show
          Gayathri Athreya added a comment - Do I need any other patches to test this than the ones I used for KULRICE-8847 ?
          Hide
          Sona Sona (Inactive) added a comment -

          These changes have already been committed.

          Show
          Sona Sona (Inactive) added a comment - These changes have already been committed.
          Hide
          Gayathri Athreya added a comment -

          So I did test this and these are the times I noted

          testRemovePrincipalFromRole - 5609 ms
          testGetRoleMemberPrincipalIds - 10530 ms
          testAssignPrincipalToRole - 6771 ms
          

          I definitely see a performance improvement but will have to wait for the testers to determine if this improves speed for them with real data. Thanks.

          Show
          Gayathri Athreya added a comment - So I did test this and these are the times I noted testRemovePrincipalFromRole - 5609 ms testGetRoleMemberPrincipalIds - 10530 ms testAssignPrincipalToRole - 6771 ms I definitely see a performance improvement but will have to wait for the testers to determine if this improves speed for them with real data. Thanks.
          Hide
          Geo Thomas (Inactive) added a comment -

          Gayathri,
          Is it possible to get a patch for this and apply to my local rice? I will give it a try with MIT data.
          Thanks
          Geo

          Show
          Geo Thomas (Inactive) added a comment - Gayathri, Is it possible to get a patch for this and apply to my local rice? I will give it a try with MIT data. Thanks Geo
          Hide
          Gayathri Athreya added a comment - - edited

          Yes, I am attaching the new RoleDaoOjb class and an associated patch. Use the springBeans xml patch and replace your roleDaoOjb with this one. Geo, this should speed up unbounded RoleLookups also BTW. Please test that too.

          Show
          Gayathri Athreya added a comment - - edited Yes, I am attaching the new RoleDaoOjb class and an associated patch. Use the springBeans xml patch and replace your roleDaoOjb with this one. Geo, this should speed up unbounded RoleLookups also BTW. Please test that too.
          Gayathri Athreya made changes -
          Attachment RoleDaoOjb.java [ 65288 ]
          Attachment _KimOjbSpringBeans_xml.patch [ 65289 ]
          Hide
          Geo Thomas (Inactive) added a comment - - edited

          I tested rice version 2.1.4-r37679 with MIT data. The unbounded Role BO lookup came up in less than a second. Earlier it used to take more than 15 minutes. Huge improvement!!!!
          I also tested PD Permission tab and Person update with different set of permissions. All looked good.
          Gayathri, We can resolve this.

          Show
          Geo Thomas (Inactive) added a comment - - edited I tested rice version 2.1.4-r37679 with MIT data. The unbounded Role BO lookup came up in less than a second. Earlier it used to take more than 15 minutes. Huge improvement!!!! I also tested PD Permission tab and Person update with different set of permissions. All looked good. Gayathri, We can resolve this.
          Hide
          Gayathri Athreya added a comment -

          Awesome, thanks Sona

          Show
          Gayathri Athreya added a comment - Awesome, thanks Sona
          Jessica Coltrin (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Shem Patterson (Inactive) made changes -
          Workflow custom [ 162998 ] Copy of custom for rice [ 215437 ]
          Shem Patterson (Inactive) made changes -
          Workflow Copy of custom for rice [ 215437 ] custom [ 225185 ]
          Shem Patterson (Inactive) made changes -
          Workflow custom [ 225185 ] Rice Workflow [ 234933 ]
          Jessica Coltrin (Inactive) made changes -
          Component/s Performance [ 13588 ]

            People

            • Assignee:
              Sona Sona (Inactive)
              Reporter:
              Gayathri Athreya
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Structure Helper Panel