Kuali Rice Development
  1. Kuali Rice Development
  2. KULRICE-3689

Improve performance of IdentityArchiveService interactions

    Details

    • Type: Improvement Improvement
    • Status: Closed Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.1.1, KFS Release 3.0.1
    • Component/s: Development
    • Labels:
      None
    • Similar issues:
      KULRICE-8448Improve Role Performance
      KULRICE-6735Document the Performance Improvements
      KULRICE-11177Need JPA related caching functionality to improve performance and support clustering
      KULRICE-14218Release 2.3.7 with KRMS performance improvements
      KULRICE-9313Perform experiment on cloning for performance improvements
      KULRICE-6557Improve Performance and Memory Consumption
      KULRICE-8918Improved KRAD collections performance
      KULRICE-12491Performance improvement for readXml in XmlHelper
      KULRICE-8956ComponentBase performFinalize Performance Improvement
      KULRICE-8250Improve EDocLite Performance
    • Rice Module:
      KIM

      Description

      Right now, every single request for entity data will result in the archive service scheduling a background thread to update or insert the data in the KRIM_ENTITY_CACHE_T table. Instead, we should probably create some sort of in-memory cache that stays active for a configurable period of time (say 1-hour or more by default) to minimize the number of writes to this table. The data shouldn't be changing so often that we need to keep the cache table up to date in real time.

      The current implementaiton is a performance concern.

        Activity

        Hide
        Jonathan Keller added a comment -

        I'm going to mark this one for 1.0.1.1 as well so we don't lose track of it since performance is an issue for the patch.

        Show
        Jonathan Keller added a comment - I'm going to mark this one for 1.0.1.1 as well so we don't lose track of it since performance is an issue for the patch.
        Hide
        Peter Giles (Inactive) added a comment -

        Some implementation notes and a question:

        I'm enqueueing the KimEntityDefaultInfoS when they come in for write, and using a scheduleAtFixedRate'd Runnable on the KSB's ScheduledThreadPoolExecutor to periodically clean out the write queue. I'm also doing a size check on the write queue when a new entry is added, and if it exceeds a certain max value I schedule it for an immediate flush via a Runnable on the KSB's thread pool. Concurrency between these threads shouldn't be a problem as they are both fed from the same ConcurrentLinkedQueue.

        My question is regarding how to flush the write queue on Rice server shutdown. Should I register a shutdown hook, or should I use one of the LifecycleS to trigger the flush on shutdown?

        Show
        Peter Giles (Inactive) added a comment - Some implementation notes and a question: I'm enqueueing the KimEntityDefaultInfoS when they come in for write, and using a scheduleAtFixedRate'd Runnable on the KSB's ScheduledThreadPoolExecutor to periodically clean out the write queue. I'm also doing a size check on the write queue when a new entry is added, and if it exceeds a certain max value I schedule it for an immediate flush via a Runnable on the KSB's thread pool. Concurrency between these threads shouldn't be a problem as they are both fed from the same ConcurrentLinkedQueue. My question is regarding how to flush the write queue on Rice server shutdown. Should I register a shutdown hook, or should I use one of the LifecycleS to trigger the flush on shutdown?
        Hide
        Peter Giles (Inactive) added a comment -

        I tried implementing ApplicationListener, but the events that I received were too early/too late in the startup/shutdown processes respectively to be of help. I'm going to try initial scheduling and final flush via KIMConfigurer on BeforeStartEvent and BeforeStopEvent and see how that works.

        Show
        Peter Giles (Inactive) added a comment - I tried implementing ApplicationListener, but the events that I received were too early/too late in the startup/shutdown processes respectively to be of help. I'm going to try initial scheduling and final flush via KIMConfigurer on BeforeStartEvent and BeforeStopEvent and see how that works.
        Hide
        Peter Giles (Inactive) added a comment - - edited

        RiceConfigEventListener.java: (added)

        • Extracted interface from ModuleConfigurer

        ModuleConfigurer.java:

        • added implements for extracted interface RiceConfigEventListener

        KIMConfigurer.java:

        • added method registerConfigEventListener(...)
        • modified onEvent(...) to dispatch to listeners

        IdentityArchiveServiceImpl.java:

        • fixed bug that caused exceptions whenever a retrieval was attempted
        • Added write queuing with:
        • scheduled periodic flushes
        • immediate flush on BeforeStopEvent
        • immediate flush w/ retry on archive lookup miss

        IdentityCurrentAndArchivedServiceImpl.java:

        • changed private method name for clarity

        common-config-defaults.xml:

        • added parameter kim.identityArchiveServiceImpl.executionIntervalSeconds
        • added parameter kim.identityArchiveServiceImpl.maxWriteQueueSize

        KIMImplementationSpringBeans.xml:

        • changed IdentityArchiveService bean to provide added constructor params

        IdentityArchiveServiceTest.java: (added)

        • tests basic IdentityArchiveService functionality
        Show
        Peter Giles (Inactive) added a comment - - edited RiceConfigEventListener.java: (added) Extracted interface from ModuleConfigurer ModuleConfigurer.java: added implements for extracted interface RiceConfigEventListener KIMConfigurer.java: added method registerConfigEventListener(...) modified onEvent(...) to dispatch to listeners IdentityArchiveServiceImpl.java: fixed bug that caused exceptions whenever a retrieval was attempted Added write queuing with: scheduled periodic flushes immediate flush on BeforeStopEvent immediate flush w/ retry on archive lookup miss IdentityCurrentAndArchivedServiceImpl.java: changed private method name for clarity common-config-defaults.xml: added parameter kim.identityArchiveServiceImpl.executionIntervalSeconds added parameter kim.identityArchiveServiceImpl.maxWriteQueueSize KIMImplementationSpringBeans.xml: changed IdentityArchiveService bean to provide added constructor params IdentityArchiveServiceTest.java: (added) tests basic IdentityArchiveService functionality

          People

          • Assignee:
            Peter Giles (Inactive)
            Reporter:
            Eric Westfall
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Structure Helper Panel