Uploaded image for project: 'Kuali Rice Roadmap'
  1. Kuali Rice Roadmap
  2. KRRM-145

Extract the batch framework from KFS into Rice

    Details

    • Type: Rice Research Item
    • Status: Closed
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: None
    • Component/s: KRAD
    • Labels:
      None
    • Rice Theme:
      Service Orientation
    • Priority Score:
      0
    • Impact if not Implemented:
      Without, each application or institution would need to implement their own batch framework. The stability and standardization without this being core is lost.
    • Priority - KFS:
      No Priority
    • Priority - KC:
      No Priority
    • Priority - KS:
      No Priority
    • Priority - Rice:
      No Priority
    • Effort Estimate:
      High ~ 1000 hrs

      Description

      KFS has a batch framework. This seems like something that many applications would like to use (do we have specific requirements for this from KRA?) We should look into what it would take to extract this into Rice.

        Attachments

          Issue Links

            Activity

            Hide
            delyea David Elyea added a comment -

            Temporarily assigning this to me for 0.9.3 as it is currently part of the extraction tasks list

            Show
            delyea David Elyea added a comment - Temporarily assigning this to me for 0.9.3 as it is currently part of the extraction tasks list
            Hide
            abyrne Ailish Byrne added a comment -

            i hope it can and that this is more important than minor, since i've already given the code out to a number of people who were using rice and missing a batch framework lmk if you want to talk about this. i'm going to bump up the priority of this issue

            Show
            abyrne Ailish Byrne added a comment - i hope it can and that this is more important than minor, since i've already given the code out to a number of people who were using rice and missing a batch framework lmk if you want to talk about this. i'm going to bump up the priority of this issue
            Hide
            ewestfal Eric Westfall added a comment -

            David, the extraction of the batch framework needs to happen asap because it's hindering some other work. I think you are the man for the job on this one Let's talk about this when we meet tomorrow morning. I've already discussed this some with Ailish/Jonathan so I will let them know what we come up with on this one as far as an extraction plan goes. Thanks!

            Show
            ewestfal Eric Westfall added a comment - David, the extraction of the batch framework needs to happen asap because it's hindering some other work. I think you are the man for the job on this one Let's talk about this when we meet tomorrow morning. I've already discussed this some with Ailish/Jonathan so I will let them know what we come up with on this one as far as an extraction plan goes. Thanks!
            Hide
            ewestfal Eric Westfall added a comment -

            Notes from our meeting (David and I) this morning:

            Non-Rice Classes depended upon by KFS Batch:

            org.kuali.kfs.sys.service.ParameterService;
            import org.kuali.kfs.sys.KFSConstants;
            import org.kuali.kfs.sys.context.SpringContext; (to get SchedulerService, beans of type Step.class, etc.)
            import org.kuali.kfs.sys.service.ParameterService;
            import org.kuali.kfs.sys.context.NDCFilter;
            import org.kuali.kfs.sys.document.dataaccess.FinancialSystemDocumentHeaderDao;

            • referenced from PurgeDocumentContentsStep

            Extraction Notes:

            1) Everything under org.kuali.kfs.sys.batch should be extracted. With the possible exception of:

            • PurgeDocumentContentsStep since it references FinancialSystemDocumentHeaderDao
            • this may also include the KFS SchedulerFactoryBean (see notes below on this)
              2) Batch classes should go into the KNS module
            • another possibility is to create these in the core module, but that would likely result in us bringing a lot more into that core module
            • need to determine if the time required to do this is worth it
              3) ParameterService and related classes will also need to be extracted since there is a bi-directional dependency between this and batch

            Implementation notes of batch in Rice:

            1) Rice will have a single scheduler
            2) Clients can customize by injecting their scheduler into Rice Configurer
            3) Remove the injection of the scheduler into KSB Configurer
            4) By default, if no scheduler injected, use the RiceSchedulerFactoryBean
            5) Jobs in Quartz need to be able to be identified by whether they are batch jobs, or ksb message retries

            • This helps to determine rendering in the KSB Quartz.do or the Batch screens
            • thought was given to if we should include all of these in a single screen and we thought having a bunch of service bus message retries in the batch screen would end up cluttering it up
              6) It needs to possible to include the batch GUI in a rice client application (in particular batchModify.do)
              7) Ultimately, there will be a batch scheduler per Rice client application
            • This means each rice client application will have it's own copy of the quartz tables (are there other tables as well?)
            • This is the same as how the KSB scheduler is implemented
              8) Add the ability to turn off just the batch piece (see questions below).

            Implementation notes for KFS:

            1) KFS will inject it's SchedulerFactoryBean into RiceConfigurer. This will allow for the useQuartzScheduling configuration to still work for KFS
            2) We considered extracting this as well, but since Rice will use a single scheduler internally, it doesn't really make sense to turn off the
            scheduler globally like this. However, if we are able to allow for turning off just the batch piece of the scheduler (and not the KSB) then KFS could just use the same scheduler and set the appropriate config property.

            Questions:

            1) Will it be possible to distinguish batch jobs from messages in the scheduler?

            • can this be implemented with JobDetail.jobDataMap by adding a type code of some sort to this map?
              2) Will batch jobs/messages need different sets of properties?
              3) If problems with above, will we need to have seperate schedulers for the KSB vs. Batch?

            Moving forward:

            1) Discuss plan with KFS representitives
            2) Attempt to extract and identify problems
            3) Work with KFS team to update KFS appropriately
            4) Is there a way in Quartz to turn off certain jobs? We would need something like this is order to allow them to turn off just the batch piece but not the KSB piece.

            Show
            ewestfal Eric Westfall added a comment - Notes from our meeting (David and I) this morning: Non-Rice Classes depended upon by KFS Batch: org.kuali.kfs.sys.service.ParameterService; import org.kuali.kfs.sys.KFSConstants; import org.kuali.kfs.sys.context.SpringContext; (to get SchedulerService, beans of type Step.class, etc.) import org.kuali.kfs.sys.service.ParameterService; import org.kuali.kfs.sys.context.NDCFilter; import org.kuali.kfs.sys.document.dataaccess.FinancialSystemDocumentHeaderDao; referenced from PurgeDocumentContentsStep Extraction Notes: 1) Everything under org.kuali.kfs.sys.batch should be extracted. With the possible exception of: PurgeDocumentContentsStep since it references FinancialSystemDocumentHeaderDao this may also include the KFS SchedulerFactoryBean (see notes below on this) 2) Batch classes should go into the KNS module another possibility is to create these in the core module, but that would likely result in us bringing a lot more into that core module need to determine if the time required to do this is worth it 3) ParameterService and related classes will also need to be extracted since there is a bi-directional dependency between this and batch Implementation notes of batch in Rice: 1) Rice will have a single scheduler 2) Clients can customize by injecting their scheduler into Rice Configurer 3) Remove the injection of the scheduler into KSB Configurer 4) By default, if no scheduler injected, use the RiceSchedulerFactoryBean 5) Jobs in Quartz need to be able to be identified by whether they are batch jobs, or ksb message retries This helps to determine rendering in the KSB Quartz.do or the Batch screens thought was given to if we should include all of these in a single screen and we thought having a bunch of service bus message retries in the batch screen would end up cluttering it up 6) It needs to possible to include the batch GUI in a rice client application (in particular batchModify.do) 7) Ultimately, there will be a batch scheduler per Rice client application This means each rice client application will have it's own copy of the quartz tables (are there other tables as well?) This is the same as how the KSB scheduler is implemented 8) Add the ability to turn off just the batch piece (see questions below). Implementation notes for KFS: 1) KFS will inject it's SchedulerFactoryBean into RiceConfigurer. This will allow for the useQuartzScheduling configuration to still work for KFS 2) We considered extracting this as well, but since Rice will use a single scheduler internally, it doesn't really make sense to turn off the scheduler globally like this. However, if we are able to allow for turning off just the batch piece of the scheduler (and not the KSB) then KFS could just use the same scheduler and set the appropriate config property. Questions: 1) Will it be possible to distinguish batch jobs from messages in the scheduler? can this be implemented with JobDetail.jobDataMap by adding a type code of some sort to this map? 2) Will batch jobs/messages need different sets of properties? 3) If problems with above, will we need to have seperate schedulers for the KSB vs. Batch? Moving forward: 1) Discuss plan with KFS representitives 2) Attempt to extract and identify problems 3) Work with KFS team to update KFS appropriately 4) Is there a way in Quartz to turn off certain jobs? We would need something like this is order to allow them to turn off just the batch piece but not the KSB piece.
            Hide
            ewestfal Eric Westfall added a comment -

            Work here was mostly completed, however merging the changes back into KFS was not feasible prior to freeze date. This extraction was not a requirement for KFS 3.0 but was rather intended to provide this functionality for other Rice applications that might want to use it in the future.

            David, to resolve this issue in the short term, let's create a branch off of KFS and a branch off of Rice where you can commit your work. That way we won't lose the work that you did and we can have someone look at bringing the batch functionality into Rice 1.1, ideally in a way that will be non-impacting to KFS (i.e. they could continue to keep their batch system and migrate at their leisure). Thanks!

            Show
            ewestfal Eric Westfall added a comment - Work here was mostly completed, however merging the changes back into KFS was not feasible prior to freeze date. This extraction was not a requirement for KFS 3.0 but was rather intended to provide this functionality for other Rice applications that might want to use it in the future. David, to resolve this issue in the short term, let's create a branch off of KFS and a branch off of Rice where you can commit your work. That way we won't lose the work that you did and we can have someone look at bringing the batch functionality into Rice 1.1, ideally in a way that will be non-impacting to KFS (i.e. they could continue to keep their batch system and migrate at their leisure). Thanks!
            Hide
            delyea David Elyea added a comment -

            Removing me as the Assignee since the work i did is too old to matter at this point. It would be easier to start over. If this ever becomes a priority the assignee can contact me if they have questions and i can try to fill them in on my original analysis.

            Show
            delyea David Elyea added a comment - Removing me as the Assignee since the work i did is too old to matter at this point. It would be easier to start over. If this ever becomes a priority the assignee can contact me if they have questions and i can try to fill them in on my original analysis.
            Hide
            jcoltrin Jessica Coltrin (Inactive) added a comment -

            This should become a roadmap item and we should look at Spring batch when we look at it.

            Show
            jcoltrin Jessica Coltrin (Inactive) added a comment - This should become a roadmap item and we should look at Spring batch when we look at it.
            Hide
            masargen Matt Sargent added a comment -

            Dup of KRRM-6

            Show
            masargen Matt Sargent added a comment - Dup of KRRM-6

              People

              • Assignee:
                Unassigned
                Reporter:
                ewestfal Eric Westfall
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: