Details

    • Type: Bug Fix Bug Fix
    • Status: Open Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Backlog
    • Component/s: Performance
    • Labels:
    • Similar issues:
      KULRICE-1214Allow ingestion of non-file-based resources
      KULRICE-12491Performance improvement for readXml in XmlHelper
      KULRICE-13834Make XML Ingester available in KRAD SampleApp
      KULRICE-9386Issues with doctype xml ingestion
      KULRICE-8448Improve Role Performance
      KULRICE-2634Implement support for better XML export, XML import and migration features
      KULRICE-6735Document the Performance Improvements
      KULRICE-2450XML Ingester does not display helpful error messages
      KULRICE-3279The XPathQualifierResolver Rule Attribute ingestion is broken and not ingesting the configuration section of the xml
      KULRICE-4624Reinvestigation into applying nice formatting and rendering for exported XML
    • Rice Module:
      KEW
    • Application Requirement:
      KFS

      Description

      From: Byrne, Ailish M
      Sent: Sunday, September 16, 2007 7:41 PM
      To: May, Teresa A; Westfall, Eric Curtis
      Subject: RE: something i've been thinking about...

      After just sitting here for the last 4 hours cleaning up crap doc types and spending a bunch of time on workgroups yesterday in addition to the time that Damon and the 50 people who reported confusion in Jira already spent, my resolve to eliminate our current workflow data strategy is strengthened¿

      After just sitting here for the last 4 hours cleaning up crap doc types and spending a bunch of time on workgroups yesterday in addition to the time that Damon and the 50 people who reported confusion in Jira already spent, my resolve to eliminate our current workflow data strategy is strengthened¿

      ________________________________________
      From: Westfall, Eric Curtis
      Sent: Friday, September 14, 2007 11:56 AM
      To: May, Teresa A; Byrne, Ailish M
      Subject: Re: something i've been thinking about...

      Regarding the XML ingestion:
      1. We have support for updating workgroups now, but it's similar to rule templates where you have to specify an allowOverwrite="true" attribute on the <workgroup> element.
      2. We have support for updating of rules now, but you have to specify a <name> element in the XML to uniquely identify it.

      Also, importing of XML for rules can be very slow (this is probably what Teresa was seeing). There's an inefficiency in the code right now where it loads rules in from the database to do a duplicate check. Also, rule imports cause the rule caches to be flushed and there's one of those per document type (times however many document types in KFS) so it can also take a little bit of time to dispatch cache flush messages across the cluster.

      So I would agree with Teresa that the XML ingestion process is a bit clunky, I think it would work to do what you're suggesting with a fresh system, it might just take awhile. Probably having the Rice team get in there and fix up the performance issues with the Rule XML ingest would give the greatest gains.

      Thanks,
      Eric

      On 9/14/07 10:47 AM, "May, Teresa A" <temay@indiana.edu> wrote:
      Ailish:

      I just want to reiterate the two concerns that I was trying to express in my earlier e-mail.

      I hadn't had any trouble until yesterday with adding new rows to tables by ingesting the XML online. Updates wouldn't work and they hadn't worked earlier in the offline process either, but if you are saying that all workflow tables would be recreated with the XML each time, this would eliminate that deficiency with the offline process. The second problem then that I was alluding to, in terms of the changes that David had been making, was the complexity of keeping the XML in line with the results of making changes through the UI. This would be dependent on what "maintaining XML" would entail.

      It is ironic that yesterday I ran into a problem with ingesting a new rule online. I had three new rules from REG to ingest into KULDEV and KULDBA. KULDEV worked fine. I had ingested two of the rules into KULDBA and then had trouble with the third. I attempted twice to ingest the rule, but the progress bar disappeared both times, there was no error, and the ingestion message on the screen was for the previous ingestion of the second XML file. I went into another session and tried a third time and then got an error that there was a duplicate rule or that it was trying to load a duplicate rule. I am not sure. When I checked the rule in KULDBA, it had been loaded twice in EN_RULE_BASE_VAL_T and both rows were the same version and were both active and current. Since I was wary of the second row and not sure if all of the elements in the other corresponding tables would have been loaded correctly that second time, I made the second row inactive and not current.

      I guess what I am saying is that my experiences with the XML ingestion process(es) makes me feel that it has been lacking and would need further consideration on how to make it work better.

      Teresa

      ________________________________________
      From: Byrne, Ailish M
      Sent: Thursday, September 13, 2007 6:43 PM
      To: Byrne, Ailish M; May, Teresa A; Westfall, Eric Curtis
      Subject: RE: something i've been thinking about...

      One pretty obvious concern that might immediately veto this idea is the time it would take. I have no idea how long it would take for workflow to ingest all our data.

      From: Byrne, Ailish M
      Sent: Thursday, September 13, 2007 10:34 AM
      To: May, Teresa A; Westfall, Eric Curtis
      Subject: RE: something i've been thinking about...

      My thought on how that would be handled is that workflow configuration (doc types, workgroups, attributes) would not be part of the base data set. Rather Gary and others would be maintaining XML that is checked into the project, and the XML would be ingested (automatically by being in the directory that workflow polls) each time a schema is overlaid. Does this make sense? I don't know if this is the best idea. Just some thoughts that I thought you both might have some good feedback on¿

      From: May, Teresa A
      Sent: Thursday, September 13, 2007 9:54 AM
      To: Byrne, Ailish M; Westfall, Eric Curtis
      Subject: RE: something i've been thinking about...

      One thing I am not clear about. When you use the ingester online, you cannot update data. It is only good for adding new data unless you blow away what is already there prior to ingesting the data and this may involve more than one table. For example, I have been adding Purchasing workgroup data that is being entered into REG. Some of this involves additions of new workgroups and I can use the export/ingester method, but some are updates and I use SQL to update the existing data. If I try the XML/ingester method, I get an error saying that the workgroup already exists. In this case, you are dealing with the workgroup table and the workgroup member table. But there are also additional workgroup tables now.

      A couple of years ago, I used the XML and startup of workflow to load all of the Travel workgroups and rules data. This data was all new. However, if I am remembering correctly, if there was
      a problem with the loaded data, I had to delete it prior to reloading the modified data. Otherwise, I got an error saying that a name had already been used. I think that some data generated this kind of error if that data already existed, and in a couple of places, duplicates were not checked but could be problematic if introduced. I had to be careful not to introduce duplicates in those places since a check was not being made.

      Has this process been modified to be more flexible with updates? What happens if changes are made similar to what David Elyea has been making where he is changing attributes, rules, templates, etc. over many tables? How will such extensive updates be handled?

      Thanks.

      Teresa

      ________________________________________
      From: Byrne, Ailish M
      Sent: Thursday, September 13, 2007 12:21 AM
      To: Westfall, Eric Curtis
      Cc: May, Teresa A
      Subject: something i've been thinking about...

      Wondering about both data migrations and production implementations - in particular our current nasty copy all workflow data from dev to dba strategy. It seems like it would be better for us to take advantage of the xml polling business, point at a real dir on the server, have the right workflow xml checked in and drop it in that dir before the app starts up after a db overlay. Does this make sense to you, or am I being crazy?

      Thanks,
      A

        Issue Links

          Activity

          Hide
          Aaron Godert (Inactive) added a comment -

          Laran was mentioning the current issues to me this morning when I bumped into him.

          I told him that 0.9.3 will bring back the embedded plugin, so then you should be able to delegate the XML building and massive data gathering work to the workflow engine asynchronously.

          Here was a thought for the interim:
          1.) Put the data gathering and workflow document route call in a single service (if it's not already in one)
          2.) Expose that service on the bus as an asynchronous service that can be called remotely
          3.) Have your "route" action in the struts action for the problem docs (i.e. Award, Budget, et) call the exposed service remotely and in async
          4.) After the struts action confirms the request for the async call was made to the KSB that's it
          5.) The KSB messaging aspects should kick off the call back to KFS remotely and ansynchronously
          6.) End result: users aren't held up and you don't have to pump a bunch of XML over into Workflow

          I think this will work. May want to check with Eric first.

          Show
          Aaron Godert (Inactive) added a comment - Laran was mentioning the current issues to me this morning when I bumped into him. I told him that 0.9.3 will bring back the embedded plugin, so then you should be able to delegate the XML building and massive data gathering work to the workflow engine asynchronously. Here was a thought for the interim: 1.) Put the data gathering and workflow document route call in a single service (if it's not already in one) 2.) Expose that service on the bus as an asynchronous service that can be called remotely 3.) Have your "route" action in the struts action for the problem docs (i.e. Award, Budget, et) call the exposed service remotely and in async 4.) After the struts action confirms the request for the async call was made to the KSB that's it 5.) The KSB messaging aspects should kick off the call back to KFS remotely and ansynchronously 6.) End result: users aren't held up and you don't have to pump a bunch of XML over into Workflow I think this will work. May want to check with Eric first.
          Hide
          Ailish Byrne added a comment -

          aarons comment is actually related to the xml generation issues. we talked about it and it is similar to an idea i had. we'll be discussing at a meeting with the folks who've been working on this on monday. thanks, aaron!

          Show
          Ailish Byrne added a comment - aarons comment is actually related to the xml generation issues. we talked about it and it is similar to an idea i had. we'll be discussing at a meeting with the folks who've been working on this on monday. thanks, aaron!
          Hide
          Ailish Byrne added a comment -

          we will probably not care about this issue at all once we can put our workflow config in the data dictionary

          Show
          Ailish Byrne added a comment - we will probably not care about this issue at all once we can put our workflow config in the data dictionary

            People

            • Assignee:
              Unassigned
              Reporter:
              Dan Lemus (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:

                Structure Helper Panel