|
Post by George on Jun 6, 2020 9:37:36 GMT -5
Robert: The array being expanded is NOT the string pool (that's separate), it is the active, assigned text lines. These entries cannot have 'spares' embedded - we'd need a whole new line-type, and have to review all code to ensure they don't get included in some functions processing. And if they're available, we'd have to add some kind of new pool of pointers to track where they are and which are available, and something like SORT would move them around and we'd have to adjust those pool pointers, and .... and on and on it goes.
Unfortunately I'm the only one who truly understands all the internals. MUEH is doing amazingly well, but still misses a lot of the nuances in there. Like his recent request for INSTANCE based Recent Files. He was pretty close, but overlooked another whole function needing updates.
I don't enjoy shooting down other's ideas, I'm not asking for the ideas to stop, just for understanding that SPFLite has grown into an incredibly convoluted piece of work as we've added and bolted on all kinds of new facilities.
Could it be re-written / re-structured to be easier to maintain? Sure, but doing that for a 50K+ source line work is not going to happen in my limited lifetime. Unless someone steps up, gets the compiler, and actually starts to work on this, you guys are all stuck with whatever I can provide.
George
|
|
|
Post by George on Jun 7, 2020 16:27:34 GMT -5
OK, Here's the latest.
This has been a very complex, multi function correction. But I think I've got things pinned down. It a combination of things.
Parsing the file data without repeated re-allocations as the buffer data is parsed.
Mainly, changing the routine which allocates a string buffer for each text line to avoid doing a full scan of the allocation control string, rather than scanning from the last allocated position. Normally this doesn't matter, but as the control string gets into the megabyte size, this can't be ignored.
So we're getting closer.
George
|
|
|
Post by George on Jun 8, 2020 12:42:48 GMT -5
I think we have lift-off. I've merged my test changes into the prod version. Just did a test, loading a file with about 1.25 million records (about 50 bytes each) and it took under 15 seconds. Another test was a 180 meg file (but only 160,000 records), it takes about 3 seconds. It's taken a pretty major re-write of the single CopyFile routine to get here. Robert: One casualty is that AUTONL will no longer handle overprint lines (we used to merge into 1 line). I tried, but could not retain that capability. I don't see that as a major issue. I'd appreciate anyone willing to try this version out. Just be aware it is a really early version, be careful. George SPFLite22.exe Removed as it has serious problems. If you downloaded it previously, I'd recommend you not use it for anything other than benchmark speed tests. Do not trust it for editing very large files. George Attachments:SPFLite22.exe (476 KB)
|
|
|
Post by George on Jun 8, 2020 15:38:59 GMT -5
Well, right now I'm not going to touch AUTO and AUTONL, frankly I've never properly understood the difference, it's kinda obscure. I'm not sure there are many users browsing Hercules print files, and if an overprint doesn't get 'properly' handled, I doubt that anyone will even notice. AUTO 'tidies up' quite a bit, I'm sure most users are happy with what it does.
George
Sorry missed your comment re: Get_EOL_STR$. The function makes no distinction between AUTO and AUTONL, it just returns CRLF.
|
|
|
Post by mueh on Jun 9, 2020 6:59:17 GMT -5
George : Good News . 20159 load times dropped significant . I did an medit of 2159 files in sum 90Mb with 2.5 Million lines and load time is 1 Min instead of 30 Min before . For single large files the improvemnt is even better . 2 Questions . 1 . Will you remove the 1.5 Million line Limit for a single file . ( after line 1499998 line data is lost ) 2. do you plan to improve performance for XFORM Add_Line 's the same way .
Thanks for great Job .
|
|
|
Post by George on Jun 9, 2020 9:31:37 GMT -5
mueh: 1.5 million limit? There is none. You're saying it won't go past 1499998? That's a serious bug I'll go check out.
MEDIT of 2159 files? You had to have done that with a macro using repetitive MEDIT commands.
XFORM_Add_Line I haven't even looked at yet, but I'm sure we can do something.
George
|
|
|
Post by mueh on Jun 9, 2020 9:46:26 GMT -5
George : FM all M command on flist having all the pathes of my Hercules source files . Here the screenshot of 1.5 Million lines Limit on single file
|
|
|
Post by George on Jun 9, 2020 11:59:38 GMT -5
MUEH: There is something VERY wrong going on. I would not use the latest Beta version for ANYTHING productive till I find it. The 1,500,000 boundary is totally dumbfounding. I have no limit like that anywhere, but it just goes bonkers at that boundary for some reason. George [UPDATE] Really tough one to pin down, but it ended up being a minor change to 1 line to correct, isn't that always the way. The new loading method of determining how many lines are in the file before making sure there are available 'slots' to store them in, exposed a flaw in how tables get expanded when they are too small. Attached is a corrected version. SPFLite22.exe (476 KB) George [\UPDATE]
|
|