|
|
Post by mueh on Jun 8, 2024 8:30:31 GMT -5
Hi George ! Some thoughts to this problem . If you think they useless ignore them . No idea about windows internals but i understand z/OS Virtual storage management . Rava's reported problem and crashes i think it's just an undetected Out of Memory with 24069 . 24158 has some DEBUG on ( file scrolling is about 25% slower ) and reports sometimes Untrapped Error #7 (Out of memory) . the blank lines he describes after Load of large file are just zero length lines . here the test case with UNDO 0 in opt to avoid additional problems . To create the test file do Edit NEW insert one blank line and execute following cmd's line r999 .1;line rr399 .1;line rr .zl;munum 1300 creates 400.000 lines with line length 1300 and Number before col 1300 and size of 550 Mb . assuming that for TXT Data and Attribute we need 1.1 GB . SPFLite as 32Bit Application has limit of 2Gb . when you now save that file and load it after SPFLite restart ( no storage fragmentation ) you will see between 332000 and 333000 starting the zero length lines . ( no Number in Col 1300) . Above creation of files shows that SPFLite can handle 400.000 lines with length 1300 . Here the munum macro to number the file . MUNUM.macro (13.86 KB) I Think that there are still REDIM PRESERVE for some 'Lookup tables' done during Load which fragment the storage during load Here a screen shot after load of file with procexp64 display .  Thanks |
|
|
|
Post by George on Jun 8, 2024 9:32:37 GMT -5
MUEH: I hadn't actually calculated the total byte size, but yes it does look to be the problem. Early SPFLite releases had sigificant problems with large files, crashes, speed slowed to a crawl etc. I spent a lot of time back then to improve things. But you can't manufacture more memory.
When you come down to it, SPFLite is supposed to be a programmers editor, not a substitute for examining huge data files. And 1300 byte records are certainly not any kind of programming files.
Really, he should get one of the free huge file editors that are available.
Or perhaps get a file-splitter, break it into pieces and view each separately.
And yes, some releases got out with debugging still on. I added a step to my new version 'how-to' checklist to verify that, but to be honest, I don't always use the checklist, I've done it all so many times.
George
|
|
|
|
Post by Robert on Jun 8, 2024 11:40:42 GMT -5
George, if you remember the old (old) days, SPF running on TSO under MVS/SP allowed each TSO user maybe 4 MB of virtual memory. After SPF took up quite a bit of memory, it didn't leave much for user data. I had a 15,000 line COBOL program, and that was pushing it to try to edit it. This is why IBM designed BROWSE the way they did. You could browse a file of any size, because they only brought into memory as much of the file as need to view the 'vicinity' around the current file location. And, because BROWSE didn't allow updates, they could always discard any piece of the file that had been read in to get some more somewhere else.
I believe Windows Win32 has a thing called memory-mapped I/O, where they treat a file like an extension of the swap file, much like what SPF did. It may be too ambitious to tackle, but certainly a solution is within the realm of possibility.
R
|
|
|
|
Post by George on Jun 8, 2024 11:52:11 GMT -5
Robert: Yeah, the good old days. I remember it was a major team project when we got our 1st 370-168 (or was it still 360 then?). A multi-month project and it came with a whopping 16Mb memory. My phone has 128Gb.
File mapping is certainly possible, but would take considerable redesign. Everything internally lives and dies on the single UDT array containing the data. It's not like the logic is performing I/O during Edit/Browse/View - there is NO I/O taking place, making it kinda tough to use a memory mapped file.
George
|
|
|
|
Post by Robert on Jun 8, 2024 12:04:17 GMT -5
The only way this could possibly be done is that the UDT array would first have to be populated with an entry for each line, but the data portion would be a NULL pointer, and you'd save the offset of each line and its length. Then, when you wanted to access the data of a line, it would have to be done through a method which would determine if the data was resident or not. If not resident, you would have to do your own memory management, and when "too much" memory had already been taken up, you would have to decide where to give up the residence of some group of lines and grab their memory to use for something else.
In that way, you would manage memory yourself in a browse session rather than using Win32 memory mapped I/O.
Perhaps the Browse code could be cloned and altered to work this way. I assume you have Edit, Browse and View all in the same code because they are similar?
When V3 settles down, it could be something you could experiment with, if you still have some ambition left. I could be interesting.
R
|
|
|
|
Post by George on Jun 8, 2024 12:49:44 GMT -5
Robert: EDIT/BROWSE/VIEW all use the same code, the only difference is checking to determine what commands are allowed. e.g. no SAVE in BROWSE/VIEW, etc.
Replacing all references to the actual Text string with some Method which would locate and return the string is technically do-able, but there will be hundreds of places, and some of the String functions do not like Function/Method return values as operands. What else? Well, the Attribute strings have to be handled the same way. And ATTR changes are handled via PEEK/POKE actions, the IO here coukld be problematic. COPY filename could be tough as well. And UNDO/REDO would be a nightmare, if even possible.
This would also require even more testing than the V3 replacement of Parse required, since now we would be altering the actual 'working' code of the command modules.
Yes this could be 'interesting', but I'd be crazy to even start on it. Again - ROI. How often has this been a restriction?
George
|
|
|
|
Post by Robert on Jun 8, 2024 13:08:49 GMT -5
This is another case where telemetry could answer a lot of questions. I wonder if adding telemetry would be a better adventure than BROWSE?
I believe there are answers to many of the issues you bring up. Remembering that IBM-style browse is totally read-only, it implies that you couldn't change attributes, so no need to POKE; although IBM Browse did allow Exclude. IBM Browse restricted a LOT of EDIT functionality. You can't change data, so you can't COPY into a BROWSE session, and since you can't change anything, there is nothing to UNDO other than Excluded lines.
I believe some of the points you brought up could be worked around with adapter functions. Likely, the adapter functions could be integrated into PB macros that inserted calls to the adapters in some cases.
The main thing to remember is that IBM Browse existed to allow viewing of literally unlimited file sizes with a very minimal resource impact.
I think your final question is the best point: how often has this been a restriction? Right now, what happens when a file is "too big"? Does SPFLite even check for this, or do you just load, load, load your file into memory until a crash happens?
R
|
|
|
|
Post by George on Jun 8, 2024 14:55:30 GMT -5
Robert: Remember that all that 'stuff' that doesn't need to be done in Browse, would have to be identified and made conditional. Right now it doesn't. Browse is simply one entry in a flag says (BROWSE | EDIT | VIEW). Trying to adapt all the commands and line commands and macro interfaces such that they recognize Browse and somehow act differently is just never going to happen.
Users with huge files that cannot be loaded will have to look elsewhere, there's lots of freeware out there to assist.
The fact that good old ISPF managed to do this is immaterial, we can only be an ISPF clone so far.
As to what happpens if a user tries and memory 'runs out'? - probably a crash.
George
|
|
|
|
Post by Robert on Jun 8, 2024 15:09:31 GMT -5
I believe that ISPF Browse doesn't really use ISPF Edit code, they completely wrote Browse from scratch. So there wasn't conditional commands per se, you just couldn't use anything that would cause a modification, like CHANGE primary cmd and I, D, R, )) (( etc.
When V3 gets to a stable point and you put out a production release, maybe I will download the source and look at it, just for jollies.
R
|
|
