|
Post by George on Feb 29, 2024 16:19:59 GMT -5
The forum has been pretty quiet lately, but just as an FYI, I have been active in the background.
Because SPFLite has been poked and prodded for many, many years, some parts have become very convoluted and ugly to maintain.
So I've tackled re-writing the main command parser to clean up this area. So far so good, it's been an interesting exercise as I re-discover all the little 'wrinkles' that were introduced over the years.
While this will probably not affect the user experience, it is a long overdue maintenance item.
George
|
|
|
Post by Robert on Feb 29, 2024 17:34:35 GMT -5
I am wondering if this is an opportunity to add any parser-related features that might have been kicked around in the past but had been dismissed because it would have called for a "total rewrite". Now that you ARE doing a total rewrite, this could be your chance.
Thoughts?
===> Here was an idea I had recently that would involve the parser. You now have the concept of MACLIB enabled. Now, suppose you have two MACLIB's, call them AA and BB. Suppose there were different versions of a macro called MYMAC.MACRO, one in both of these libraries AA and BB, and you want to be able to get to either one at will. How?
The idea was to allow a "MACLIB qualifier". Suppose you want the version of MYMAC in library AA or BB. To issue this command on the primary command line, you could say something like this:
AA:MYMAC
or
BB:MYMAC
This would also give you a way to issue qualified macro names from a key-mapped key.
This way, you could 'define' MACLIB's without having to make any particular MACLIB the "default" one to access the macros in that library.
Since you are now working on the parser, maybe this is something you could add, if you thought it was worth doing. Main issue is detecting the "MACLIB qualification" syntax. You might have to do a "look-ahead" checking for:
name1 : name2
and then seeing if name1 was the name of a MACLIB symbol.
R
|
|
|
Post by George on Mar 1, 2024 11:21:56 GMT -5
Robert: Could be interesting. First I'll get things working to handle the existing syntax (which is mandatory).
I'm working through the commands now and each one seems to require some modification/correction to the parser. It shows just how many internal 'exceptions to the rule' have been introduced over the years. I've also found code supporting operands that aren't even documented. Those I have to watch for as some are 'internal only' and probably can't be just tossed out.
George
|
|
|
Post by Robert on Mar 1, 2024 11:38:16 GMT -5
George, that sounds like nature's way of telling you to make notes, and go back and update the doc. Undocumented features = unused features. New users will never know they exist. And, I am about as old a user as you got, and even *I* don't know about those undocumented features. All I really know is what is in the doc. I read the Help and act accordingly.
R
|
|
|
Post by George on Mar 1, 2024 14:47:41 GMT -5
Robert: That's why I have to check carefully. Some commands are issued internally with secret Kwds, for various reasons. It's not a big problem, just something to watch for. These internal Kwds will NEVER be documented in HELP.
George
|
|
|
Post by Robert on Mar 6, 2024 16:30:54 GMT -5
Any news? Awful lot of crickets out here ...
R
|
|
|
Post by George on Mar 7, 2024 9:19:06 GMT -5
Robert: I have the basic parser re-written, and the underlying manipulators (Search, Change, Join etc.), but each Primary command needs tweaking to accept the new method. In most cases it's fairly trivial, but some commands are quite a struggle. I'm working through them A-Z and am tackling LOCATE right now. It's gotta be the worst one so far, over 50 keyword type parameters mixed in with some having trailing numeric operands. It's a nightmare.
I'd like to respecify and re-write it, but we're stuck with what we've created.
Interesting during all this, I've found several places where the Doc is incomplete, or has conflicting information, so these are corrected as I go.
Fun, fun, fun!
George
|
|
|
Post by Robert on Mar 7, 2024 10:55:15 GMT -5
Yes, LOCATE is a charmer, but it's very useful. It's no wonder it's a challenge, but a worthwhile one.
I seem to recall a developer that had such a distaste for "make-work", and now, he is the one making work for himself. You must have a good reason, I am sure.
Parsers can be fun, the main thing to remember is to be as consistent as possible. That way, you don't "out-clever" yourself and end up creating brand-new bugs.
R
|
|
|
Post by Robert on Mar 7, 2024 11:33:05 GMT -5
George, any guess why the gang has been so quiet? I can't recall when there's been such a long time with essentially no user commentary coming in. Nobody has any questions? Nobody has found any bugs? Nada?
R
|
|
|
Post by George on Mar 7, 2024 14:29:48 GMT -5
Robert: The parser seems fine, the problem is commands (like LOCATE) which 'break the rules of "position independent" operands' which was always a hallmark of ISPF syntax.
But as to why tackle this? Winter cabin fever? Boredom? Who knows? The parsing has never been great, and as I go through the commands one by one migrating them to the 'new way', many of them have been simplified greatly. Simplification has to be better.
The problem is testing. There is simply no way I can ever test out each command and all it's variations of operands. This will need a lot of assistance from our forum testers.
George
|
|
|
Post by Robert on Mar 7, 2024 14:43:02 GMT -5
Maybe what you need is a macro to generate all combinations of operands of all commands, and then run them as a script.
R
|
|
|
Post by George on Mar 7, 2024 15:39:22 GMT -5
Robert: Yeah. Sure. I looked at that once. It's an enormous undertaking. Just look at the LOCATE command that we were talking about. Figure out what the test data would have to look like to encompass all the search types, all the different line types required (not all of which could be saved in STATE) and then the scripts with all the LOCATE commands issued one at a time. I'm not even sure how the script would be able to check if the command was successful or not.
Now multiply that by the 100+ commands and all THEIR variations, a lot of which would require unique test data files.
I should live so long. Sadly, this is just a dream.
George
|
|
|
Post by Robert on Mar 7, 2024 16:48:38 GMT -5
Yes, the macro would end up creating thousands of command lines.
I believe the answer to "how the script could check if the commands were successful" is for the script NOT to check.
Instead, you would need to add some internal support for this functionality. What I mean is, let's say you have a macro whose purpose is to create a primary-command test suite. When you run the test suite, its first command would be some internal command, for YOUR use only, which would set a flag indicating that you were testing the operation of SPFLite and NOT operating it normally. Then, after each command were attempted, the command parser code would determine if the command were valid or not, then perhaps write out a "log file" with the results of parsing each line.
At this point, I am assuming that your goal here is to validate that the parser is working correctly, and not that SPFLite itself is working correctly. Let us assume that the bulk of your (unchanged) code still works, so we are not trying to validate each and every little tidbit of code, right?
I believe that the basic validation idea can in fact be done, and that it's NOT just a dream. Nor do you need a miraculously lengthened lifespan to get it all done.
I think you are looking at it the wrong way. I don't believe that it is an "enormous undertaking". Not if it's done right.
R
|
|
|
Post by George on Mar 8, 2024 12:53:35 GMT -5
Robert: No, the problem is not just proving the parser is 'doing its job'. This change is NOT just affecting the parser.
When we're talking about the original or the new parser, there is simple NO one single defined interface between the code in the actual command and the data created by the parser. You can't just swap in a new parser and expect the command code to accept it as a 100% perfect substitute. So I can't revise/replace the parser without going into each and every command and 'walking the code' to see what has to be adapted.
We have commands ranging from END with no operands, to others with just a simple ON/OFF and those like CHANGE with every possible search option under the sun and similar modify options, and of course LOCATE with it's myriad options and custom parser.
So yes, we have to test that SPFLite (the whole thing) is working. It's true that masses of underlying support routines remain unchanged, but the command code sits between those routines and the parser, and with either the new or old parser, a lot of the command code still has a lot of unique operand validation to perform that no generalized parser could incorporate.
All I can do is plug along. Attempting some kind of automated validation is a total diversion.
George
|
|
|
Post by Robert on Mar 8, 2024 13:04:46 GMT -5
That is regrettable. My experience with parsers and lexers is that they ALWAYS define a single interface. For compiler projects, they pretty much have to. That is because most compilers use parser generators like YACC and scanners like LEX to do the work. When these are used, you CAN swap in a new parser, just like that. That is standard operating procedure.
I know that using such tools would be hard in SPFLite, since most of them are written in C or C++.
But, I do wonder how you are going to manage plugging along this way. You started off by saying there was no way you could test the parser and all its possible combinations. Now you say that even doing that much is not enough. But attempting an automated validation is a waste of time to you, and so you are going to count on the users to debug your new code, since you see no way to test it yourself. That is large burden to give the users, especially when they don't even know what they are trying to find wrong.
I can't and won't argue with you, but there are times I wish you did things differently.
I would ask that you issue a new production release that represents the code prior to your parser change plus all outstanding beta changes, so that if your new code is not stable, users will have an official release to fall back on.
R
|
|