|
Post by George on Oct 15, 2022 20:25:11 GMT -5
FM needs to get the Profile STATE value to know if it is possible to get the file's LINE count from the STATE file. This is done as it collects the file's data.
Yes, there is a collision between the DEFAULT list and EFT, the doc will have to point that out.
Yes, EFT entries are memory resident.
When we added file properties to FM (Title, Resolution, Size etc.) it was a big performance hit for FM, that's why I'm leery of every extra bit of processing.
The DEFAULT list only gets checked the same as EFT will, there's no need to list everywhere it's used. George
|
|
|
Post by Stefan on Oct 16, 2022 4:11:50 GMT -5
FWIW... I worry that this EFT idea may not come to pass. Robert,I probably mis-understood, but I think most users will treat EFT as " providing a subdivision when fileExtension is too broad". So most users' EFT file will be relatively short compared to all the file types they deal with. Hence what if every time SPFLite is presented with a file, it checks the EFT file first. Logic goes like this... IF EFT-match THEN profnam = Eftnam ELSE profnam = fileExtension ENDIFIF EXISTS(profnam in useDEFAULT list ) THEN profnam = "DEFAULT" ELSE IF NOT EXISTS(profnam in CFG ) THEN pop up dialog as now to ask user Valid "profile" options are: (a) Use DEFAULT (b) Use an existing profile (list) (c) Make new Profile ENDIF For "(a) Use DEFAULT" add file-extension to current DEFAULT list. For (b) and (c) options... IF profnam = file-extension THEN Make new profile in CFG ELSE add entry for fully-qualified-filename to EFT and let the user manually tweak the EFT list if required ENDIFENDIF
Basically, if there are clashes between the current DEFAULTS list and new EFT entries, EFT wins. Let the users manually modify their EFT or the DEFAULT list if it bothers them.
George, If I understand your FM discussion correctly, you want to know line count for which you need know State(ON/OFF), for which you need to know which Profile applies to the line entry. My first thought was "sod the line count". Rationale being, most users probably don't run with STATE ON anyway so are used to not seeing the counts. I can't recall ever 'using' those counts for anything, but that's just me.
However, to reduce the FM-refresh performance hit, what if you were to introduce a "State On" flag (Y/N/?) for the FM list, populated when you first build the FM list?
Then on FM-Refresh you only need to 'dig around for State and Profile info' for those entries where the flag is "?". Any performance 'hit' would still be there but would be limited to 'first' load of a filelist only.
Alternatively, (and I half assume you already do this)... ...perhaps, once obtained, you could maintain the line count for each FM list entry, and just modify it every time a file/entry exists EDIT, rather than re-reading all the entries' line counts all over again on FM Refresh?
|
|
|
Post by George on Oct 16, 2022 10:21:25 GMT -5
Stefan: Your EFT logic is 99% exactly what is happening, except for the last couple about creating new Profiles (that's simply because I haven't touched it yet)
Robert: I have repeatedly said 'let's get this out and see about performance after'. That doesn't remove my concerns, just defers them till we can test it out.
How many profiles do I have? I have over 90 right now, but then I'm constantly testing and mucking about with test files, looking for bugs. Like you, 15-20 would be all I'd normally need.
Caching Profiles would certainly help, but it's a very problematic change, profile fetching and storing actions happen all over the place.
e.g. right now some code could say Prf.HEX = &True (say the Profile is DEFAULT) the LOCK status is checked, the internal HEX field is updated and if UNLOCKED, the CFG file is updated. If LOCKED the CFG is untouched, only the single tabs' Profile is altered. I guess the cache version then gets updated only along with the CFG file. This is another of those 'add a few lines' in each of the profile variable SET routines.
Since this is performance related, and we don't even know WHETHER it will be needed, this will have to wait for that.
Where does your code stand? Still testing? When can we push out a Beta?
George
|
|
|
Post by George on Oct 16, 2022 12:28:28 GMT -5
Robert: Just FYI, all Profile stuff is already maintained and accessed from an Object. My Reference above to Prf.HEX was such a reference, fetching the HEX value from the Prf object. But Objects don't necessarily make adding a cache simpler.
Right now most code just accesses settings like that as if it were a simple variable (for example, as Prf.HEX). I would have to find all Prf.xxx references and choose whether to continue to use Prf.xxx (to get the active Profile value) or to change the request to something else like Prf.Cxxx to get the cached, locked Profile value. (This means a whole new set of Property GET routines.) And still have to modify all the Property SET routines to handle updating CFG or not, Cache or not). Icky!
There are no nice tidy solutions.
|
|
|
Post by George on Oct 16, 2022 14:28:55 GMT -5
Profiles are swapped in and out all the time. Think MEdit where all the files may not be the same Profile type.
A Profile object is simply the current collection of Profile variables based on where and how it's used. e.g. Every open tab (including FM) has it's own Profile object representing THAT tabs 'current' profile. i.e. Every tab has a 'Prf' object defined. So, within that tab, any code that asks for Prf.HEX gets the value from THAT tabs Prf Object. Any command like HEX ON/OFF does the SET to the Prf object within that tab. (N.B. Prf here is not something special, it's just the name of the Profile object as I have declared it within the tab's own data object. (Which if you looked at SPFLite code) is pointed to by the GLOBAL TP variable, which is a pointer to an object.
Objects are fine, and provide great data isolation, but a Profile cache will be a global resource, so handling it has to be done carefully, especially if it is done within the Prf objects which are local.
|
|
|
Post by George on Oct 16, 2022 15:56:45 GMT -5
Robert: You've totally lost me. I do not grasp the <- syntax diagram type stuff, to me it's just meaningless confusion.
The profile for a tab if fetched from the CFG file. That is the basis for all Profile related decisions in that tab.
If a Profile variable is altered, the Tab's version of the Profile data is ALWAYS altered. And if the profile is UNLOCKED, that variable in the profile is altered in the CFG database as well.
So, to me, your suggestion that I might be able to 'revise the definition of a Profile Object to reflect this bi-modal behavior" is meaningless. Just what would does that mean? You're pointing in a direction, and I don't have a compass.
I mean the Profile data is already bi-modal, it can be read, and it can be altered (under LOCK control), so what are you suggesting to change?
Mark me very confused.
George
|
|
|
Post by Stefan on Oct 17, 2022 8:57:17 GMT -5
Sounds like you have reached an accord over performance. Let's get the thing out there in a beta and see if there are spanners in the works.
You can always add a line to potential beta testers that "...FM performance may be affected, so you're advised to keep your previous SPFLITE2.EXE so you can just copy it back into your C:\Program files...\SPFLITE2 folder if/when you feel the pain"
|
|
|
Post by George on Oct 17, 2022 9:05:12 GMT -5
Robert: No problem if REGEX is needed. If you do, it's probably better to use the PCRE version rather then the built in PB version.
Calling is simple, here's a sample:
t = PCRERegexCompile("\.[0-9]{6}\-[0-9]{6}\.[0-9]{6}") ' Setup PCRE Search for .999999-999999.999999 PCRERegexTest(str, 1, j, k) ' Look like a Backup file?
t = Null string if the compile OK, or an error message string The test scans the str, starting at position 1, and returns j=found location and k=length
Problem is you'd need the code for the two functions as well as some global and initialization code - awkward
Maybe use PB's version, with a view to swapping later to the PCRE version. The PB version is very close:
REGEXPR mask$ IN target$ [AT start&] TO iPos& [, iLen&]
George
|
|
|
Post by George on Oct 17, 2022 13:34:33 GMT -5
Sounds good. Going from the PB version to the PCRE version is trivial.
George
|
|
|
Post by Stefan on Oct 18, 2022 3:38:18 GMT -5
George, good news. I have been trying to figure out how to match a pattern like ABC*.TXT when it doesn't start with a delimiter. <snip> . . . . . For users very concerned about performance, there is a simple solution: Don't write patterns like ABC*.TXT but use \*ABC*.TXT instead. R R,
This may be a stupid thing to suggest, but...
With any user input, I try to accommodate the simplest (most lazy) way a user may provide the required data.
The approach I usually deploy when I need something 'delimited' but the user may or may not enter it that way, is to add my own delimiter where I need it. So if I wanted ABC*.TXT to be provided as \*ABC*.TXT, I add the '\*' myself where required before I process the entry.
In many cases, changing the actual 'entry' value is undesirable, so I leave it asis and use a construct like... pfx$ = IIF$( LEFT$(entry$,2) = "\*", "", "\*" )
and thereafter process the data as pfx$+entry$, e.g. FIND(pfx$+entry$), dotPos=INSTR(pfx$+entry$,".")-2, etc
Maybe this makes the REGEx less complex and possibly faster?
As an aside, when I first started with REGEX, I struggled to ensure that 'special' characters within text needed to be 'escaped', until I came across the \Q...\E sequence. More readable too than lots of '\' chars.
|
|
|
Post by Stefan on Oct 31, 2022 5:28:54 GMT -5
Robert, I'm excited about EFT and have gone with beta version 22301, but I'd appreciate some help, please. Despite having read the "Lyrics" example in the latest Quick-Start documentation, I'm struggling with the EFT syntax. Take a CFGMaint export file.
I should like to use a Profile called TXT_CFG instead of the normal TXT profile, irrespective of the folder structure in which the file is stored. Sample Filename: CFGMaint EXP 2022-10-27 10.54.TXT I thought I'd use the "CFGMaint EXP" part of the name as the 'trigger' and created a TXT_CFG profile.
So far my EFT entries have grown to this, but the file still loads with the TXT profile. "\CFGMaint EXP*.TXT" = TXT_CFG "\CFGMaint EXP" = TXT_CFG "\CFGMaint*.TXT" = TXT_CFG "\CFGMaint" = TXT_CFG "\CFGMaint*" = TXT_CFG
"\CFGMAINT EXP*.TXT" = TXT_CFG "\CFGMAINT EXP" = TXT_CFG "\CFGMAINT*.TXT" = TXT_CFG "\CFGMAINT" = TXT_CFG "\CFGMAINT*" = TXT_CFG "CFGMAINT EXP*.TXT" = TXT_CFG "CFGMAINT EXP.TXT" = TXT_CFG "CFGMAINT*.TXT" = TXT_CFG "CFGMAINT" = TXT_CFG "CFGMaint*" = TXT_CFG
I also tried it with TXTCFG (without the underscore) and without spaces around the = (equals) sign, but I've run out of straw to grasp.
What EFT statement should I deploy to get this file to load with the named profile, please?
|
|
|
Post by George on Oct 31, 2022 8:32:32 GMT -5
Stefan: I'll leave this for Robert. I'm sure it's the spaces in the name, and he does have some optional wild-card chars that address the issue.
George
|
|
|
Post by George on Oct 31, 2022 10:57:43 GMT -5
Robert: You should be able to test with ANY backup files, the filename format is fixed.
George
|
|
|
Post by Stefan on Oct 31, 2022 12:57:14 GMT -5
R, Thank you! I wasn't looking for bugs, just inspiration. The many duplicate entries came about because I was working through various options, including the possibility that the rules might be case-sensitive. Every time a rule didn't work, I added another rule to see if that would be better. I didn't delete the 'duff' ones as they formed a useful history of what had already been tried, and, as they hadn't matched, they were doing no harm at this time. The "Quick-Start doc did mention that ' .' was a delimiter and you're right, I missed the one in the 'time' part of the filename. There's no "Probable" in your conclusion, I knew it was "DEFINITELY user error" Thank you for the solution.
I tried it and it worked, even if it slightly missed the requirement of matching only the CFGMAINT EXPort files, but the EFT statement
"CFGMaint EXP*.*.TXT" = TXT_CFG does the job admirably. I think the final documentation should include a few examples of different EFT statements, deliberately including some that won't work. Then you can use them to explain why each one works or doesn't work, high-lighting easily avoidable "gotchas".
However, it started me thinking... I reckon the 'dot' rule will potentially catch out many a user, given it is the only character out of the (./:) trio which can occurs 'naturally' WITHIN a filename.
This makes it tricky to create reliable EFT statements, because at the time the user creates an EFT entry, s/he may not know if future filenames contain dots, and if so, how many. Your "\LYRICS\*.TXT" = LYRICS example matches "Billy Joel - Piano Man" but would fail to match "You're the Boss - B.B. King" or any track by "R.E.M.".
That's probably not very desirable, and neither is having multiple EFT patterns to match the various scenarios. Food for thought?
|
|
|
Post by Stefan on Oct 31, 2022 15:39:32 GMT -5
R.
Yes, I agree with all your reasoning. I especially like your saying! And I support the EFT-Tester idea.
Hoever, I don't think I understand your "2 ways" example's relevance to the "dot issue".
Given that the dot is ALWAYS a delimiter, you cannot wild-card it. It is always there.
As the creator, you understand this very much better than I, and I sincerely hope I'm wrong. Taking your LYRICS concept from the Quick-Start guide as an example...
...it seems to me that if you wanted to accommodate Lyrics from songs whose Artists' names included 0, 1, 2 or even 3 dots, you need to code 4 different EFT statements, one to accommodate each number of dots. That would address the 'known' file names but you still couldn't be sure that you match ALL relevant files correctly. A filename with 5 dots would fail to match and receive the wrong profile.
And dots can turn up anywhere, The track titles might also contain dots (Mr. Bojangles, etc). And in the path too. There's nothing to stop me creating a folder call A.B.C. I think the dot being a delimiter is at best a Nightmare, at worst an EFT deal breaker. I do not envy you your task, but genuinely wish you luck and success.
|
|