|
|
Post by George on Aug 14, 2019 16:08:22 GMT -5
OK, I've been talking about a new way of storing Config data, so here are some details on what I'm proposing. Please keep in mind that most of us are just fine with the way things are, but some users are feeling restricted in various ways. What I'm proposing keeps things almost unchanged for the majority and gives new options to users IF they want to take advantage of them. FIRST For those of us, like me, who are fine with the way things are, then be assured that: - All your settings etc. will remain unchanged and will be migrated to
the new system.
- OPTIONS, PROF EDIT etc. will all look and act just the same as always.
- The \Documents\SPFLite\ folder will be 'tidier' as several files and
one folder will become obsolete.
- There is NO requirement to move, alter or otherwise change the current
\Documents\SPFLite folder other than deleting the obsolete items
mentioned in the previous point.
SECOND For those looking for more control over where SPFLite stores all the Config files, you will have it. For those who want to run multiple SPFLite sessions with different configuration options, you will have it DETAILS - The current INI file, along with the KBD, SPR, BKP, and SPS files and all the
\PROFILE\xxx.INI files will be collapsed into a single .CFG file.
- I know there are users who feel they should be able to 'peek' inside INI
files and alter the settings in there. That will no longer be possible,
everything in the new CFG file is visible and modifyable via the OPTIONS and
PROF EDIT functions.
- If you really want to peek inside, I recommend the tool at
sqlitebrowser.org
- This CFG file will reside in a "Home" folder which can reside anywhere in
your system, even on a Network drive.
- The current sub-folders (AUTO, CLIP. FILELIST, JOBS, MACROS, RUN and STATE)
will reside in a "Data" folder which can be the same as the "Home" folder,
or in a separate location.
- For those wanting multiple copies of SPFLite, the current -TITLE and -INI
command line options have been combined into a new -INSTANCE option. This
new INSTANCE support will:
- Alter the Title bar the same way as -TITLE did.
- Provide the Instance with it's own private set of configuration
options. Thus it could have different settings for anything specified
via the OPTIONS command. - Allow the Instance to have its own KBD configuration, or share the
normal Default KBD file. - Similarly share or not share the SET variables and/or the Retrieve stack.
- Allow the Instance to have its own private "Data" folder.
- All Instances will share the common pool of File Profiles.
SUMMARY - There will be ONE "Home" folder per system. It will be shared by all
SPFLite sessions and can handle shared updates from multiple sessions.
- There need only be one "Data" folder unless a user wants to use Instances
and chooses to maintain different versions for some reason.
- When instances are used, they each have independent Last used files lists,
last screen position and size, they may have different KBD settings, or
different SET variables. They can have different Fonts, colors, FM settings
etc. Basically they can have a completely unique configuration setup.
COMMENTS I'd appreciate your comments, questions etc. I'm proposing this to make things easier for you to customize your work environment. I'm perfectly willing to adapt and modify this if there are aspects that end up doing the opposite. George |
|
|
|
Post by nicc on Aug 15, 2019 15:01:14 GMT -5
Like it.
I'm wondering if you could extend the db usage to include tables for variable pools like in ISPF? But would you even need to? Could user macros use such a concept? I will be looking into SQLite access via thinBasic - some day.
|
|
|
|
Post by George on Aug 15, 2019 15:34:22 GMT -5
I'm sure linking in access via the macro language would be fairly trivial, just some variation on the GLBL support already there. Just need some rules on pool names etc.
I'm only using SQLite in brain-dead simple mode. Just a series of tables with key/value pairs. Once I got over the initial learning curve, it actually becomes fairly simple. And SQLite itself has been great, does whatit says, and seems amazingly fast considering everything you do needs to be interpreted. I'm going to try some timings, but it seems to be faster on startup than the old INI files. Even with the CFG file on a LAN drive, which I thought would really kill it, and startup is still only a couple seconds. And startup does between 800 and 1000 parameter look-ups.
George
|
|
|
|
Post by heiner55 on Aug 16, 2019 7:01:51 GMT -5
I also prefer simple text files.
|
|
|
|
Post by George on Aug 16, 2019 11:57:04 GMT -5
SQLite is probably the most widely deployed database engine, it is literally used everywhere, browsers, cell phones etc. It is not a server/client style; there is no Server to run. No DB maintenance to perform. It is simply a pre-compiled C library, I can't picture what you call a 'point of no return'.
The SQLite DLL shipped with SPFLite is simply not going to suddenly stop working. No more so than the PCRE DLL we ship, or the thinBasic DLL we ship. If not SQLite, then why are the other two OK?
As to staying with simple text files - why?
So you can peek in them? What does that buy you?
So you can change them? We always strongly discourage that. e.g. you already can't change KBD at all.
Everything in the CFG file is displayable and modifiable via SPFLite. But if you're that desperate to peek, get "DB Browser for SQLite", I believe it will also let you change things if you know a little SQL. Then you can muck things up to your hearts content.
I've not heard a valid reason yet to avoid SQLite. I'm listening, convince me.
George
|
|
|
|
Post by heiner55 on Aug 16, 2019 23:29:53 GMT -5
I am using Linux.
They are using simple text files.
|
|
|
|
Post by George on Aug 17, 2019 11:24:35 GMT -5
I'll reply more fully, but just yank the PCRE or thinBasic DLLs and watch how reduced the SPFLite functionality becomes.
George
|
|
|
|
Post by George on Aug 17, 2019 13:00:44 GMT -5
1. SQLite may indeed be widely used, but it has never been used in SPFLite
before, nor anything like it. Even if *you* know how it works, I don't - and
likely many or most of our users don't either. That means NO users have any
experience dealing with the consequences and ramifications of having a database
engine bolted into SPFLite as part and parcel of using a text editor. Don't
think there will *be* consequences and ramifications? Yes there will. Just
wait.
2. I am glad that SQLite is a pre-compiled C library, for sake of
implementation simplicity. I have no objection to that aspect per se. However,
that is not the problem, since thinBasic was likely much harder to integrate,
and yet you did it.
3. Why do I say a 'point of no return'? Simple. Right now, you have text
files for configuration data. Like them or hate them, they are known
quantities. What is the data representation of an SQLite db file? I have no
idea, personally. Suppose that some day you find some significant problem with
how it works? Once everyone's config data is in SQLite, how will you get that
information OUT? Suppose SQLite failed and corrupted its data; you would be
lost. Suppose you wanted to go back to an earlier SPFLite version that didn't
use a database? You would be stuck. You'd have to reinstall an old version and
(maybe) recreate all the config information from scratch. That is what I mean
by a point of no return. If you have users commit to SQLite - and evidently
doing so without even testing whether or not it will be successful in the user
community first - and something goes wrong, your entire user base is screwed.
That's not a nice thing to do.
4. You say "the SQLite DLL shipped with SPFLite is simply not going to suddenly
stop working." You don't know that. I just installed a Windows 10 update, and
two applications that worked in the prior version of Windows 10 did just that -
they suddenly stopped working. This happened to me two days ago. You cannot
guarantee the same will not happen with SQLite. Well, yes, if it did, the
people that created it will (*hopefully*) create a fix or workaround eventually,
but how long will *that* take? In the meantime, every SPFLite user would be
dead in the water.
5. You asked, "No more so than the PCRE DLL we ship, or the thinBasic DLL we
ship. If not SQLite, then why are the other two OK?" Why are the other two OK?
For a very important reason. SPFLite can function perfectly well without
thinBasic and without PCRE. Yes, if they failed, you would have reduced
functionality (being unable to run macros or to use R strings) but then you
would still HAVE functionality. Plus, thinBasic and PCRE don't change much and
the risk of failure is small. But if ANYTHING does go wrong with SQLite, you
have no editor. You're dead in the water. Your entire product would go dead,
based on outside software that YOU did not write and have no control over. The
only way a user could protect themselves against this possibility is to
permanently maintain an older, obsolete version of SPFLite as a fallback.
6. You asked, "As to staying with simple text files - why? So you can peek in
them? What does that buy you? So you can change them? We always strongly
discourage that. e.g. you already can't change KBD at all." This is a
multi-part question so I will answer this a piece at a time.
6.1. No one said that text files HAD to be simple. The KBD file has a checksum
(right?) and a rigid format that would be hard to alter manually. There are
text configuration formats that lend themselves to SPFLite usage that are not as
'simplistic' as you suggest. One is XML and one is JSON. The advantage of XML
is that Windows comes standard with an XML interpreter. You don't need outside
software for it, and MS will guarantee that any given version of its XML will
work with its current Windows release - it has to, because Windows itself uses
XML. XML is not the only possibility, but it *is* a possibility. XML has
extremely wide usage over decades and is well known.
6.2. Do I want to "peek" at the configuration files? Of course I do, and so
does every SPFLite user. Don't think they don't. You say that "Everything in
the CFG file is displayable and modifiable via SPFLite", meaning everything
*will* be, eventually. That may be true, but it also means that you could only
get to the config data through whatever UI you write. Please forgive me for
saying this, George (really) but your UI windows never have been all that easy
to work with. The KBD and Profile ones are good examples of this; they still
has issues years after being stabilized. Being forced to deal with every single
config value through a GUI interface is not a prospect I look forward to. I am
sorry to have to say it - and I don't choose to criticize you lightly - but I am
being totally honest with you here.
6.3. I am NOT "desperate to peek", but peeking serves a legitimate function.
Not only for information purposes, but to copy and paste into other applications
or for other reasons. How do I know if your UI will even allow that? How COULD
I know, since you are designing it in secret without any user input or feedback
at this time?
6.4 You say that I should get "DB Browser for SQLite" if I want to "peek" and
"change" things. That falls so far short of my needs and interests. I don't
WANT to get "DB Browser for SQLite". Not even if you paid me. I have ZERO
interest in learning how SQLite works. Yes, I do happen to know a little SQL,
but that knowledge is decades old. And again, the ability to apply SQL to
SPFLite parameters is of zero value to me. The parameters are neither complex
nor numerous enough to require a database query to find them. Nor do I want to
"muck things up". I am sure if you asked your user community how many of them
want to tamper with the configuration parameters so much that it "mucked up"
SPFLite to the point of being inoperative, the answer would be (wait for it)
ZERO.
7. Finally, using a database to store parameters in an attempt to manage
multi-instance and multi-system concurrent access is the wrong solution to the
wrong problem. You don't need a database. You need a Windows Service module to
act as an interface and to manage concurrency.
George, you know me, and have known me for years. I don't claim that all of the
advice I have ever given you in that time was good, but hopefully you would say
most of it was. Taking into account all of my IT experience and all that I know
about SPFLite, I genuinely believe that incorporating a database into it is the
wrong answer. Based on what I know at present, if you went ahead with this
plan, I would likely not install it or any subsequent version of SPFLite that
included a database, since (in my heart of hearts) I would not trust it. I'm
sorry to have to put it that way, but that's how I honestly feel.
|
|
|
|
Post by George on Aug 17, 2019 15:15:07 GMT -5
It's obvious Robert and I differ on moving to a Database from a bunch of INI files, and if there are a lot of others out there that feel the same way, than I will drop this, I don't want to upset a lot of users who have concerns about this change.
I've spent a lot of time with prototype code to ensure it will do what I want, and everything looks just fine.
I personally don't understand the paranoia about 'something may go wrong' just because it's using something mysterious called a 'database', but I won't go that route if its going to cause concern with a lot of you.
So please, voice your concerns, opinions etc. You're the ones who will be using this, you should be providing input.
George
|
|
|
|
Post by nicc on Aug 17, 2019 15:31:16 GMT -5
It is not as though SQLite soetimes turns up on your computer - my laptop has 449 *.db files of which 6 are my own. The rest are Win10, Firefox, OpenOffice, CCleaner, Norton, Spybot, Malwarebytes etc. Now, possibly some of those .db files are not sqlite databases but I believe some applications use a different extension for their sqlite databases so 400+ is a reasonable count.
As I see no reason why you would need anything beyond INSERT, SELECT, DELETE and UPDATE statements any version of sqlite would do - even pre-version 3 and no new release of sqlite is going to be put in the wild without basic functionality working.
|
|
|
|
Post by George on Aug 18, 2019 12:52:14 GMT -5
Robert: All reasonable items.
1. No problem at all.
2. This is certainly possible. I have a skeleton INI to CFG conversion tool, it would not be much to add a prompt for a new folder structure, since the new 'SQL' version will have the ability to have the config files stored anywhere. Default of course would be to keep everything where is currently is (\Documents\SPFLite\)
3. Giving the EXE a new name is another 'no problem'. In my prototype testing I have all my new stuff in the same folders as all the current stuff. Co-existence is more than possible. The new parameter problem is also solvable, my conversion tool already has a table of all existing INI parameters, I'd need an additional column to be able to put everything back into its original [Type] structure.
I recognize this all takes a lot of work, but this is not some quick and dirty upgrade, there are an incredible number of small change items to be done. The initialization code alone will be tough as it has to handle all variations from first time user to conversion of an existing user to a user simply doing a reset to start over etc. Throw in all the cmd line overrides and it gets crazy.
I've tried adapting the INI type support to handle multiple instances, different locations etc. and it gets much messier.
It really won't be bad, really.
George
|
|
|
|
Post by George on Aug 20, 2019 15:26:36 GMT -5
Question: If I create some kind of Export / Import ability for the configuration, what should the exported data look like.
I was thinking a plain text file that is mostly like an INI file. This allows it to be processed simply by anyone.
e.g.
[[Tablename1]]
[SectionName1]
keyword1=value1
keyword2=value2
[SectionName2]
kwd1=value1
kwd2=value2
[[Tablename2]]
[SectionName1]
kwd1=value1
kwd2=value2
TableNames are things like the Options data, KBD data, SET values, Profiles, etc.
SectionNames are just like the existing [xxxx] names in the existing INI files
Should Export/Import be built-in, or separate EXE files?
Also, I think Roberts suggestion to do a rename is probably wise. So how about some suggestions for the name.
Robert suggested SPFLiteSQ. SPFLiteDB. SPFLiteNS (Next Step). SPFLiteXL.
George
|
|
|
|
Post by George on Aug 21, 2019 12:04:52 GMT -5
- I can already migrate existing INI stuff to the new CFG file, all that happens is a new SPFLite.CFG file appears in the normal \Documents\SPFLite folder.
- The result is that Both old and new versions of SPFLite can co-exist, but of course any config changes made by one version are not seen by the other. For a short 'check-out' period this should not be a problem.
- I can also re-create a whole 'set' of old config files from the new CFG file. Don't ask me why or how someone would want to use this, but it exists because I just know it will be asked for, needed or not.
- I had thought, that you and others were concerned that having the data stored in a CFG file, that the contents were not visible, and that made you uncomfortable. I pictured the Export mostly as a way to make everything visible; making it resemble a stock INI file was just to output it in a format that was easily understandable and usable for Edit purposes.
- Quite honestly, I think Export/Import aren't worth it. As a tool to let people "see" what's in the CFG file, it's an utter waste just to satisfy someone's curiosity. As a backup/restore tool for the configuration its a lot of work to duplicate what any existing data backup tools can do. Want a backup? Use a backup program, copy and/or ZIP the whole folder, whatever. I don't think every application should be expected to provide its own set of Backup and Restore functions. We didn't do this for all the old INI style config files in the past, is everyone's confidence in SQLite that poor?
- So I will create an Export, simply to provide this 'visibility'. I'm going to skip the Import for now. I'm not doing this to provide some 3rd intermediate form of config storage.
- And if we could get agreement on dropping this whole visibility thing, I'd kill Export as well.
So there will be: SPFLiteINI2CFG.EXE which will build a new SPFLite.CFG file from existing INI files. SPFLiteCFG2INI.EXE which will re-create a set of INI files from an existing SPFLite.CFG. SPFLiteCFGExport.EXE which will create a TXT file of the contents of SPFLite.CFG as I described it above. The first one above will automatically be invoked by the 1st execution of the new SPFLite version. So existing users will see a short delay and then the old familiar SPFLite screen will appear. George |
|
|
|
Post by George on Aug 21, 2019 12:20:14 GMT -5
Re: Names. I think we should keep SPFLite, even though it's put on weight. We'd need changes to the web Site, Forums, etc. and might lose whatever small recognition factor we currently have.
But the EXE can certainly become SPFLiteR2.EXE.
There's no need at all to have separate folders for old and new versions, for whatever overlap period is needed during migration, the existing structure is just fine.
All the files stay the same except for:
New version will add SPFLite.CFG and will ignore and not use SPFLite.INI/KBD/BKP/SPR/SPS or the entire \Profiles folder.
So there's no real collisions.
George
|
|
|
|
Post by George on Aug 21, 2019 14:06:20 GMT -5
Just to demonstrate how useless creating an Export of the CFG file is for those curious to see the contents of the file, here's the output from the 1st cut at an export program. And this is a basic CFG, I have not created any alternate Instances, KBD maps etc. or any of the new capabilities which would blow up it's size considerably. Other than looking at it and saying "interesting" what exactly would anyone DO with it? George SPFLite.CFG.Export (56.29 KB) |
|
