|
Post by George on Jan 7, 2024 13:43:54 GMT -5
I just wish I could get a grip on why you see this kind of problem, and many others over the years, where nobody else is reporting them. As I said, I simply can't DO ANYTHING, it frustrating, but that's the way it is.
George
|
|
|
Post by Robert on Jan 8, 2024 13:56:46 GMT -5
George, I just tested the 24.008 beta. I did my FCLIP macro against a file of 14,000 lines. Observations:
1. No crash (so far)
2. All primary commands run much slower. If I issue a command like DELETE "ABC" ALL, the command first disappears, then there is about a 2 second wait, then the delete is performed. This is new behavior.
3. I issued a Power Type command on this file (it's a CLIP session), and when I was done with it, I got the message again, The Files name(s) in the UNDO control area do not appear to be valid, Bad Fn: <nothing>, Current UNDO SAVE function will be skipped.
R
|
|
|
Post by George on Jan 8, 2024 15:38:08 GMT -5
OK Robert, I had another report from MUEH and confirmed something is weird. File size (other than processing time) just SHOULDN'T matter, but it obviously seems to.
Oh BTW, a DELETE xxx ALL is one of the slowest commands normally, let alone on a big file. Try a FIND XXX ALL and see the difference. The deleting of individual lines like that is a painful process.
Fun. Fun. Fun.
George
|
|
|
Post by Robert on Jan 8, 2024 19:17:48 GMT -5
Right, I know that DELETE ALL XXX in a large file is slow, but what it's doing now is odd, where the command disappears first, then a delay, then the command finishes. I think in some ways, what you had originally, with some kind of semaphore, might have actually been (mostly) correct. Maybe what's needed is a test to make sure the subtask's work is actually done before simply assuming it is. Perhaps some kind of completion flag.
Idea:
COMPLETED = FALSE START_TASK() LOOP WAIT_TASK(250_MS) IF COMPLETED THEN EXIT_LOOP END LOOP Is this possible? Would it help?
R
==> Oops, I see it wouldn't help. If you were in a wait loop, the main process still would be waiting on the subtask to finish. My bad, this won't fix it. Sigh.
|
|
|
Post by George on Jan 9, 2024 15:25:13 GMT -5
There is something rotten in Denmark, otherwise known as UNDO/REDO. It's some kind of relationship between filesize and MINLEN, but I've been unable to find anywhere where the volume of data/lines would make a difference -- but it does. Spent the day on this, and I'm just floundering right now.
|
|
|
Post by Robert on Jan 9, 2024 16:39:03 GMT -5
Perhaps file size is only indirectly related. Could it be that the larger a file is, the more time it takes, and so it's really the size of the workload and how long it takes to finish? And not some arbitrary line count? Maybe MINLEN acts to make lines longer, so that makes the I/O time longer? I don't use MINLEN, but I still have gotten issues.
I believe the underlying issue is time, not size. It's probably system dependent, too, so a fast machine with only SSD's has fewer of these hiccups. Just kicking ideas around ...
R
|
|
|
Post by George on Jan 10, 2024 12:32:58 GMT -5
Robert: Well, I think I've tracked it down, and it's a long time lurker. Never had such a frustrating one to find. This is the code that reads back in the Data Attribute array. Note: bufw and TW() are data type WSTRING .
FNum = FREEFILE ' Get file # OPEN me.UndoTWFnGet(i) FOR BINARY ACCESS READ LOCK SHARED AS FNum GET$$ # FNum, LOF(# FNum), bufw ' Read entire file PARSE bufw, TW(), $$CRLF ' Assign back to the TW() array CLOSE # FNum '
If you give up, scroll down.
George
FNum = FREEFILE ' Get file # OPEN me.UndoTWFnGet(i) FOR BINARY ACCESS READ LOCK SHARED AS FNum GET$$ # FNum, (LOF(# FNum) / 2), bufw ' Read entire file PARSE bufw, TW(), $$CRLF ' Assign back to the TW() array CLOSE # FNum '
LOF returns the filesize of the OPEN file in BYTES.
GET$$ wants the number of WSTRING characters to read into bufw.
The GET$$ was being asked to read twice the number of characters. Don't understand quite why, but the array gets filled with X'FFFF' characters.
This could account for a bunch of mysterious code corruption problems.
|
|
|
Post by Robert on Jan 10, 2024 13:31:36 GMT -5
WOW, that sure looks like you found it. LOF always returns a byte count, but GET$ takes an ANSI char count and GET$$ takes a WIDE char count. I suspect the first half had the real data and the second half had FFFF because there was no more data to assign to the bufw array. It has to put SOMETHING in it, and PB chose that, because FFFF is not a meaningful Unicode character.
I am puzzled as to why this data is Unicode in the first place. What is going on here? I thought internally all data was ANSI. Could you explain this?
I am also wondering about the PARSE call. Parsing thousands of lines ending in CRLF and allocating many, many strings could be very time consuming. Aren't there 'string builder' calls that would run faster?
R
|
|
|
Post by George on Jan 10, 2024 17:55:23 GMT -5
Robert: The attributes are stored in a parallel WSTRING to the actual data's STRING values? Why? Well, we need 16 bits to hold all the various attributes, that fits in a WSTRING character quite nicely. And when needed you can still do string manipulation functions on the WSTRING in concert with the normal data STRING manipulations.
PARSE of a buffer into a string array IS the fastest method, it's all done by PB assembler code, no Basic code will beat it.
Stringbuilder support is the opposite, very efficient building of strings.
What amazes me is that the error has been there forever. Why do we not see problems in smaller size files? I guess it's just memory allocation 'luck-of-the-draw' layout whether it hurts or not.
MUEH is still reporting an FCLIP problem, so we're not out of the woods yet.
George
|
|
|
Post by Robert on Jan 10, 2024 18:29:53 GMT -5
You know, this issue about the UNDO file name being corrupted HAS been there, and we HAVE seen memory corruption errors, attributable or not, for some time. But, it's been so sporadic, you've never been able to pin it down.
I sure hope this GET$$ thing is the answer. That would be a huge relief.
R
|
|
|
Post by George on Jan 12, 2024 12:27:23 GMT -5
Stefan: Do you have any outstanding 'niggles' with the (TabBNDS) and (Comments) support?
George
|
|
|
Post by George on Jan 12, 2024 12:31:10 GMT -5
Robert: Well, certainly a lot of bug fixes in the UNDO/REDO support. Like you, I hope this cleans up some of these random problems that we can't seem to duplicate. Memory corruption stuff is just so difficult to catch because the crash is usually in some 'innocent victim' routine rather than the real culprit.
George
|
|
|
Post by Robert on Jan 13, 2024 22:03:48 GMT -5
Just tried beta 24.012. Again got UNDO file names invalid message. "Bad Fn" displays as null string.
Sigh.
===> More information. I continued editing the Clip file, ignoring the error message. I inserted a blank line and deleted it, just to force the error message again. This time, the "Bad Fn" was NOT blank. Instead, it showed as "BYELORUSSIAN-UKRAINIAN I". This is the name of a Cyrillic letter, and appears on the first screen of the edit display. It thus appears that the main data is corrupting the UNDO file name area. If you are interested in this example, let me know and I can try to ship you the data and an exact editing sequence to cause this.
P.S. I saved the Clip session, closed it and then reopened it. I no longer got the error messages.
R
|
|
|
Post by George on Jan 14, 2024 10:21:06 GMT -5
Robert: Yes, send me stuff. If I can repeat it I should be able to find it. Hard to believe after all the latest review and fixup there's still something rotten in there.
George
|
|
|
Post by Robert on Jan 14, 2024 17:17:09 GMT -5
OK, I emailed it off to you. Was able to reproduce the incident 3 times consistently.
R
|
|