Bug in Directory processing

Well ok, rename "bug" to "issue" if you prefer. I called it a bug, because some number of releases ago I used to get almost identical symptoms with the internal loop message, which was then followed by an actual crash (this was attempting "Find Files" after a long period with many files open - an issue that was subsequently fixed). The difference now is that okaying the pop-up means that execution continues, whereas previously it used to crash. My bad. Perhaps the file name ought not to contain the word "crash"?

Regarding your point about Notepad, I just tried it and the folder appeared in the Open dialogue almost instantaneously.

Robert: Without writing a pile of custom code, reading folders in 'chunks' or some other threaded process is non-trivial. All the API functions (whether Windows or PB) work the same, like - Issue the Start search parameters, then Do while Not-Done, Loop.

So tell me, how do you 'thread' that? The FM screen display wants to display a list, not some moving target of a list which is being dynamically added to by some other thread.

No, threading won't help here without some kind of major internal re-structuring, and basically that ain't going to happen.

What I have to do is add some timing traces and see just where this current process is spending all its time.

I created a folder with 12,000 files in it, and yes, it's painful to load. So until I figure out just where the time is being wasted, all solutions are off the table, we just don't know yet what we're trying to fix.

George

Clive: I can see the problem quite easily with my test folder. My problem is simply "where the heck is it spending all that time?"

The PB compiler has a nice PROFILE feature that measures where you 'spend' your time, but right now it sems to be not working 100%, so I'm struggling with that. One way or the other, I'll track this down.

What I'm concerned about is that recent versions have converted the FM 'List' into a list of Objects that hold all the file information. I'm hoping that the overhead is not related to switching to Objects from simple arrays of TYPES. That would be really tough to overcome without dropping the Object structure and going back to TYPE arrays.

We'll see.

George

Robert: I tried your suggestion reading with a small test program. The slowdown is not in the reading.

It reads the folder, allocates a new FM Data Object for each one, and stores the basic FD directory data. (The full program also completes a lot of other Object fields with formatted data, which this doesn't)

And this runs basically instantly. And the BigFolder is on a real hard drive, not an SSD.



So, I'll just have to continue my search for who/what/where the overhead is coming from.


Here's the test program.

George

GLOBAL ghDebug AS DWORD
GLOBAL gFMD()              AS iFMData                             ' Latest Dir scan results (Object pointers)

FUNCTION PBMAIN () AS LONG
LOCAL i, j AS LONG, t AS STRING
LOCAL FD AS DIRDATA
LOCAL FP AS ASCIIZ * %MAX_PATH
DIM gFMD(1 TO 100) AS GLOBAL iFMData

   FP = "E:\BigFolder\*.*"
Debug "Starting"
   t = DIR$(FP, 22 TO FD)
   DO WHILE ISNOTNULL(t)
      INCR i
      IF i > UBOUND(gFMD()) THEN
         REDIM PRESERVE gFMD(1 TO 2 * i) AS GLOBAL iFMData
      END IF
      LET gFMD(i) = CLASS "cFMData"
      gFMD(i).FD = FD
      t = DIR$(NEXT, TO FD)
   LOOP
Debug "Ended"
debug "Read " + FORMAT$(i) + " records"
debug "Record " + FORMAT$(i) + " = " + gFMD(i).Filename
   MSGBOX "OK"
END FUNCTION



OK, I've identified one routine that is certainly accounting for the vast majority of the overhead.

The routine that reads file folders is driven by a requirement created by FILELISTs, that is, the user can request multiple folder reads, with multiple filemasks, and this can result in duplicate entries in the final displayed list.

So, before adding an entry to the final displayed file list, a call is made to a routine whose sole function is to say YES/NO if the file is already in the list. This is the routine that's creating the extreme overhead.

And as the # of files increases, this routine aggravates things more and more.



And it's retty hard to optimize, the list is not ordered in any way at this point.

     METHOD IsAFDup(fn AS STRING) AS LONG
     '---------- See if a filename is already in the gFMD array
    REGISTER i AS LONG
        FOR i = 1 TO gFMDCtr                                     ' Spin through the table
           IF fn = gFMD(i).FullPath THEN METHOD = %True: EXIT METHOD
        NEXT i                                                   '
     END METHOD

Not sure how to handle this yet, still thinking.

George

Robert:
Clive:
OK, I've altered the data reading routine to skip the DUP check when there is only one path being loaded (e.g. a FilePath FM mode, or a FILELIST with only 1 entry)

The improvement is way more than enough to satisfy our needs. I tried with both a hard drive and an SSD drive, barely any difference. It loads and displays the 12,000 item folder is about 2 seconds.

This will be in the next release.

George

Stefan: PowerBasic provides a Collection Object, which performs mostly the same way, except it stores key/value pairs. So that I could say:

CollectionNaaaame.Add("crater", 1) to add the name (all names would have a 'value' of 1

and then ask

IF CollectionName.Contains("crater') THEN ... True if crater exists.

Never used it this way, have to benchmark it for performance, but most PB provided tools are bloody efficient.

I'll try it out just to see. Let you know.

George

[UPDATE]
Only did a quick test in a small test program, but the Collection seems pretty fast.

I did a test with my BigFolder (12,000 files)

Using a simple text array, and doing a serial search for DUPS before adding a new entry, it takes around 3100 ms.

Using a PB Collection Object, and doing a Contains search before adding a new entry, it takes around 300 ms.

So the PB code is pretty fast.

But right now I'll stick with just avoiding the DUP search for single path loads.  But now knowing the performance of the PB Collection Object it will certainly be considered for other uses.

[\UPDATE]