simplest way to overcome slow-scrolling problem is not to scroll the whole column by single lines, but to simply display the next "page" when the cursor is at the bottom. The cursor would then position itself at the top of the next list of 20 files - no repeated redraws needed. Do the reverse when the cursor goes past the top of the displayed list
Eddy's file selector works like that and I don't like this mode very much. Anyway, a similar thing can be accomplished in SC by pressing Shift/Ctrl/down-arrow and Shift/Ctrl/up-arrow.