In the bowels of a business unit, a director got a great deal on a third party software package. He bought it, without talking to corporate IT, and then was upset when it couldnt gracefully integrate with any of the corporate IT assets. Eager to throw good money after bad, the director hired his nephews consultancy to build an integration tool to make his new toy work.

A few months later, the users complained about performance, and somehow, fixing this thing became Jeffs problem. The process was simple enough: slurp enterprise data out of a text file, and pass the data on to the third-party tool. It didnt take Jeff long to figure out why it performed poorly:

Private Sub ProcessFile()

    ' prepare to do stuff

    Do Until blnLastTime = True
        
        Set fileReader = fso.OpenTextFile(strFileName)
        
         If fileReader.AtEndOfStream = True Then
            blnLastTime = True
         Else
            strTextLine = fileReader.ReadLine
         End If

        ' actually do stuff
         
        fileReader.Close
        Delete_Line (strFileName)
    Loop
    fileReader.Close

End Sub



Private Sub Delete_Line(strFile)

    Set fileReader = fso.OpenTextFile(strFile)
    
    If fso.FileExists(strFile & "2") Then
        fso.DeleteFile (strFile & "2")
    End If
    
    Set fileWriter = fso.CreateTextFile(strFile & 2)
    
    If fileReader.AtEndOfStream = False Then
        fileReader.ReadLine
    End If
    
    If fileReader.AtEndOfStream = False Then
            strLine = fileReader.ReadAll
            fileWriter.Write (strLine)
    End If
    
    
    fileReader.Close
    fileWriter.Close

    fso.DeleteFile strFile, True
    fso.CopyFile strFile & "2", strFile, True
    fso.DeleteFile strFile & 2, True
    
End Sub

Start by opening a file foo.txt. Read a single line from the file. Send it to the third party app. Close the file. Open foo.txt again. Open another file, called foo.txt2. Read the first line from foo.txt, again. Throw that away. Read the remainder of foo.txt, and write it to foo.txt2. Copy foo.txt2 back over foo.txt. Now, go back to the top of the loop and read a single line from foo.txt again.

So, for a 10,000 line file, this would perform 30,000 file open operations, write nearly 50 million lines, delete 20,000 files, and perform 10,000 copy operations. It didnt take Jeff very long to rewrite this to simply read the file, one line at a time. The runtime dropped from a few hours to less than a minute.

http://thedailywtf.com/Articles/Line-by-Line.aspx