Odtnhj

I am trying to iterate through a file with millions of lines containg some data and I am retrieving it. Unfortunately this is going very slow and I was wondering how I could make it more efficient.
At the moment I am loading two files and I am iterating by each line.

The code:

# Retrieve session data
UBList = array([line.split('t') for line in source])
SID = set(UBList[:,1])
n_unique_sessions = len(Counter(UBList[:,1]))
source.close()
sessions = 
session_info = 
purchases = array([line.split('t') for line in open('Data/order_overview.txt', 'r').readlines()])

for sid in SID:
 print sid
 s = [line for line in UBList if line[1]==sid]
 uline = [line for line in s if line[3]=='17']
 tline = [line[2] for line in s]
 t_format = "%Y-%m-%d %H:%M:%S.%f"
 s_start = datetime.strptime(tline[0], t_format)
 s_end = datetime.strptime(tline[-1], t_format)
 s_length = (s_end - s_start).total_seconds()
 d_time = [line[2] for line in s if line[3]=='4']
 if len(uline) > 0: 
 uid = uline[0][12]
 else: 
 uid = 'NotFound'
 num_queries = len([line for line in s if line[3]=='27'])
 num_purchases = nonzero(purchases[:,0]==sid)[0].shape[0]
 sessions.update(sid: (uid, num_queries, num_purchases, s))
 f = open('Results/' + sid + '_' + uid + '_' + str(num_queries) + '_' + str(num_purchases) + '_' + str(s_length) + '.txt', 'w')
 f.writelines(['t'.join(line) for line in s])
 f.close()

Would something like this speed things up?

somevar = dict([sid, for sid in SID])
for line in UBList:
sid = line[1]
dSID[sid].append('t'.join(line))

Also is it possible to get the next line after a certain criteria is met? For example I find a line get its value with the next and do a calculation. And add the results if the line with the criteria was found multiple times.

If source is your file object, don't read it all to memory beforehand. You can process it and read at the same time, and Python is clever enough to process data while waiting for next chunk to be read. To achieve that, use list generators, not list comprehensions.

You create a lot of arrays you don't actually need, like s. Replace them with generators and get a speedup. Creating standard Python array is a heavy operation, and if it's bigger than a 4kB, generators should be better.