lrParser - combat statistics parser.

[Yurai]

Hey people, sorry for taking a long time since last I checked. I was attending javazone last week, combined with other miscellaneous stuff.

Murugan, constructive criticism is always appreciated! I was actually planning a more detailed breakdown of abilities to be displayed in a dialog along with a pie chart when double clicking a table row. The tooltip, due to its nature, should probably be limited to a shortlist.

tachikoma, thank you for your kind words. I'm also really happy that there are people who want to contribute. I should make another attempt at setting up a subversion repository for code sharing (my attempt to use the one curseforge provided didn't go so well, as chronicled earlier in this thread), so collaboration will be easier.

In regards to thread manipulation, I might just port the entire thing over to the recently released java 7, which has a lot better support for threading and concurrency.

[tachikoma]

I keep getting double dmg in the results, when I hand copy the log files into my reading dirrectory while the parser is running it looks fine, when I'm in the darkhold running parser it shows double or tripple dmg.

[Yurai]

That is the same problem others have reported, but the fact that you tried two different cases and it only happens while ffxiv is the one outputting is an important clue. I think it might be related to the thread that notices new files and puts them in the queue. ffxiv reuses file names for the logs, so I use timestamps to compare whether a file should be re-added to the queue or not. That might be the culprit, if ffxiv "touches" the files some time after outputting them. There are a couple of ways I can try to fix this, either by storing a checksum for each file and using that for comparison, or by writing a small token in the start of parsed files to mark them as read (ffxiv will replace the file when it tries to reuse the file name, and make the token vanish).

I should have some time to work on this on friday. I'll prioritize this first.

[tachikoma]

I think I fixed the double reading issue, not sure if this solves it yet as my work PC doesn't have FFXIV on it to test live. I'm pretty sure the issue is with reading and writting to the Log queue without it being synchronized properly. I tried using synchronizing calls in teh places that access the Log enum but that didn't work all the time (though it did seem to help). I then tried doing this:

public enum Log {
INSTANCE;
private BlockingQueue<File> queue;

private Log() {
queue = new LinkedBlockingQueue<File>();
}...

Seems to be working right now. I PM'd you on curse forge the other day hope we can get in contact.

[Yurai]

Good thinking, though that kind of leaves the question as to why it only occurs in a live setting and not when parsing a static directory of pre-generated files. If you could do some live testing as well, that would be great.

Having a queue that ensures concurrency is probably a good idea in either case, I'll change it to PriorityBlockingQueue. If testing shows that this is sufficient and I can avoid having to tamper with the files, that would be excellent. Well done!

[Yurai]

I keep trying to add the code to the curseforge subversion so we can collaborate on it, but the damn thing won't authorize me no matter what I do... I'll try finding an alternate place for code sharing.

[tachikoma]

Did more DH runs testing and its doubling results almost 100% of the time when live, I close the parser and restart the parse w/o clearing log and it parses up the current files perfectly. This was after making the Log enum a LinkedBlockingQueue. So that didn't solve the issue.

[tachikoma]

I am thinking that the FFXIV program does something messed up when writting to the dirrectory. I think it must write out the file in stages this is the only thing I can think of that would cause the file to be double read. And it seems consistent like it happens 100% of the time right now. I'm going to set a delay on the parser so that it doesn't read any files till they are a certain amount of time old maybe a 2min. delay.

[Yurai]

Hmm, I'm pretty sure the file is complete when it appears in the folder (never seen them increase in size), but it might do some post-processing in the header or something that updates the file's timestamp. I think it should be safe to parse the file when it's there, we just have to make sure we don't pick it up again. Computing a checksum would be the safest bet, but that might have a detrimental impact on performance.

[Yurai]

Hmm, now that I look at the monitoring thread again, I think I see why this is happening. Pretty gross oversight on my behalf. Could you try doing the following changes and see if it fixes the problem in a live setting for you:

Add the following global variable in LogMonitor.java and initialize it in the constructor:
HashMap<String, Long> timestamps = null;

Constructor:
timestamps = new HashMap<String, Long>();

Replace the entire contents of the run() method with the following:

do {
for (File f : logDir.listFiles()) {
if (f.isFile() && !alreadyParsed.containsKey(f.getName())) {
Log.INSTANCE.addFileToLog(f);
alreadyParsed.put(f.getName(), f);
timestamps.put(f.getName(), f.lastModified());
} else if (f.isFile() && f.lastModified() > timestamps.get(f.getName())) {
Log.INSTANCE.removeFileFromLog((File) alreadyParsed.get(f.getName()));
Log.INSTANCE.addFileToLog(f);
timestamps.put(f.getName(), f.lastModified());
}
}

try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
} while (true);