ED Astrometrics: Maps and Visualizations

Update to the above. I've narrowed the problem to decoding compressed data (although encoding seems fine). While I continue to investigate the underlying problem, I have a work-around in place of having the map-maker script use BMPs to load the logos and other image elements, rather than PNGs. If that works, the maps should begin updating again soon.

It's been a while since I've had to track something like this down. What a pain... LOL.
 
So the work-arounds seem to be working, and the maps are updating again. I figured I'd write up a "post mortem" just in case anyone is interested in this sort of thing, though the problem is not actually corrected yet. I'm just working around it.

This is the sort of thing that makes me crazy, as a system administrator. I mean, worse than space madness. ;) But even so, it's also fun and interesting to do this kind of detective work. It simply gets maddening when it's almost always hitting a server or process that you want to quickly restore to service, with no immediately apparent solution. Thankfully it doesn't happen often, but there's always a risk with updating software on a system that's been humming along automatically just fine.

It all started with an OS update (yum update in CentOS) last week. I forgot to check that the images were still updating for several days, but no biggie, they're just images. :D Marx mentioned that the nebula maps weren't up to date and seemed to be missing some data. This involved two problems, first that I didn't have the spreadsheets pulling on the right schedule, but secondly that it could no longer read the XLSX file from Google Docs. It complained of some sort of stream error. I figured at first that this might be a problem with Google mangling it or using a file feature that the Perl modules don't understand. I was only using Excel format because I've worked with the Perl module for that before and it contains all of the sheets in one file.

At this point I also noticed that the other maps hadn't been updating due to a PerlMagick (ImageMagick) error, so I figured that was a separate problem, and focused on the spreadsheet first. I switched it to use ODS format instead, and got a nearly identical error. (What?) So to sidestep this first issue, I switched to CSV format (no reliance on Perl modules) and this script started crashing on the image manipulation too.

Now here's where PerlMagick (a wrapper for the ImageMagick libraries) can be irritating. The crash and error message usually happen long after the actual problem has occurred. Like, maybe hours later, in the case of the map-maker script. It would die on executing a Composite or Write command on one of the images, saying that no image was present. Turning on some additional warning messages (that might otherwise be invisible), I discovered that it was actually failing right at the start, when it tried to load PNG image assets. Most of the other operations would happily (and silently) try to perform operations on an empty data set. The warnings indicated that it thought that the PNGs were corrupted on disk. I knew that wasn't true.

So what do all of these problems have in common? They're all perl modules that are failing to read complex data formats and parse them. I figured the most likely culprit would be compression, since I'm not sure what else they could be doing in common that wouldn't also affect everything else running in Perl. As a test, I started switching the scripts to use BMP and TGA formats internally, since they're uncompressed (though TGA has a compression option, it's rarely used). This worked, thus giving me more evidence that it's an compression decoding issue. The command-line ImageMagick still works, and converts back to PNG before uploading the images to the web server.

So what I know is this-- low level compression in Perl is failing, but everything else is still running fine. I've tried rolling back the linux packages that pertain to Perl, ImageMagick, and Zlib, but nothing has solved this yet. In the meantime, the scripts are doing fine working with other formats. So at least I can relax now. :D
 
Last edited:
I am certainly interested in post mortems. Not that I work in that profession but more than once reading such reports helped me with a problems some time later :)
 
Turning on some additional warning messages (that might otherwise be invisible), I discovered that it was actually failing right at the start, when it tried to load PNG image assets. Most of the other operations would happily (and silently) try to perform operations on an empty data set. The warnings indicated that it thought that the PNGs were corrupted on disk. I knew that wasn't true.
That certainly sounds annoying. You'd think a problem as serious as failing to load a file would merit a visible-by-default error message at least! 🤔

Marx mentioned that the nebula maps weren't up to date and seemed to be missing some data. This involved two problems, first that I didn't have the spreadsheets pulling on the right schedule, but secondly that it could no longer read the XLSX file from Google Docs.
I thought you were pulling POI data directly from EDSM now, but I guess not? Comparing your interactive map to the EDSM one, the sets of POIs are close but not identical. (I think your POIs are a strict subset of the EDSM ones, but it's hard to be sure with all the overlapping pins and such.)
 
I thought you were pulling POI data directly from EDSM now, but I guess not?
He was talking about the nebula data for the nebula maps (see the Maps and Charts, not the interactive map), which is pulled from here. Some real nebulae are still missing from the map, because I forgot to fill out the reference systems - should be done relatively soon though.
 
Last edited:
That certainly sounds annoying. You'd think a problem as serious as failing to load a file would merit a visible-by-default error message at least! 🤔
Now that I know that it does this, I'm tempted to modify every..single..function call to perlMagick with a wrapper that checks to see if it threw a warning, and display it myself. At least problems might be easier to identify in the future if I do that.
 
Top Bottom