Question: mapknitter export stuck

pataxte is asking a question about general
Follow this topic

by pataxte | May 24, 2020 17:01 | #23692


Trying to export a couple of maps on mapknitter but they have gotten stuck in the compositing stage. They are high res (@warren helped me with this a couple of weeks ago). Export seemed to be working fine, and then stuck over the last couple of days. Map is "2020_bean_bed"

thx



15 Comments

Thank you for reporting this! I can confirm that I see the unfinished exports in the queue: unfinished.png

I'll investigate and open an issue with the development team about it!

Is this a question? Click here to post it to the Questions page.

Thank you @icarito! Please let me know and any advice regarding what caused this issue and how to avoid it in the future would be great.


Hi, just responding to the stalled exports issue; @icarito is right that we are still trying to add more error reporting so this is not as mysterious, but on checking those exports, it seems they are exporting at roughly 0.02 cm/px and that is resulting in a quite gigantic image -- 84743 x 84743 pixels -- so those surely failed to export due to running out of memory. I think in general it's going to be tough to run exports at greater than 10k x 10k pixels -- that's just a really giant TIFF file -- and maybe ~15k x 15k would be the extreme limit.

The extremely low resolution at common (~50cm/px) scales here is still just a result of this being a map of a very tiny real-world area, so it means all our usual expectations about what resolutions will work are just shifted to a smaller scale. It should still work, but we need to figure out what the best resolutions are -- going back to my old response in https://publiclab.org/notes/pataxte/05-07-2020/mapknitter-exports-are-very-small-scale-unreadable#c26803 perhaps something like 0.2cm/px is ideal?

Is this a question? Click here to post it to the Questions page.


Reply to this comment...


Yes I went thru the logs and it appears that the exporter failed, likely out of memory because these exports are so high resolution (84k x 84k). @warren has taken note and we aim for the exporter to alert the user when it fails in this way. I suggest you try to tweak the settings to export in a lower resolution.

OK, thanks. How do I see the resolution of the exports? I'm a novice so I don't know where to look to tell that my images are very high resolution and therefor going to get stuck.

And, how can I stop those exports that are (still!) stuck? thx

Is this a question? Click here to post it to the Questions page.


Hi, The resolution is the number that you enter when you initiate the export.

The label says: "Choose a scale in 'centimeters per pixel' (where a smaller 50cm pixel is higher resolution - comparable to Google Maps - or a larger 200cm pixel is lower resolution)".

By default it offers 73 for me. Try a higher number until it works. I'm sorry there's nothing you can do about those failed exports until the development community adds better error reporting to the user interface.


Hi again, I'm trying to sort this out while my head is in it, thus so many questions! Any scale of lower resolution than about 15 doesn't work, the resulting map is super tiny, indecipherable. But even though I lock all the individual images (n=4), the map produced seems to change the layering so that things don't line up. For example, here is what it looks like on mapknitter just before exporting:

Screen_Shot_2020-05-27_at_2.38.24_PM.png

But when it's exported, the images on top are different, even if all locked individually, or the entire page is locked via icon in upper left corner. See here - notice rectangular garden bed in this and previous image for easy comparison:

Screen_Shot_2020-05-27_at_2.31.37_PM.png

How can I get the export to look like the map that I see in mapknitter? thank you!

Is this a question? Click here to post it to the Questions page.


I see what you are saying @pataxte , and it might take us several days to sort through this...in the meantime, sorry for your trouble! Could you check if the order that you select the images for export relates to what's shown on the exported map?

Is this a question? Click here to post it to the Questions page.


Good question @liz ! I did this and the order in which the images are selected does not make a difference, at least in this case the result is always the same, and not optimum as the two examples show above. I did notice though that there appears to be a default 'original' layout (see pic #2 above), and no matter if the images are locked in a different layering, as soon as I move between tabs (About, Images, Export, Chat) it reverts to the original layering. So it makes sense that it also reverts to that order when exporting. @warren and others just put the layer button back into Mapknitter a few weeks ago, maybe it doesn't communicate with the lock command? Also, the small vertical menu in the upper left corner that includes the download and trash buttons seems to have disappeared for me. It re-appears if I log out and shut mapknitter down, but after working a while it disappears again. As always, thanks so much for everyone's help/guidance on these issues.

Is this a question? Click here to post it to the Questions page.


Thank you so much for your detailed reporting and perseverance! Tuesdays are currently our day to prioritize code issues and i will bring this up tomorrow.


Ah, just responding here too - i believe we actually want a LOWER number than 70 cm/px -- in my response to the other comment and in https://publiclab.org/notes/pataxte/05-07-2020/mapknitter-exports-are-very-small-scale-unreadable#c26803 i suggest 0.2 -- sorry, it's a little counter-intuitive because a larger cm/px value actually means a lower resolution.

As to the question of ordering the images, I'll look into that now, I thought we had fixed it but perhaps @pataxte's point is that the ordering you carefully tweak with the menu is then voided when you do other actions, so we still have an outstanding issue? Looking into it now. Thanks!

Is this a question? Click here to post it to the Questions page.


Just following up - i see that:

  1. it seems that re-ordering manually does not result in a different request being sent to the export system; when I ran this and then moved the smaller image to the back and re-ran, it still had image with id = 85 as the first image in the request. I'll look into this:

Screenshot_2020-06-02_at_11.32.08_AM.png

Output:

  1. however, I also saw that interacting with other page elements like tabs could in some cases deselect the images, and also re-order them (possibly back to original order?) so that could be a second issue. I'll open issues in GitHub for both but will prioritize the first as I believe it'll be easier to fix. Thank you!

Is this a question? Click here to post it to the Questions page.


Just following up with a link to the GitHub issue for ordering! https://github.com/publiclab/mapknitter/issues/1328


awesome! looks like you already closed this issue, but i'm not sure if mapknitter.org may or may not be already reflecting the update. Please let us know when it's safe for @pataxte to try again :)


Reply to this comment...


Log in to comment