Global Mapper v25.0

Georeferencing historical maps of undocumented local datum

wmiswm
wmiswm Global Mapper UserTrusted User
I am not sure in which forum section I should put this post in, if not appropriate, please move this to the appropriate section. Thank you very much.

I am georeferencing some sets of historical maps with undocumented local datum. The maps have accurate grid information printed on the maps. This means, I can read the co-ordinates of any point on the maps in the unknown datum. On the maps, I can also identify plenty of ground control points (GCPs) with known co-ordinates in known datum such as wgs84.

I understand that I can geo-reference the maps directly with GCPs of wgs84. However, georeferencing by direct triangulation will have several undesirable issues.
1. it is assumed that all GCPs are accurate. The co-ordinates of all GCPs are preserved in the triangulation. This means that any error of any of the GCP, no matter introduced in the original historical map production, or during georeferencing, will not be corrected.
2. Any single GCP will not contribute to the global accuracy of the output. Any single GCP is treated as a local reference point and will have local effect only, ie affecting the triangulation of its immediate vicinity only. In other words, I cannot use even say 200 GCPs to correct for a single GCP error.
3. In areas where suitable GCPs cannot be available, the triangulation is "extrapolated" from nearby areas. But extrapolation will proportionally magnify any error according to distance. This means that the error is proportionally increased from the nearest available GCPs.
4. Maps of the same unknown datum will be treated as independent maps and each have to be georeference with the same set of GCPs repeatedly.

I am not sure whether Global Mapper can use another algorithm in the above scenario. By supplying a number of GCPs with the corresponding co-orinates in the unknown datum together with the known datum such as WGS84, Global Mapper will reverse engineer the parameters of the unknown datum, correct for any deviation by statistical calculations, and find out the most statistically conforming datum parameters. Then the user can georeference the printed map using the reverse engineered "native" datum, and transform the georeference map into say WGS84 format.

The merits of this algorithm are that, each GCP will contribute to the overall accuracy of the final output statistically. The error of individual GCP, if any, can be corrected by statistical handling. The more numbers of GCPs available, the more overall accurate will be the final output. The areas without available GCP will be as accurate as those areas with plenty of GCPs. Also, when the reverse engineered datum parameters become available, maps of the same undocumented datum can be easily georeferenced using the "native" datum.
«1

Comments

  • global_mapper
    global_mapper Administrator
    edited March 2014
    You can get rid of the effects of using triangulation for an individual layer rectification by changing the Rectification method under the Options menu on the rectification dialog. Likely Polynomial is what you are more accustomed to.

    If you have a list of control points in the new datum and WGS84 you can also create a custom datum based on that transformation file. You do this using the 'Add Datum' button on the Projection tab of the Configuration dialog. Go to the Help for the Projection tab of the Configuration dialog (press Help on the Configuration dialog then select the Projection link) for more information about the formatting of that file.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    From the scanned tiff, I can read a large number of GCPs, each with co-ordinate in the undocumented datum and WGS84. What I previously did was that, I chose WGS84 in the projection tab, then I manually selected the GCPs one by one, and typed in the WGS84 co-ordinate. In the "Option" -> Rectification Method, I chose "Automatic".

    Then I have the issues I mentioned in my previous post.

    Please confirm my understandings of your suggestions, and correct me if I am wrong:
    1. open a scanned image file and the Image Rectifier will open
    2. in the "Option" -> Rectification Method, choose Polynomial
    3. in the projection tab, choose the correct native projection that the map was produced (say UTM), choose the correct Zone, then in the datum tab (which is an undocumented datum), choose Add Datum
    4. in the "Add Datum" pop up menu, key in an arbitary name, abbreviation etc, choose the correct spheroid, say International 1929.
    5. then choose "Custom shift base on Control Point File", and choose the GCP file as appropriate.
    6. then press "OK" and go back to the Image Rectifier page.
    7. locate and add in a few ground control points and type in their native co-ordinates in the undocumented datum.
    8. finish the Image Rectifier and go to the main Global Mapper window
    9. output to global mapper package file in WGS84
    10. quit Global Mapper and re-open the gmp file to have the transformed map in WGS84

    Am I correct with this procedure?
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    BTW, the projection specified on the map is "Cassini", but in Global Mapper, there are "Cassini Malaysia" and "Cassini Soldner". Which is the correct one to use?
  • global_mapper
    global_mapper Administrator
    edited April 2014
    Yes if your GCP file has lat/lon coordinates in the existing unknown datum for the map and WGS84 then creating a custom datum from that file, then adding a few control points in the native projection of the layer (if known) would work according to the procedure that you described. Note that you wouldn't need to close Global Mapper to re-open the GMP (package) file, you can just use File->Unload All, then load the new package file in.

    Cassini Soldner is the base projection system which supports any parameters, whereas Cassini Malaysia is a particular Cassini-Soldner projection definition commonly used in Malaysia. So you would likely want Cassini-Soldner with any appropriate parameters. What part of the world is the map in?

    Another option for rectifying the image which may be much easier is to first use the File->Download Online Imagery/Data menu command to load some data for the same area, like perhaps World Imagery or World Street Maps, then just select common control points from the online source rather than use the unknown datum and projection system. If you just select a good projection for the area (like UTM) and a few common control points from a known good source that should work without all of the extra work to set up a custom datum and trying to determine the original projection system.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Hi Mike,

    Thank you very much for your instructions. The maps are some sets of historical maps of Hong Kong, produced about 100 to 40 years ago. All the maps are high quality maps based on accurate triangulation and high altitide photography. Unfortunately, the datum used was Hong Kong 1929. It may sound rediculous, but it is a fact, Hong Kong does not have document preservation law. The datum details are not recoverable now.

    What are known about the map parameters are those printed as footnote on the maps:
    Projection: Cassini
    Spheroid: Clarke 1858
    Unit of measurement: Foot
    Meridian Origin: 114 degree 10'42.80" East of Greenwich
    Latitude of origin: 22 degree 18'43.68" North
    Scale Factor at origin: Unity
    False Co-ordinate of origin: 132033.92 ft Easting
    62565.96 ft Northing
    Datum: Hong Kong 1929

    The above are all the known facts of the maps. Do you have any further advice on the full utilization of these available facts?
  • global_mapper
    global_mapper Administrator
    edited April 2014
    With that information you could fully define the Cassini Soldner projection used. I believe that the Hong Kong 1963 datum is a re-triangulation of the Hong Kong 1929 datum so it is likely close enough unless you have very detailed maps. Of course if you just rectify against some modern data with several points in common and set the map projection to the Cassini Soldner with those parameters and really any datum you should get an extremely accurate rectification.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Global Mapper
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Hi Mike,

    I actually have tried to define the HK1929 manually, but not very sure about whether I have done the procedure correctly. The procedure I have taken is as below:

    1. Open GM and click on the "Display Settings / Pprojection" tab
    2. choose "Cassini-Soldner"
    3. choose "Feet (International)"
    4. click "Add Datum"
    5. enter the datum name and abbreviation
    6. choose "Clarke 1858"
    7. click "apply" and goes back to the Projection window
    8. enter the Central Meridian value as 114.178555555
    9. enter the Central latitude value as 22.3121333333
    10. enter the False Easting value as 40243.938816
    11. enter the False Northing value as 19070.104608
    12. click "apply"
    13. save the projection to file

    Then I started rectification of a scanned image with this datum entry. And then I output the map to gmp in WGS84. Then I locate the co-ordinate of an arbitary poi and check its co-ordinate against modern maps. Unfortunately, the so obtained co-ordinate deviates from the co-ordinate from modern maps by about 300m. Is there anything or parameters that I have wrongly entered?
  • global_mapper
    global_mapper Administrator
    edited April 2014
    You would need to provide some datum conversion parameters when defining the custom Hong Kong 1929 datum in addition to just selecting the ellipsoid (Clarke 1858) that it's based. I suspect the Hong Kong 1929 is nearly identical to Hong Kong 1963, so rather than creating your own custom datum, just select the built-in Hong Kong 1963 datum and see if that gets you pretty close.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Hi Mike, the issue is that my historical maps are referenced with the hk1929 grid readings, and unfortunately I do not have reliable conversion parameters for hk1929 to HK1963. I have actually tried to rectify the tiff using HK1963, but found that the whole map is shifted far out of sight.

    My maps are in 1:10000 scale, so are very accurate and detailed. Approximate conversion is not quite acceptable for these maps (for identifying and locating historical objects). You mentioned that I have to supply some conversion parameters in the datum definition in order to make it work, and I am quite sure that the parameters are not available if not derived from scratch. Is there anything that I can do?
  • global_mapper
    global_mapper Administrator
    edited April 2014
    Since HK1963 is just a re-triangulation of the same control points as HK1929 I was hoping you could use them basically interchangeably, but perhaps not. If HK1929 is different enough to matter at that scale and you don't have any known parameters them I'm afraid the approach to take would be to either define a custom HK1929 using a file with known HK1929 and WGS84 common points, or just rectify in a different datum against a different known data set, like perhaps modern high resolution imagery where you can pick accurate control points that you can also identify in the original data.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Hi Mike again, on trying to get some information of HK1963 to HK1929 conversion, I found two contradictory descriptions of the origin of the HK1963 datum. The value of the origin co-ordinate description is not matching.

    Hong Kong 1963 Grid
    Hong Kong 1963

    When I tried to call the build in HK1963 datum in GM, by opening GM -> Display setting/Projection -> Projection: Cassini Soldner -> Datum: HK1963 -> Planar Unit: Feet (International)
    I then notice that all the four parameters in the "parameter" box is zero. Should I enter the four parameters manually according to HK1929 or HK1963 (do not sure which of the above link is reliable!).
  • global_mapper
    global_mapper Administrator
    edited April 2014
    The 4 parameters for Cassini-Soldner would be the same regardless of datum as those are the projection parameters. Each datum consists of an ellipsoid and transformation parameters specified on the Add Datum dialog. That is the step you missed before, but I don't believe any transformation parameters for the HK1929 datum are published anywhere. Hopefully HK1963 is close enough that you can just use it though.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Hi Mike, I also believe that the HK1963 is largely based on the same parameters as the HK1929, but with the origin or griding shifted.

    I just did an experiment. I used the HK1963 datum, Cassini Soldner projection, Feet (International), and entered the 4 parameters of HK1929 (origin, false n/e etc), I mean HK1929. Then I rectified an image according to this new setting, and output to gmp in WGS84, unload all loaded data, and import the just created gmp. I then checked an arbitary poi with known modern wgs84 co-ordinate. The modern co-ordinate is shifted to the north by about 100m (means that the so produced map is shifted to south of its true location by about 100m).

    Do you have any clue to correct for this shift?
  • global_mapper
    global_mapper Administrator
    edited April 2014
    If the shift is consistent you can change the Cassini-Soldner definition to change the false northing by 100m to adjust for the shift, then just use that modified projection for the rectifications.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Hi Mike, I have another question. How can I edit the custom datum I entered earlier?
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    I found the custom_datums.txt file now. My next question is, can I create a custom datum which is based on a build in datum? What I wish to do it to add a custom HK1929 datum which is based on the build in HK63 datum transformation parameters.
  • global_mapper
    global_mapper Administrator
    edited April 2014
    I have updated the adding of a custom datum so if you first select the Hong Kong 1963 datum, then press Add Datum, the parameters will default to what the selected datum used. You can then just modify those.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • global_mapper
    global_mapper Administrator
    edited April 2014
    I have updated the adding of a custom datum so if you first select the Hong Kong 1963 datum, then press Add Datum, the parameters will default to what the selected datum used. You can then just modify those.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • global_mapper
    global_mapper Administrator
    edited April 2014
    Forgot to paste the build. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • global_mapper
    global_mapper Administrator
    edited April 2014
    Forgot to paste the build. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Thank you very much. I have checked a few poi co-ordinates and seems that I can accurately transform the hk29 to wgs84 now. I shall come back should I encounter further problem. Thank you very much again.
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    hi Mike, seems that your download link is not working.
  • global_mapper
    global_mapper Administrator
    edited April 2014
    Does the link itself not work or is it running what is in the zip that's the problem? The .zip files aren't full installs, just new builds that you have to extract over your existing installation to update them.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    The link is working now, thank you very much. And I have solved my datum issues, thank you very much again.

    But I have several issues to report.

    1. open gmw, in control panel, right click map, choose rectify, choose right most projection tab, assign the correct datum projection parameters etc, apply the new parameters to re-interpret the entered GCPs, choose yes etc and goes back to GM main window, save gmw. Close GM, re-open the just saved gmw, go to control panel to check datum and projection etc, unfortunately the new rectification parameter settings remains as the incorrect old values. Seems that saving as gmw cannot over-write the previous wrong rectification parameters. However, exporting directly to Geotiff without going through gmw will have the correct rectification settings.

    2. my scanned map is about 6GB big. Export to PNG and save the .map .prj etc together. Close GM and then load the just saved .map file, the map (PNG) is rotated 180 degrees, namely north becomes south, and east becomes west, curious enough. Output to Geotiff does not have problem.

    3. open the so rectified 6GB geotiff by GM, output to gmp in wgs84, wait for more than one hour to complete the output process. Unfortunately the resulting gmp is just about 1GB in size and cannot be opened by GM. However, change to wgs84 lat long projection in configuration -> projection, then output geotiff will produce the correct wgs84 geotiff.
  • global_mapper
    global_mapper Administrator
    edited April 2014
    Can you provide the GMW file with the specified datum and parameters? When you save the GMW and reload is it the control points that are being lost or is the projection/datum being changed to something else?

    For the PNG export, if you delete or rename the .map file does the PNG come in properly? I'm wondering if perhaps there is just an issue with the Ozi .map export as that is rather new.

    What error do you get trying to load the 1GB GMP file? I'm not surprised that the GMP is smaller as the GMP will internally store a compressed GeoTIFF file for the image so it might shrink down a bit.

    Also can you get the latest build and see if you get the same results? There were some recent changes for embedding custom datum definitions right in GMW and GMP files that may help. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Hi Mike,
    Let us concentrate on the gmw rectification issue first, the others will take me many hours for testing and we can deal with them some days later one by one.

    The bug is described as below. I rectify a raster image with projection parameters of file A.prj, then save the workspace as A.gmw. Then I quit GM, and re-open A.gmw. Then I go to the rectification window, and re-interpret the GCPs with parameters of the file B.prj. I then accept and exit to the main GM window. The I check the configuration -> projection and confirm that the parameters are that of B.prj. I then re-enter the rectification window to re-confirm that the GCPs are in fact interpreted according to the parameters of B.prj. Everything seems good up to this point. I then save the workspace as B.gmw and exit GM.

    I then re-open B.gmw, and go into the rectification windows; and I also go into the configuration -> projection window to read the projection parameters. Unfortunately, both windows show that B.gmw retains the projection parameters of A.prj instead of B.prj.

    I can reproduce this bug with any prj and image files, and the latest GM build above gave me the same result.
    Please see what can be done. Thank you very much.
  • global_mapper
    global_mapper Administrator
    edited April 2014
    I'm so far unable to reproduce this. Can you provide me with the a.prj and b.prj and a.gmw and b.gmw so I can see what they have? I tried with 2 PRJ files, one with UTM zone 15 NAD27 and the other with the same projection but NAD83. They both saved and restored from the .gmw file and with the rectification just fine.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    I'll send you via email after the Easter holiday, thank you very much.
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited April 2014
    Hi Mike, I have sent you an email with a set of files for your analysis. Thank you very much.
  • global_mapper
    global_mapper Administrator
    edited April 2014
    I took a look and found that the issue was that the a.prj and b.prj were so close that the differences weren't captured by the embed of the projection definition in the .gmw file. The central meridian and origin latitude were stored to a thousandth of an arc second precision, which is about 3 cm on the ground. I have increased the precision by 2 digits so now you get those stored to 1/100,000th of an arc second, or about 0.3 mm on the ground in difference between lat/lon projection defined values before you see a difference in the PRJ file.

    I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation