Background
Inspiration for blogging about my mapping application on the trial HCP is based on an old blog I published back in 2011. That blog covered ideas for thematic mapping with SAP systems and had a focus on mapping the worldwide user base of SCN, using each country's total SCN points for the theme of the map. My original intention for that blog was to cover thematic mapping of a particular country with location based data. The old blog was broken, with missing images and formatting, in the move to the SCN Jive platform. As I have now found that I can run the open source software I used in my original setup on the HCP, I decided to revisit the ideas on mapping location data for a particular place. This time out I used the town where I live (Solihull, UK) as the location with the intention of combining a variety of SAP technologies that I have used over the intervening years to create either maps or QRcodes. Although to begin with I have recreated a missing image from the original blog, which shows my general intention in how I use of maps and QRcodes.
Image may be NSFW.
Clik here to view.
The above image is generated entirely with GeoServer (an open source server for sharing geospatial data) running in the HCP trial account. It is based on the original image in my broken blog where I was trying out the ideas of thematic maps for Japan. It was while I was living in Japan that I picked up an interest in topics such as GIS(maps) and QRcodes/Barcodes. I have found these topics useful over the years as a use case/personal project to try and integrate them with and learn more about SAP technology. The QRcode in the above image is a static PNG file which is combined with the map to produce the final image by GeoServer. I use another open source program ZXing running on the HCP to dynamically create QRcodes in my application. I will cover how I use GeoServer and ZXing in the following sections but I start with an image of my final application below. The image below shows a choropleth map of Solihull next to a cluster map of crime data. The data is in HANA and I use XS to generate the choropleth and cluster maps, with GeoServer providing the base map and ZXing used to create the QRcode.
Image may be NSFW.
Clik here to view.
Over the next few sections I will cover each part of the application running in the HCP.
Neo GeoServer
I originally deployed geoserver into the NetWeaver cloud platform (it also had an alternate name of Neo) in 2012. Using previous versions of GeoServer I found myself hacking Java JAR files of the GeoServer package to get it running with the NetWeaver cloud Java runtime.
Since GeoServer moved to Java 7 with some of the latest releases and the now named HANA Cloud Platform (HCP) made a Tomcat Java runtime available, deploying GeoServer to the HCP is a lot more straightforward.
The way I deployed GeoServer is contained in this document - link. Some of the highlights (issues Image may be NSFW.
Clik here to view.
extracted below.
The key step for me was to actually pick the Tomcat server from Eclipse as the automatic choice did not always pick Tomcat from my experience. ( I found Tomcat was the automatic choice when deploying from the command line with the Tomcat SDK ).
Image may be NSFW.
Clik here to view.
Issues I faced were with increasing the timeout of the server while deploying to the cloud. The nature of how I use the application meant I had rather large GeoServer data directory to upload to the cloud. So I extended the timeout to allow the server to start.
Image may be NSFW.
Clik here to view.
Also GeoServer contains the SLF4J (a java logging framework) jar files and the Tomcat runtime also has these for logging. Once deployed I had a multiple bindings error in the logs. According to my interpretation of the slf4j website mentioning the issue, the JVM should still be able to pick a binding (at random!) so I took no further actions on this. http://www.slf4j.org/codes.html#multiple_bindings
Running GeoServer in the HCP considerations
GeoServer can connect to spatially enabled databases and as HANA is spatially enabled it should be in theory possible to make a connection. There is no standard connection for HANA from the Geotools/GeoServer download sites directly. However as GeoServer is open source software it is possible to create your own. I personally do not have the Java skills to do this (yet, maybe never, or never say never...erm).
I did find Marcel Akkerman’s github repo and with an HANA jdbc driver it could be used with GeoServer. So as I only ever hack example code, I had to try it out and got so far but sadly not a working example. I never attempted to use it in the actual HCP but via a tunnel method. I had GeoServer running on my local Ubuntu virtual machine. By changing a line in Marcel’s github repo and building the project with maven I managed to get GeoServer initially connected to HANA via the tunnel method. Screenshot below shows how far I got by defining a GeoServer store and selecting the tables in my HCP HANA database. (it falls over pretty quickly after that). I did try and contact Marcel via twitter to check if he had made progress on his implementation as the github repo is now a few years old, Marcel didn’t reply but if by chance you read this Marcel I would be interested in your experiences with HANA and GeoServer.
Image may be NSFW.
Clik here to view.
I would state connecting GeoServer to HANA directly would be my ideal approach but as that is a step too far (got a feeling I’ll come back to that in the future). I have alternate ways to use GeoServer in the HCP cloud as a standalone runtime only type installation. I use my local GeoServer as a working machine and when I am ready use bash scripts to publish to the cloud. As GeoServer has a REST interface I found it easier to use this interface to upload a lot of Ordnance Surveyopendata shapefiles to the cloud. Now that I mention Ordnance Survey opendata this brings me onto.
Open Data and open source geospatial software
While I was living in Japan I did find out that there was an open data strategy in the UK with the UK government opening up large amounts of data. I have tried out various sources of open geographic data such as the Ordnance Survey data and the Office for National Statistics geoportal was key for me to create the choropleth maps based on its geographical reference data.
If you are interested in starting out or checking out open source geospatial software then I would recommend the collection of software found on the the OSGeo Live collection as I mentioned in my original blog in 2011. This collection of software was how I got a head start into GIS systems when I lived in Japan. The latest edition is OSGeo-Live 9.0and is a bootable DVD, USB or virtual machine. The quick start guides were great for me to get up and running with GeoServer and QGIS (which I cover later on) and it is straightforward to setup (if you check the manual and help :)). For GeoServer in particular I have also found the documentation on the GeoServer site has pretty much covered all the items I wanted to achieve and a couple of key items found on the GeoServer training material offered by http://www.geo-solutions.it/
That covers my use of GeoServer, now onto QRcodes.
A Zebra Crossing in the cloud
I covered a method to create QRcodes (in November 2010) in one of my previous blogs based on the open source project called ZXing and running the app in an SAP NetWeaver Java Application Server. I used the NWDS to create my project and made an EAR (Enterprise ARchive) file available for download in that blog (the EAR file can be downloaded by following this link). I found that with a bit a manipulation of that EAR file I could get this running in the HCP. And now in 2016 , two Java apps can run on the HCP at the same time!! So this allows me to create maps and QRcodes with my application. At the very start of the NetWeaver cloud (as it was known) trial, then accounts had some limitations and only one app was allowed to run for Java, which reminds me….
I did deploy ZXing for QRcodes to the NetWeaver cloud as a demo during the limited trial access to the cloud back then. As I recall this, it does prompt me to mention and thank Matthias Steiner for his help and assistance over the time I have been using the HCP. Also a thank you to the SAP cloud team, even when I asked a n00bie (shooting myself in the foot type) question on the forums, there was an understanding and positive answer from HCP team. Also last time out with ZXing I tweeted a QRcoded message to Matthias asking why there was a only a one-off 90 day trial. Times have changed, and it seems a long time since the 90 day limits of NetWeaver cloud and amazing amount of possibilities with the HCP. Although it is difficult to know where to start next! So I’ll carry on with maps or a qrcode as a reason to hack around with the HCP. So below is a recreation of that tweeted QRcoded message asking why there was a limitation for the NetWeaver cloud. The message is way out of date now as there is ongoing access to the HCP trial but the QRcode is generated by ZXing running in the HCP.
Image may be NSFW.
Clik here to view.
Deploying a Zebra Crossing
I extracted the WAR file from the EAR file I had previously created, as I didn’t need all that enterprise packaging that an EAR file contains Image may be NSFW.
Clik here to view.
Importing the WAR file into Eclipse and then deploying to cloud caused issues that the deployed servlet could not be found. Googling the subject I added a fix to the web.xml to link back to the servlet, also when I changed the project name, I updated the QRcode.jsp to reflect the name change. The details of how I did this can be found in this document.
Limits of trial
I did have some issues deploying GeoServer to the cloud and mainly due to how I use the application on the HCP. There is a difference between structured and unstructured data and the data limits are within the HANA and MAXdb database and not as I had assumed 1gb of space no matter where the data is located. Another consideration for GeoServer is permanent file creation at the operating system level on the HCP trial. GeoServer configuration is stored in a data directory and any changes are not persisted at restart in the HCP. I use GeoServer's REST interface to work around this on the HCP trial and will cover that later on in this blog.
I enjoy using the HCP and it’s great that I can use the platform and all the applications on the trial for free (and not limited to 90 days as before :)). So I have limited the use of GeoServer cache in my trial however mapping applications can use a large amount of cached data to help performance of the map. ( I have not tried as I don't want to pay Image may be NSFW.
Clik here to view. but there are options to use AWS S3 for GeoWebCache)
At the moment I am checking out the Java monitoring and JMX monitoring capabilities of the HCP to assist me in monitoring my application.
Application setup
Now I had my base Java applications for maps and QRcodes running in the HCP it was time to adapt an application to use them. As I had already loaded 18 million crimes from 2011-2013 into HANA on the HCP it was an ideal dataset to use with its location based data. The data set in my HCP trial account covers crime and policing in England, Wales and Northern Ireland and downloaded from http://data.police.uk/data/ . The data is made available under the UK Open Government Licence.
Solihull Metropolitan Borough Council Crime Stats
As just mentioned back in 2013 I used Lumira to connect to the trial HCP to map England and Wales crime data and I went on to create a Lumira extension to map Solihull’s crime data. Example screenshot below. The method I used to create the maps with Lumira is covered in my blog post from 2014 (link). The same Lumira extension I created but one based on Japan can be found on the github open source collection of Lumira extensions here.
When I first created a map of Solihull below in Lumira I did notice one area which from the National Statistics site is registered as a Lower Super Output Areas (LSOA) and code is E01010109. LSOAs are a geographic hierarchy designed to improve the reporting of small area statistics
Image may be NSFW.
Clik here to view.
The above did not show any town names or identifiable features, so I used the openstreetmap based site http://overpass-turbo.eu/ to extract place names to show on the map.
Image may be NSFW.
Clik here to view.
The LSOA E01010109 area is somewhat lost in the above image and also it indicates that there are no local towns/villages in that area. The area is wide and I was curious to find out was there any particular cluster of crime locations in LSOA E01010109?
I know LSOAs have a mean average population of 1,500 so what could be in the E01010109 LSOA area as there appears to be no towns or villages. This is where the Ordnance Survey open map data could come in useful for me to add some more detail to my maps.
How I used the Ordnance Survey Open Data with GeoServer is in this document here.
There are many mapping services on the internet and some free and others not so free. One thing with most of these services is that it is their styled map and some have usage limits or you have to pay. Something that would be useful with maps, is the ability to change a map or mark a feature more prominently to highlight a story or area of interest.
My styling skills are limited and I did mention in my original mapping blog that when I did try some colour coordination, I struggled to match my tie with my shirt. In the interim years I have managed to lose the ties but the styling skills are still somewhat lacking. However using the great resource that is the Ordnance Survey data and style sheets I have had a go at altering the maps appearance, with an objective to make a point.
That point being Birmingham Airport, now I am picking on Birmingham Airport from E01010109 LSOA area but there are more commercial areas in that particular LSOA. A map below showing Birmingham Airport in the E01010109 LSOA area. The map is being styled and the (PNG) image created by GeoServer running in the HCP and using GeoServer’s CQL filter capability to highlight the LSOA area.
Image may be NSFW.
Clik here to view.
Simply by changing my CQL filter I could highlight another area and by an example if you follow this link another LSOA area of Solihull MBC will be highlighted.
Back to my HCP crime application and the E01010109 LSOA selected from the choropleth map on the left. The cluster map on the right indicates crime is located around Birmingham Airport however there are other clusters spread around the LSOA.
Image may be NSFW.
Clik here to view.
HANA Spatial Projections
I needed a way to convert the National Statistics site’s open data shapefiles to projections that can be used with HANA and my XS based choropleth map code in the HCP trial. Background to map projections can be found on wikipedia. The map projections straight out of the box/cloud in the HCP are rather limited (there appears to be ways to create your own in an non trial HANA system but from my experience with the cloud based HANA spatial projections are limited).
Image may be NSFW.
Clik here to view.
I decided to work with these initial projections shown above. However to get the National Statistics provided shapefiles loaded into the cloud, then some form of transformation was required as they were in different map projections.
SQL Anywhere & QGIS
I have used SQL Anywhere for a while now and think it is a genius database and impressed by its capabilities. Back in 2014 I mapped the crime data with SQL Anywhere and blogged about it here.
As stated previously I do use QGIS (A Free and Open Source Geographic Information System) and up to a certain release it could be connected to SQL Anywhere. QGIS latest releases are no longer compatible with SQL Anywhere. However using the latest version of Ubuntu Linux then a base install of QGIS will fetch version 2.0.1 and this is still compatible with SQL Anywhere. You can install the QGIS SQL Anywhere plugin with sudo apt-get install qgis-sqlanywhere and then select the plugin via the plugin manager in QGIS.
Using SQL Anywhere’s spatial ST_TRANSFORM it is possible to convert/transform map projections. Also QGIS can convert map projections. Image below of my installation of QGIS reading the LSOA shapefile I downloaded from the National Statistics. The key step for me was to enrich the data with the reference tables from National Statistics site with SQL Anywhere. This allowed me to select only Solihull's LSOA areas in my application.
Image may be NSFW.
Clik here to view.
With QGIS I can convert the enriched SQL Anywhere tables back to ESRI shapefiles with SRID 4326, which allows me to import them into HCP and leads me onto….
XS Application
I had the base XS services to produce choropleth maps when I created the SCN world application and blogged about that in 2015. The SCN world of points shapefile contained both the map and the SCN point values I wanted to map. How I loaded this into the HCP is covered in my blog here. With my crime map I did want to separate the data from the actual map shapefiles.
As you have made it this far through my blog, I will share the actual link to the application below. I have opened it up to all SCN users with a valid SAP ID logon account. As I learnt from my SCN World application I can open up the application to all SAP ID users. I would value your comments if you do have time to check out the application at the link below.
https://s5hanaxs.hanatrial.ondemand.com/p1248461150trial/hihanaxs/neogeocrime/xs/solihullcrime.html
I do use Stoyen Manchev’s 8 easy steps to develop XS app blog as a base or reference for everything I try with XS. As a result of using this base XS code, most of my XS services use a sales order variable even though there ain’t no sales order on my site. The code is based on examples from Stoyen and also Trinoy and Kevin who I thanked in my SCN world blog as well. Follow this link for the code to generate the maps with XS services and the javascript source code can be accessed in the link to my app above.
Putting it all together on a Leaflet
I used a great open source Javascript libraryLeaflet.js to bring all aspects of my application onto a map. Using leaflet I can make XS service calls to generate the choropleth and cluster maps. As the choropleth is useful to identify areas of high crime, it does not provide the ability to drill down into the individual crime locations. A leaflet plugin calledPruneClusterallows me to show a large dataset via a cluster of crime locations..
I have also chosen to useoverpass turboon my site to highlight certain aspects of the location related to a crime type. I have previously used overpass turbo with Lumira whenI highlighted the university of Oxford buildings/sites and linked this to the location of bike theft.This time I used a Leaflet.js plugin/extensionleaflet-layerJSONto show various points of interest on a map where I have related them to the crime. So for example a shopping trolley icon will be displayed for any shops, if shoplifting is the selected crime type. Also bike crime and vehicle crime will show parking areas for these types of crimes.
I call the overpass turbo site via the HTTP destination configuration setup with the HCP. I have a xshttpdest file and use XS service to call an HANA spatial query (using SHAPE.ST_ENVELOPE) to calculate a bounding box of the LSOA to pass to the overpass turbo site. The code I used for the HTTP destination is available via this link.
An example screenshot below highlighting these methods described above. The map on the left is the aggregated crime data shown in a choropleth map and the map on the right is the cluster of crime locations. The choropleth map controls the cluster map and is kept in sync with another leaflet plugin calledleaflet.sync.The bike icons are bicycle parking locations in Solihull from the overpass query. An attempt by me to highlight bicycle crime next to bicycle parks.
Image may be NSFW.
Clik here to view.
Scanning with Intent part II
The way I use the QRcodes in my app is to use them to start a navigation app on my phone. This will automatically start the navigation on my phone and works as follows.
In the same blog I created QRcodes with ZXing I also tried out scanning the QRcode back into SAP via an ZXing based app on my HTC Android phone. As a final part to this blog I will cover the attempts I have made to revisit the scanning of QRcodes but now with my Nexus 5 Android phone. I reused the intent code I shared in the original blog with template code from the Android SDK and checked it worked. I then modified the code to use a navigation intent from the results of the barcode intent.
https://developers.google.com/maps/documentation/android-api/intents#intent_requests
https://github.com/zxing/zxing/wiki/Scanning-Via-Intent
Image may be NSFW.
Clik here to view.
The image above is a screenshot of the final app in the Android SDK.
I had complained in my original QRcode scanning blog on how difficult it had been with my original Android phone to capture screen shots in comparison to Apple products available at the time. Screenshots are now a whole lot easier on my Nexus Android phone and can be generated with two key presses. However has I have the SDK (again) installed I can now quite easily capture a video of my phone in action. A video of my apps scanning a QRcode and starting navigation via map apps installed on my phone is below (video recording direct to the phone via adb shell screenrecord /sdcard/Download/sapqrmap.mp4)
An Intent to use HAT
Before I cover how I used the Hybrid Application Toolkit (HAT) I will thank Simmaco Ferriero's for his blogs about installing HAT on a Mac and how to use the Kapsel barcode scanner. With my own Mac I had started to install all the component parts with independant downloads. Following Simmaco’s installation blog on a Mac I had success with Homebrew ( the missing package manager for OS X) method of installation. I followed Simmaco’s barcode scanner blog only for the barcode scanner parts and not the integration to Gateway. My intention was to use a third party plugin for cordova for Android like Intents but the plugin is stated to work for multiple devices. I only use an Android phone in my examples.
I used Simmaco’s Kapsel barcode scanning blog as a guide but chose the starter application (icon shown below - I also used this icon for the app on the phone). I added the Kapsel barcode scanner as per the blog to the device configuration but also added the geolocation option from the Cordova options page. I added the code from the blog to add the barcode button to the view and the code for the javascript controller and deployed to my running local HAT instance.
Image may be NSFW.
Clik here to view.
I then chose to deploy to my phone to check that the base application would at least start the barcode scanner on my phone.
I built the Android APK with
cordova build
This created an APK called android-debug.apk
I then manually transferred the apk to my phone and tested this successfully, as the barcode scanner started.
I then added the custom plugin to call the navigation.
I had installed plugman (link to a reference for plugman in another of Simmaco’s blogs).
The third party plugin I wanted wasuk.co.workingedge.phonegap.plugin.launchnavigatorand I installed it with the following command.
plugman install --plugin=uk.co.workingedge.phonegap.plugin.launchnavigator --platform=android --project=QRcodeNavi/hybrid/platforms/android --plugins_dir=/Users/robert/SAPHybrid/plugins
Installing "uk.co.workingedge.phonegap.plugin.launchnavigator" for android
I then repeated the HAT setup but added the custom plugin path
Image may be NSFW.
Clik here to view.
HAT has to be ready and running (run.sh) and it will use the directory setup in the config of HAT.
./run.sh
Input your HAT Connector certificate password: **********
===================================================================
Hybrid App Toolkit (v1.12.5) Connector is listening on port 9010.
I created a new WEB IDE project from the starter application and added the same barcode and geolocation plugins as before. However this time I added the WEB IDE setup for custom plugins to pick up the navigation plugin.
Image may be NSFW.
Clik here to view.
I updated the controller code to call the launchnavigator plugin with the results of the barcode scan. I used a Javascript substring command to get the lat,lon details only from the URL shared on my crime site. This way I can use both the native Android app and the SAP Mobile app with my site.
Image may be NSFW.
Clik here to view.
I then built the APK with the cordova command and copied the APK to a temporary directory.
I used the following command for the first Installation onto my phone via a USB connection
adb install /var/tmp/QRapk/android-debug.apk
If I changed the code and wanted to push the same APK to my phone then the installation failed with [INSTALL_FAILED_ALREADY_EXISTS]
To get around that I now use the -r flag
adb install -r /var/tmp/QRapk/android-debug.apk
A video for both mobile apps for scanning and navigation with QRcodes.
Thank you for reading and I end with some credits.
Credits
Crime Data
As covered in my previous blog (link above) I use the crime data from http://data.police.uk/data/. The data is made available under theUK Open Government Licence.
Census Data
I use the official labour market statistics site for the census data which can be found at the following web address
https://www.nomisweb.co.uk/census/2011
Choropleth Map Data
For the choropleth maps on my site then I used the Office for National Statistics to download the geographical reference data here at this link ONS Geo Portal
Contains Ordnance Survey data © Crown copyright and database right 2015
Contains National Statistics data © Crown copyright and database right 2015
Base Map Data
The base map data is from Ordnance Survey Open Data shapefiles and available with the Open Government License. https://www.ordnancesurvey.co.uk/business-and-government/products/opendata-products.html
Contains OS data © Crown copyright [and database right] (2016)
Leaflet - Javascript library for maps
My application relies on Leaflet.js to bring all the contents together in one place on a map. Thank you Vladimir. Also thanks to the extensions I use for leaflet, leaflet.sync, leaflet prunecluster and leaflet-layer.json
OverpassTurbo
Overpass-turbo.eu I use the api from this stie to display Openstreetmap data on my map. License link for the API. So many options for extracting the data and the way I use it is to highlight certain features on a map to correspond to a chosen crime type.
.