7 ms·
Earth on AWS – Open geospatial data
- malux85 9y agoWow what great timing! Just as we are scaling up our imagery DL projects, this is cool!
- phy07 9y agoOn a somewhat related topic - can anyone recommend a geocoder available through AWS? There are several AWS marketplace solutions available on the link at the bottom of the original article.[1] Only Geolytica and Forward Geocoder seem to be available to new customers, and both have < 5 reviews. [1] https://aws.amazon.com/mp/gis/#geocoding https://aws.amazon.com/mp/gis/#geocoding
- Bedon292 9y agoMight not be what you are looking for but https://wiki.openstreetmap.org/wiki/Nominatim https://wiki.openstreetmap.org/wiki/Nominatim is a geocoder that runs on OSM data.
- tuukkah 9y agoPelias (and Mapzen Search) is so much better: http://pelias.io/ http://pelias.io/
- SOLAR_FIELDS 9y agoThanks for sharing. Geocode and RevGeo are generally considered a Hard Problem (TM) in GIS so it is nice to see great projects such as this.
- rkda 9y agoYou might want to check out the Data Science Toolkit http://www.datasciencetoolkit.org/ http://www.datasciencetoolkit.org/
- freyfogle 9y agoHi, I'm one of the makers of the OpenCage Geocoder: https://geocoder.opencagedata.com https://geocoder.opencagedata.com We provide a single, simple API that behind the scenes aggregates numerous open geocoders, including nominatim, DSTK, and others. Please give us a try, there is a free testing tier you can use as long as you like.
- brightball 9y agoWow...that’s a treasure trove of useful data all in one place. Major thanks to Amazon.
- rement 9y agoHere is another source https://earthexplorer.usgs.gov/ https://earthexplorer.usgs.gov/. This one is the RAW data for many of the tilesets.
- pknerd 9y agoPardon if the question sounds dumb; will we have real time data of a certain region, for instance getting info about clouds?
- maxerickson 9y agoNot real time, but the Landsat, Sentinel-2, MODIS and GEOS data are all updated on a continuing basis. GEOS are from geostationary satellites pointed at the US and are updated a couple times an hour: http://www.ssd.noaa.gov/imagery/index.html http://www.ssd.noaa.gov/imagery/index.html
- zorm 9y agoGOES-16 imagery could be up to every 30 seconds over specific regions, every 5 minutes over CONUS, and every 15 minutes over entire disk now.
- nether 9y agoYou have real-time NEXRAD data, where the reflectivity is basically cloud cover. NEXRAD of course only covers the US.
- reaperducer 9y agoAnd Cuba! https://radar.weather.gov/ridge/radar_lite.php?rid=gmo&product=&loop=no https://radar.weather.gov/ridge/radar_lite.php?rid=gmo&produ...
- gravypod 9y ago> and identifies the people, locations, organizations, counts, themes, sources, emotions, counts, quotes, images and events driving our global society every second of every day. They're really serious about counting, aren't they.
- notwedtm 9y agoDodge, duck, dip, dive and dodge!
- mentos 9y agoWould love to see Amazon make an integration for Unreal Engine 4 or their Lumberyard video game engine so a game developer can easily import detailed swaths of the earth.
- flippmoke 9y agoThis already exists with Unity via Mapbox - https://www.mapbox.com/unity/ https://www.mapbox.com/unity/
- _pdp_ 9y agoI know this will be cool if combined with machine learning but to do what? :)
- yoloswagins 9y agoLook at cars trends of cars parked on streets.
- brootstrap 9y agolol is this a serious comment??? You have terabytes (petabytes) or data in front you of but cant think of a single thing to do with it???? Oh i know, we'll just 'machine learning' our 'big data' and get great business insights.
- borplk 9y agoWhat are some interesting things to do with this?
- jdavis703 9y agoVarious financial analysis' such as counting the number of cars in retailers parking lots, looking for crop shortages among commodity traders, estimating damage from natural disasters to estimate insurance company's exposure. I'm sure there's also more altruistic uses, such as providing better forcasting and advise for farmers in developing countries.
- cjalmeida 9y agoDisaster response is one of the altruistic cases. You can use deep learning to measure impact of hurricanes and better allocate resources
- BatFastard 9y agoBut this data can be months or years old, doesn't seem like this would have much value to financial analysts.
- nigma 9y agoIt depends on the revisit time, spatial resolution, region of interest, cloud coverage and product type. For example Landsat 8 images the entire Earth every 16 days, Sentinel-2 revisit time is ~5 days with 2 satellites and MODIS provides daily data but at moderate spatial resolution (> 250m). We expect both the spatial resolution and revisit time to improve as more companies are launching satellite constellations.
- BatFastard 9y agoI did not appreciate that fact. Thanks
- jefft255 9y agoIn my lab there is a masters student working on monitoring deforestation for palm fields in Indonesia using Google Earth Engine, which is similar to Earth on AWS. There is a whole scientific field devoted to analysing this kind of data: remote sensing. It's underrated in the hacker community honestly.
- lukejduncan 9y agoI wonder how this compares to Planet Labs dataset.
- rkda 9y agoAre you referring to their Open California dataset? https://www.planet.com/products/open-california/ https://www.planet.com/products/open-california/ It's larger as Open California only has the datasets from Landsat 8, Sentinel, and Planet's own satellites.
- deleted 9y ago[deleted]
- cloverich 9y agoIn case you don't scroll all the way down, there's a list of articles and video's titled "Use cases" at the bottom of the page which appear to cover (at least) how some of this data has been used.
- anigbrowl 9y agoI wish all technology announcements included that and put it up front. I'm often surprised at the number of interesting-sounding things I follow from HN's front page only to end up with no idea of what they're good for or why I might want to invest time in learning about them. Your most enthusiastic customers can sometimes be the people who didn't know what was possible until your product came along.
- colordrops 9y agoLooks like a lot of this data comes from free sources. It's not clear from their site what the licensing is though.
- cerealbad 9y agohow can you keep everyone safe if you can't see where they are, what they are doing or what is inside their head?
- Moocat87 9y agoI'd like to see the National Snow and Ice Data Center's data (soil moisture, sea ice cover/concentration, snow cover [looks like MODIS is already available], permafrost, glacier outlines) on AWS. I know there are people there that want to see it happen, but it's a matter of cost. What incentives/programs does Earth on AWS offer to assist stewards of public data to make it available on AWS? Additionally, I think some of this data is normally behind URS/Earthdata Login, what did the politics of making the data available on AWS without URS look like?
- zorm 9y agoNOAA is working on making this happen through the big data project: http://www.noaa.gov/big-data-project http://www.noaa.gov/big-data-project On the NOAA side, there tend not to be loginwalls so that hasn't been much of a concern. I work with one of the partners on this project, so if you have specific datasets or use case ideas feel free to drop me an email at zflamig uchicago.edu.
- brailsafe 9y agoI had the chance to speak with the/a rep from Amazon who at the time was working to make the Landsat8 data available on AWS exactly a year ago at a conference. From what I remember, AWS covers the hosting cost of the datasets in exchange for being able to incentivize the use of AWS in working with them. The data storage and transfer costs, as well as logistics are enormous. I don't recall how the transfer was being managed but it was certainly describe more or less as a partnership.
- zitterbewegung 9y agoThis makes me want to take this open source weather forecasting model and run it on AWS. http://planetwrf.com http://planetwrf.com
- nether 9y ago> The planetWRF model is no longer available for downloading. Welp.
- zorm 9y agoPeople are doing this already. See https://depts.washington.edu/learnit/techconnect/cloudday/wordpress/wp-content/uploads/Kevin-Jorissen_Amazon_HPC-on-AWS-cfnCluster-and-WRF.pdf https://depts.washington.edu/learnit/techconnect/cloudday/wo... for some good info on this.
- ktta 9y agoThis feels like another service Google can replicate and be much better at it considering their past experience.
- finstell 9y agoSure, they can replicate much better and then shut it down later on.
- ktta 9y agoI mean the second they decide to license their Google Earth data including the cleanups they do on it, their offering will be unparalleled. There's no other org who has gone through this expensive process other than google. Even if they decide to shut it down soon, the value companies and scientists get out of it will be worth it.
- Doctor_Fegg 9y ago> the second they decide to license their Google Earth data including the cleanups they do on it Haha. Good luck with that. Google doesn't give its geodata away.
- ktta 9y agoWe'd all be surprised, sure. But Google's biggest bet is cloud and I think they are willing to sacrifice a few things for the big win.
- Boothroid 9y agoPresumably you are talking about the vector dataset? I think most of the raster imagery comes from commercial aerial imagery sources (certainly they have no monopoly whatsoever on that). I think there are other global vector datasets that are broadly comparable, no? Streetview excepted. Maybe also worth pointing out that Google's record in geo isn't without its failures.
- tzakrajs 9y ago
- rburhum 9y agoLooking at the comments, most people don't understand what this is. In the geospatial industry, there are many organizations that produce free open data. For example the NAIP image data comes USDA and has been paid for by the US govt so the city/state can used it for agriculture - hence why the images are not just RGB, but they also include an infrared band so they can be used for agriculture algorithms like NDVI results. For that particular dataset the license is very liberal. In case you are curious about that particular problem, you can find more info here: https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery/ https://www.fsa.usda.gov/programs-and-services/aerial-photog... The problem with dealing with datasets of this size is that just the mere collection and storage of it, is a problem of resources. This AWS link here is saying that they have grabbed all these datasets from various govt and non-profits and are hosting them in raw form so you can use them. Because the data comes from so many different institutions, the license is different - but practically speaking super liberal. It is not competing with any previous commercial service from any vendor, nor it is meant to be a solution of any kind... Just big public spatial datasets hosted at AWS.
- andy_ppp 9y agoAlso, call me cynical but this is about running machine learning on impossibly large datasets meaning huge profits for AWS.
- joelhaasnoot 9y agoBasically a glorified mirror with some marketing/use cases.
- querious 9y agoWhat I'm psyched about is OpenStreetMaps data queryable with Athena. It's traditionally kind of a pain to convert PBFs to a queryable format.
- SOLAR_FIELDS 9y agoOut of pure curiosity, how so? I deal with Protobuf regularly, and as long as a decent library exists to dump to JSON that is domain specific to your use case it is trivial. Is that the only thing missing here?
- Boothroid 9y agoGlobal OSM is 40Gb or so - there are various libraries to translate it but as you can imagine, the sheer size of the dataset causes challenges. You also have to make choices about how you translate the attributes - for example, if you want to pull certain tags from the key:value field into separate columns in a table. Yet another issue involves source and target geometries - there can be inconsistencies in how features of the same type are recorded in OSM in terms of geometry, and so getting disparate input types translated into a single output type involves choices. Yes you can easily (after a wait!) get global OSM translated into something else, but making that something else exactly what you need can take effort.
- rmc 9y agoFor starters, the OSM PBF file format is not a protobuf file! Instead it's a collection of protobuf files inside each other! You can read more in the fileformat: https://wiki.openstreetmap.org/wiki/PBF_Format https://wiki.openstreetmap.org/wiki/PBF_Format There are other problems, specific to OSM and not PBF/protobuf, like needing to store the locations of nodes until the end of file because they could be referenced anywhere in the file.
- maxerickson 9y agoHave you looked at Overpass API? (it provides direct access to OSM data using a DSL: http://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL http://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL ) For tiny purposes the public servers are sufficient and there seem to be quite a few people running private servers.
- kiproping 9y agoI am waiting for the day we can get satellite images that are so fine you can see people or animals.
- Boothroid 9y agoYou can get this now if you are military or have plenty of spare cash lying around. The problem is that to get this level of resolution your sensor needs to be nearer the earth, and thus your platform has a shorter lifespan because it will be subject to greater atmospheric drag, and thus its per-picture cost will be comparatively very high. This might prompt the question, why not put it farther away with a bigger lens? Well, there is an upper limit on the size/weight of the lens that you can lob up to any given orbit, and thus it's less feasible to get this level of resolution from a higher orbit. You also have the issue of swath width to think about - generally the higher your resolution the smaller your imaging area, which might limit the usefulness and thus the price you can charge for your imagery. I think drone aerial imagery holds more promise than satellite imagery. Who knows though, perhaps with fancy new image processing algorithms and sensors we will get the level of resolution you are talking about from satellite imagery at reasonable cost over time. Edit: or with bigger cheaper rockets.
- rmocnik 9y agoExploring thru these datasets can be quite addictable. Especially with service like http://apps.sentinel-hub.com/sentinel-playground/ http://apps.sentinel-hub.com/sentinel-playground/
- wenbert 9y agoI'm always interested in these kinds of data. A few months ago, I was looking at different open sources to geocode a lot of addresses around the world. I have tried openstreetmap and some VM from datasciencetoolkit - both have poor results. Are there other sources aside from Google? Google appears to be the most accurate.
- arctux 9y agoCheck out https://openaddresses.io https://openaddresses.io It has ~477 million freely-licensed addresses.
- dsnuh 9y agoWill the datasets be open to contribution from members of the public or are these readonly mirrors? Seems like Blue Horizon and Prime Now amongst many other of their offerings that would be use cases for up to the minute data?
- kkmx 9y agoLooks like many of the datasets were obtained from federal organizations in which case it should actually be under public domain.
- tantalor 9y agoHow does this compare to other offerings like Google Earth Engine[1], GCP Landsat[2], or GCP Sentinel-2[3]? [1] https://earthengine.google.com/ https://earthengine.google.com/ [2] https://cloud.google.com/storage/docs/public-datasets/landsat https://cloud.google.com/storage/docs/public-datasets/landsa... [3] https://cloud.google.com/storage/docs/public-datasets/sentinel-2 https://cloud.google.com/storage/docs/public-datasets/sentin...
- william6 9y agomost people are indeed clueless to what this is.
- jaipilot747 9y agoSlightly tangential, but is there a "modern" alternative to GDAL for working with raster data? The last time I tried, stitching together tiles and cutting it to state boundaries took an inordinate amount of time (upwards of 15 minutes for 6 tiles from Landsat-7/8). Though, I'm half convinced it was because I was doing something very suboptimal.. Also, iirc, it was single threaded.
- llccbb 9y agoNo, GDAL is still the best. I also suspect you were doing something suboptimal. As far as modern wrappers for GDAL, `rasterio` is the most pythonic. Part of sgillies suite including shapely, rasterio, and fiona.