water temp: explore adding locations layer to map #684
Comments
|
QW2019_national_GIS.zip |
|
Using https://github.com/usgs/wdfn-graph-server it would: Returns a png containing a hydrograph for parameterCode(xxxxxx) for the NWIS site(siteid) for the last seven days. P00011 or P00010 would give us temperature. |
|
|
|
I spoke with modelers and Pat tonight. They like the idea of adding the temp locations on the map. Steve Markstrom suggested plotting Mean Daily Value from Water Watch (or NWIS) with symbology to match the modeled values for QW points. Would provide some validation verification.. Get a sense for uncertainty. Essentially grabbing the previous daily value and symbolizing it. No popups etc. That sounds more like what Marty was suggesting. |
|
I think trying out what this looks like and whether we can color based on the previous day's temperature would be a nice addition if you have some time to work on it after the Q&A. @mwernimont |
|
I converted the shapefile of stations with temperature sensors locations to geojson and uploaded it to http://maptiles-prod-website.s3-us-west-2.amazonaws.com/geojson/QW_Temp_2019.json to play with just plotting those locally, here is what it looks like with some questionable style choices, for example:
going to see if i can get a dot click to open up a graph api image response, like https://waterdata.usgs.gov/nwisweb/graph?agency_cd=USGS&site_no=09380000&parm_cd=00060&period=7 |
|
thanks to some Marty help ™️ this morning, we got this up on test. the style isn't really well thought out at the moment (dot color and pop up itself), but this was the 'low cost' idea i was babbling about monday.
we might want to decide if this is useful to bring some additional detail to the viz, if we want all the station points or maybe we filter the layer first by those that 1) only intersect the GF segment layer and/or 2) select specific station representatives where we have large clumps. the user can't really discern when they are so heavily clumped. or we could allow more zooming in too, to help with that issue. one thing too Marty pointed out was that we sometimes don't actually get an accurate response for the last 7 day request, that's just because the server doesn't have the data to share, you might see that if you click around long enough
working on getting it out to beta too to see how mobile looks 😬 |
|
@mhines-usgs Casey Lee pointed me to a national app where I downloaded the Temp sites they say are continuous for 2018 -2020. http://kswsc.cr.usgs.gov:3838/cjlee/qw_inventory/ We can take a look and compare the sites that we have with what they have. I agree that many of the sites in the south in my data layer didn't have values for the last week. |
|
A second option would be to highlight the water quality monitoring Network of 110 sites across the Nation. https://nrtwq.usgs.gov/nwqn/#/ |
|
Third option which I'm leaning toward is a download from QW watch. I extracted only those that are Site Type Stream. |
|
Ok, Working now on exchanging the GeoJSON for this last suggested one instead so we can see how that looks. |
|
Thanks Megan. I think it should be Alright. The sites in that list should only be streams, no canals, ditches, or groundwater sites. Since they are reported in QW Watch real-time page they should be live. Even the waterwatch site has disclaimers for missing data: "Site operated on a seasonal basis or currently is not operating. No values are available for the last 6 hours." Which we could adapt for our app to say "Some sites are operated on a seasonal basis or currently is not operating. No values are available for the previous week. The data used to produce these graphs are provisional.". |
|
sounds good. I loaded the new data in and updated the test tier now with that data. I need to take a closer look tomorrow and make sure the data are correct, initially when I converted from xlsx->csv, the site_no field converted to numeric and dropped the leading zeroes, hopefully I got them all back where needed and did not add any where I didn't need, but will take another gander tomorrow and redo the conversion more programmatically to confirm. I will also look at a few ideas for display options then too and make sure the layer shows up for toggling in the layer list. in regards to the text addition, I would guess that text should be present on the first ? page for 'what does this map show me' -- and maybe we could also include a short mention in the About this map text. Thinking something like It shows the latest available daily estimates of natural water temperature for approximately 58,000 stream reaches across the lower forty-eight states. or something like that! suggestions welcome! --suggestion: It shows the latest available daily estimates of natural water temperature for approximately 58,000 stream reaches across the lower forty-eight states. The real-time temperature monitoring sites are visible at finer zoom levels, and clicking will open a graph of the past 7 days of temperature monitoring data along-side the modeled temperature estimates. |
|
I think these look good. I found a few that don't have data pulls for 00010 in Montana. I think we should increase the zoom level 1-2 times more to allow people to see what is going on in the dense city areas like Washington DC or Philly, PA. |
|
I'm getting errors like this: I checked this site and it has continuous data but it didn't have any since mid-day on Dec 1. but Nov 24 - 30 it had data.... I got an error for 11312685 that "site/Parameter-code combination not found. But there are two forms of temperature there, YSI and ADCP. 073816503 says graph request failed. But there are data in NWIS. Still have a lot of graph request fails near New Orleans. 02098206 in SC combination not found, but there def are data. We can ask Jim about the functionality of this tool. Maybe certain kinds of water temp are harder to pick up.. ADCP etc. We could switch to the 110 site list of the NQWM network points if we wanted to keep it simpler... |
|
Yeah, I saw that too and that's why I want to revisit what I did and make sure that the site numbers are actually correct, too. |
|
The ones I checked in NWIS were all correct station numbers. I could not see a reason except for one site that didn't have data for yesterday. The rest all seemed to have data. |
|
I have one more option for us. Jordan pointed me to the next-gen graphs. How hard would it be to switch to these graph renderings? https://labs.waterdata.usgs.gov/about-graph-image-api/index.html |
|
I'll give it a look |
|
So its possible A couple caveats I am seeing though: The other is there doesn't seem to be a combined temp C and F parameter code for water temp? https://help.waterdata.usgs.gov/parameter_cd?group_cd=% So would we just show C or F? or am I missing the code to do both in my searches? |
|
Nevermind, Megan pointed out the F is on the right. My eyes are horrible apparently |
|
I am also just getting lots and lots of failed calls even on sites that had worked previously. Even on their own api call testing area it failed. So not sure if we want to put this in or bring someone in to help figure out why so many fails would be happening? |
|
The old figure is working well, then I don't think we should try something new, esp if it's not working well. |
|
Well, are there more errors in the new service? We certainly could ask. Still, if the old one works, let's leave it. |
|
All I can really say for certainty is I adjusted our call parameters to the old service to call to the new one and was get timeout errors on most every call. Maybe the way we call the old service isn't the best for the new service etc etc. So it is nice to have currently working option, but maybe in the future if we want the new service for sure in the app we bring in outside help to make sure we are doing everything perfectly. |
|
Ok. I think the graphs look less fuzzy. Thank you. The panning is good too. Any luck with figuring out the reason for so many errors IF there is data? |
|
@mhines-usgs @mwernimont @lindsayplatt I think we can move to Production. What implications would that have IF we are still tweaking the station list for the Graphs some? Otherwise I think we are good to go. |
|
as long as we replace the json file in s3 with a file by the same name, we can update that anytime even after the prod release. |
|
the errors are not anything we are able to fix, unfortunately. it should be something that team can look into and addresses when they decide it is a priority. |
|
ok |
|
@mhines-usgs @mwernimont Joe plans to approve at lunch. Could you do a final check of our sites against the sites in Water Watch QW page? There were a couple in WI that Joe found in our mapper but they don't have water temperature at the site at all. [9:23 AM] Nielsen, Joseph P USGS Current Conditions for USGS 05429700 YAHARA RIVER @ FORTON STREET BRIDGE @ STOUGHTON,WI USGS Current Conditions for USGS 05427530 ROCK RIVER AT NEWVILLE, WI I pullled them from this website: https://waterwatch.usgs.gov/wqwatch/ Can you verify the list I gave to Marty is what we have in the mapper? Can you check sites that have data in QW watch against this list of sites that are in this excel file? |
|
@jenniferRapp we are on it and will report back! |
|
Downloaded the xslt file you gave us and the sites in question by Joe are in there. So they are in our json data as well. So a little recheck of the QW lists and data creation is probably needed as we don't know how many sites we don't want ended up in our data. Looking into it. |
|
Ok, here are the steps I used, I have never used this site before to retrieve station info so I'd like you to review it:
When that opens a dialog window to save, add .tsv on the end, open in Excel the RDB header looks like US Geological Survey Then filter by Site Type to remove the types Jen wants to exclude https://maps.waterdata.usgs.gov/mapper/help/sitetype.html
which leaves
|
|
I would just join the two tables together to see which sites are common to both. Most of those site types have to come out. Lake, ditch, es.. nothing but streams. |
|
Sorry, I'm not understanding the first comment? Join which tables? So only leave stations that are ST? |
|
The one I pulled and is the excel file, or the file that is IN the mapper now, to the sites you pulled today that show the temperature data on the website should be joined together. I think I only kept ST last time. Jen |
|
for posterity, sent jen the csv to review, here is the csv version and the geojson version that is ready to replace the file in s3 |
|
I used the file Megan shared that represents the current sites in the Mapper "common_temp" and checked against a pull of data straight from the webpage displaying current conditions for Temperature water watch sites. Any sites that had old temp data (Sept, May) were omitted. Any sites that were not listed on that website I deleted too. New file to delete from our current QW monitoring points = |
|
ok, I removed those 274 locations from the common_temp.csv and recreated the geojson and uploaded to s3! 🚀 |
|
I'm checking them and there are almost no sites that respond with no data in WI or in the Northeast (NY north). I did find a few that need zeros up front which were missing. Would it be crazy for us to split up the country and click on each site to make sure it has data or it says "no data were available for specified period"? I think the ones that come up with a mismatch in parameter code might be issues with the site number.... I'm checking more states later tonight. I'm just keeping a list of stations that need zeros in front. I wonder if the method for returning zeros didn't work that well. I'll send a list of stations that need a zero added to their number. https://doimspp-my.sharepoint.com/:w:/g/personal/jrapp_usgs_gov/EWw8ZDhUlsBInejV5FX7i_IBup80Fp8LIKGN6P8kae6c2A?e=ya4dY2 |
|
@mhines-usgs Could we check the station numbers based on their lat longs? IF they are east of ~Wyoming they should have a zero in front. Can you think of a good way to verify? I'm finding a lot that are off. |
|
yes, I'll double check. Sorry if I goofed that up. |
|
This is a real challenge when working with National lists of stations. I want to suggest to Emily and wdfn that they distribute data with USGS016023000 format because it would do wonders for people trying to keep the station IDs as text and not lose the zeros. Sorry I didn't do a better job maintaining them for you when I provided the lists. Hopefully we can get these verified and updated in a simple way. |
|
i have a meeting soon and will fix them up afterwards, fyi. |
|
pushed an update, clicking around now to feel somewhat confident... i found one so far that is still not working, but it appears to be an equipment malfunction |
|
still found a few :hair-pulling: so will keep fixing. |
|
another malfunction: 01120790 |
|
Sorry about the hair pulling. I intentionally left in the Equp malfunction sites because I assume teh WSC are fixing them. There are only a few. As long as the station IDs are good, we should be fine, @mhines-usgs |
|
that's cool. the popups just look the same, so it's not immediately obvious! thanks for confirming :) I hope Don like it!! 🤞 |
|
I hope he likes it too. No indication from Pat either way. :| But I like it and I think we have done a great job building it. Is it pushed to Production now? Thanks, sorry about the last minute requests. Started with Joe after he looked at the prod cite yesterday am... |
|
yep, the latest updates are visible on prod. it's ok, i just am a bit distracted with other work but am around if you notice any other issues. im doing one more scan on the list. |
|
Thanks Megan, |
|
@mwernimont would you let me know the status of the pop-ups and whether the 'newer' plots are going to work? I think that my preference is to set the display back to the original 'old' water temp plots. |
in response to people wanting to get more details about actual temperatures, explore how complicated it would be to add station locations in a new layer that links to the actual water temperature for stations either through an api request to some external station data we can leverage, or even populate a pop up with 7 day water temp graph from the WDFN graph api?
Perhaps show Calibration gages? Steve Markstrom may know.
The text was updated successfully, but these errors were encountered: