Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interactive Metric Logging not writing back to AzureML Experiments (not sure if this is a bug / not a feature) #395

Open
buswrecker opened this issue Oct 15, 2020 · 1 comment

Comments

@buswrecker
Copy link

Bug/Problem Description

I'm trying to log metrics interactively; as i'm writing code interactively on a local machine; ie not within an estimator or a submit_experiment with local compute. I understand that this can be done if wrapped within an estimator; but that's not what i'm looking for; there are instances, where i don't want to wrap my interactive code as an estimator, but i definitely would like to log items while i'm testing / building - ie a cool graph/metric etc.

To Reproduce

library(azuremlsdk)
ws <- load_workspace_from_config()

experiment_name <- "train-r-local-test"
exp <- experiment(ws, experiment_name)
runid <- start_logging_run(exp)
message("set x to 100")
x <- 100

message("logging accuracy")
log_metric_to_run("Accuracy", 0.23)
log_metric_to_run("RMSE", 0.00001)

complete_run(runid)

Expected Behaviour

Accuracy and RMSE values to pop up in my experimentation page in AzureML; this doesn't happen - see below
image

Additional Context

We are trying to integrate AzureML R SDK within a Shiny Application; and we would want metrics of the ShinyApp to be logged into AzureML's experimentation page; and subsequently registering models into the AzureML Model Management.

Yes - we could design the Shiny app to trigger AzureML Submit_Experiments into a Remote Compute cluster; but for some of the interactive work on the App; this is not appropriate, the latency to submit experiments into a remote compute for logging one or two metrics exceeds a threshold that would be rather non-interactive for the ShinyApp user.

@buswrecker
Copy link
Author

The RunID is required in the log_metric_to_run function

The correct code for this to run

runid <- start_logging_run(exp)
message("set x to 100")
x <- 100

message("logging accuracy")
log_metric_to_run("Accuracy", 0.23, runid)
log_metric_to_run("RMSE", 0.00001, runid)

This should be documented in ?start_logging_run

Examples:

     ## Not run:
     
     ws <- load_workspace_from_config()
     exp <- experiment(ws, name = 'myexperiment')
     run <- start_logging_run(exp)
     log_metric_to_run("Accuracy", 0.9)
     complete_run(run)
     ## End(Not run)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant