Aci Deployment failed with exception: Error in entry script

[csetzkorn]

My use case is to deploy a locally trained model - perfectly valid use case AFIK. For reproducibility I trained this simple model:

library(datasets)
library(caret)

setwd("C:/Data")

index <- createDataPartition(iris$Species, p=0.80, list=FALSE)
testset <- iris[-index,]
trainset <- iris[index,]

model = train(Species ~ ., 
                  data=trainset, 
                  method="rpart", 
                  trControl = trainControl(method = "cv"))

saveRDS(model, "model.rds")

The model can be deployed using code, which is great:

I try to use this scoring script:

library(jsonlite)

init <- function()
{
  model_path <- Sys.getenv("AZUREML_MODEL_DIR")
  model <- readRDS(file.path(model_path, "model.rds"))
  message("iris classfication model loaded")
  
  function(data)
  {
    vars <- as.data.frame(fromJSON(data))
    prediction <- predict(model, newdata=vars)
    toJSON(prediction)
  }
}

and deploy a web service to score the model (or you call it infer these days):

library(azuremlsdk)

interactive_auth <- interactive_login_authentication(tenant_id="xxx")

ws <- get_workspace(
        name = "amazing_work_space", 
        subscription_id = "xxx", 
        resource_group ="xxx", 
        auth = interactive_auth
)

model <- get_model(ws, name = "iris_classification")

r_env <- r_environment(name = 'myr_env',
                       version = '1')

inference_config <- inference_config(
  entry_script = "score.R",
  source_directory = ".",
  environment = r_env)

aci_config <- aci_webservice_deployment_config(cpu_cores = 1, memory_gb = 0.5)

aci_service <- deploy_model(ws, 
                            'xxx', 
                            list(model), 
                            inference_config, 
                            aci_config)

wait_for_deployment(aci_service, show_output = TRUE)

I get:

Running.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. Failed Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 6f62fc59-6b42-4b1e-957b-b0460de9e49e More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details.", "details": [ { "code": "CrashLoopBackOff", "message": "Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details." } ] }

So there is something wrong with my scoring script? Any help would be very much appreciated please! Thanks.

[csetzkorn]

I have the suspicion that, as I run R version 4.0.5 (2021-03-31) locally and my model is fitted using this version, Azure ML crumbles as predefine images only support R 3.6?!

[csetzkorn]

I also tried to fudge the entry script (which I believe can be on my dev machine?). this should be independent of the version of R and the model:

library(jsonlite)

init <- function()
{
  model_path <- Sys.getenv("AZUREML_MODEL_DIR")
  #model <- readRDS(file.path(model_path, "model.rds"))
  message("iris classfication model loaded")
  
  function(data)
  {
    #vars <- as.data.frame(fromJSON(data))
    #prediction <- predict(model, newdata=vars)
    #toJSON(prediction)
    return('{"result": "Hello, world"}')
  }
}`

`
error(s):

Running........................................................
Failed
Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 7c9d3034-c4bb-4f6c-bdae-14857afb14d0
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details."
}
]
}

Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 7c9d3034-c4bb-4f6c-bdae-14857afb14d0
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details."
}
]
}

Error in py_call_impl(callable, dots$args, dots$keywords) :
WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 7c9d3034-c4bb-4f6c-bdae-14857afb14d0
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details."
}
]
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 7c9d3034-c4bb-4f6c-bdae-14857afb14d0\nMore information can be
`

[dataders]

@csetzkorn can you triple backtick your code blocks like below (without the terminal slash I needed to make a meta-code block)? the r at the end of the first line will do R syntax highlighting.

Right now it looks like you're only single backtick-ing.

```r
library(datasets)
library(caret)

foo <- "bar"
```\

[csetzkorn]

@swanderz - thanks. I think I did this now? I tried to use the code UI button. Sorry do not raise many issues in Github usually.

[csetzkorn]

Today I took some more time to use the pre-trained model you use in this sample:

https://github.com/Azure/azureml-sdk-for-r/tree/master/samples/deployment/deploy-to-aci

I renamed it model_ms.rds and saved it locally in C:/Data/model_ms.rds reflected in the slightly adapted code below. Firstly the (attempted) deployment of the web service takes 30+ minutes. Not sure if this is normal?

Secondly, even this failed (see error below).

I will give up on using azureml-sdk-for-r for now. It does not appear to be fit for purpose and I spend many hours on getting something simple like this to work. I think I am better off rolling my own container.

It follows some code (of course I would use environment variables in prod).

Model deployment - works fine - no issue (it appears in http://ml.azure.com/):

library(reticulate)
library(azuremlsdk)

interactive_auth <- interactive_login_authentication(tenant_id="xxx")

ws <- get_workspace(
        name = "xxx", 
        subscription_id = "xxx", 
        resource_group ="xxx", 
        auth = interactive_auth
)

model <- register_model(
        ws, 
        model_path = "C:/Data/model_ms.rds", 
        model_name = "model_ms.rds"
)

```\

This is the scoring/entry code taken from the sample code (I renamed the model file name):

```r
Copyright(c) Microsoft Corporation.
# Licensed under the MIT license.

library(jsonlite)

init <- function()
{
  model_path <- Sys.getenv("AZUREML_MODEL_DIR")
  model <- readRDS(file.path(model_path, "model_ms.rds"))
  message("model is loaded")
  
  function(data)
  {
    plant <- as.data.frame(fromJSON(data))
    prediction <- predict(model, plant)
    result <- as.character(prediction)
    toJSON(result)
  }
}
```\

This is my deployment code, which follows your example:

```r
library(reticulate)
library(azuremlsdk)

interactive_auth <- interactive_login_authentication(tenant_id="xxx")

ws <- get_workspace(
        name = "xxx", 
        subscription_id = "xxx", 
        resource_group ="xxx", 
        auth = interactive_auth
)

model <- get_model(ws, name = "model_ms.rds")

# Create environment
r_env <- r_environment(name = "r_env")

# Create inference config
inference_config <- inference_config(
  entry_script = "score.R",
  source_directory = ".",
  environment = r_env)

# Create ACI deployment config
deployment_config <- aci_webservice_deployment_config(cpu_cores = 1,
                                                      memory_gb = 1)

# Deploy the web service
service_name <- paste0('aciwebservice-', sample(1:100, 1, replace=TRUE))
service <- deploy_model(ws, 
                        service_name, 
                        list(model), 
                        inference_config, 
                        deployment_config)
wait_for_deployment(service, show_output = TRUE)
```\

So I am not sure what else to try? This is very hard to debug. Any feedback would be very much appreciated. Thanks!

Errors (deployment via UI does not work either!):

Running.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Failed
Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 9fd14d0f-d2f6-4153-993c-bf42e9e5e2f6
More information can be found using '.get_logs()'
Error:
{
  "code": "AciDeploymentFailed",
  "statusCode": 400,
  "message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details.",
  "details": [
    {
      "code": "CrashLoopBackOff",
      "message": "Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details."
    }
  ]
}

Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 9fd14d0f-d2f6-4153-993c-bf42e9e5e2f6
More information can be found using '.get_logs()'
Error:
{
  "code": "AciDeploymentFailed",
  "statusCode": 400,
  "message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details.",
  "details": [
    {
      "code": "CrashLoopBackOff",
      "message": "Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details."
    }
  ]
}

Error in py_call_impl(callable, dots$args, dots$keywords) :
  WebserviceException: WebserviceException:
        Message: Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 9fd14d0f-d2f6-4153-993c-bf42e9e5e2f6
More information can be found using '.get_logs()'
Error:
{
  "code": "AciDeploymentFailed",
  "statusCode": 400,
  "message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details.",
  "details": [
    {
      "code": "CrashLoopBackOff",
      "message": "Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details."
    }
  ]
}
        InnerException None
        ErrorResponse
{
    "error": {
        "message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 9fd14d0f-d2f6-4153-993c-bf42e9e5e2f6\nMore information can be

[csetzkorn]

Is there any feedback please?