Network Reliability Engineering Community

Syringe container crashloopbackoff

In my local minikube environment, I’m seeing
syringe-6478894-v4hxn 0/1 CrashLoopBackOff 5 6m
and the lesson list won’t display.
From the describe pods, it looks like an issue assigning the appropriate network:



kubectl describe pods syringe-6478894-v4hxn
Name:               syringe-6478894-v4hxn
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               minikube/10.0.2.15
Start Time:         Mon, 29 Apr 2019 18:18:17 -0400
Labels:             antidote_role=infra
                    app=syringe
                    pod-template-hash=6478894
Annotations:        foobar=
                    k8s.v1.cni.cncf.io/networks-status=[{
    "name": "",
    "ips": [
        "10.32.0.10"
    ],
    "default": true,
    "dns": {}
}]
Status:         Running
IP:             10.32.0.10
Controlled By:  ReplicaSet/syringe-6478894
Containers:
  syringe:
    Container ID:   docker://a01b660e0e6619c6fa11b8a9b754309c720a283ba8224246c5fe5f8663235845
    Image:          antidotelabs/syringe:latest
    Image ID:       docker-pullable://antidotelabs/syringe@sha256:c596223a61ce98822873aa5b83c1b2664a4534808c05070a6c439a3620adc4a5
    Ports:          50099/TCP, 8086/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 29 Apr 2019 18:36:41 -0400
      Finished:     Mon, 29 Apr 2019 18:36:41 -0400
    Ready:          False
    Restart Count:  8
    Environment:
      SYRINGE_DOMAIN:           localhost:30002
      SYRINGE_LESSONS:          /antidote/lessons
      SYRINGE_DOMAIN:           localhost
      SYRINGE_TIER:             local
      SYRINGE_LESSON_REPO_DIR:  /antidote
      SYRINGE_LESSONS_LOCAL:    true
    Mounts:
      /antidote from host-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from syringesa-token-snldg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  host-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /antidote
    HostPathType:  
  syringesa-token-snldg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  syringesa-token-snldg
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               23m                default-scheduler  Successfully assigned default/syringe-6478894-v4hxn to minikube
  Warning  FailedCreatePodSandBox  22m                kubelet, minikube  Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "1aea5a8bb0da626c1ca6a17b7e04e0d3f6d656f477b528eef4dc41c88986a1f8" network for pod "syringe-6478894-v4hxn": NetworkPlugin cni failed to set up pod "syringe-6478894-v4hxn_default" network: Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod syringe-6478894-v4hxn in out of cluster comm: Get https://10.96.0.1:443/api/v1/namespaces/default/pods/syringe-6478894-v4hxn: dial tcp 10.96.0.1:443: getsockopt: connection refused, failed to clean up sandbox container "1aea5a8bb0da626c1ca6a17b7e04e0d3f6d656f477b528eef4dc41c88986a1f8" network for pod "syringe-6478894-v4hxn": NetworkPlugin cni failed to teardown pod "syringe-6478894-v4hxn_default" network: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod syringe-6478894-v4hxn in out of cluster comm: Get https://10.96.0.1:443/api/v1/namespaces/default/pods/syringe-6478894-v4hxn: dial tcp 10.96.0.1:443: getsockopt: connection refused]
  Warning  FailedCreatePodSandBox  20m                kubelet, minikube  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fdcfac87092bd299b272c09b43dad4ce6331d4ce0a395f477540c01f0106587f" network for pod "syringe-6478894-v4hxn": NetworkPlugin cni failed to set up pod "syringe-6478894-v4hxn_default" network: Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "weave-net": unable to allocate IP address: Post http://127.0.0.1:6784/ip/fdcfac87092bd299b272c09b43dad4ce6331d4ce0a395f477540c01f0106587f: dial tcp 127.0.0.1:6784: connect: connection refused
  Normal   SandboxChanged          20m (x4 over 22m)  kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Normal   Created                 20m (x4 over 23m)  kubelet, minikube  Created container
  Normal   Started                 20m (x4 over 23m)  kubelet, minikube  Started container
  Normal   Pulled                  20m (x4 over 23m)  kubelet, minikube  Successfully pulled image "antidotelabs/syringe:latest"
  Normal   Pulling                 17m (x6 over 23m)  kubelet, minikube  pulling image "antidotelabs/syringe:latest"
  Warning  BackOff                 2m (x83 over 23m)  kubelet, minikube  Back-off restarting failed container```

Looks like that was a temporary issue, likely as the network was stood up at the beginning. The repeating error is that the pod is causing, so kubectl logs should show the way.

I’ll bet I know the problem - I just merged a change to Syringe that renames some of the required env variables. I just pushed the necessary changes to selfmedicate to account for this, please do a git pull on your local repo and redeploy syringe like so:

kubectl delete deployment syringe
kubectl create -f manifests/syringe-k8s.yaml
1 Like

That worked, in so far as syringe is now running, however the lesson lists are still not loading.

16%20PM

Can you please post syringe logs with kubectl logs?


time="2019-04-29T23:00:33Z" level=info msg="syringed (dev-df1d2436c6d5f882826e919b53e61dda1be88165) starting."

time="2019-04-29T23:00:33Z" level=debug msg="Syringe config: {\"CurriculumDir\":\"/antidote\",\"Tier\":\"local\",\"Domain\":\"localhost\",\"GRPCPort\":50099,\"HTTPPort\":8086,\"DeviceGCAge\":0,\"NonDeviceGCAge\":0,\"HealthCheckInterval\":0,\"TSDBExportInterval\":0,\"TSDBEnabled\":false,\"LiveLessonTTL\":30,\"CurriculumLocal\":false,\"CurriculumRepoRemote\":\"https://github.com/nre-learning/nrelabs-curriculum.git\",\"CurriculumRepoBranch\":\"master\"}"

time="2019-04-29T23:00:33Z" level=debug msg="Searching /antidote/lessons for lesson definitions"

time="2019-04-29T23:00:37Z" level=info msg="Imported 0 lesson definitions."

time="2019-04-29T23:00:37Z" level=info msg="Syringe API started." HTTP Port=8086 gRPC Port=50099

time="2019-04-29T23:00:37Z" level=info msg="No syringe-managed namespaces found. Starting normally."

10.32.0.13 - - [29/Apr/2019:23:00:41 +0000] "GET /exp/lessondef HTTP/1.1" 404 10

time="2019-04-29T23:00:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:01:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:01:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:01:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:02:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:02:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:02:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:03:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:03:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:03:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:04:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:04:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:04:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:05:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:05:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:05:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:06:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:06:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:06:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:07:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:07:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:07:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:08:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:08:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:08:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:09:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:09:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:09:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:10:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:10:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:10:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:11:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:11:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:11:41Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:12:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:12:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:12:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:13:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:13:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:13:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

10.32.0.13 - - [29/Apr/2019:23:14:07 +0000] "GET /exp/lessondef HTTP/1.1" 404 10

10.32.0.13 - - [29/Apr/2019:23:14:11 +0000] "GET /exp/lessondef HTTP/1.1" 404 10

time="2019-04-29T23:14:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:14:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:14:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:15:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:15:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:15:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:16:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:16:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:16:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:17:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:17:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:17:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:18:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:18:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:18:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:19:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:19:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:19:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC."

time="2019-04-29T23:20:37Z" level=debug msg="Recording periodic influxdb metrics"

time="2019-04-29T23:20:37Z" level=warning msg="Unable to push periodic metrics to Influx: Post http://influxdb:8086/write?consistency=&db=syringe_metrics&precision=s&rp=: dial tcp: lookup influxdb on 10.96.0.10:53: no such host"

time="2019-04-29T23:20:42Z" level=debug msg="No syringe-managed namespaces found. No need to GC." 

Looks like the repo hasn’t been cloned to the right location. Note in the selfmedicate.sh script right now that the nrelabs-curriculum and the antidote-selfmedicate repositories must both be present on the system, and next to each other (i.e. both sibling directories of each other)

Even odder: they are,
agrochowski-mbp:Lessons agrochowski$ ls

antidote-selfmedicate nrelabs-curriculum

self-medicate.sh has the correct path - LESSON_DIRECTORY="…/nrelabs-curriculum"
I should say, this did come properly up once. My machine crashed however, and this is after my attempts to recover.

There should be two periods, not three.

there are two, not sure why it pasted like that.
LESSON_DIRECTORY="../nrelabs-curriculum"

Are you using the master branch of the nrelabs-curriculum repo? If so, have you done a git pull there recently? Several recent updates to that repo have changed the way lessons are defined.

I forked on friday, just merged in the new changes to my fork and will try again.